Papers by Benjamin Chen
Condensing Multilingual Knowledge with Lightweight Language-Specific Modules (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to boost performance in multilingual models but scalability is difficult to manage. |
| Approach: | They propose a method that incorporates language-specific (LS) modules to boost model performance. |
| Outcome: | The proposed method outperforms state-of-the-art methods while outperforming existing methods. |
A Unified View of Evaluation Metrics for Structured Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | a framework that unifies evaluation metrics for structured prediction tasks is presented . metric design decisions are motivated by specific characteristics of tasks, and we suggest modifications to existing metrics to meet those motivations. |
| Approach: | They propose a framework that unifies a variety of evaluation metrics for different structured prediction tasks. |
| Outcome: | The proposed framework can be used to create new metrics based on the output structure of a number of tasks. |
When Do Decompositions Help for Machine Reading? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on decompositions of complex questions has focused on multi-step reasoning . but, in machine reading, it is unclear when decomposing is helpful . |
| Approach: | They conduct experiments on decompositions in machine reading to unify recent work . they find that decomposing complex questions can be helpful in zero or limited-data settings . |
| Outcome: | The proposed model can learn decompositions implicitly even with limited data, the study shows . the results are consistent with previous work on decomposing complex questions . |
Community lexical access for an endangered polysynthetic language: An electronic dictionary for St. Lawrence Island Yupik (N19-4)
Copied to clipboard
| Challenge: | a new electronic dictionary for St. Lawrence Island Yupik is developed to facilitate language-learning on the island . the endangered language is spoken primarily on St. lisa's St.liss island, Alaska . |
| Approach: | They propose a morphologically-aware electronic dictionary for St. Lawrence Island Yupik . the dictionary is set in an uncluttered interface and uses HTML, Javascript, and CSS . |
| Outcome: | The proposed dictionary is set in an uncluttered interface and is available in English and in Yupik . it is based on the morphologically-aware version of the Badten et al. paper dictionary . |
Help! Need Advice on Identifying Advice (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained systems are able to capture advice better than rule-based systems, but advice identification is challenging. |
| Approach: | They analyze a dataset of advice posts on two reddit forums and annotate whether they contain advice. |
| Outcome: | The proposed models show that pre-trained models capture advice better than rule-based systems, but advice identification is challenging. |
RORA: Robust Free-Text Rationale Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing metrics rely on degree to which rationale supports a label, but they fail to evaluate rationales that inadvertently leak the label. |
| Approach: | They propose a RObust free-text RAtionale evaluation against label leakage that quantifies the new information supplied by a rationale to justify the label. |
| Outcome: | The proposed evaluation outperforms existing methods in evaluating human-written, synthetic, or model-generated rationales, particularly demonstrating robustness against label leakage. |
Learning to Rank for Plausible Plausibility (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to encoding sentences using contextualized encoders are inconsistent . |
| Approach: | They propose to use a cross entropy log-loss objective to improve plausibility . they propose a margin-based loss leads to a more plausible model of plausability . |
| Outcome: | The proposed loss is intuitively wrong when applied to plausibility tasks . the proposed loss leads to a more plausible model of plausability . |
MultiMUC: Multilingual Template Filling on MUC-4 (2024.eacl-long)
Copied to clipboard
William Gantt, Shabnam Behzad, Hannah An, Yunmo Chen, Aaron White, Benjamin Van Durme, Mahsa Yarmohammadi
| Challenge: | We present multilingual parallel template filling datasets for MUCs . systems were required to extract one template per incident, containing details about perpetrators, victims, weapons used . |
| Approach: | They introduce MultiMUC, the first multilingual parallel corpus for template filling . they obtain automatic translations from a strong multilingual machine translation system . |
| Outcome: | The proposed dataset includes translations of the classic MUC-4 template filling benchmark into Arabic, Chinese, Farsi, Korean, and Russian. |
SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding (2026.findings-acl)
Copied to clipboard
Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, Chi Ruan, Benjamin Schneider, Weixu Zhang, Xiang Li, Andy Zheng, Yuyu Zhang, Ping Nie, Wenhu Chen
| Challenge: | Existing benchmarks for agentic repository-level code understanding overlook long tail topics and rely on memorized knowledge. |
| Approach: | They propose a repository-level agentic code understanding benchmark that uses long-tail repositories with executable environments to enforce topical balance. |
| Outcome: | Empirically, a Qwen3-8B model trained with the proposed benchmark outperforms GPT-4o by 2.3 points. |
Constrained Language Models Yield Few-Shot Semantic Parsers (2021.emnlp-main)
Copied to clipboard
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, Benjamin Van Durme
| Challenge: | Large pretrained language models excel at generating natural language, but they are not efficient for task specific semantic parsing. |
| Approach: | They propose to use large pretrained language models as few-shot semantic parsers . they paraphrase inputs into a controlled sublanguage resembling English . |
| Outcome: | The proposed model can generate surprisingly accurate models on multiple tasks with minimal code and data. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Predicting Entity Salience in Extremely Short Documents (2024.emnlp-industry)
Copied to clipboard
| Challenge: | False positive: ES is a natural language understanding task that determines which entities are most salient to a passage . Falsity: Popsicle, Frank Epperson and San Francisco are salient entities . |
| Approach: | They propose a lightweight and data-efficient approach for entity salience detection on short documents . they propose he use of a human-labeled dataset to evaluate entity salient on short questions . |
| Outcome: | The proposed approach achieves competitive performance over state-of-the-art models at significant cost and latency advantages. |
LOME: Large Ontology Multilingual Extraction (2021.eacl-demos)
Copied to clipboard
Patrick Xia, Guanghui Qin, Siddharth Vashishtha, Yunmo Chen, Tongfei Chen, Chandler May, Craig Harman, Kyle Rawlins, Aaron Steven White, Benjamin Van Durme
| Challenge: | LOME is a system for performing multilingual information extraction with large ontologies. |
| Approach: | They propose a system for multilingual information extraction with a framenet parser . LOME is available as a Docker container on Docker Hub and a lightweight version is available on the web . |
| Outcome: | The proposed system outperforms or is competitive with the (monolingual) state-of-the-art . it can be used to build knowledge graphs with large ontologies and across multiple languages . |
Learning to Retrieve Iteratively for In-Context Learning (2024.emnlp-main)
Copied to clipboard
Yunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia, Richard Shin, Jason Eisner, Benjamin Van Durme
| Challenge: | In-context learning is a powerful tool for learning large language models. |
| Approach: | They propose an iterative retrieval framework that empowers retrievers to make iterable decisions through policy optimization. |
| Outcome: | The proposed framework outperforms existing methods on semantic parsing datasets with 4M additional parameters for state encoding. |
Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting (N19-1)
Copied to clipboard
J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, Benjamin Van Durme
| Challenge: | Lexically-constrained sequence decoding allows for explicit positive or negative phrase-based constraints to be placed on target output strings in machine translation or monolingual text rewriting tasks. |
| Approach: | They propose a vectorized dynamic beam allocation algorithm which extends work in lexically-constrained decoding to work with batching. |
| Outcome: | The proposed method improves on natural language inference, question answering and machine translation tasks by fivefold . |
Hierarchical Entity Typing via Multi-level Learning to Rank (2020.acl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a canonical information extraction task that assigns spans to one of a handful of types. |
| Approach: | They propose a hierarchical entity classification method that embraces ontological structure at training and during prediction. |
| Outcome: | The proposed method outperforms previous work on strict accuracy and significantly outperformed previous work. |
Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent expressive speech-to-speech translation systems have achieved impressive expressivity preservation performances by cascading unit-to speech (U2S) generator to the speech- to-unit translation model. |
| Approach: | They propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST) They aim to address this limitation by incorporating a distillation with no label (DINO) self-controlled training strategy into the model’s pretraining process. |
| Outcome: | The proposed model significantly improved the expressive speech-to-speech translation system in noisy environments while maintaining competitive performance in clean environments. |
Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations (2026.findings-acl)
Copied to clipboard
| Challenge: | Natural Language Inference (NLI) datasets often exhibit label variation. |
| Approach: | They extend LiTEx taxonomy to two NLI datasets and jointly analyze label variation and label variation. |
| Outcome: | The proposed model combines explanations as a lens to analyze variation in NLI annotations and examine individual differences in reasoning. |
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists (2025.emnlp-main)
Copied to clipboard
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun
| Challenge: | AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. |
| Approach: | They propose an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. |
| Outcome: | The proposed pipeline synthesizes accurate tasks and tasks from a dataset of 5,404 tasks covering four scientific disciplines and 756 Python packages. |
Uncertain Natural Language Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Existing categorical bin assignments for NLI are not effective for entailment inference. |
| Approach: | They propose a refinement of Natural Language Inference that targets direct prediction of subjective probability assessments. |
| Outcome: | The proposed model can be used in pre-training and correlates well with humans. |
Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles (2024.findings-naacl)
Copied to clipboard
Weiting Tan, Haoran Xu, Lingfeng Shen, Shuyue Stella Li, Kenton Murray, Philipp Koehn, Benjamin Van Durme, Yunmo Chen
| Challenge: | Recent work shows that large language models can generalize to machine translation using zero-shot examples with in-context learning. |
| Approach: | They investigate the factors contributing to this gap by matching the writing styles of the target corpus. |
| Outcome: | The proposed methods can be enhanced without the need for parallel demonstration examples. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
Iterative Document-level Information Extraction via Imitation Learning (2023.eacl-main)
Copied to clipboard
| Challenge: | Documents may feature zero or more instances of a template of any given type, and the task of template extraction entails identifying the templates in a document and extracting each template’s slot values. |
| Approach: | They propose to use iterative extraction to extract complex relations, i.e., N-tuples representing a mapping from named slots to spans of text within a document. |
| Outcome: | The proposed model leads to state-of-the-art results on two established benchmarks and a strong baseline on the new BETTER Granular task. |
Argument Summarization and its Evaluation in the Era of Large Language Models (2025.emnlp-main)
Copied to clipboard
Moritz Altemeyer, Steffen Eger, Johannes Daxenberger, Yanran Chen, Tim Altendorf, Philipp Cimiano, Benjamin Schiller
| Challenge: | Large Language Models (LLMs) have revolutionized various Natural Language Generation tasks, including Argument Summarization (ArgSum). |
| Approach: | They propose a prompt-based evaluation scheme and validate it through a human benchmark dataset. |
| Outcome: | The proposed evaluation scheme outperforms existing methods and is validated by a human benchmark dataset. |
Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction (2021.emnlp-main)
Copied to clipboard
Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, Benjamin Van Durme
| Challenge: | Zero-shot cross-lingual information extraction (IE) is a technique for training data in a source language but not in . |
| Approach: | They explore techniques including data projection and self-training to improve zero-shot cross-lingual information extraction (IE) IE is a construction of an IE model for some target language given existing annotations exclusively in English. |
| Outcome: | The proposed techniques show that they perform better than any single strategy. |
Guided K-best Selection for Semantic Parsing Annotation (2022.acl-demo)
Copied to clipboard
Anton Belyy, Chieh-yang Huang, Jacob Andreas, Emmanouil Antonios Platanios, Sam Thomson, Richard Shin, Subhro Roy, Aleksandr Nisnevich, Charles Chen, Benjamin Van Durme
| Challenge: | a prototype model trained on a small amount of data is not available, leading to limited prediction performance. |
| Approach: | They propose a human-in-the-loop process that generates a set of valid candidates and allows users to quickly traverse the set and filter incorrect parses. |
| Outcome: | The proposed process can be used to efficiently traverse the candidate set and select the correct parse, with minimal modification when necessary. |
Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation (2022.findings-acl)
Copied to clipboard
| Challenge: | a low-resource task-oriented semantic parser is limited by privacy requirements for unlabeled natural utterances. |
| Approach: | They propose a setup for low-resource task-oriented semantic parsing based on user interactions . they use structured canonical utterances, then simulating corresponding natural language to improve performance. |
| Outcome: | The proposed setup improves on a low-resource task-oriented semantic parser using utterances collected through user interactions. |
An Empirical Study on Finding Spans (2022.emnlp-main)
Copied to clipboard
| Challenge: | Various information extraction tasks require a span finding component, which either directly yields the output or serves as an essential component of downstream linking. |
| Approach: | They propose methods for span finding, the selection of consecutive tokens in text for some downstream tasks. |
| Outcome: | The proposed methods perform better on masked language models and pre-trained encoders than on encoder-decoder models. |