Papers by Dan Roth
ReCode: Robustness Evaluation of Code Generation Models (2023.acl-long)
Copied to clipboard
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang
| Challenge: | Existing work on robustness in text or code tasks has focused on classification, while robustness for code generation tasks is an uncharted area. |
| Approach: | They propose a robustness evaluation benchmark for code generation models that customizes over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. |
| Outcome: | The proposed model performs better on human annotators and on SOTA models with human annnotators. |
TALEN: Tool for Annotation of Low-resource ENtities (P18-4)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task that requires a large amount of training data and annotators who do not speak the language are hard or impossible to find. |
| Approach: | They propose a web-based interface for named entity annotation in low-resource settings . TALEN includes in-place lexicon integration, TF-IDF token statistics, Internet search, and entity propagation . |
| Outcome: | The proposed interface performs better than a popular annotation tool and is more accurate and recall-rich than the current one. |
Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination? (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) suffer from hallucinations and unfaithful reasoning due to keyword/entity biases. |
| Approach: | They propose a new probing method and benchmark to quantify this phenomenon by using a keyword/entity biases-based probing technique called EUREQA. |
| Outcome: | The proposed method achieves 62% accuracy on multi-hop and complex QA benchmarks. |
TRANSIENTTABLES: Evaluating LLMs’ Reasoning on Temporally Evolving Semi-structured Tables (2025.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that large language models are limited in their ability to reason over time due to static datasets. |
| Approach: | They present a dataset that includes 3,971 questions derived from over 14,000 tables . they introduce a template-based question-generation pipeline that harnesses LLMs to refine questions . |
| Outcome: | The proposed model improves on the TRANSIENTTABLES dataset . it demonstrates that the model can reason over time, even when it is not static . |
ner and pos when nothing is capitalized (D19-1)
Copied to clipboard
| Challenge: | Named entity recognition and part of speech tagging require capitalization in training. |
| Approach: | They propose to modify only the casing of the train or test data using lowercasing and truecasing methods to modify the cassing of a model with high performance on both cased and uncased text. |
| Outcome: | The proposed model improves mention detection on noisy out-of-domain Twitter data by 8%. |
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing questions that explicitly describe the process for deriving the answer are often implicit. |
| Approach: | They propose a question answering benchmark where the required reasoning steps are implicit in the question and should be inferred using a strategy. |
| Outcome: | The proposed model is short, topic-diverse, and covers a wide range of strategies. |
Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach (D19-1)
Copied to clipboard
| Challenge: | 0Shot-TC is a challenging NLU problem to which little attention has been paid by the research community. |
| Approach: | They propose to use a standardized evaluation system to classify text snippets without seeing task specific training data. |
| Outcome: | The proposed model is based on a set of standardized evaluations and state-of-the-art baselines. |
Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty (2025.findings-emnlp)
Copied to clipboard
Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, Dan Roth
| Challenge: | Existing methods to gauge model’s uncertainty through self-consistency in responses to the target query are misleading: an LLM may confidently provide an incorrect answer to a target query, yet give a confident and accurate answer to that same query when answering a knowledge-preserving perturbation of the query. |
| Approach: | They propose a method that uses multi-agent interaction to estimate black-box LLMs' uncertainty. |
| Outcome: | The proposed method outperforms existing self-consistency based methods and improves hallucination detection. |
Characterizing and Measuring Linguistic Dataset Drift (2023.acl-long)
Copied to clipboard
Tyler Chang, Kishaloy Halder, Neha Anna John, Yogarshi Vyas, Yassine Benajiba, Miguel Ballesteros, Dan Roth
| Challenge: | Existing metrics for dataset drift have not considered specific dimensions of linguistic drift that affect model performance. |
| Approach: | They propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift. |
| Outcome: | The proposed metrics are more effective than previous metrics at predicting out-of-domain model accuracies compared to popular fine-tuned embedding distances . |
Compositional Data and Task Augmentation for Instruction Following (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that understand spatial concepts and compositional language are inadequate for executing natural language instructions in a physically grounded domain. |
| Approach: | They propose to use knowledge-free auxiliary signals to help the model understand compositional instructions and provide supervision for the instruction's components. |
| Outcome: | The proposed model correctly identifies the source block while the existing model fails on this example. |
CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing commonsense reasoning datasets target different knowledge types, modalities, and formats, but how to help machines acquire and infer over commonsensical knowledge is still unclear. |
| Approach: | They propose a commonsense reasoning benchmark to motivate commonsensing progress from two perspectives: (1) Evaluating whether models can distinguish knowledge quality by predicting if the knowledge is enough to answer the question or not. |
| Outcome: | The proposed model outperforms existing models in evaluating their generalization capabilities across tasks while demonstrating that distinguishing knowledge quality remains challenging for current models. |
Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that current models are prone to generating unfaithful summaries . a proposed method is effective in identifying and correcting extrinsic hallucinations . |
| Approach: | They propose a model-agnostic post-processing technique to correct unfaithful summaries . they generate alternative candidates where names and quantities are replaced with compatible ones . |
| Outcome: | The proposed method corrects extrinsic hallucinations in unfaithful summaries. |
Event Linking: Grounding Event Mentions to Wikipedia (2023.eacl-main)
Copied to clipboard
| Challenge: | a new task for natural language understanding is called Event Linking . the context where an event is mentioned lacks the details of this event . |
| Approach: | They propose a new task to link an article's event mention to the most appropriate Wikipedia page . they collect a training set from Wikipedia and evaluate two models to test the task . |
| Outcome: | The proposed model is based on a dataset and a real-world news domain . it is expected that the most appropriate Wikipedia page will provide rich knowledge about the mention . |
MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents (2026.tacl-1)
Copied to clipboard
Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, Reut Tsarfaty
| Challenge: | Automated agents powered by large language models are becoming more ingrained into how people seek information . but evaluation benchmarks for LLMs rarely feature natural questions that are time-consuming . a new benchmark, MoNaCo, aims to address this gap by eliciting and manually answering time-wasting questions . |
| Approach: | They propose a benchmark of 1,315 natural and time-consuming questions that require dozens of intermediate steps to solve. |
| Outcome: | MoNaCo benchmarks achieve at least 61.2% F1 in real-world time-consuming questions hampered by low recall and hallucinations . Frontier LLMs evaluated on MoN achieving at least 61% F1, harmed by low memory and halluzinations. |
Event Semantic Classification in Context (2024.findings-eacl)
Copied to clipboard
| Challenge: | In this work, we focus on the semantic classification of events in context to help machines gain a deeper understanding of events. |
| Approach: | They propose to integrate event semantics into downstream tasks to help machines understand events better. |
| Outcome: | The proposed model improves the understanding of events in context. |
On the Strength of Character Language Models for Multilingual Named Entity Recognition (D18-1)
Copied to clipboard
| Challenge: | Character-level patterns have been widely used in English Named Entity Recognition systems. |
| Approach: | They propose to use corpus-agnostic character-level language models to capture name tokens . they demonstrate they can capture name and non-name tokens in a diverse set of languages . |
| Outcome: | The proposed model improves the performance of an off-the-shelf NER system for multiple languages. |
Event Time Extraction and Propagation via Graph Attention Networks (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on grounding events into a precise timeline has been limited due to the inherent ambiguity of language and the requirement for information propagation over inter-related events. |
| Approach: | They propose a 4-tuple temporal representation for entity slot filling to ground events into a timeline using a graph attention network approach. |
| Outcome: | The proposed approach yields 7.0% match rate over contextualized embedding approaches and 16.3% higher match rate compared to sentence-level manual event time argument annotation. |
LLM-Symbolic Integration for Robust Temporal Tabular Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for temporal tabular question answering are inconsistent and fail to provide the variability needed to thoroughly evaluate models. |
| Approach: | TEMPTABQA-C uses a synthetic dataset and symbolic representation to generate and execute SQL queries. |
| Outcome: | TEMPTABQA-C improves on previous methods for temporal tabular question answering . incorporating adaptive fewshot prompting with tailored examples improves performance . lack of robustness, scalability, and interpretable solutions is key obstacle . |
Task-Oriented Dialogue as Dataflow Synthesis (2020.tacl-1)
Copied to clipboard
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, Alexander Zotov
| Challenge: | Existing approaches to task-oriented dialogue represent dialogue state as a dataflow graph . microsoft's SMCalFlow dataset features complex dialogues about events, weather, places, and people . |
| Approach: | They propose a dataflow graph-based dialogue agent that maps each user utterance to a program that extends this graph. |
| Outcome: | The proposed framework improves representability and predictability in natural dialogues . it uses dataflow graphs and metacomputation to map user intents to a program . |
Bootstrapping Transliteration with Constrained Discovery for Low-Resource Languages (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to transliteration generation require a large number of training examples. |
| Approach: | They propose a bootstrapping algorithm that uses constrained discovery to improve generation . they show that the model can be used with as few as 500 training examples . |
| Outcome: | The proposed method improves on nine languages written in a unique script. |
Paired Examples as Indirect Supervision in Latent Decision Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new method to learn compositional structured models is needed . end-task supervision provides only a weak indirect signal on values the latent decisions should take. |
| Approach: | They propose a way to leverage paired examples that provide stronger cues for learning latent decisions . they use a DROP dataset to acquire paired questions that provide strong cue signals . |
| Outcome: | The proposed approach improves compositional question answering on a DROP dataset. |
QANom: Question-Answer driven SRL for Nominalizations (2020.coling-main)
Copied to clipboard
Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, Ido Dagan
| Challenge: | Traditionally, SRL annotations focus on verbal predicates, but other types of predicate are frequent in natural language. |
| Approach: | They propose a semantic scheme for capturing predicate-argument relations for nominalizations, termed QANom, using crowdsourcing and QA-driven annotations. |
| Outcome: | The proposed scheme outperforms existing annotations and is useful for downstream tasks. |
Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models display sensitivity to input perturbations, causing concerns about trust among users. |
| Approach: | They propose a methodology to examine how input perturbations affect language models across various scales, including pre-trained models and large language models. |
| Outcome: | The proposed methods enhance the model’s robustness to input perturbations and if exposure to one perturbation enhances or diminishes its performance with respect to other perturbations. |
Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of event processing do not understand the essentiality of step events towards a goal event. |
| Approach: | They propose to deconstruct a goal event into a discrete representation of finer-grained (step) events, which are not equally important to the goal. |
| Outcome: | The proposed model can understand the essentiality of different step events towards a goal event. |
RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks (2025.emnlp-industry)
Copied to clipboard
Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda, Hansa Meghwani, Jyotika Singh, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Existing evaluation methods do not explicitly measure this distinction, hindering effective dataset curation and real-world focused model development. |
| Approach: | They introduce a region-based score to quantify a dataset's reliance on global versus local visual information. |
| Outcome: | The proposed model-based score systematically compares model performance on image patches versus full images to determine if tasks require holistic image understanding or can be solved with partial or localized visual cues. |
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts (2024.findings-acl)
Copied to clipboard
Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, Dan Roth
| Challenge: | Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. |
| Approach: | They propose to use flowcharts as visual contexts to assess the capabilities of visual question-answering multimodal language models in reasoning. |
| Outcome: | The proposed benchmarks evaluate models' ability to follow visual information without pre-existing knowledge on a suite of open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. |
There’s a Time and Place for Reasoning Beyond the Image (2022.acl-long)
Copied to clipboard
| Challenge: | Currently, most work in this area is focused on reasoning with local evidence, but there is a gap between a state-of-the-art joint model and human performance. |
| Approach: | They propose a model that can be used to infer, associate, and reason with contextual information from other sources to establish a more complete picture. |
| Outcome: | The proposed model can identify the news articles related to the image through segment-wise understandings of the signs, buildings, crowds, and more. |
Gold Standard Annotations for Preposition and Verb Sense with Semantic Role Labels in Adult-Child Interactions (C18-1)
Copied to clipboard
| Challenge: | Existing corpus of child-directed speech augments existing corpus for semantic role labels . sense and number of arguments were open to multiple interpretations due to rapidly changing discourse . |
| Approach: | They propose to augment an existing corpus of child-directed speech to provide supervised learning of semantic role labels. |
| Outcome: | The resulting corpus is a gold standard for supervised learning of semantic role labels in child-directed speech. |
“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding (D19-1)
Copied to clipboard
| Challenge: | a new study examines temporal commonsense and compares it to human performance on a dataset . a previous study focused on duration, frequency, stationarity and ordering, but not all aspects of temporal similarity have been studied. |
| Approach: | They define five classes of temporal commonsense and use crowdsourcing to develop a new dataset that serves as a test set. |
| Outcome: | The proposed dataset shows that the best current methods are far behind human performance by 20%. |
Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations (2024.naacl-long)
Copied to clipboard
Sihao Chen, Hongming Zhang, Tong Chen, Ben Zhou, Wenhao Yu, Dian Yu, Baolin Peng, Hongwei Wang, Dan Roth, Dong Yu
| Challenge: | Sentence embeddings are typically learned to recognize the semantic relation between two text inputs. |
| Approach: | They introduce a contrastively-learned contextual embedding model for fine-grained semantic representation of text. |
| Outcome: | The proposed model is able to produce contextual embeddings corresponding to different atomic propositions, i.e. semantic equivalence between propositions across different text sequences. |
Label Semantics for Few Shot Named Entity Recognition (2022.findings-acl)
Copied to clipboard
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, Dan Roth
| Challenge: | Named entity recognition (NER) is a fundamental natural language understanding task that requires large amounts of high quality annotated in-domain data. |
| Approach: | They propose a neural architecture that leverages the semantic information in the names of the labels to give the model additional signal and enriched priors. |
| Outcome: | The proposed model is especially effective in low resource settings. |
ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing static benchmarks do not guarantee that models can use the provided evidence for answering, which is essential to avoid hallucination when the required knowledge is new or private. |
| Approach: | They propose to automatically perturb existing static one for dynamic evaluation by using a chatGPT framework and a set of open-domain QA datasets. |
| Outcome: | The proposed framework generates new test cases on two open-domain QA datasets and is human-readable and useful to trigger hallucination in LLMs. |
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing text overlap based evaluation metrics are limited to matching tokens, either lexically or via embeddings. |
| Approach: | They propose a metric to evaluate the content quality of a summary using question-answering (QA) QA-based methods directly measure a summary’s information overlap with a reference, making them fundamentally different from text overlap metrics. |
| Outcome: | The proposed metric outperforms current state-of-the-art metrics on most evaluations using benchmark datasets while being competitive on others due to limitations of state- of-the art models. |
Design Challenges for a Multi-Perspective Search Engine (2022.findings-naacl)
Copied to clipboard
| Challenge: | a document retrieval system fails to deliver diverse and direct responses to controversial questions . classical document retrievals provide a ranked list of references to relevant but not necessarily trustworthy web documents . |
| Approach: | They propose a perspective-oriented document retrieval paradigm to address these challenges . they propose sponses with different perspectives within topically-related web documents . |
| Outcome: | The proposed system is based on a user survey and a prototype . it will be used to assess the utility and understanding of the system . |
Pairwise Representation Learning for Event Coreference (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing work induces mention representations independently by extracting features from the sentence that contains the mention, without using the context of the other mention. |
| Approach: | They propose a Pairwise Representation Learning scheme for the event mention pairs that jointly encodes a pair of text snippets so that the representation of each mention in the pair is induced in the context of the other one. |
| Outcome: | The proposed scheme outperforms state-of-the-art representations on cross-document and within-document benchmarks. |
What is Your Article Based On? Inferring Fine-grained Provenance (2021.acl-long)
Copied to clipboard
| Challenge: | a new study of claim provenance seeks to trace and explain the origins of claims . a critical reader must be able to assess where the information comes from and where it originates from . |
| Approach: | They propose a method to model and reason about the provenance of multiple interacting claims . they propose generating metadata for the source article based on context and search signals . |
| Outcome: | The proposed method improves on baselines by identifying key external information in the source article. |
Temporal Common Sense Acquisition with Minimal Supervision (2020.acl-main)
Copied to clipboard
| Challenge: | Temporal common sense is crucial for understanding natural language, but its acquisition is challenging . human annotation on such concepts is costly and often not made explicit in text . |
| Approach: | They propose a method that exploits explicit and implicit mentions of temporal common sense to build a temporal similarity language model. |
| Outcome: | The proposed model gives better predictions of various dimensions of temporal common sense than the standard BERT. |
Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQuAD 2.0 (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has attempted to improve extractive QA performance by enriching the dataset with unanswerable questions. |
| Approach: | They build an out-of-domain corpus of competitive and non-competitive questions . they compare the results with the results of the Recognizing Textual Entailments task . |
| Outcome: | The proposed model fails even in the case of simpler questions . the proposed model can be used to address more realistic situations in reading comprehension . |
Capturing the Content of a Document through Complex Event Identification (2022.starsem-1)
Copied to clipboard
| Challenge: | Recent work grouped granular events into more general events, called complex events . however, this approach assumes that a given complex event is always described in consecutive sentences . |
| Approach: | They propose a context-augmented representation learning approach that uses contextual information to model pairwise relation between granular events. |
| Outcome: | The proposed approach outperforms baselines on the complex event identification task. |
Weaver: Interweaving SQL and LLM for Table Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches that combine SQL and LLM rely on rigid workflows . Tables play a critical role across various domains such as finance, healthcare and scientific research . |
| Approach: | Weaver is a modular pipeline that integrates SQL and LLM for table-based question answering. |
| Outcome: | Weaver outperforms state-of-the-art methods on four Table QA datasets. |
Coreference Reasoning in Machine Reading Comprehension (2021.acl-long)
Copied to clipboard
| Challenge: | Existing datasets for machine reading comprehension do not reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Approach: | They propose to use existing coreference resolution datasets to train machine reading comprehension models to better reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Outcome: | The proposed method improves the performance of state-of-the-art models on a set of coreference-related datasets. |
TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification (D18-1)
Copied to clipboard
| Challenge: | Existing methods to determine whether a claim is supported by evidence are decoupled from determining the truth value of the claim. |
| Approach: | They propose a system that decouples evidence finding from determining the truth value of a claim . they propose identifying appropriate evidence for a given claim and determining its truth value . |
| Outcome: | The proposed system decouples evidence finding from determining the truth value of a claim . it can identify evidence candidates and determine the truth of the claim based on predicted evidence . |
Incorporating Question Answering-Based Signals into Abstractive Summarization via Salient Span Selection (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for generating summarizations using QA-based supervision produce higher quality summaries than baseline methods. |
| Approach: | They propose a method for incorporating question-answering signals into a summarization model by automatically marking document NPs as salient based on whether they are answered in the gold summaries. |
| Outcome: | The proposed method generates higher-quality summaries than baseline methods on benchmark summarization datasets. |
End-Task Oriented Textual Entailment via Deep Explorations of Inter-Sentence Interactions (P18-2)
Copied to clipboard
| Challenge: | Existing datasets for textual entailment (TE) have been used to study TE. |
| Approach: | They propose a deep explorations of inter-sentence interactions for textual entailment task that uses a convolution to make important words in P and H play a dominant role in learnt representations. |
| Outcome: | Experiments show that the pretrained DEISTE on SciTail gets 5% improvement over prior state of the art and that it generalizes well on RTE-5. |
Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (2024.findings-emnlp)
Copied to clipboard
Suyash Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth
| Challenge: | Existing datasets for tabular question answering focus on text within cells, but real-world data is multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content. |
| Approach: | They propose a dataset to assess whether current AI models can perform knowledge-aware reasoning on multimodal structured data. |
| Outcome: | The proposed dataset is a robust benchmark for advancing AI’s comprehension and capabilities in analyzing multimodal structured data. |
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. |
| Approach: | They propose a syntactic probing framework for detecting and quantifying such contamination in large language models. |
| Outcome: | The proposed framework generates syntactic variants of test queries for four widely used NL2SQL datasets. |
Enhancing Temporal Understanding in LLMs for Semi-structured Tables (2025.findings-naacl)
Copied to clipboard
| Challenge: | Temporal reasoning over tabular data presents significant challenges for large language models (LLMs), as evidenced by recent research. |
| Approach: | They propose a method that enhances LLMs' temporal reasoning over tabular data by using standard prompts and introduce a novel approach, C.L.E.A.R. |
| Outcome: | The proposed method improves evidence-based reasoning across models and indirect supervision with auxiliary unstructured data significantly boosts model performance in these tasks. |
Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say “I Don’t Know” (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models struggle to recognize their knowledge limits in closed-book question answering . disagreements between prompting regimes are indicative of potential errors, study finds . |
| Approach: | They evaluate three task-equivalent prompting regimes: Direct, Assistive, Incremental . disagreements between prompting régimes provide a precise signal of internal uncertainty . |
| Outcome: | The proposed decomposed prompting outperforms standard uncertainty baselines as an error detector. |
Learning to Decompose and Organize Complex Tasks (2021.naacl-main)
Copied to clipboard
| Challenge: | Using a novel end-to-end pipeline, we propose a solution that consumes a complex task and induces 'dependency graphs' from unstructured text to represent sub-tasks and their relationships. |
| Approach: | They propose a pipeline that consumes a complex task and induces 'dependency graphs' from unstructured text to represent sub-tasks and their relationships. |
| Outcome: | The proposed pipeline outperforms state-of-the-art graph induction pipelines in a dataset of complex tasks with their sub-task graphs. |
Zero-shot Event Extraction via Transfer Learning: Challenges and Insights (2021.acl-short)
Copied to clipboard
| Challenge: | Existing methods for event extraction require expensive annotation and are not extensible to new event ontologies. |
| Approach: | They propose to use textual entailment and/or question answering queries to extract a zero-shot event from a set of TE and/ or QA queries. |
| Outcome: | The proposed method achieves acceptable results on ACE-2005 and ERE, but there is still a large gap from supervised approaches. |
Aligning LLMs for Multilingual Consistency in Enterprise Applications (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) remain unreliable for global enterprise applications due to performance gaps between high-resource and mid/low-resourced languages . |
| Approach: | They propose a batch-wise alignment strategy that aligns model outputs across languages . this method improves non-English accuracy by up to 23.9% without compromising English performance . |
| Outcome: | The proposed approach improves non-English accuracy by up to 23.9% without compromising English performance, model reasoning, or retrieval quality. |
Evidence-based Trustworthiness (P19-1)
Copied to clipboard
| Challenge: | Existing methods to estimate the trustworthiness of information sources are local in that they apply to a given claim. |
| Approach: | They propose a framework for estimating the trustworthiness of information sources in an environment where multiple sources provide claims and supporting evidence. |
| Outcome: | The proposed models show that they improve on baselines and show that the proposed models are more accurate than baselines. |
Event-Centric Natural Language Processing (2021.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will provide an introduction to various methods for automating the extraction, conceptualization and prediction of events and their relations. |
| Approach: | This tutorial will provide an introduction to various methods for automating events and their relations, and a wide range of NLU and commonsense understanding tasks. |
| Outcome: | This tutorial will provide an introduction to various methods for automating extraction, conceptualization and prediction of events and their relations, and a wide range of NLU and commonsense understanding tasks. |
Temporal Reasoning on Implicit Events from Distant Supervision (2021.naacl-main)
Copied to clipboard
| Challenge: | a novel temporal reasoning dataset evaluates the degree to which systems understand implicit events . state-of-the-art models struggle when predicting temporal relationships between implicit and explicit events - a recent paper . |
| Approach: | They propose a temporal reasoning dataset that evaluates the degree to which systems understand implicit events. |
| Outcome: | The proposed model outperforms baseline systems on TRACIE by 5% and 11% on MATRES, an explicit event benchmark. |
From Spatial Relations to Spatial Configurations (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing spatial representations are not sufficient for describing complex spatial configurations. |
| Approach: | They propose to integrate existing spatial representation languages with an annotation schema to extend the capabilities of existing ones. |
| Outcome: | The proposed language can represent a large set of spatial concepts crucial for reasoning . it integrates with the Abstract Meaning Representation (AMR) annotation schema and annotates text from diverse datasets . |
Improving Temporal Relation Extraction with a Globally Acquired Statistical Resource (N18-1)
Copied to clipboard
| Challenge: | Existing temporal extraction systems that extract temporal relations can be improved by using a resource that provides prior knowledge of the temporal order that events usually follow. |
| Approach: | They propose to use a probabilistic knowledge base acquired in the news domain to extract temporal relations between events from the New York Times articles over a 20-year span. |
| Outcome: | The proposed system and resource are both publicly available. |
Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner (2022.findings-naacl)
Copied to clipboard
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henghui Zhu, Xinchi Chen, Peng Xu, Zhiheng Huang, Andrew Arnold, Dan Roth
| Challenge: | Large language models have achieved high performance on various natural language benchmarks, but the explainability of their output remains elusive. |
| Approach: | They propose an architecture called iterative retrieval-generation reasoner that generates an entailment tree that explains a given hypothesis by using premises from C. |
| Outcome: | The proposed model outperforms existing benchmarks on premise retrieval and entailment tree generation with around 300% gain in overall correctness. |
MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking, which can elicit harmful or unsafe behaviors. |
| Approach: | They propose a multilingual guardrail with reasoning for prompt classification that integrates culturally and linguistically nuanced variants and supervised fine-tuning. |
| Outcome: | The proposed guardrail outperforms baselines across in-domain and out-of-domain languages by more than 15%. |
RESIN: A Dockerized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System (2021.naacl-demos)
Copied to clipboard
Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan Windisch Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Heng Ji
| Challenge: | We present a new information extraction system that can construct temporal event graphs from news documents. |
| Approach: | They propose a temporal event graph extraction system that can extract news documents . they extend the system from sentence-level event extraction to cross-document cross-media event extraction . |
| Outcome: | The proposed system can extract temporal event graphs from news documents in multiple languages and multiple data modalities. |
Open Domain Question Answering with Conflicting Contexts (2025.findings-naacl)
Copied to clipboard
Siyi Liu, Qiang Ning, Kishaloy Halder, Zheng Qi, Wei Xiao, Phu Mon Htut, Yi Zhang, Neha Anna John, Bonan Min, Yassine Benajiba, Dan Roth
| Challenge: | Open domain question answering systems often rely on information retrieved from large collections of text to answer questions. |
| Approach: | They evaluate and benchmark three powerful Large Language Models with a dataset . they find that 25% of unambiguous open domain questions can lead to conflicting contexts . |
| Outcome: | The proposed model can't be used to answer questions with conflicting contexts . it can be fine tuned to provide richer information into the model's training . |
ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering (2026.acl-long)
Copied to clipboard
| Challenge: | Large language model (LLM) agents often face strict input context limits, preventing efficient consideration of large toolsets. |
| Approach: | They propose a tool that allows LLMs to merge tools with auto-correction and toolScopeRetriever to rank and select only the most relevant tools for each query. |
| Outcome: | Evaluations on three state-of-the-art LLMs and three open-source tool-use benchmarks show gains of 8.38% to 38.6% in tool selection accuracy. |
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios (2022.naacl-demo)
Copied to clipboard
Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, Heng Ji
| Challenge: | Existing methods for event prediction are incomplete and noisy. |
| Approach: | They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework . |
| Outcome: | The proposed framework can be applied to a wide variety of newsworthy scenarios. |
What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception (2024.naacl-long)
Copied to clipboard
| Challenge: | Question answering models can often be black boxes, as their reasoning process is mostly opaque. |
| Approach: | They analyze the effect of rationales generated by QA models on user feedback and how well they enable users to understand and trust model answers. |
| Outcome: | The proposed model can be used to improve model responses by removing feedback from end users and enhancing model outputs by using natural language feedback. |
Enhancing LLM Capabilities Beyond Scaling Up (2024.emnlp-tutorials)
Copied to clipboard
| Challenge: | general-purpose large language models (LLMs) are expanding in scale and access to unpublic training data. |
| Approach: | This tutorial aims to examine the capabilities of general-purpose large language models . authors discuss adaptation of LLMs to address conflicts, defense against attacks . |
| Outcome: | This tutorial aims to examine the evolution of general-purpose large language models (LLMs) the authors argue that the evolution is dependent on the availability of training data and the scale of the models. |
Evaluating NL2SQL via SQL2NL (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks do not address robustness to linguistic variation in NL2SQL models . |
| Approach: | They propose a schema-aligned paraphrasing framework that leverages SQL-to-NL to generate semantically equivalent, lexically diverse queries while maintaining alignment with the original schema and intent. |
| Outcome: | The proposed framework generates semantically equivalent, lexically diverse queries while maintaining alignment with the original schema and intent. |
What do we expect from Multiple-choice QA Systems? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that good performance on a dataset might not correlate well with human’s expectations from models that “understand” language. |
| Approach: | They propose to train a top performing multiple choice question answering model against expectations from models that "understand" language. |
| Outcome: | The proposed training paradigm leads to a model that performs on par with the original model while better satisfying our expectations. |
Where Have I Heard This Story Before? Identifying Narrative Similarity in Movie Remakes (N18-2)
Copied to clipboard
| Challenge: | Existing methods to identify instances of similar narratives are limited by annotated data. |
| Approach: | They propose a task for identifying instances of similar narratives from a collection of narrative texts. |
| Outcome: | The proposed approach yields an 8% absolute improvement over a baseline on a novel dataset of plot summaries of 577 movie remakes from Wikipedia. |
Is Killed More Significant than Fled? A Contextual Model for Salient Event Detection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on identifying the salient information in a text has used a limited representation of events that omits essential information. |
| Approach: | They propose a highly contextual model of event salience that uses a rich representation of events and integrates document-level information. |
| Outcome: | The proposed model improves on an event salience dataset by 2-4% on standard metrics and addresses flaws in existing evaluation methodologies. |
QuASE: Question-Answer Driven Sentence Encoding (2020.acl-main)
Copied to clipboard
| Challenge: | Question-answering (QA) data often encodes essential information in many facets . a growing interest of QA has led to many large-scale QA datasets available to the community . |
| Approach: | They propose a question-answer driven sentence encoding framework to learn representations from QA data. |
| Outcome: | The proposed framework learns representations from QA data, using BERT or other state-of-the-art contextual language models. |
Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data (2026.findings-acl)
Copied to clipboard
Xuanming Zhang, Shwan Ashrafi, Aziza Mirsaidova, Amir H. Rezaeian, Miguel Ballesteros, Lydia Chilton, Zhou Yu, Dan Roth
| Challenge: | Recent work has explored reasoning efficiency via test-time scaling and early exit strategies. |
| Approach: | They propose an anytime reasoning framework and the Anytime Index to improve model quality . they also propose an inference-time self-improvement method to produce better intermediate solutions . |
| Outcome: | The proposed method improves on NaturalPlan, AIME, and GPQA datasets and improves reasoning quality and efficiency under budget constraints. |
Talking Point based Ideological Discourse Analysis in News Events (2025.findings-acl)
Copied to clipboard
Nishanth Sridhar Nakshatri, Nikhil Mehta, Siyi Liu, Sihao Chen, Daniel Hopkins, Dan Roth, Dan Goldwasser
| Challenge: | Existing models of ideological discourse analysis fail to capture the key elements that shape real-world narratives and lack the ability to integrate contextual information required for understanding abstract ideological views. |
| Approach: | They propose a framework motivated by the theory of ideological discourse analysis to analyze news articles related to real-world events. |
| Outcome: | The proposed framework can generate ideology-specific viewpoints (partisan perspectives) it can be used to generate event snapshots, a visual way of interpreting event discourse. |
Comparing Biases and the Impact of Multilingual Training across Multiple Languages (2023.emnlp-main)
Copied to clipboard
Sharon Levy, Neha John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, Dan Roth
| Challenge: | Currently, studies on bias and fairness in natural language processing focus on a single language and/or across few attributes (e.g. gender, race). However, biases can manifest differently across languages for individual attributes. |
| Approach: | They adapt existing sentiment bias templates in English to Italian, Chinese, Hebrew, and Spanish for race, religion, nationality, and gender. |
| Outcome: | The proposed model favors groups that are dominant in each language's culture, indicating bias amplification, after multilingual finetuning. |
Design Challenges in Low-resource Cross-lingual Entity Linking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing techniques for grounding mentions of entities in a foreign language do not rise to the challenges introduced by text in low-resource languages (LRL) and fail to generalize to text not taken from Wikipedia, on which they are usually trained. |
| Approach: | They propose a cross-lingual XEL technique that uses search engines to locate and search for foreign language entries in Wikipedia. |
| Outcome: | The proposed system shows an increase of 25% in gold candidate recall and 13% in end-to-end linking accuracy over state-of-the-art baselines. |
Understanding Spatial Relations through Multiple Modalities (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing work on common sense reasoning and understanding of spatial relations is limited. |
| Approach: | They propose a spatial model that uses both textual and visual information to predict spatial relations between two entities in an image. |
| Outcome: | The proposed model improves prediction accuracy and coverage and deals with unseen subjects, objects and relations. |
REaR : Retrieve, Expand and Refine for Effective Multitable Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | REaR is retriever-agnostic and improves dense/ sparse retrievers on complex table QA datasets. |
| Approach: | a new framework separates semantic relevance from structural joinability for efficient multi-table retrieval. adam scott and eric liu introduce REaR, a three-stage, LLM-free framework. |
| Outcome: | a new framework improves retrieval quality and performance on complex table QA datasets . it separates semantic relevance from structural joinability and prunes weakly related candidates . the framework is retriever-agnostic and delivers performance competitive with state-of-the-art LLM-augmented retrieval systems . |
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (N18-1)
Copied to clipboard
| Challenge: | Using a dataset of 6,500+ questions, we found that human solvers achieved an F1-score of 88.1%. |
| Approach: | They propose a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. |
| Outcome: | The proposed reading comprehension challenge is based on a reading comprehension dataset with 6,500+ questions and 1000+ paragraphs across 7 domains. |
RobustQA: Benchmarking the Robustness of Domain Adaptation for Open-Domain Question Answering (2023.findings-acl)
Copied to clipboard
Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, Dan Roth
| Challenge: | Existing ODQA datasets consist mainly of Wikipedia corpus, and are insufficient to study models’ generalizability across diverse domains. |
| Approach: | They propose a benchmark to evaluate ODQA's domain robustness using Wikipedia corpus . they annotate QA pairs in retrieval datasets with rigorous quality control . |
| Outcome: | The proposed benchmark improves model performance on annotated QA pairs in retrieval datasets with rigorous quality control. |
CogCompTime: A Tool for Understanding Time in Natural Language (D18-2)
Copied to clipboard
| Challenge: | Existing systems that extract temporal information from text can be useful for natural language understanding. |
| Approach: | They propose a system that extracts temporal information from text and normalizes it to a standard format. |
| Outcome: | The proposed system achieves state-of-the-art performance and incorporates the most recent progress. |
Towards Open-Domain Topic Classification (2022.naacl-demo)
Copied to clipboard
| Challenge: | Existing supervised classification models are insensitive to class names, but are no longer effective in open-domain tasks where the taxonomy is unbounded. |
| Approach: | They propose a topic classification system that accepts user-defined taxonomy in real time . they train a pretrained language model on a new Wikipedia dataset and train it on Wikipedia . |
| Outcome: | The proposed system improves over existing zero-shot models and performs competitively with weakly-supervised models trained on in-domain data. |
Using LLM for Improving Key Event Discovery: Temporal-Guided News Stream Clustering with Event Summaries (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using hierarchical Dirichlet processes, we characterize news articles associated with key events from news streams. |
| Approach: | They propose a generic framework for news stream clustering that analyzes the temporal trend of news articles to automatically extract the underlying key news events that draw significant media attention. |
| Outcome: | The proposed framework produces more coherent clusters based on event summaries . the proposed framework is a first step in a new field of news analysis . |
PerspectroScope: A Window to the World of Diverse Perspectives (P19-3)
Copied to clipboard
| Challenge: | PerspectroScope is a web-based system that lets users query a discussion-worthy natural language claim . |
| Approach: | They propose a web-based system which lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim. |
| Outcome: | The proposed system lets users query a discussion-worthy natural language claim and extract and visualize various perspectives in support or against the claim. |
Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputs (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Text-to-SQL technology bridges natural language (NL) questions and database querying. |
| Approach: | They propose a method for evaluating LLM-generated NLRs using Combo-Eval and a dataset for NLR benchmarking. |
| Outcome: | The proposed method reduces LLM calls by 25-61% and improves performance across scenarios with and without ground truth references. |
“Who said it, and Why?” Provenance for Natural Language Claims (2020.acl-main)
Copied to clipboard
| Challenge: | generating and publishing content is so easy, we are bombarded with information and are exposed to all kinds of claims. |
| Approach: | They propose a formal definition of provenance graph for a given natural language claim . they evaluate the approach using two benchmark datasets to capture provenance . |
| Outcome: | The proposed method shows initial success in capturing provenance and its effectiveness on claim verification. |
H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for tabular reasoning combine textual and symbolic reasoning in a two-stage process to address these limitations. |
| Approach: | They propose an algorithm that integrates symbolic and semantic (textual) approaches in a two-stage process to address these limitations. |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods across three tabular question-answering and fact-verification datasets, underscoring its effectiveness and efficiency. |
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Spectral-normalized identity priors (SNIP) is a structured pruning approach for a Transformer model. |
| Approach: | They propose a structured pruning approach which penalizes an entire residual module toward an identity mapping. |
| Outcome: | The proposed method improves on 5 GLUE benchmark tasks while maintaining comparable performance. |
What Do Users Care About? Detecting Actionable Insights from User Feedback (2022.naacl-industry)
Copied to clipboard
| Challenge: | a large amount of data can be used to extract actionable insights from user feedback . however, the data is unstructured and voluminous, and is underutilized for most users . |
| Approach: | They propose an unsupervised method for finding actionable insights from user feedback . they cluster data into groups containing coherent insights, followed by theme detection . |
| Outcome: | The proposed approach outperforms baselines on two real-world user feedback datasets and one academic dataset. |
Joint Multilingual Supervision for Cross-lingual Entity Linking (D18-1)
Copied to clipboard
| Challenge: | Entity Linking (XEL) systems ground entity mentions written in any language to Wikipedia . XEL is challenging for most languages due to limited availability of resources as supervision . |
| Approach: | They develop a cross-lingual XEL approach that combines supervision from multiple languages jointly. |
| Outcome: | The proposed approach significantly improves on the current state-of-the-art in 8 languages. |
Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end questionanswering assume that pre-trained language can decompose complex tasks into more straightforward sub-tasks. |
| Approach: | They propose to use distant supervision to train decomposition-based transformers for large-scale parallel news. |
| Outcome: | The proposed model improves on semantic parsing and on hotpotQA and strategyQA datasets by 20% to 30%. |
On the Effects of Transformer Size on In- and Out-of-Domain Calibration (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained transformer language models are notoriously expensive to train . prior work has developed smaller, more compact models to reduce training costs . |
| Approach: | They propose to develop smaller, more compact transformer language models which can be calibrated in-domain . they show that smaller models can achieve competitive calibration compared to larger models . |
| Outcome: | The proposed models achieve competitive calibration and better calibration than larger models on a wide range of tasks. |
Partial Or Complete, That’s The Question (N19-1)
Copied to clipboard
| Challenge: | Existing annotation schemes aim at acquiring completely annotated structures, but partial annotations can be costly and hinder learning. |
| Approach: | They propose a method to find out that learning from partial structures can sometimes outperform learning from complete ones. |
| Outcome: | The proposed method outperforms existing methods in three different structured learning tasks. |
Text Classification with Few Examples using Controlled Generalization (N19-1)
Copied to clipboard
| Challenge: | Current training data for text classification is limited, resulting in limited generalization capacity. |
| Approach: | They propose a feed-forward network that can generalize from unlabeled parsed corpora to produce task-specific semantic vectors. |
| Outcome: | The proposed approach is especially effective in low-data scenarios compared to state-of-the-art methods. |
Joint Constrained Learning for Event-Event Relation Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Understanding natural language involves recognizing how multiple event mentions structurally and temporally interact with each other. |
| Approach: | They propose a joint constrained learning framework that enforces logical constraints within and across multiple temporal and subevent relations of events by converting constraints into differentiable learning objectives. |
| Outcome: | The proposed framework outperforms SOTA methods on benchmarks for temporal relation extraction and event hierarchy construction. |
Commonsense Reasoning for Natural Language Processing (2020.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will outline the various types of commonsense knowledge and discuss techniques to gather and represent commonsence knowledge. |
| Approach: | This tutorial will provide researchers with the critical foundations and recent advances in commonsense representation and reasoning. |
| Outcome: | This tutorial will outline the various types of commonsense and discuss techniques to gather and represent commonsence knowledge while highlighting the challenges specific to this type of knowledge (e.g., reporting bias). |
Generalization in Instruction Following Systems (2021.naacl-main)
Copied to clipboard
| Challenge: | Understanding and executing natural language instructions in a grounded domain is one of the hallmarks of artificial intelligence. |
| Approach: | They propose a learning strategy that involves data augmentation to improve the model's performance. |
| Outcome: | The proposed learning strategy outperforms state-of-the-art models in the blocks world domain while satisfying our expectations much better. |
Evaluating LLMs’ Mathematical Reasoning in Financial Document Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in natural language understanding, but their capability for complex mathematical reasoning with a hybrid of structured tables and unstructured text remain uncertain. |
| Approach: | They propose a prompting technique tailored to semi-structured documents that matches or outperforms baselines performance while providing a nuanced understanding of LLMs' abilities. |
| Outcome: | The proposed prompting technique outperforms baseline prompting techniques while providing a nuanced understanding of LLMs' abilities. |
Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learning (2023.acl-long)
Copied to clipboard
Alexander Hanbo Li, Mingyue Shang, Evangelia Spiliopoulou, Jie Ma, Patrick Ng, Zhiguo Wang, Bonan Min, William Yang Wang, Kathleen McKeown, Vittorio Castelli, Dan Roth, Bing Xiang
| Challenge: | Existing methods for data-to-text generation focus on specific types of structured data. |
| Approach: | They propose a method that provides a unified representation that can handle various forms of structured data such as tables, knowledge graph triples, and meaning representations. |
| Outcome: | The proposed method improves zero-shot and few-shot scenarios and can adapt to new structured data. |
Generic Temporal Reasoning with Differential Analysis and Explanation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing temporal reasoning models drop to random guessing on TODAY, suggesting that they heavily rely on spurious information rather than proper reasoning for temporal predictions. |
| Approach: | They propose a task called TODAY that evaluates whether systems can correctly understand the effect of incremental changes in temporal relation distributions. |
| Outcome: | The proposed task outperforms existing models, including GPT-3.5, on in-domain benchmarks while allowing for more appropriate annotations. |
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have demonstrated their strong performance on IQ test questions, achieving high scores across many languages. |
| Approach: | They propose a dataset to evaluate cognitive multimodal reasoning and problem-solving skills of large models. |
| Outcome: | The proposed dataset contains 2,728 multiple-choice questions and 4,642 images spanning 26 categories. |
Learning Constraints and Descriptive Segmentation for Subevent Detection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Event mentions in text correspond to real-world events of varying degrees of granularity . task of subevent detection aims to resolve this granulem issue by recognizing membership of events . |
| Approach: | They propose a task of event-based text segmentation as an auxiliary task to improve learning for subevent detection. |
| Outcome: | The proposed method outperforms baseline methods on subevent detection, HiEve and IC datasets while achieving decent performance on EventSeg prediction. |
“I’d rather just go to bed”: Understanding Indirect Answers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Humans produce and interpret complex utterances even in simple scenarios. |
| Approach: | They present a large-scale English language corpus with 34,268 (polar question, indirect answer) pairs to enable progress on this task. |
| Outcome: | The proposed corpus contains 34,268 (polar question, indirect answer) pairs, and reaches 82-88% accuracy for a 4-class distinction, and 64-85% for 6 classes. |
Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale (2023.acl-long)
Copied to clipboard
| Challenge: | 70% of attention heads and 20% of the feed forward networks can be removed with minimal decline in task performance. |
| Approach: | They propose to investigate whether in-context learning is not uniform across all components of a large language model. |
| Outcome: | The proposed model can remove 70% of attention heads and 20% of feed forward networks with minimal decline in task performance. |
Taxonomy Expansion for Named Entity Recognition (2023.emnlp-main)
Copied to clipboard
Karthikeyan K, Yogarshi Vyas, Jie Ma, Giovanni Paolini, Neha John, Shuai Wang, Yassine Benajiba, Vittorio Castelli, Dan Roth, Miguel Ballesteros
| Challenge: | Training a Named Entity Recognition model involves fixing a taxonomy of entity types . however, requirements evolve and a model may need to recognize additional entity types. |
| Approach: | They propose a method that uses only partially annotated datasets to train a model to recognize additional entity types. |
| Outcome: | The proposed approach performs better with partially annotated datasets than other approaches . the gap between the proposed approach and other approaches is large in additional datasets . |
Conflicts in Texts: Data, Implications and Challenges (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Conflicts in data could reflect complexity of situations, changes that need to be explained and dealt with, difficulties in data annotation, and mistakes in generated outputs. |
| Approach: | This survey categorizes conflicting information into three key areas . they identify the areas where conflicting data can be ignored and undermine models' reliability and trustworthiness. |
| Outcome: | The findings highlight key challenges and future directions for developing conflict-aware NLP systems that can reason over and reconcile conflicting information more effectively. |
AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trials are expensive and time-consuming, and accurate trial prediction is key to advancing medical treatments. |
| Approach: | They propose a framework that combines reasoning capabilities of large language models with the explainability of classical machine learning to generate, evaluate, and refine tabular features without human input. |
| Outcome: | The proposed framework performs better than SOTA methods on clinical trial prediction tasks within a limited number of iterations. |
A Distributional and Orthographic Aggregation Model for English Derivational Morphology (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to derived word generation model derivational morphology to generate words with particular semantics are not effective. |
| Approach: | They propose a novel aggregation model that learns derivational transformations as orthographic functions and as functions in distributional word embedding space. |
| Outcome: | The proposed model learns to choose between the hypothesis of each system and the hypothesis from the model. |
Aligning to Constraints for Data-Efficient Language Model Customization (2025.findings-naacl)
Copied to clipboard
Fei Wang, Chao Shang, Shuai Wang, Sarthak Jain, Qiang Ning, Bonan Min, Vittorio Castelli, Yassine Benajiba, Dan Roth
| Challenge: | General-purpose language models (LMs) are aligned to diverse user intents, but fall short when it comes to specific applications. |
| Approach: | They propose a framework that uses constraints to automatically produce supervision signals for user alignment with constraints. |
| Outcome: | The proposed framework can produce supervision signals for user alignment with constraints. |
PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications (2025.emnlp-industry)
Copied to clipboard
Hitesh Laxmichand Patel, Amit Agarwal, Srikant Panda, Hansa Meghwani, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Existing evaluation metrics for Multimodal Large Language Models (MLLMs) are inadequate to assess their robustness to irrelevant or distracting visual context. |
| Approach: | They propose a patch-context-robustness index to measure MLLMs' robustness to visual context variations. |
| Outcome: | The proposed score measures the robustness of MLLMs to visual contexts across 15 vision-language benchmarks. |
MAPWise: Evaluating Vision-Language Models for Advanced Map Queries (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models excel at tasks requiring joint understanding of visual information and natural language. |
| Approach: | They propose to use choropleth maps to answer questions from three geographical regions in the United States, India, China as question templates. |
| Outcome: | The proposed model outperforms other models in the area of visual language and visual question answering. |
ESTER: A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent event-centric reading comprehension datasets focus mostly on event arguments or temporal relations. |
| Approach: | They propose a machine reading comprehension dataset that leverages natural language queries to reason about the five most common event semantic relations. |
| Outcome: | The proposed dataset shows that current SOTA systems achieve 22.1%, 63.3% and 83.5% for token-based exact-match, **F1** and event-based **HIT@1** scores. |
DeAL: Decoding-time Alignment for Large Language Models (2025.acl-long)
Copied to clipboard
James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-An Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, Dan Roth
| Challenge: | Large Language Models (LLMs) are expected to generate content aligned with human preferences. |
| Approach: | They propose a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). |
| Outcome: | The proposed framework allows the user to customize reward functions and enables Decoding-time Alignment of LLMs. |
Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chart question answering (CQA) is a crucial area of Visual Language Understanding. |
| Approach: | They evaluate the robustness and consistency of current Visual Language Models on a dataset encompassing diverse question categories and chart formats. |
| Outcome: | The proposed models handle varying levels of chart and question complexity and are robust across different visual representations of the same underlying data. |
Toward Robust Evaluation for Multilingual Grammatical Error Correction: Can Large Language Models Replace Human References? (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has shown that using aclosest-gold reference yields more accurate performance estimates, but producing such references for each system individually is costly. |
| Approach: | They propose a method for generating closest-gold references by prompting a large language model with system outputs and a standard reference-based evaluations show weak or no correlation. |
| Outcome: | The proposed method outperforms state-of-the-art models on 14 languages across 14 benchmarks. |
On the Limitations of Reference-Free Evaluations of Generated Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that evaluation metrics which accurately estimate the quality of generated text are limited in their ability to evaluate generated text. |
| Approach: | They argue that reference-free metrics are limited in their ability to evaluate generated text . they recommend that they be used as diagnostic tools for analyzing and understanding model behavior . |
| Outcome: | The proposed evaluation metrics are limited in their ability to evaluate generated text . they can be optimized at test time, can be biased against models with similar outputs . |
On Reference (In-)Determinacy in Natural Language Inference (2025.findings-naacl)
Copied to clipboard
Sihao Chen, Chaitanya Malaviya, Alex Fabrikant, Hagai Taitelbaum, Tal Schuster, Senaka Buthpitiya, Dan Roth
| Challenge: | Using reference determinacy, models fail to recognize context mismatch in NLI examples . |
| Approach: | They propose a benchmark to identify reference ambiguity in NLI examples . they propose RD as a possible assumption for natural language inference . |
| Outcome: | The proposed benchmark identifies reference ambiguity in natural language inference examples . 80% false contradiction and >50% entailment predictions are found . |
Leveraging LLM For Synchronizing Information Across Multilingual Tables (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research has sought to improve cross-language synchronization of Wikipedia tables using rule-based methods, but they struggle with complexity and generalization. |
| Approach: | They propose to use a dataset to simulate the process of updating outdated Wikipedia tables and introduce a task decomposition strategy that enhances coherence and accuracy. |
| Outcome: | The proposed model outperforms baselines in Information Updation (1.79%) and Information Addition (20.58%), highlighting its strength in dynamically updating and enriching data across architectures. |
Towards Long Context Hallucination Detection (2025.findings-naacl)
Copied to clipboard
Siyi Liu, Kishaloy Halder, Zheng Qi, Wei Xiao, Nikolaos Pappas, Phu Mon Htut, Neha Anna John, Yassine Benajiba, Dan Roth
| Challenge: | Large language models are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context. |
| Approach: | They propose a dataset specifically designed for long-context hallucination detection. |
| Outcome: | The proposed architecture outperforms existing models while providing faster inference. |
Yes, No or IDK: The Challenge of Unanswerable Yes/No Questions (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing datasets for Yes/No QA are lacking information needed to answer a Yes/Non question. |
| Approach: | They extend the Yes/No QA task by adding questions with an IDK answer to a BoolQ dataset and create out-of-domain test sets for the task. |
| Outcome: | The proposed dataset includes paragraphs together with naturally occurring questions whose answer is either "Yes" or "No". |
In and Out-of-Domain Text Adversarial Robustness via Label Smoothing (2023.acl-short)
Copied to clipboard
| Challenge: | Existing studies show that state-of-the-art NLP models are vulnerable to adversarial attacks . label smoothing has been proven effective in a variety of applications and modalities . |
| Approach: | They propose to use label smoothing to improve adversarial robustness in pre-trained models against various popular attacks. |
| Outcome: | The proposed method significantly improves adversarial robustness in pre-trained models against various popular attacks. |
New Frontiers of Information Extraction (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Information extraction (IE) is the process of automatically extracting structural information from unstructured or semi-structured data. |
| Approach: | This tutorial will provide an introduction to recent advances in IE by answering several important research questions. |
| Outcome: | The tutorial will address several important research questions and outline directions for further investigation. |
Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieving relevant tables are not sufficient as many questions require retrieving multiple tables and joining them through a join plan that cannot be discerned from the user query itself. |
| Approach: | They propose a method that uncovers useful join relations during table retrieval. |
| Outcome: | The proposed method outperforms the state-of-the-art methods for table retrieval by up to 9.3% in F1 score and for end-to-end QA by up 5.4% in accuracy. |
PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for Natural Language Inference (NLI) only recognize textual entailment relations on sentence-level . however, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence . |
| Approach: | They propose a system to recognize whether one text is textually entailed by another . they use a corpus of over 45K propositions annotated by human raters to study the textual entailment relation of each proposition in a sentence individually. |
| Outcome: | The proposed dataset can be used to understand the compositionality of NLI labels. |
LogiCoL: Logically-Informed Contrastive Learning for Set-based Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current dense retrievers struggle with queries with logical connectives, a use case that is often overlooked but important in downstream applications. |
| Approach: | They propose a logically-informed contrastive learning objective for dense retrievers that learns to respect the subset and mutually exclusive set relation between query results. |
| Outcome: | The proposed model improves retrieval performance and consistency on entity retrieval tasks. |
Extending Multilingual BERT to Low-Resource Languages (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual BERT (M-BERT) has been a huge success in both supervised and zero-shot cross-lingual transfer learning. |
| Approach: | They propose a simple but effective approach to extend multilingual BERT to any new language and show an increase in F1 on M-BERT and new languages. |
| Outcome: | The proposed approach improves on languages already in M-BERT and out of it on other languages. |
TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current machine reading comprehension benchmarks have no questions that test temporal phenomena . a new study studies reading comprehension for temporal relations . |
| Approach: | They propose a reading comprehension benchmark built on news snippets and 21k human-generated questions querying temporal relationships. |
| Outcome: | The new reading comprehension benchmark TORQUE achieves an exact-match score of 51% on the test set . the benchmark is built on 3.2k news snippets with 21k human-generated questions . |
A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for summarization evaluations that approximate human judgments are lacking for accuracy and reliability. |
| Approach: | They propose methods for calculating confidence intervals and running hypothesis tests for correlations using bootstrapping and permutation. |
| Outcome: | The proposed methods show that the confidence intervals are wide, demonstrating high uncertainty in the reliability of automatic metrics. |
Devil’s Advocate: Anticipatory Reflection for LLM Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Introspection-driven approach equips LLM agents with introspection, enhancing consistency and adaptability in solving complex tasks. |
| Approach: | They propose a zero-shot approach that equips LLM agents with introspection, enhancing consistency and adaptability in solving complex tasks. |
| Outcome: | The proposed approach improves performance and efficiency by reducing the number of trials and plan revisions by 45%. |
Few-Shot Novel Concept Learning for Semantic Parsing (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing deep learning algorithms typically require thousands of examples to learn novel concepts. |
| Approach: | They propose an algorithm for learning novel concepts by representing them as programs over existing concepts. |
| Outcome: | The proposed approach outperforms end-to-end neural semantic parsers in a few-shot novel concept learning setting. |
Bootstrapping Small & High Performance Language Models with Unmasking-Removal Training Policy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (LMs) have shown promising ability on handling various downstream tasks including textual classification and question answering. |
| Approach: | They propose to use BabyBERTa to train child-directed speech without unmasking words while masking parameters to improve grammatical accuracy. |
| Outcome: | The proposed model achieves grammatical ability comparable to RoBERTa-base model, which is trained on 6,000 times more words and 15 times more parameters. |
LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new benchmark for natural language inference is proposed for long premises . lawngNLI can train and test systems for implication-based case retrieval and argumentation. |
| Approach: | They propose a new natural language inference benchmark LawngNLI from U.S. legal opinions with automatic labels with high human-validated accuracy. |
| Outcome: | The proposed benchmark can train and test systems for implication-based case retrieval and argumentation. |
Event Causality Identification with Synthetic Control (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to event causality identification have primarily utilized linguistic patterns and multi-hop relational inference, risking false causality . |
| Approach: | They propose to use the Rubin Causal Model to identify event causality by generating a twin from existing corpora. |
| Outcome: | The proposed method can identify causal relations more robustly than previous methods, including GPT-4, which is demonstrated on a causality benchmark, COPES-hard. |
DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization (2022.acl-short)
Copied to clipboard
Zheng Li, Zijian Wang, Ming Tan, Ramesh Nallapati, Parminder Bhatia, Andrew Arnold, Bing Xiang, Dan Roth
| Challenge: | Empirical analyses show that pre-trained sequence-to-sequence models can achieve a 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts. |
| Approach: | They propose to distill and quantize pre-trained sequence-to-sequence models to reduce memory and latency requirements. |
| Outcome: | Empirical results show that the proposed model achieves 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts on multiple summarization and QA datasets. |
Multi-lingual Entity Discovery and Linking (P18-5)
Copied to clipboard
| Challenge: | This tutorial reviews the framework of cross-lingual EL and motivates it as a broad paradigm for the Information Extraction task. |
| Approach: | This tutorial will review the framework of cross-lingual EL and motivate it as a broad paradigm for the Information Extraction task. |
| Outcome: | The aim of this tutorial is to review the framework of cross-lingual EL and motivate it as a broad paradigm for the Information Extraction task. |
MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation (2026.findings-acl)
Copied to clipboard
Jyotika Singh, Fang Tu, Miguel Ballesteros, Weiyi Sun, Sandip Ghoshal, Michelle Yuan, Yassine Benajiba, Sujith Ravi, Dan Roth
| Challenge: | Large language models suffer performance degradation when user instructions and context are distributed over multiple conversational turns. |
| Approach: | They propose a framework that condenses chat history in the background without disrupting the user experience. |
| Outcome: | The proposed framework reduces token counts by up to 72% in 10-turn dialogues while remaining robust to distractors and irrelevant turns. |
Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing QA-based summarization metrics must automatically determine whether the QA model’s prediction is correct or not. |
| Approach: | They benchmark lexical answer verification methods used by current QA-based metrics and two more sophisticated text comparison methods, BERTScore and LERC. |
| Outcome: | The proposed methods outperform the other methods in some settings while remaining statistically indistinguishable from lexical overlap in others. |
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias (2026.acl-long)
Copied to clipboard
| Challenge: | VLM-as-a-Judges pay limited attention to the image content when making decisions, and often blindly favor the more informative answer, even when they can recognize it conflicts with the image. |
| Approach: | They propose a judging paradigm that first corrects inconsistencies with the image content in candidate answers, and then compares the answers against this corrected version. |
| Outcome: | The proposed paradigm reduces informativeness bias by up to 17%, resulting in performance gains of up to 9.8%. |
Zero-Shot On-the-Fly Event Schema Induction (2023.findings-eacl)
Copied to clipboard
| Challenge: | a new approach to event processing uses large language models to generate source documents that can be curated without manual data collection. |
| Approach: | They propose a framework that generates a graphical representation of events in documents . they show that the model is more complete than previous supervised methods . |
| Outcome: | The proposed model is more complete than human-curated schemas in most scenarios. |
Can we Retrieve Everything All at Once? ARM: An Alignment-Oriented LLM-based Retrieval Method (2025.acl-long)
Copied to clipboard
| Challenge: | Existing RAG solutions address the alignment problem in a limited manner . ARM explores relationships among data objects, enabling a retrieve-all-at-once solution for complex queries . |
| Approach: | Experimental results show that ARM improves alignment of open-domain questions with available data . ARM explores relationships among data objects, enabling a retrieve-all-at-once solution for complex queries. |
| Outcome: | Experimental results show that ARM outperforms existing RAG methods on complex open-domain questions. |
Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have boosted performance on some WS benchmarks, but the source of improvement is not clear. |
| Approach: | They propose a method that uses twin sentences for evaluation and two new baselines that account for artifacts in WS benchmarks. |
| Outcome: | The proposed evaluation method is suboptimal for the Winograd Schema . it uses twin sentences to account for commonsense reasoning abilities . |
Analogous Process Structure Induction for Sub-event Sequence Prediction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on event understanding is focusing on procedural (or horizontal) tasks such as predicting the next event given an observed sequence. |
| Approach: | They propose an Analogous Process Structure Induction framework which leverages analogies among processes and conceptualization of sub-event instances to predict the whole sub- sequence of previously unseen open-domain processes. |
| Outcome: | The proposed framework can predict the whole sub-event sequence of previously unseen open-domain processes. |
SocREval: Large Language Models with the Socratic Method for Reference-free Reasoning Evaluation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing reference-free reasoning evaluation metrics rely on human-annotated reasoning chains as references, but require fine-tuning with human-derived chains before evaluation. |
| Approach: | They propose to use GPT-4 to automatically evaluate reasoning chain quality by leveraging the Socratic method. |
| Outcome: | Empirical results show that the proposed approach significantly improves existing reference-free reasoning evaluation metrics. |
Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses (2020.acl-main)
Copied to clipboard
| Challenge: | Empirical research in natural language processing has adopted a narrow set of principles for assessing hypotheses . alternative approaches to assess hypothese rely on p-value computation, which suffers from several known issues. |
| Approach: | They propose to compare different methods for assessing hypotheses . they argue that practitioners should first decide their target hypothesis before choosing a method . |
| Outcome: | The proposed method differs from other methods, but is not widely used in NLP . the proposed method is based on a p-value computation, but has a small gap in accuracy . |
How Good (really) are Grammatical Error Correction Systems? (2021.eacl-main)
Copied to clipboard
| Challenge: | Standard evaluations of Grammatical Error Correction systems use a fixed reference text generated relative to the original text. |
| Approach: | They propose to use a gold reference text to evaluate Grammatical Error Correction systems that is generated relative to the original text and is independent of the system output. |
| Outcome: | The proposed evaluations show that the system performs 20-40 points better than standard evaluations. |
A Multi-Axis Annotation Scheme for Event Temporal Relations (P18-1)
Copied to clipboard
| Challenge: | Existing temporal relation (TempRel) annotation schemes have low inter-annotator agreements even between experts, suggesting that the current annotation task needs a better definition. |
| Approach: | They propose to annotate temporal relation (TempRel) annotation schemes based on event start-points instead of a conventional 60’s-80’s model. |
| Outcome: | The proposed model improves IAA from the conventional 60’s to 80’s and can be used by crowdsourcing to alleviate labor intensity. |
MultiOpEd: A Corpus of Multi-Perspective News Editorials (2021.naacl-main)
Copied to clipboard
| Challenge: | a news editorial is a genre of persuasive text where argumentation structure is usually implicit. |
| Approach: | They propose an open-domain news editorial corpus that supports automatic perspective discovery by identifying and abstracting natural language perspectives from editorials. |
| Outcome: | The proposed system supports automatic perspective discovery tasks in news editorials. |
LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding (2026.acl-long)
Copied to clipboard
Zhivar Sourati, Zheng Wang, Marianne Menglin Liu, Yazhe Hu, Mengqing Guo, Sujeeth Bharadwaj, Kyu J. Han, Tao Sheng, Sujith Ravi, Morteza Dehghani, Dan Roth
| Challenge: | Conventional retrieval-augmented generation (RAG) methods encode content in isolated chunks during ingestion, losing structural and cross-page dependencies, and retrieve a fixed number of pages at inference. |
| Approach: | They propose a Layout-Aware Dynamic RAG framework that encodes content in isolated chunks during ingestion and retrieves a fixed number of pages at inference. |
| Outcome: | Experiments on MMLongBench-Doc, LongDocURL, DUDE, and MP-DoxVQA show that LAD-RAG improves retrieval, achieving over 90% perfect recall on average without any top-k tuning, and outperforming baseline retrievers by up to 20% in recall at comparable noise levels. |
PerKGQA: Question Answering over Personalized Knowledge Graphs (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for question answering over knowledge graphs have focused on generalizable or generic knowledge, which assumes there is a predefined global KG for all queries. |
| Approach: | They propose to use a non-parametric technique that employs case-based reasoning and a parametric approach using graph neural networks to query a predefined knowledge graph (KG) |
| Outcome: | The proposed methods outperform strong baselines on an academic and an internal dataset by 6.5% and 10.5%. |
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing definitions of system-level correlations are inconsistent with how they are used to evaluate systems. |
| Approach: | They propose to calculate correlations only on pairs of systems separated by small differences in automatic scores . they propose to use the full test set instead of the subset of summaries judged by humans . |
| Outcome: | The proposed changes improve the accuracy of the estimated correlations on pairs of systems separated by small differences in automatic scores. |
Seeing Things from a Different Angle:Discovering Diverse Perspectives about Claims (N19-1)
Copied to clipboard
| Challenge: | a number of fact checking techniques are used to identify and eliminate biases in text data. |
| Approach: | They propose to use search engines to expand and diversify a dataset of claims, perspectives and evidence to address a selection bias. |
| Outcome: | The proposed approach outperforms existing methods in a language understanding task. |
PAR2-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering (2026.acl-industry)
Copied to clipboard
| Challenge: | Multi-hop question answering is a practical bottleneck in industry applications . large language models (LLMs) fail frequently when evidence coverage is incomplete or reasoning trajectories drift . |
| Approach: | They propose a training-free two-stage framework that separates coverage from commitment . it performs breadth-first anchoring to build a high-recall evidence frontier . compared with IRCoT, it achieves 23.5% higher answer accuracy . |
| Outcome: | The proposed framework outperforms strong baselines in MHQA benchmarks and achieves 23.5% higher answer accuracy and 10.5% NDCG gains in retrieval quality. |
Do Language Embeddings capture Scales? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models possess significant linguistic, common sense and factual knowledge, but are short of the capability required for general common-sense reasoning. |
| Approach: | They propose to train pretrained language models with a method of canonicalizing numbers . they address a task which is also pre-requisite for general common-sense reasoning . |
| Outcome: | The proposed model can answer questions about common sense and linguistics, but lacks the capability to answer questions on scalar attributes. |
Summary Cloze: A New Task for Content Selection in Topic-Focused Summarization (D19-1)
Copied to clipboard
| Challenge: | Recent work on topic-focused summarization has focused on generic summarizing due to the size of the available datasets. |
| Approach: | They propose a method to study content selection in topic-focused summarization by narrowing the scope to a sentence-level task. |
| Outcome: | The proposed method can generate a large-scale summary cloze dataset from Wikipedia. |
Foreseeing the Benefits of Incidental Supervision (2021.emnlp-main)
Copied to clipboard
| Challenge: | Real-world applications often require improved models by leveraging a range of cheap incidental supervision signals. |
| Approach: | They propose a unified PAC-Bayesian motivated informativeness measure that characterizes the uncertainty reduction provided by incidental supervision signals. |
| Outcome: | The proposed measure quantifies the value added by incidental supervision signals to sequence tagging tasks. |
PRAISE: Enhancing Product Descriptions with LLM-Driven Structured Insights (2025.acl-demo)
Copied to clipboard
| Challenge: | Accurate and complete product descriptions are laborious to sift through manually. |
| Approach: | They propose a system that uses Large Language Models to extract, compare, and structure insights from customer reviews and seller descriptions. |
| Outcome: | The proposed system can extract, compare, and structure insights from customer reviews and seller descriptions. |
CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context (2024.lrec-main)
Copied to clipboard
Yangruibo Ding, Zijian Wang, Wasi U. Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, Bing Xiang
| Challenge: | Pre-trained language models (LMs) for code have shown promising performance in code completion tasks but ignore the rich semantics in other files within the same project. |
| Approach: | They propose a framework that jointly learns the in-file and cross-file context on top of code LMs and a static-analysis-based tool that locates and retrieves the most relevant project-level cross- file context for code completion. |
| Outcome: | The proposed framework improves existing code LMs with a 33.94% relative increase in exact match and 28.69% in identifier matching when the cross-file context is provided. |
Constrained Labeled Data Generation for Low-Resource Named Entity Recognition (2021.findings-acl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) in lowresource languages has been a challenge for years . Existing methods suffer from low quality of annotated data in target language . |
| Approach: | They propose a method that uses projected annotations to generate pseudo supervised data with a transformer language model and a constrained beam search. |
| Outcome: | The proposed method achieves state-of-the-art or competitive performance in low-resource languages. |
Robust Cross-Lingual Hypernymy Detection Using Dependency Context (N18-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual hypernymy detection are sparse and can be trained on related languages with negligible loss of performance. |
| Approach: | They propose a family of unsupervised approaches for cross-lingual hypernymy detection which learns sparse, bilingual word embeddings based on dependency contexts. |
| Outcome: | The proposed approach significantly improves performance on this task, compared to approaches based only on lexical context. |
JTPRO: A Joint Tool–Prompt Reflective Optimization Framework for Language Agents (2026.findings-acl)
Copied to clipboard
Sandip Ghoshal, Anshul Mittal, Jyotika Singh, Miguel Ballesteros, Weiyi Sun, Fang Tu, Shailender Singh, Yassine Benajiba, Fahad Shah, Sujeeth Bharadwaj, Sujith Ravi, Dan Roth
| Challenge: | Large language model agents struggle with ambiguous tool descriptions and underspecified tool schemas that ignore tool-specific nuances. |
| Approach: | They propose a framework for improving tool-calling reliability in trace-supervised settings by rolling out-driven reflection. |
| Outcome: | The proposed framework outperforms baselines and reflective prompt optimizers by 5%–20% on OSR. |
Zero-shot Label-Aware Event Trigger and Argument Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on event extraction relies on labor-intensive annotation, ignoring semantic meaning of event types' labels. |
| Approach: | They propose a zero-shot event extraction approach that first identifies events with existing tools and then maps them to a given taxonomy of event types in a no-shot manner. |
| Outcome: | The proposed approach doubles the performance of previous approaches on a ACE-2005 dataset . it leverages label representations induced by pre-trained language models and maps events to the target types . |
Do Image–Text Metrics Respect Semantic Invariances? (2026.findings-acl)
Copied to clipboard
Amit Agarwal, Hitesh Laxmichand Patel, Meizhu Liu, Jyotika Singh, Karan Dua, Hansa Meghwani, Matthew Rowe, M. Avendi, Yassi Abbasi, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Reference-free image–to–text evaluators are now standard for scoring image–caption alignment, yet it is unclear whether they respect semantic invariances. |
| Approach: | They propose an invariance probe on five popular evaluators under semantics-preserving perturbations along three axes: spatial edits, object changes, and socio-linguistic framing. |
| Outcome: | The proposed invariance probe shows that spatial edits and simple phrasing changes shift scores by ()6% on average and cause ranking flips in up to (),37% of cases. |
How Large Are Lions? Inducing Distributions over Quantitative Attributes (P19-1)
Copied to clipboard
| Challenge: | Current NLP systems have little knowledge about quantitative attributes of objects and events. |
| Approach: | They propose to use web data to create a resource consisting of distributions over physical quantities associated with objects, adjectives, and verbs. |
| Outcome: | The proposed method compares favorably with state-of-the-art results on existing datasets for relative comparisons of nouns and adjectives and on a new dataset. |
An Improved Neural Baseline for Temporal Relation Extraction (D19-1)
Copied to clipboard
| Challenge: | Existing datasets are small and/or have low inter-annotator agreements. |
| Approach: | They propose a new neural system that achieves 10% absolute accuracy improvement over the previous best system. |
| Outcome: | The proposed system achieves 10% absolute improvement over the previous best system on two benchmark datasets. |
Label Semantic Aware Pre-training for Few-shot Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models for text classification use label semantics but few studies have attempted to give models access to informative representations of labels. |
| Approach: | They propose to use label semantics to train generative models by performing secondary pre-training on labeled sentences from a variety of domains. |
| Outcome: | The proposed approach improves generalization and data efficiency of text classification systems while maintaining comparable performance to state-of-the-art models. |
Program-of-Thought Reveals LLM Abstraction Ceilings (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models exhibit reasoning ability when supervised with chain-of-thought (CoT) traces. |
| Approach: | They evaluate large language models with CoT traces and fine-tune them with Program-of-Thought supervision. |
| Outcome: | The proposed model performance degrades sharply under numeric perturbations under isomorphic variants. |
Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing models for event temporal relation extraction are based on data-driven machine learning . however, TEMPREL extraction is not accurate under distribution shifts. |
| Approach: | They propose to conduct counterfactual analysis to attenuate the effects of two types of training biases: the event trigger bias and the frequent label bias. |
| Outcome: | The proposed model extracts TempRel and timelines more faithfully compared to SOTA methods . it is based on two perspectives: one is to extract genuinely based upon contextual description . the other is to provide proper uncertainty estimation and abstain from extraction when no relation is described in the text . |
Joint Reasoning for Temporal and Causal Relations (P18-1)
Copied to clipboard
| Challenge: | a cause must occur earlier than its effect, temporal and causal relations are closely related . a joint inference framework is developed for studying temporal, causal relations . |
| Approach: | They propose a joint inference framework for temporal and causal relations . they use constraints inherent in time and causality to enforce constraints . |
| Outcome: | The proposed framework improves extraction of temporal and causal relations from text. |
Cross-lingual Entity Alignment with Incidental Supervision (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to match entities in multilingual knowledge graphs are insufficient, resulting in inconsistent seed alignment between KGs. |
| Approach: | They propose a model that integrates multilingual KGs and monolingual text corpora in a shared embedding scheme and a self-learning based alignment learning process to induce correspondence between entities and lexemes. |
| Outcome: | The proposed model significantly outperforms state-of-the-art methods on benchmark datasets and significantly outpersts existing methods. |
CogCompNLP: Your Swiss Army Knife for NLP (L18-1)
Copied to clipboard
Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhili Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan, Qiang Ning, Shaoshi Ling, Dan Roth
| Challenge: | a corpus-reader module supports popular corpora, feature extraction and annotation modules for semantic and syntactic tasks. |
| Approach: | They propose a library that provides modules to address different challenges . they provide a corpus-reader module that supports popular corpora in the NLP community . |
| Outcome: | The proposed library simplifies the process of design and development of NLP applications by providing modules to address different challenges. |
Indirectly Supervised Natural Language Processing (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on indirect supervision addresses challenges in ML for NLP . conventional approaches to NLP use taskspecific labeled examples of a large volume . indirect supervision is useful for a wide range of NLP tasks, but it is not enough for decoders . |
| Approach: | This tutorial aims to address questions about indirect supervision in machine learning . authors discuss indirect supervision from T′ that handles T with outputs spanning from a moderate size to an open space . |
| Outcome: | This tutorial aims to answer questions about how to provide supervision for ML tasks . it will discuss indirect supervision from T′ that handles T with outputs spanning from a moderate size to an open space . |
Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge (2023.findings-acl)
Copied to clipboard
Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, Bing Xiang
| Challenge: | Open-ended Visual Question Answering (VQA) requires models to reason over visual and natural language inputs using world knowledge. |
| Approach: | They propose a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. |
| Outcome: | The proposed pipeline expands the knowledge coverage from in-domain training data by 4.1% on OK-VQA, without additional computation cost. |
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners (2024.emnlp-main)
Copied to clipboard
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie Su, Camillo Taylor, Dan Roth
| Challenge: | a new hypothesis-testing framework is developed to assess whether large language models possess genuine reasoning abilities or primarily depend on token bias. |
| Approach: | They propose a framework to assess whether large language models have genuine reasoning abilities or primarily depend on token bias. |
| Outcome: | The proposed framework outlines a list of hypotheses where token biases are readily identifiable . the results suggest that most LLMs still struggle with logical reasoning . |
ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon the insights from disciplines such as user experience research and human behavioral psychology to ensure that the results are reliable. |
| Approach: | They propose a framework for human evaluation of generative large language models that takes into account usability, aesthetics and cognitive biases. |
| Outcome: | The proposed framework is based on the framework proposed by Deutsch and alnajjar . it is aimed at ensuring that human evaluation is accurate in the age of generative AI . |
ExpertQA: Expert-Curated Questions and Attributed Answers (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study examined the attribution and factuality of language models in domains . experts from various fields are using large language models for information-seeking scenarios . |
| Approach: | They evaluate language models' attribution and factuality by bringing domain experts in the loop . they collect expert-curated questions from 484 participants across 32 fields of study . |
| Outcome: | The results show that language models can provide factually correct answers in high-stakes fields, but they can also be harmful to experts. |
Zero-Shot Open Entity Typing as Type-Compatible Grounding (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to entity typing have limited flexibility to transfer across text genres and generalize to new type taxonomies. |
| Approach: | They propose a zero-shot entity typing approach that requires no annotated data and can flexibly identify newly defined types. |
| Outcome: | The proposed system outperforms state-of-the-art supervised NER systems on a broad range of datasets and on 'biological domain' it is competitive with supervised systems and outperformed on out-of training datasets. |