ERROR

A Closer Look at Claim Decomposition (2024.starsem-1)

Challenge:	Lack of access to model details has raised concerns about data contamination among researchers.
Approach:	They conduct the first systematic analysis of work using OpenAI’s GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination.
Outcome:	The proposed models have been exposed to 4.7M samples from 263 benchmarks during the first year after their release.

Copied to clipboard

Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, Benjamin Van Durme

Challenge:	Recent work uses claim decomposition to determine how well supported a claim is for applications in factual precision of generated text, entailment of human generated text and claim verification.
Approach:	They propose an LLM-based approach to generating decompositions inspired by Bertrand Russell’s theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposing quality over previous methods.
Outcome:	The proposed method improves on the FActScore and a Bertrand Russell-inspired approach to generating decompositions inspired by neo-Davidsonian semantics and improves decomposability quality.

Evaluating Research Novelty Detection: Counterfactual Approaches (D19-53)

Copied to clipboard

Reinald Kim Amplayo, Seung-won Hwang, Min Song

Challenge:	Despite its importance, this direction of research has not been explored as much.
Approach:	They propose to use counterfactual simulations to evaluate paper novelty detection models . they ask models to differentiate papers at time t and counterf actual paper from future time .
Outcome:	The proposed models can be compared against a set of papers with a given date and with different annotations.

LETI: Learning to Generate from Textual Interactions (2024.findings-naacl)

Copied to clipboard

Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

FPI: Failure Point Isolation in Large-scale Conversational Assistants (2022.naacl-industry)

Challenge:	Existing techniques fine-tune on input-output pairs or with numerical rewards that gauge the output quality are not effective.
Approach:	They propose to fine-tune pre-trained language models with binary labels and a Python interpreter to get textual feedback from the inputs.
Outcome:	The proposed model outperforms the base model on unseen problems and achieves comparable or better performance on humanEval.

Copied to clipboard

Rinat Khaziev, Usman Shahid, Tobias Röding, Rakesh Chada, Emir Kapanci, Pradeep Natarajan

Adaptively Scheduled Multitask Learning: The Case of Low-Resource Neural Machine Translation (D19-56)

Challenge:	Large-scale conversational assistants can cause errors in their modules . a machine learning system can analyze large volumes of data and isolate the source of error .
Approach:	They propose a machine learning system that embeds incoming request and context using pre-trained transformer models and encodes additional metadata features to output failure point predictions.
Outcome:	The proposed system obtains 92.2% of human performance while scaling to analyze the entire traffic in 8 different languages of a large-scale conversational assistant.

Copied to clipboard

Poorya Zaremoodi, Gholamreza Haffari

Biasly: An Expert-Annotated Dataset for Subtle Misogyny Detection and Mitigation (2024.findings-acl)

Challenge:	Neural Machine Translation suffers from the lack of bilingual data in low-resource scenarios.
Approach:	They propose to inject inductive biases into Neural Machine Translation (NMT) using auxiliary syntactic and semantic tasks.
Outcome:	The proposed approach improves translation quality by reweighing training data of main and auxiliary tasks based on their contributions to generalisability of main task.

Copied to clipboard

Brooklyn Sheppard, Anna Richter, Allison Cohen, Elizabeth Smith, Tamara Kneese, Carolyne Pelletier, Ioana Baldini, Yue Dong

Challenge:	the Biasly dataset captures misogyny in movies in ways unique within the literature.
Approach:	The Biasly dataset captures misogyny in North American film by combining annotations of movie subtitles with common NLP algorithms.
Outcome:	The Biasly dataset captures misogyny expressions in North American film . it contains annotations of movie subtitles and text generation for rewrites .

Universal Language Model Fine-tuning for Text Classification (P18-1)

Copied to clipboard

Jeremy Howard, Sebastian Ruder

MASEval: Extending Multi-Agent Evaluation from Models to Systems (2026.acl-demo)

Challenge:	Existing approaches to computer vision require task-specific modifications and training from scratch.
Approach:	They propose a method that can be applied to any task in NLP and propose to open-source it.
Outcome:	The proposed method outperforms the state-of-the-art on six text classification tasks, reducing error by 18-24% on majority of datasets.

Copied to clipboard

Cornelius Emde, Alexander Rubinstein, Anmol Goel, Ahmed Heakl, Sangdoo Yun, Seong Joon Oh, Martin Gubri

Challenge:	MASEval provides a framework-agnostic, system-level comparison across any agent framework and benchmark.
Approach:	They propose a Python library that treats the entire agentic system as the unit of analysis.
Outcome:	The proposed framework treats the entire agentic system as the unit of analysis.

Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances (2022.acl-short)

Copied to clipboard

Suvodip Dey, Ramamohan Kummara, Maunendra Desarkar

SEAL: Interactive Tool for Systematic Error Analysis and Labeling (2022.emnlp-demos)

Challenge:	Dialogue State Tracking (DST) is a task-oriented conversational agent that keeps track of key information exchanged during a conversation.
Approach:	They propose a new evaluation metric called Flexible Goal Accuracy to address shortcomings of JGA.
Outcome:	The proposed metric improves on existing metrics and improves performance of turn-level and non-cumulative belief state models.

Copied to clipboard

Nazneen Rajani, Weixin Liang, Lingjiao Chen, Margaret Mitchell, James Zou

Visual Data in NLP

Challenge:	Existing models that fail on tail data or rare groups are difficult to identify due to lack of explicit labels.
Approach:	They propose a systematic error analysis and labeling tool that uses a two-step approach to identify high-error slices of data and then give human-understandable semantics to those underperforming slices.
Outcome:	The proposed tool identifies high-error slices of data and gives human-understandable semantics to those underperforming slices.

FastV-RAG: Towards Fast and Fine-Grained Video QA with Retrieval-Augmented Generation (2026.acl-long)

Copied to clipboard

Gen Li, Peiyu Liu

TellMeWhy: Learning to Explain Corrective Feedback for Second Language Learners (D19-3)

Challenge:	Existing methods for retrieval-augmented generation are inefficient and often fail to maintain high answer quality.
Approach:	They propose an efficient VLM-based RAG framework built on a speculative decoding pipeline and a similarity-based filtering strategy to mitigate errors.
Outcome:	The proposed framework reduces inference latency without sacrificing correctness . it achieves comparable or higher accuracy than standard approaches while speeding up inference by approximately 2x .

Copied to clipboard

Yi-Huei Lai, Jason Chang

Framing Named Entity Linking Error Types (L18-1)

Challenge:	Write & Improve and Grammarly typically use canned text to explain grammatical errors, but corrective feedback with the most useful explanations may contain collocations, grammar, and contextsensitive examples.
Approach:	They propose to analyze sentences with corrections to identify error types and problem words and to extract grammar patterns, collocations and example sentences.
Outcome:	The proposed system can be used to customize explanations based on the context of the error.

Copied to clipboard

Adrian Braşoveanu, Giuseppe Rizzo, Philipp Kuntschik, Albert Weichselbraun, Lyndon J.B. Nixon

Knowledge Representation

Challenge:	Named Entity Linking (NEL) and relation extraction forms the backbone of Knowledge Base Population tasks.
Approach:	They propose a taxonomy to frame common errors and apply it to four well-known Named Entity Linking systems.
Outcome:	The proposed taxonomy was applied to four well-known Named Entity Linking systems on three gold standards.

The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data (2026.eacl-demo)

Copied to clipboard

Sunith Vallabhaneni, Thomas Berkane, Maimuna S. Majumder

Challenge:	largelanguage models (LLMs)-powered web agents can be useful for research in areas such as social science, public health, and economics.
Approach:	They propose a model-agnostic multi-agent system that auto-mates the process of validating and remediatingweb-sourced datasets.
Outcome:	The proposed system outperforms baseline approaches and achieves datacompleteness and precision up to 73.3%.

Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator (2025.coling-industry)

Copied to clipboard

Frederic Thomas Kirstein, Terry Lima Ruas, Bela Gipp

Challenge:	Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors.
Approach:	They propose a framework that uses a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment.
Outcome:	The proposed framework achieves high correlation with human judgment and a consistent rating and adaptability to custom error guidelines.

ProcessBench: Identifying Process Errors in Mathematical Reasoning (2025.acl-long)

Copied to clipboard

Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin

SchAman: Spell-Checking Resources and Benchmark for Endangered Languages from Amazonia (2022.aacl-short)

Challenge:	Existing models fail to generalize to more challenging math problems, authors say . existing benchmarks related to assessing language models' reasoning process are limited .
Approach:	They propose a tool to measure language models' ability to identify erroneous steps in reasoning . they use two types of models: process reward models and critic models .
Outcome:	The proposed model outperforms existing models in evaluating language models' reasoning process . the best open-source model has demonstrated the critique capability competitive with the proprietary model .

Copied to clipboard

Arturo Oncevay, Gerardo Cardoso, Carlo Alva, César Lara Ávila, Jovita Vásquez Balarezo, Saúl Escobar Rodríguez, Delio Siticonatzi Camaiteri, Esaú Zumaeta Rojas, Didier López Francis, Juan López Bautista, Nimia Acho Rios, Remigio Zapata Cesareo, Héctor Erasmo Gómez Montoya, Roberto Zariquiey

Zero-Shot Text Classification with Self-Training (2022.emnlp-main)

Challenge:	Spell-checking as a generation task requires large amount of data, which is not feasible for endangered languages such as the languages spoken in Peru.
Approach:	They propose to use augmented misspelling data to train neural spell-checking models for four endangered languages of Peru: Shipibo-Koniba, Asháninka, Yánesha, yine .
Outcome:	The proposed model achieves better scores in most of the errors and languages in the four indigenous languages of Peru: Shipibo-Koniba, Asháninka, Yánesha, yine.

Copied to clipboard

Ariel Gera, Alon Halfon, Eyal Shnarch, Yotam Perlitz, Liat Ein-Dor, Noam Slonim

MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification (2024.findings-emnlp)

Challenge:	Recent advances in large pretrained language models have increased attention to zero-shot text classification.
Approach:	They propose a plug-and-play method to bridge this gap by requiring only class names along with an unlabeled dataset.
Outcome:	The proposed model can be trained on a natural language inference dataset and performs on dozens of unseen tasks without the need for domain expertise or trial and error.

Copied to clipboard

Kai Sun, Yushi Bai, Ji Qi, Lei Hou, Juanzi Li

Information Retrieval

Challenge:	Existing benchmarks for multimodal reasoning in large multimodal models are underperforming on multimodal tasks.
Approach:	They propose a benchmark for multimodal reasoning in large multimodal models, MM-MATH . MM's process evaluation employs LMM-as-a-judge to automatically analyze solution steps . diagram misinterpretation is the most common error, they find .
Outcome:	The proposed model achieves only 31% accuracy, compared to 82% for humans.

Erratum: Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)

Copied to clipboard

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg

Disentangling Memory and Reasoning Ability in Large Language Models (2025.acl-long)

Challenge:	During production of this paper, an error was introduced to the formula on the bottom of the right column of page 1020.
Approach:	the formula was changed in the last two terms of the paper .
Outcome:	the correct formula is now available on the web.

Copied to clipboard

Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, Yongfeng Zhang

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards (2026.findings-acl)

Challenge:	Existing LLMs operate as an opaque process without explicit separation between knowledge retrieval and reasoning steps, making the decision-making process unclear and disorganized.
Approach:	They propose a language model inference paradigm that decomposes the complex inference process into two distinct and clear actions: (1) memory recall: which retrieves relevant knowledge, and (2) reasoning: which performs reasoning steps based on the recalled knowledge.
Outcome:	The proposed paradigm decomposes the inference process into two distinct and clear actions, memory and reason, guiding the model to distinguish between steps that require knowledge retrieval and those that involve reasoning.

Copied to clipboard

Yupeng Chang, Yuan Wu, Yi Chang

Placing M-Phasis on the Plurality of Hate: A Feature-Based Corpus of Hate Online (2022.lrec-1)

Challenge:	Critic-free reinforcement learning with verifiable rewards (RLVR) is a practical paradigm for aligning Large Language Models.
Approach:	They propose a framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments.
Outcome:	Experiments show that RLVR improves training stability and performance compared to critic-based methods . compared with other approaches, RL VR improves in cold-start regimes with binary verifiers .

Copied to clipboard

Dana Ruiter, Liane Reiners, Ashwin Geet D’Sa, Thomas Kleinbauer, Dominique Fohr, Irina Illina, Dietrich Klakow, Christian Schemer, Angeliki Monnier

Ethical NLP

Challenge:	HS-related corpora over-simplify the phenomenon of hate by labelling user content with binary classes, e.g., hate/neutral . this ignores the complex and subjective nature of HS, which limits the real-life applicability of classifiers trained on these corporales.
Approach:	They present a corpus of 9k German and french user comments from migration-related news articles.
Outcome:	The proposed corpus is annotated with 23 features that become descriptors of various types of speech, ranging from critical comments to implicit and explicit expressions of hate.

Knowledge-Augmented Language Model Verification (2023.emnlp-main)

Copied to clipboard

Jinheon Baek, Soyeong Jeong, Minki Kang, Jong Park, Sung Hwang

A Multi-Orthography Parallel Corpus of Yiddish Nouns (2020.lrec-1)

Challenge:	Recent Language Models (LMs) generate factually incorrect answers to queries . authors propose to augment LMs with knowledge retrieved from external source .
Approach:	They propose to augment LMs with knowledge retrieved from external sources . they validate the output and knowledge of the knowledge-augmented LM with a separate verifier .
Outcome:	The proposed model can generate factually incorrect answers on multiple questions . a verifier detects retrieval errors and can correct them by retrieving new knowledge or generating new text .

Copied to clipboard

Jonne Saleva

Challenge:	Existing corpora of Yiddish text are limited to a single, potentially non-standard orthography . non-phonetically spelled Hebrew words are the largest cause of error, according to our study .
Approach:	They propose a multi-orthography parallel Yiddish corpus based on Wiktionary scraping . they also demonstrate how the system can be used to bootstrap a transliteration model .
Outcome:	The proposed system achieves error rates between 16.79% and 28.47% on the test set.

Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes (2024.naacl-long)

Copied to clipboard

Rose Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, Dorottya Demszky

Numerical Reasoning

Challenge:	Our work explores the potential of large language models (LLMs) to close the novice-expert knowledge gap in remediating math mistakes.
Approach:	They propose a method that uses cognitive task analysis to translate an expert’s latent thought process into a decision-making model for remediation.
Outcome:	The proposed model can bridge the novice-expert knowledge gap by using cognitive task analysis to translate an expert’s latent thought process into a decision-making model for remediation.

“All that Glitters”: Techniques for Evaluations with Unreliable Model and Human Annotations (2025.findings-naacl)

Copied to clipboard

Michael Hardy

VariErr NLI: Separating Annotation Error from Human Label Variation (2024.acl-long)

Challenge:	Using standard metrics in the presence of poor labels masks label and model quality . evaluation techniques accounting for unreliable labels reveal important flaws, including spurious correlations and nonrandom racial biases .
Approach:	They analyze human labels, GPT model ratings, and transformer encoder model ratings . they show that standard metrics in the presence of poor labels mask label and model quality .
Outcome:	The proposed methods mask label and model quality even in the presence of poor models.

Copied to clipboard

Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, Barbara Plank

Entity Exchange in the Wild: A Diagnostic Study of LLM Based Real-World Conversational Entity Extraction (2026.acl-industry)

Challenge:	Existing work on label variation and annotation errors has focused on them in isolation.
Approach:	They propose a 2-round annotation procedure to separate human label variation from annotation errors by pairing valid explanations with annotators' validations.
Outcome:	The proposed procedure is based on the NLI task in English and contains 7,732 valid judgements on 1,933 explanations for 500 re-annotated items.

Copied to clipboard

Soumya Jain, Ayush Kumar

Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets (2022.emnlp-main)

Challenge:	Prior work has examined the impact of transcription noise and cross-turn reasoning, but it has not systematically analyzed how entity-exchange phenomena themselves shape extraction performance.
Approach:	They evaluate 16 large language models on 6,387 real-world customer–agent conversations spanning 12 entity types across numeric, alphanumeric, temporal, and free-text categories.
Outcome:	The proposed model improves on the extracted entities across all three axes yielding average gains of up to 6.4% across models.

Copied to clipboard

Philippe Laban, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong

PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning (2024.naacl-long)

Challenge:	Existing methods for evaluating progress in natural language generation tasks are expensive, difficult to reproduce, and non-reusable.
Approach:	They propose a new automatic evaluation method for NLG called Near-Negative Distinction that repurposes prior human annotations into NND tests.
Outcome:	The proposed method achieves higher correlation with human judgments than standard NLG evaluation metrics.

Copied to clipboard

Xuekai Zhu, Biqing Qi, Kaiyan Zhang, Xinwei Long, Zhouhan Lin, Bowen Zhou

Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling (2026.eacl-long)

Challenge:	Large language models excel in various tasks, but their huge size and inaccessibility of parameters present challenges for practical deployment.
Approach:	They propose to use CoT data to distill task-specific ability from large language models to smaller models . they use reasoning programs to suppress errors in distilled data and improve distillation quality .
Outcome:	The proposed model outperforms LLMs on arithmetic reasoning, symbolic reasoning, and general ability.

Copied to clipboard

Benjamin Elder, Anupama Murthi, Jungkoo Kang, Ankita Rajaram Naik, Kiran Kate, Kinjal Basu, Danish Contractor

Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation (2021.findings-emnlp)

Challenge:	Large language models rely on external tools and APIs to perform tasks specified in natural language.
Approach:	They propose a benchmark that transforms SQL queries from BIRD-SQL into executable API sequences.
Outcome:	The proposed benchmark evaluates 10 LLMs and 4 ReACT agents with low task completion rates and 50% task completion rate.

Copied to clipboard

Jarana Manotumruksa, Jeff Dalton, Edgar Meij, Emine Yilmaz

Error Analysis and the Role of Morphology (2021.eacl-main)

Challenge:	Existing models rely on a traditional cross-entropy loss function during training, which may not be optimal for improving the joint goal accuracy.
Approach:	They propose a Turn-based Loss Function that penalises the model if it inaccurately predicts a slot value at the early turns more so than in later turns to improve joint goal accuracy.
Outcome:	The proposed techniques improve the state-of-the-art model by approximately 7-8% relative reduction in error and achieve a new state- of-the art joint goal accuracy with 59.50 and 54.90 on MultiWOZ2.1 and MultiWOz2.2, respectively.

Copied to clipboard

Marcel Bollmann, Anders Søgaard

Quadapter: Adapter for GPT-2 Quantization (2022.findings-emnlp)

Challenge:	Using morphological features does improve error prediction across tasks, but is less pronounced in morphology-complex languages.
Approach:	They propose to use morphological features to improve error prediction across four different tasks and up to 57 languages to test their hypothesis.
Outcome:	The proposed model is more discriminative in morphologically simple languages than in simple ones.

Copied to clipboard

Minseop Park, Jaeseong You, Markus Nagel, Simyung Chang

DRK: Discriminative Rule-based Knowledge for Relieving Prediction Confusions in Few-shot Relation Extraction (2022.coling-1)

Challenge:	Quantization-aware training (QAT) fine-tunes model parameters in response to the numerical error arising from quantization.
Approach:	They propose a quantization adapter that scales activations channel-wise to make activations quantization-friendly by scaling them channel- wise.
Outcome:	The proposed method prevents overfitting and improves quantization performance.

Copied to clipboard

Mengru Wang, Jianming Zheng, Fei Cai, Taihua Shao, Honghui Chen

Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation (2020.coling-main)

Challenge:	Existing methods to identify relation type in low-resource scenario fall into prediction confusions owing to the limited inference ability over shallow text features.
Approach:	They propose a discriminative rule-based knowledge method to identify the relation type between entities in a given text in the low-resource scenario.
Outcome:	The proposed method improves on four types of meta tasks with a 6.0% accuracy gain on average.

Copied to clipboard

Takumi Gotou, Ryo Nagata, Masato Mita, Kazuaki Hanawa

Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction (2021.emnlp-main)

Challenge:	a paper aims to improve performance measures for grammatical error correction . conventional measures treat all errors equally, but some are easier to correct .
Approach:	They propose a way to determine the difficulty of error correction and to motivate researchers . paper examines performance measures for grammatical error correction using a scorer and weighting algorithm .
Outcome:	The proposed measures agree with our intuition of correction difficulty . the results show that the measures are more complex than conventional measures .

Copied to clipboard

Xuming Hu, Chenwei Zhang, Yawen Yang, Xiaohe Li, Li Lin, Lijie Wen, Philip S. Yu

Low-Resource NLP

Challenge:	Existing methods to extract relation facts from limited labeled corpora are laborintensive to obtain . Existing approaches use self-training to generate pseudo labels that will cause gradual drift problem or leverage meta-learning scheme which does not solicit feedback explicitly.
Approach:	They propose a Gradient Imitation Reinforcement Learning method to encourage pseudo label data to imitate gradient descent direction on labeled data and bootstrap its optimization capability through trial and error.
Outcome:	The proposed method handles two major scenarios in low-resource relation extraction when no unlabeled data is available.

Prejudge-Before-Think: Enhancing Large Language Models at Test-Time by Process Prejudge Reasoning (2025.findings-emnlp)

Copied to clipboard

Jianing Wang, Jin Jiang, Yang Liu, Mengdi Zhang, Xunliang Cai

Negative language transfer in learner English: A new dataset (2021.naacl-main)

Challenge:	Experimental results from competition-level complex reasoning demonstrate that bootstrapping with process prejudge can significantly enhance the reasoning ability of LLMs.
Approach:	They propose a new process prejudge strategy for LLM reasoning that bootstraps with process prejudgment .
Outcome:	The proposed method can be bootstrapped with process prejudge in LLM reasoning . it allows the model to anticipate errors rather than relying on trial and error.

Copied to clipboard

Leticia Farias Wanderley, Nicole Zhao, Carrie Demmans Epp

Self-Discriminative Learning for Unsupervised Document Embedding (N19-1)

Challenge:	This dataset contains annotated error causes for learner writing errors that tie learner mistakes to structures from their first language.
Approach:	They propose a learner English dataset enhanced with annotated error causes and concrete examples of learner errors that relate to their first languages.
Outcome:	The proposed dataset will be used to analyze learner errors related to language transfer from the learners’ first language.

Copied to clipboard

Hong-You Chen, Chin-Hua Hu, Leila Wehbe, Shou-De Lin

Low-Resource NLP

Challenge:	Existing methods for document embedding learning do not consider inter-document relationships.
Approach:	They propose to exploit the inter-document information and directly model the relations of documents in embedding space with a discriminative network and a novel objective.
Outcome:	The proposed method has errors that are 5 to 13% lower than state-of-the-art models and is even more pronounced in scarce label setting.

Evidence-based Factual Error Correction (2021.acl-long)

Copied to clipboard

James Thorne, Andreas Vlachos

NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)

Challenge:	Existing methods to correct factual errors are limited to labeled claims . a recent task of fact verification has attracted significant attention .
Approach:	They propose a task of factual error correction that performs edits to a claim so that the generated rewrite is better supported by evidence.
Outcome:	The proposed method produces accurate factual error corrections for 5x more instances in human evaluation and a .125 increase in SARI score.

Copied to clipboard

Abhilasha Ravichander, Siddharth Dalmia, Maria Ryskina, Florian Metze, Eduard Hovy, Alan W Black

Natural Language Interfaces

Challenge:	Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems.
Approach:	They show that component components that precede an answering engine can introduce varied and considerable sources of error.
Outcome:	The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use.

LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)

Copied to clipboard

Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang

A Challenge Set and Methods for Noun-Verb Ambiguity (D18-1)

Challenge:	Current medical benchmarks have limitations in question design, data sources and evaluation methods.
Approach:	They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records .
Outcome:	The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios.

Copied to clipboard

Ali Elkahky, Kellie Webster, Daniel Andor, Emily Pitler

Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing - A Tale of Two Parsers Revisited (D19-1)

Challenge:	English part-of-speech taggers make egregious errors related to noun-verb ambiguity, despite having achieved 97%+ accuracy on the WSJ Penn Treebank since 2002.
Approach:	They propose to use a WSJ dataset to identify 30,000 examples of noun-verb ambiguity . they find that english part-of-speech taggers make egregious errors related to nouns and verbs .
Outcome:	The proposed model improves on the WSJ Penn Treebank by 14% and 52% relative to the previous model.

Copied to clipboard

Artur Kulmizev, Miryam de Lhoneux, Johannes Gontrum, Elena Fano, Joakim Nivre

Representation Learning

Challenge:	In recent years, dependency parsing has shifted from discrete features to neural networks and continuous representations.
Approach:	They propose to use deep contextualized word embeddings to pack information about global sentence structure into local feature representations to make the two approaches virtually equivalent in terms of accuracy and error profile.
Outcome:	The proposed model improves the accuracy and error profile of transition-based and graph-based dependency parsers on 13 languages.

Reanalyzing the Most Probable Sentence Problem: A Case Study in Explicating the Role of Entropy in Algorithmic Complexity (2021.eacl-main)

Copied to clipboard

Eric Corlett, Gerald Penn

Localization of Fake News Detection via Multitask Transfer Learning (2020.lrec-1)

Challenge:	Existing descriptive complexity measures are ineffective at describing algorithms' behaviour, and can make an apparently tractable problem seem NP-complete.
Approach:	They propose to use statistical measures to give an updated analysis of the complexity of the NP-complete most probable sentence problem for pCFGs.
Outcome:	The proposed method can be applied to word sense disambiguation and inference tasks.

Copied to clipboard

Jan Christian Blaise Cruz, Julianne Agatha Tan, Charibeth Cheng

Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models (2024.acl-long)

Challenge:	Existing methods for detecting fake news require large labeled datasets and expert-curated corpora, which low-resource languages may not have.
Approach:	They construct a benchmark dataset for fake news detection in Filipino using curated corpora and transfer learning techniques.
Outcome:	The proposed method can achieve 91% accuracy on a fake news dataset, reducing error by 14% compared to established baselines.

Copied to clipboard

Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji-Rong Wen, Rui Yan, Yongbin Li

Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach (2024.findings-emnlp)

Challenge:	Despite the advances in large language models, they still face difficulties with multi-step reasoning tasks.
Approach:	They propose a method that randomly masks certain tokens within the chain of thought to improve model accuracy by 5% over standard supervised fine-tuning.
Outcome:	The proposed method improves accuracy and accuracy by 5% over standard fine-tuning with a few codes modified.

Copied to clipboard

Dongyue Li, Ziniu Zhang, Lu Wang, Hongyang Zhang

Exploring the Capacity of a Large-scale Masked Language Model to Recognize Grammatical Errors (2022.findings-acl)

Challenge:	a new algorithm to estimate fine-tuning performance for a target task is proposed . conventional subset selection methods require repeated training on subsets of auxiliary tasks .
Approach:	They propose an algorithm to fine-tune a language model for a target task by optimally using auxiliary tasks' information.
Outcome:	The proposed method can estimate fine-tuning performance on CPUs in seconds.

Copied to clipboard

Ryo Nagata, Manabu Kimura, Kazuaki Hanawa

Goal-oriented Vision-and-Dialog Navigation via Reinforcement Learning (2022.findings-emnlp)

Challenge:	a language model-based error detection method can learn errors with a small training sample.
Approach:	They propose a language model-based method for grammatical error detection with feedback comments.
Outcome:	The proposed method can learn errors with a little training data and improve recall faster than non-language models.

Copied to clipboard

Yan Cao, Keting Lu, David DeFazio, Shiqi Zhang

Visual Data in NLP

Challenge:	Existing methods for vision-and-dialog navigation are limited and do not adapt to domain changes.
Approach:	They propose a problem where an agent computes dialog-navigation policies from trial and error.
Outcome:	The proposed agent outperforms baselines in success rate in photo-realistic simulations.

Invertible Tree Embeddings using a Cryptographic Role Embedding Scheme (2020.coling-main)

Copied to clipboard

Coleman Haley, Paul Smolensky

Representation Learning

Challenge:	Unlike previous attempts, this method does not come at the cost of intractable representation size; it works well when there is sufficient randomness in the representation scheme for simple data and providing an upper bound on its error.
Approach:	They propose a method for embedding trees in a vector space based on Tensor-Product Representations (TPRs) that allows for inversion: the retrieval of the original tree structure and nodes from the vectorial embeddment.
Outcome:	The proposed method can provide invertibility with error 1% that previous methods would require 8.6 1057 dimensions to represent.

Where Did That Come From? Sentence-Level Error-Tolerant Attribution (2025.findings-emnlp)

Copied to clipboard

Ori Ernst, Aviv Slobodkin, Meng Cao, Sihui Wei, Jackie CK Cheung

MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators (2025.coling-main)

Challenge:	Existing task definitions exclude unsupported or hallucinated content leaving them unattributed . authors propose a new definition for sentence-level error-tolerant attribution .
Approach:	They propose a new definition for sentence-level error-tolerant attribution that extends attribution to include incorrect or hallucinated content.
Outcome:	The proposed approach reduces annotation time and facilitates hallucination fixing.

Copied to clipboard

Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, Dacheng Tao

Machine Translation

Challenge:	Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment.
Approach:	They propose a framework that automatically post-edits the original translation based on each error, thereby filtering out non-impactful errors.
Outcome:	The proposed framework improves reliability and quality of error spans against GEMBA-MQM, across eight LLMs in both high- and low-resource languages.

What’s under the hood: Investigating Automatic Metrics on Meeting Summarization (2024.findings-emnlp)

Copied to clipboard

Frederic Kirstein, Jan Philip Wahle, Terry Ruas, Bela Gipp

Information Extraction & Text Mining

Challenge:	Existing evaluation metrics do not capture meeting-specific errors, leading to ineffective assessment.
Approach:	They examine the relationship between established metrics and human evaluations to determine what challenges and errors are captured by correlating metric scores with human evaluation.
Outcome:	The proposed measures show weak correlations with human evaluations and a third of the correlations show error masking.

CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues (2025.coling-main)

Copied to clipboard

Sebastian Steindl, Ulrich Schäfer, Bernd Ludwig

Self-Error-Instruct: Generalizing from Errors for LLMs Mathematical Reasoning (2025.acl-long)

Challenge:	Large-scale Wizard-Of-Oz dialogue datasets lack certain types of utterances, which would make them more realistic.
Approach:	They propose to use a large language model to create and repair communication errors in an automatic pipeline.
Outcome:	The proposed method is based on linguistic theory and uses a state-of-the-art Large Language Model (LLM) to create the error and repair it.

Copied to clipboard

Erxin Yu, Jing Li, Ming Liao, Qi Zhu, Boyang Xue, Minghui Xu, Baojun Wang, Lanqing Hong, Fei Mi, Lifeng Shang

On the Robustness of Cognate Generation Models (2022.lrec-1)

Challenge:	Existing approaches to learning from errors synthesize training data by extrapolating from isolated bad cases, thereby failing to generalize the extensive patterns inherent within these cases.
Approach:	They propose a framework that synthesizes more generalized training data from isolated bad cases by extrapolating from isolated cases.
Outcome:	The proposed framework synthesizes more generalized training data to address these model weaknesses.

Copied to clipboard

Winston Wu, David Yarowsky

Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (D19-1)

Challenge:	We examine different types of noise generated by human errors and how these noisy inputs affect the performance of cognate generation models.
Approach:	They evaluate two popular neural cognate generation models’ robustness to human-plausible noise.
Outcome:	The proposed models are robust to deletion, duplication, swapping, keyboard errors, and a new type of error, phonological errors.

Copied to clipboard

Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, Andy Cedilnik

Factual or Contextual? Disentangling Error Types in Entity Description Generation (2023.acl-long)

Challenge:	a lack of high quality conversational data is limiting progress in dialog systems . we present a dataset of 13,215 task-based dialogs .
Approach:	They propose a task-based dialog dataset which includes 13,215 task-related dialogs . they use a two-person, spoken "Wizard of Oz" approach and a "self-dialog" approach .
Outcome:	The taskmaster-1 dataset contains 13,215 task-based dialogs comprising six domains.

Copied to clipboard

Navita Goyal, Ani Nenkova, Hal Daumé III

TGEA: An Error-Annotated Dataset and Benchmark Tasks for TextGeneration from Pretrained Language Models (2021.acl-long)

Challenge:	Existing evaluation practices only distinguish between model generated referring expressions being accurate (ground-truth) versus inaccurate (not groundtruth).
Approach:	They propose to integrate indicators for factual inconsistencies and contextual incongruities into automated evaluations of language models to assess the differences in error types across familiar vs unfamiliar entities.
Outcome:	The proposed evaluation paradigm disentangles factuality and congruity errors in natural contexts.

Copied to clipboard

Jie He, Bo Peng, Yi Liao, Qun Liu, Deyi Xiong

Exploiting Sentence Order in Document Alignment (2020.emnlp-main)

Challenge:	Using pretrained language models, we propose an error-annotated dataset for text generation . we use carefully selected prompt words to guide GPT-2 to generate candidate sentences .
Approach:	They propose an error-annotated dataset with multiple benchmark tasks for text generation from pretrained language models.
Outcome:	The proposed dataset covers 24 types of errors according to common sense and linguistics.

Copied to clipboard

Brian Thompson, Philipp Koehn

Challenging Large Language Models with New Tasks: A Study on their Adaptability and Robustness (2024.findings-acl)

Challenge:	a document alignment method that exploits sentence order information is beneficial even when the end goal is sentence-level bitext.
Approach:	They propose a document alignment method that incorporates sentence order information in both candidate generation and candidate re-scoring.
Outcome:	The proposed method outperforms the most recent document alignment method on Sinhala–English documents.

Copied to clipboard

Chenxi Li, Yuanhe Tian, Zhaxi Zerong, Yan Song, Fei Xia

Characterizing and Measuring Linguistic Dataset Drift (2023.acl-long)

Challenge:	Existing evaluation approaches for large language models (LLMs) rely on existing tasks and benchmarks, raising concerns about test set contamination and the genuine comprehension abilities of LLMs.
Approach:	They propose to evaluate LLMs by designing new tasks, automatically generating evaluation datasets for the tasks, and conducting detailed error analyses to scrutinize LLM's adaptability to new tasks.
Outcome:	The proposed method examines LLMs’ adaptability to new tasks, their sensitivity to prompt variations, and their error tendencies.

Copied to clipboard

Tyler Chang, Kishaloy Halder, Neha Anna John, Yogarshi Vyas, Yassine Benajiba, Miguel Ballesteros, Dan Roth

Traffic Light or Light Traffic? Investigating Phrasal Semantics in Large Language Models (2024.findings-emnlp)

Challenge:	Existing metrics for dataset drift have not considered specific dimensions of linguistic drift that affect model performance.
Approach:	They propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift.
Outcome:	The proposed metrics are more effective than previous metrics at predicting out-of-domain model accuracies compared to popular fine-tuned embedding distances .

Copied to clipboard

Rui Meng, Ye Liu, Lifu Tu, Daqing He, Yingbo Zhou, Semih Yavuz

Selective Question Answering under Domain Shift (2020.acl-main)

Challenge:	Phrases are fundamental linguistic units through which humans convey semantics.
Approach:	They assess the capacity of API-based large language models to comprehend phrase semantics . they use three human-annotated datasets to analyze their results .
Outcome:	The proposed model outperforms embedding-based methods in phrase semantic reasoning tasks . the proposed model does not show significant advantage over fine-tuned methods .

Copied to clipboard

Amita Kamath, Robin Jia, Percy Liang

Natural Language Interfaces

Challenge:	Deployed question answering (QA) models need to know when to abstain from answering questions that diverge from their training data.
Approach:	They propose a selective question answering under domain shift in which a QA model is tested on a mixture of in-domain and out-of-domain data and must answer (i.e., not abstain on) as many questions as possible.
Outcome:	The proposed method answers 56% of questions while maintaining 80% accuracy.

PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion (2024.findings-acl)

Copied to clipboard

Yiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao, Nan Duan

Improving Open Information Extraction via Iterative Rank-Aware Learning (P19-1)

Challenge:	Recent evaluations of Large Language Models (LLMs) focus on their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs.
Approach:	They propose a PowerPoint Task Completion benchmark to assess LLMs’ ability to create and edit PPT files based on user instructions.
Outcome:	The proposed system outperforms open-source and closed LLMs with 75.1% accuracy in single-turn dialogue testing but only achieves 6% session accuracy.

Copied to clipboard

Zhengbao Jiang, Pengcheng Yin, Graham Neubig

Information Extraction & Text Mining

Challenge:	Open information extraction (IE) is the task of extracting open-domain assertions from natural language sentences.
Approach:	They propose an additional binary classification loss to calibrate the extraction likelihood . they propose an iterative learning process where extractions generated by the open IE model are incrementally included as training samples to help the model learn from trial and error.
Outcome:	Experiments on open information extraction (IE) show that the extraction likelihood is not well calibrated when comparing quality of extracted assertions.

FinGrAct: A Framework for FINe-GRrained Evaluation of ACTionability in Explainable Automatic Fact-Checking (2025.findings-emnlp)

Copied to clipboard

Islam Eldifrawi, Shengrui Wang, Amine Trabelsi

The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning (2023.acl-long)

Challenge:	despite the importance of actionability, no prior research has evaluated its effectiveness.
Approach:	They propose a fine-grained evaluation framework that can access the web to assess actionability in AFC explanations.
Outcome:	The proposed framework surpasses state-of-the-art evaluators in achieving highest correlation with human judgments while showing lowest egocentricbias.

Copied to clipboard

Zhuang Li, Lizhen Qu, Philip Cohen, Raj Tumuluri, Gholamreza Haffari

The statistical advantage of automatic NLG metrics at the system level (2021.acl-long)

Challenge:	Prior studies have focused on translating utterances from high-resource languages to low-resourced languages.
Approach:	They propose an active learning approach that exploits the strengths of both human and machine translations by iteratively adding small batches of human translations into the machine-translated training set.
Outcome:	The proposed approach reduces errors and bias in the translated data, resulting in higher parser accuracies than the current model trained on machine translations.

Copied to clipboard

Johnny Wei, Robin Jia

Multi-Agent Collaboration via Cross-Team Orchestration (2025.findings-acl)

Challenge:	Statistically, humans are unbiased, high variance estimators, while metrics are biased, low variance estimator.
Approach:	They compare automatic metrics to humans and a derived, perfect segment-level annotator by applying a bias-variance-noise decomposition to adjust the error to a noise-free, infinite test set setting.
Outcome:	The proposed method outperforms humans and a derived, perfect segment-level annotator in two settings.

Copied to clipboard

Zhuoyun Du, Chen Qian, Wei Liu, Zihao Xie, YiFei Wang, Rennai Qiu, Yufan Dang, Weize Chen, Cheng Yang, Ye Tian, Xuantang Xiong, Lei Han

Challenge:	Large Language Models (LLMs) have significantly impacted various domains, especially through organized LLM-driven autonomous agents.
Approach:	They propose a framework that enables orchestrated teams to jointly propose various task-oriented solutions and interact with their insights in a self-independence while cross-team collaboration environment for superior solutions generation.
Outcome:	Experiments show that the framework can generate better software quality compared to state-of-the-art frameworks.

Linguistically-Informed Self-Attention for Semantic Role Labeling (D18-1)

Copied to clipboard

Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, Andrew McCallum

Pareto Optimal Learning for Estimating Large Language Model Errors (2024.acl-long)

Challenge:	Existing models of semantic role labeling use no explicit linguistic features. prior work has shown that syntax trees can dramatically improve SRL decoding.
Approach:	They propose a neural network model that incorporates syntax using only raw tokens . they show that LISA out-performs the state-of-the-art with contextually-encoded word representations a 1.0 F1 on newswire and 2.0 F1 in out-of domain text .
Outcome:	The proposed model outperforms the state-of-the-art model with word embeddings and predicted predicates.

Copied to clipboard

Theodore Zhao, Mu Wei, J. Preston, Hoifung Poon

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error (2024.acl-long)

Challenge:	Existing approaches for improving the correctness of LLMs include prompt engineering, retrieval methods, and a generative model.
Approach:	They propose a method that generates a risk score to estimate the probability of error in an LLM by integrating multiple sources of information.
Outcome:	The proposed method is well correlated with the true LLM error rate, thus facilitating error correction.

Copied to clipboard

Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su

Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020.coling-main)

Challenge:	Existing work on tool-augmented LLMs focuses on the broad coverage of tools and the flexibility of adding new tools.
Approach:	They propose a biologically inspired method for tool-augmented LLMs that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory.
Outcome:	The proposed method improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and outperforms GPT-4.

Copied to clipboard

Isaac Caswell, Theresa Breiner, Daan van Esch, Ankur Bapna

HiMATE: A Hierarchical Multi-Agent Framework for Machine Translation Evaluation (2025.findings-emnlp)

Challenge:	Large text corpora are increasingly important for a wide variety of NLP tasks.
Approach:	They propose to train automatic language identification models on up to 1,629 languages . they find that human-judged accuracy for web-crawl text corpora is only around 5% for many lower-resource languages.
Outcome:	The proposed models achieve over 90% average F1 on 1,629 languages . human-judged accuracy for web-crawl text corpora is only around 5% for many lower-resource languages - suggesting a need for more robust evaluation.

Copied to clipboard

Shijie Zhang, Renhao Li, Songsheng Wang, Philipp Koehn, Min Yang, Derek F. Wong

Machine Translation

Challenge:	Existing LLM-based evaluation methods fail to accurately identify error spans and assess their severity.
Approach:	They propose a Hierarchical Multi-Agent Framework for Machine Translation Evaluation based on the MQM error typology and a hierarchical multi-agent system enabling granular evaluation of subtype errors.
Outcome:	The proposed framework outperforms baselines in error span detection and severity assessment.

Knowledge-based Consistency Testing of Large Language Models (2024.findings-emnlp)

Copied to clipboard

Sai Rajan, Ezekiel Soremekun, Sudipta Chattopadhyay

It’s Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning (2024.findings-acl)

Challenge:	Large language models (LLMs) are being increasingly utilized in real-world applications.
Approach:	They propose an automated testing framework which leverages a knowledge graph to construct test cases.
Outcome:	The proposed framework generates 19.2% error inducing inputs and reveals 16.5% knowledge gap across all tested LLMs.

Copied to clipboard

Nishant Balepur, Shramay Palta, Rachel Rudinger

Intended Target Identification for Anomia Patients with Gradient-based Selective Augmentation (2024.findings-emnlp)

Challenge:	Recent research aims to unlock the reasoning capabilities of large language models (LLMs) chain-of-thought (COT) prompting can help LLMs reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored.
Approach:	They propose a task where large language models reason toward incorrect answers using chain-of-thought prompting.
Outcome:	The proposed task underperforms the strategy of choosing the correct answer on commonsense and scientific reasoning datasets.

Copied to clipboard

Jongho Kim, Romain Storaï, Seung-won Hwang

Failure makes the agent stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions (2026.findings-acl)

Challenge:	Identifying the intended target item from patient’s circumlocution involves the two challenges of term failure and error.
Approach:	They propose to robustify the model from unseen and SPE terms and enhance it with unseense terms by using gradient-based selective augmentation (GradSelect).
Outcome:	The proposed model outperforms existing models on the Tip of the Tongue dataset and shows that it can handle anomia patients by addressing the outlined challenges.

Copied to clipboard

Junhao Su, Yuanliang Wan, Junwei Yang, Hengyu Shi, Tianyang Han, Yurui Qiu, Junfeng Luo

Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)

Challenge:	Existing approaches to self-reflection rely on heuristic prompting or unidirectional reasoning traces.
Approach:	They propose a structured reflection method that transforms the "from error to repair" process into a first-class, controllable, and trainable action.
Outcome:	The proposed method improves multi-turn tool-call success rates and error recovery while reducing redundant calls.

Copied to clipboard

Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova

Time to Revisit Exact Match (2025.findings-emnlp)

Challenge:	Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling .
Approach:	They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods .
Outcome:	The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds .

Copied to clipboard

Auss Abbood, Zaiqiao Meng, Nigel Collier

Question Answering

Challenge:	Temporal question answering is an established method for evaluating temporal reasoning in large language models.
Approach:	They propose a numerical estimation task where all questions require a numeric, temporal answer, allowing us to evaluate models beyond EM.
Outcome:	The proposed model responses are based on a numerical estimation task and are distilled from Test of Time and TempTabQA.

The Use of Text Alignment in Semi-Automatic Error Analysis: Use Case in the Development of the Corpus of the Latvian Language Learners (L18-1)

Copied to clipboard

Roberts Darģis, Ilze Auziņa, Kristīne Levāne-Petrova

To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation (2024.acl-long)

Challenge:	Using error annotation methods, the corpus of the Latvian language learners can be adapted for other languages with relatively free word order.
Approach:	They propose a method for creating error annotated corpora using text correction, automated morphological analysis, automated text alignment and error annotation.
Outcome:	The proposed method has been approbated in the development of the corpus of the Latvian language learners.

Copied to clipboard

Abdul Waheed, Karima Kadaoui, Muhammad Abdul-Mageed

GMEG-EXP: A Dataset of Human- and LLM-Generated Explanations of Grammatical and Fluency Edits (2024.lrec-main)

Challenge:	Existing models for multilingual automatic speech recognition (ASR) are computationallyintensive and lack proper comprehensive evaluations.
Approach:	They propose to distill knowledge from large teacher models into smaller student variants that are more efficient.
Outcome:	The proposed model outperforms existing models on standard benchmarks and dialectal data.

Copied to clipboard

S. Magalí López Cortez, Mark Josef Norris, Steve Duman

Explainability & Interpretability in NLP

Challenge:	Recent work has explored the ability of large language models (LLMs) to generate explanations of existing labeled data.
Approach:	They propose a dataset to examine the ability of large language models to explain revisions in sentences by comparing human- and LLM-generated explanations of grammatical and fluency edits to a human evaluation criteria.
Outcome:	The proposed explanations address grammatical and fluency edits and are compared with a dataset built from the GMEG (Grammarly Multi-domain Evaluation for GEC) dataset.

Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline (2025.emnlp-main)

Copied to clipboard

Meng Lu, Ruochen Zhang, Carsten Eickhoff, Ellie Pavlick

Why do LLaVA Vision-Language Models Reply to Images in English? (2024.findings-emnlp)

Challenge:	Multilingual large language models (LLMs) exhibit factual inconsistencies across languages . authors identify two primary sources of error: insufficient engagement of reliable English-centric mechanism for factual recall, and incorrect translation from English back into the target language for the final answer.
Approach:	They propose two vector interventions to redirect the model toward better internal paths for higher factual consistency.
Outcome:	The proposed interventions increase the recall accuracy by over 35 percent for the lowest-performing language.

Copied to clipboard

Musashi Hinck, Carolin Holtermann, Matthew Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shao-Yen Tseng, Vasudev Lal

VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Rooms (2025.emnlp-main)

Challenge:	Including an image in a multimodal query significantly increases the likelihood of the model returning an English response regardless of the language of the query.
Approach:	They propose a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models’ internal representations of image and text inputs.
Outcome:	The proposed approach reduces the multilingual error by switching the language backbone for a bilingual language model.

Copied to clipboard

Seungwon Lim, Sungwoong Kim, Jihwan Yu, Sungjae Lee, Jiwan Chung, Youngjae Yu

Cognitive Modeling

Challenge:	Existing studies on embodied agents have addressed the importance of exploration in environments where tasks and solutions are not predefined.
Approach:	They propose a virtual escape room that evaluates AI models in a dynamic environment . they propose to integrate memory management and reasoning into the simulation .
Outcome:	The proposed model improves in dynamic and exploration-driven environments by integrating memory management and reasoning.

Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling (2025.acl-long)

Copied to clipboard

Jiayi Zeng, Yizhe Feng, Mengliang He, Wenhui Lei, Wei Zhang, Zeming Liu, Xiaoming Shi, Aimin Zhou

Can VLMs Recall Factual Associations From Visual References? (2025.findings-emnlp)

Challenge:	Current error-handling works are performed in a passive manner, with explicit error- handling instructions.
Approach:	They propose a new benchmark to analyze LLMs' performance on a mis-prompt benchmark and a dataset to promote further research.
Outcome:	The proposed benchmark shows that current LLMs show poor performance on proactive error handling, and that SFT improves on error handling instances.

Copied to clipboard

Dhananjay Ashok, Ashutosh Chaubey, Hirona Jacqueline Arai, Jonathan May, Jesse Thomason

MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning (2025.emnlp-main)

Challenge:	a systematic deficiency in the multimodal grounding of Vision Language Models is identified . VLMs can recall factual associations when provided a textual reference to an entity .
Approach:	They identify a systematic deficiency in the multimodal grounding of Vision Language Models . they show that VLMs struggle to link their internal knowledge of an entity with its image representation .
Outcome:	The study shows that VLMs struggle to link internal knowledge with image representations . the findings provide recommendations for future research .

Copied to clipboard

Jingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, Han Zhao

LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages (2023.emnlp-main)

Challenge:	Existing reward models assume a global reward function, limiting personalization and pluralistic alignment.
Approach:	They propose a framework that leverages binary preference datasets to enhance personalized preference learning.
Outcome:	The proposed framework captures diverse human preferences without fine-grained annotations and significantly improves personalized preference learning on downstream tasks.

Copied to clipboard

Milind Agarwal, Md Mahfuz Ibn Alam, Antonios Anastasopoulos

When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following (2025.findings-emnlp)

Challenge:	Currently, existing systems cannot accurately identify most of the world's 7000 languages due to lack of data and computational challenges.
Approach:	They propose a misprediction-resolution hierarchical model, LIMIT, that reduces error by 55% on a children's stories dataset and by 40% on 'fLORES-200' benchmark.
Outcome:	The proposed model reduces error by 55% on the MCS-350 and 40% on the FLORES-200 benchmarks.

Copied to clipboard

Keno Harada, Yudai Yamazaki, Masachika Taniguchi, Edison Marrese-Taylor, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo

Learning to love diligent trolls: Accounting for rater effects in the dialogue safety task (2023.findings-emnlp)

Challenge:	a large number of languages are increasingly used to evaluate their ability to follow multiple instructions simultaneously.
Approach:	They propose two benchmarks to evaluate LLMs' ability to follow multiple instructions simultaneously . they use many instruction-following eval and style-aware Mostly Basic programming problems .
Outcome:	The proposed models predict performance on unseen instruction combinations and not used during training with 10% error.

Copied to clipboard

Michael Ilagan

Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts (2025.emnlp-main)

Challenge:	Xu et al., 2018: chatbots generate offensive utterances, which must be avoided . he proposes a solution that can learn from user interactions in a way that is robust to trolls .
Approach:	They propose a method to learn from user feedback in a way that is robust to trolls . they propose multiple users rate each utterance, then perform latent class analysis to infer correct labels.
Outcome:	The proposed solution can infer training labels with high accuracy when trolls are consistent, even when a majority are trolled.

Copied to clipboard

Georgios Chochlakis, Peter Wu, Tikka Arjun Singh Bedi, Marcus Ma, Kristina Lerman, Shrikanth Narayanan

Learning API Functionality from In-Context Demonstrations for Tool-based Agents (2025.findings-emnlp)

Challenge:	Existing approaches to model complex subjective tasks in natural language are limited by significant variation in annotations.
Approach:	They propose a simple in-context learning binary filtering baseline that estimates the reasonableness of a document-label pair.
Outcome:	The proposed approach can be integrated into annotation pipelines to enhance signal-to-noise ratios.

Copied to clipboard

Bhrij Patel, Ashish Jagmohan, Aditya Vempaty

Challenge:	Documentation is often missing, outdated, privatized, or inconsistent in tool-based agents.
Approach:	They propose a task that allows agents to learn API functionality directly from in-context demonstrations.
Outcome:	The proposed model improves the task success rate of an API-based agent by providing explicit function calls and natural language critiques.

Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset (2026.acl-long)

Copied to clipboard

Qian Chen, Xianyin Zhang, Yanzhi Liu, Lifan Guo, Feng Chen, Chi Zhang

Learning From Free-Text Human Feedback – Collect New Datasets Or Extend Existing Ones? (2023.emnlp-main)

Challenge:	Large Vision-Language Models (LVLMs) have expanded capabilities beyond text understanding . a novel Chinese financial multimodal evaluation benchmark is used to evaluate LVLM capabilities .
Approach:	They propose a Chinese financial multimodal evaluation benchmark to evaluate LVLMs' capabilities . the model has an overall accuracy of 66.11% and an average score of 77.18 .
Outcome:	The proposed model achieves an overall accuracy of 66.11% on the question answering task and an average score of 77.18 on detection, recognition, and information extraction tasks.

Copied to clipboard

Dominic Petrak, Nafise Moosavi, Ye Tian, Nikolai Rozanov, Iryna Gurevych

Natural Language Interfaces

Challenge:	Existing datasets for learning from free-text human feedback are scarce.
Approach:	They manually annotate a subset of a popular dialogue dataset with error and user response types using an improved version of the Integrated Error Taxonomy and a newly proposed user response type taxonomies.
Outcome:	The proposed dataset provides new insights into dataset composition, error types, user response types, and the relations between them.

Error Analysis of Multilingual Language Models in Machine Translation: A Case Study of English-Amharic Translation (2024.emnlp-main)

Copied to clipboard

Hizkiel Alemayehu, Hamada Zahera, Axel-Cyrille Ngonga Ngomo

EXPO-SQL: Execution-based Clause-level Policy Optimization for Text-to-SQL (2026.findings-acl)

Challenge:	Multilingual large language models have significantly advanced machine translation, yet challenges remain for low-resource languages like Amharic.
Approach:	They evaluated the performance of NLLB-200 and M2M in English-Amharic bidirectional translation using the Lesan AI dataset.
Outcome:	The proposed models outperformed the existing models in English-Amharic bidirectional translation using the Lesan AI dataset.

Copied to clipboard

Jaehoon Lee, CheolWon Na, Suyoung Bae, Jin-Seop Lee, Jihyung Lee, YunSeok Choi, Jee-Hyong Lee

Programming Languages in NLP

Challenge:	Existing RL methods assign query-level rewards to all clauses, treating correct and incorrect clauses equally.
Approach:	They propose a method which provides fine-grained supervision through clause-level rewards.
Outcome:	Experiments on widely-used Text-to-SQL benchmarks show that EXPO-SqL outperforms existing methods by fine-grained clause-level learning.

Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities (2024.emnlp-main)

Copied to clipboard

Sachit Menon, Richard Zemel, Carl Vondrick

Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models (2025.acl-long)

Challenge:	Large language models have shown promising results in arithmetic and symbolic reasoning by expressing intermediate reasoning in text as a chain of thought, yet struggle to extend this capability to answer text queries that are easily solved by visual reasoning.
Approach:	They propose a method to unlock the visual reasoning capabilities of multimodal large language models by using a metaphorical ‘whiteboard’ to draw out reasoning steps as images and return these images back to the model for further processing.
Outcome:	The proposed method shows that it can be used on four difficult tasks that involve visual and spatial reasoning with no demonstrations or specialized modules.

Copied to clipboard

Junjie Wu, Gefei Gu, Yanan Zheng, Dit-Yan Yeung, Arman Cohan

TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators (2025.findings-acl)

Challenge:	Long-context language models have impressive capabilities in long-contrast understanding tasks, but long-text referencing remains underexplored.
Approach:	They propose a benchmark to assess long-context referencing capability of LCLMs . they use three subsets to test the model's ability to identify key indexes based on contextual relationships .
Outcome:	The proposed benchmark assesses the long-context referencing capability of LCLMs.

Copied to clipboard

Jianling Li, ShangZhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie WangHaojie, Jianrong Wang, Xu Han, Zhiyuan Liu, Maosong Sun

Recognizing Social Cues in Crisis Situations (2024.lrec-main)

Challenge:	Triton is a high-level Python-like programming language for building efficient GPU kernels.
Approach:	They propose a TritonBench benchmark that provides a comprehensive evaluation of Tritonic operators on widely deployed GPUs.
Outcome:	The proposed benchmarks show that current LLMs struggle to generate efficient Triton operators on widely deployed GPUs aligned with industry applications.

Copied to clipboard

Di Wang, Yuan Zhuang, Ellen Riloff, Marina Kogan

Text Classification

Challenge:	During natural disasters, observations of other people's behavior can play an essential role in a person's decision-making.
Approach:	They propose a task to categorize social cues in tweets during crisis situations using an annotated dataset of 6,000 tweets.
Outcome:	The proposed task is challenging for existing systems and a manual task is based on a dataset of 6,000 tweets labeled with eight social cue categories.

Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement (2025.findings-acl)

Copied to clipboard

Xiaoqing Zhang, Yuhan Liu, Flood Sung, Xiuying Chen, Shuo Shang, Rui Yan

Programming Languages in NLP

Challenge:	Recent research indicates that large language models (LLMs) have demonstrated remark-able capabilities in various programming-related domains, such as code generation and code refinement.
Approach:	They propose a framework that combines exploration with refinement to reduce test-time computation overhead.
Outcome:	The proposed framework outperforms SOTA and AgentCoder on humanEval and MBPP benchmarks while reducing test-time computation overhead and scalability.

Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets (2025.acl-long)

Copied to clipboard

Dongyue Li, Ziniu Zhang, Lu Wang, Hongyang R. Zhang

The Sonar Moment: An Audio Geo-Localization Benchmark for Audio-Language Models (2026.findings-acl)

Challenge:	Existing methods for fine-tuning language models are efficient when adapting to a single dataset.
Approach:	They propose to use an ensemble method for fine-tuning a language model to multiple datasets instead of a single adapter per task.
Outcome:	The proposed method improves performance on multiple datasets while preserving low-rank adaptation properties.

Copied to clipboard

Ruixing Zhang, Zihan Liu, Leilei Sun, Tongyu Zhu, Weifeng Lv

To Err Is Human, How about Medical Large Language Models? Comparing Pre-trained Language Models for Medical Assessment Errors and Reliability (2024.lrec-main)

Challenge:	AGL1K is the first audio geo-localization benchmark for audio language models (ALMs) it is based on a crowd-sourced platform and is available in 72 countries and territories.
Approach:	They propose a benchmark for audio geo-localization that quantifies the informativeness of each recording and a metric that quantizes the information of each audio clip.
Outcome:	The proposed benchmarks cover 72 countries and territories and can be used to improve audio geo-localization.

Copied to clipboard

Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Meliha Yetisgen

Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decoding (2025.acl-long)

Challenge:	a 1999 report found that at least forty thousand deaths are a result of preventable medical errors.
Approach:	They test pre-trained language models to characterize their error generation and reliability in medical assessment ability.
Outcome:	The results show that pre-trained models can generate errors and perform better than human models.

Copied to clipboard

Hidetaka Kamigaito, Hiroyuki Deguchi, Yusuke Sakai, Katsuhiko Hayashi, Taro Watanabe

The Retrieval Bottleneck: Scaling Laws for Reinforcement Learning in RAG (2026.acl-long)

Challenge:	Existing studies show that MBR decoding improves model generation performance . however, the theoretical underpinnings of these results remain uncertain .
Approach:	They propose a theoretical interpretation of MBR decoding from the perspective of bias–diversity decomposition.
Outcome:	The proposed method improves the quality estimation of hypotheses by decomposing bias and diversity into two main factors.

Copied to clipboard

Shu Zhou, Jinman Leng, Yufei Song, Xin Wang, Tao Fan, Hao Wang

Information Retrieval

Challenge:	Retrieval-augmented generation (RAG) has become the dominant paradigm for building knowledge-intensive language systems.
Approach:	They propose a sigmoidal scaling law that shows that retrieval quality determines the asymptotic performance ceiling.
Outcome:	The proposed model achieves strong performance on knowledge-intensive benchmarks while retaining the predictable scaling long available for pre-training but previously absent in RAG-RL.

Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs (2025.acl-long)

Copied to clipboard

Xiang Zhang, Juntai Cao, Chenyu You, Dujian Ding

Low-Resource NLP

Challenge:	Chain-of-Thought (CoT) prompting has emerged as a practical workaround, but most CoT-based methods rely on a single generic prompt like “think step by step” with no task-specific adaptation.
Approach:	They propose a theoretical framework that explains why some prompts succeed while others fail by using a generic generic prompt like "think step by step" they show that prompts function as selectors, extracting specific task-relevant information from the model's full hidden state during CoT reasoning.
Outcome:	The proposed framework explains why some prompts succeed while others fail.

CURA: Clinical Uncertainty Risk Alignment for Language Model–Based Risk Prediction (2026.acl-long)

Copied to clipboard

Sizhe Wang, Ziqi Xu, Claire Najjuuko, Charles Alba, Chenyang Lu

AccessEval: Benchmarking Disability Bias in Large Language Models (2025.emnlp-main)

Challenge:	Clinical language models (LMs) are increasingly applied to support clinical risk prediction from free-text notes, yet their uncertainty estimates are poorly calibrated and clinically unreliable.
Approach:	They propose a framework that aligns clinical LM-based risk estimates and uncertainty with individual error likelihoods and cohort-level ambiguities.
Outcome:	The proposed framework improves accuracy on clinical risk prediction tasks without compromising discrimination.

Copied to clipboard

Srikant Panda, Amit Agarwal, Hitesh Laxmichand Patel

METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models (2026.acl-long)

Challenge:	Large Language Models exhibit disparities in how they handle real life queries.
Approach:	They propose a large-scale benchmark to evaluate large language models across six real-world domains and nine disability types.
Outcome:	The proposed model outputs show higher factual error, more negative tone, and increased stereotyping with social perception compared to neutral queries.

Copied to clipboard

Pengfeng Li, Chen Huang, Chaoqun Hao, Hongyao Chen, Xiao-Yong Wei, Wenqiang Lei, See-Kiong Ng

Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models (2026.findings-acl)

Challenge:	Existing benchmarks evaluate contextual causal reasoning in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy.
Approach:	They use a unified context to benchmark large language models' contextual causal reasoning skills.
Outcome:	The proposed benchmarks show that LLMs are susceptible to distraction by irrelevant but factually correct information at lower level of causality.

Copied to clipboard

Felix Herron, Solange Rossato, Alexandre Allauzen, François Portet

Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning (2026.findings-acl)

Challenge:	a number of studies have been done to improve ASR for speaker groups, but there is still room for improvement . authors propose a framework typifying two types of error in phoneme embeddings .
Approach:	They propose a framework typifying two types of error that can occur in phoneme modeling . they propose random error/high variance in phonemes embedding vs systematic error/embedding bias .
Outcome:	The proposed framework typifies errors in phoneme modeling in ASR systems . it shows that training only on a single, typically disadvantaged SG improves performance .

Copied to clipboard

Haolin Liu, Dian Yu, Sidi Lu, Yujun Zhou, Rui Liu, Zhenwen Liang, Haitao Mi, Chen-Yu Wei, Dong Yu

LLMs as Lab Engineers: A Benchmark for Analytical Method Lifecycle Management (2026.findings-acl)

Challenge:	Existing reinforcement learning methods rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.
Approach:	They propose a process reward model that rewards correct steps only when they detect errors . they propose VPPO, which rewards the correct prefix and an erroneous suffix .
Outcome:	a new approach outperforms sparse-reward RL and prior PRM-guided baselines on Pass@1 and Pass@K . a process reward model (PRM) outperformed sparser-rebound RL on multiple reasoning benchmarks .

Copied to clipboard

Xiaoyi Chen, Mahsa Monshizadeh, Chaoqi Zhang, Jianjun Lang, Yang Wu, Genevieve Mortensen, Xiaozhong Liu, Haixu Tang

Benchmarking Fine-Grained Error Detection in Multimodal Reasoning (2026.acl-long)

Challenge:	General-purpose commercial models outperform domain-specialized ones, while RAG and reasoning significantly improve performance.
Approach:	They propose a benchmark to evaluate LLMs' capabilities in analytical chemistry scenarios.
Outcome:	The proposed framework outperforms existing benchmarks focused on factual knowledge and provides practical guidance for analytical chemistry challenges.

Copied to clipboard

Chi-Min Chan, Han Zhu, Chunyang Jiang, Jiaming Ji, Juntao Dai, Wei Xue, Sirui Han, Yike Guo

MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models (2026.acl-long)

Challenge:	Multimodal Process Reward Models (MPRMs) have emerged as a pivotal framework for enhancing the reasoning capabilities of Multimodal Large Language Models.
Approach:	They propose a benchmark specifically designed to evaluate MPRMs’ proficiency in detecting erroneous reasoning steps across diverse error categories.
Outcome:	The proposed model achieves up to 4.8% performance improvement through test-time scaling.

Copied to clipboard

Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu, Zhiqi Huang