Papers with agreement
Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing studies have shown that transformers are able to predict subject-verb agreement, demonstrating their ability to uncover an abstract representation of the sentence in an unsupervised way. |
| Approach: | They propose to compare how transformers handle subject-verb and object-past participle agreements in French using probing and counterfactual analysis methods. |
| Outcome: | The proposed model handles subject-verb and object-past participle agreements in a way consistent with their modeling in theoretical linguistics. |
Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT (2022.coling-1)
Copied to clipboard
| Challenge: | Existing research shows that humans are prone to making agreement errors with specific constructions. |
| Approach: | They compare the performance of BERT-base and that of humans using crowdsourcing . they find that meaningfulness is stronger for BERT than for humans . |
| Outcome: | The proposed model performs better than humans on a crowdsourcing experiment . |
Different types of syntactic agreement recruit the same units within large language models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models can reliably distinguish grammatical from ungrammatically sentences, but how gramatical knowledge is represented within the models remains an open question. |
| Approach: | They use a functional localization approach inspired by cognitive neuroscience to identify the LLM units most responsive to 67 English syntactic phenomena in seven open-weight models. |
| Outcome: | The proposed model is most responsive to 67 English syntactic phenomena and consistently supports model performance. |
CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents (2026.acl-demo)
Copied to clipboard
| Challenge: | Automated Scientific Discovery (ASD) systems rely on parametric knowledge to generate and run code-based experiments. |
| Approach: | They propose a system that distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples. |
| Outcome: | The proposed system produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples. |
Modelling Narrative Elements in a Short Story: A Study on Annotation Schemes and Guidelines (2020.lrec-1)
Copied to clipboard
| Challenge: | Text-processing algorithms that annotate main components of a story are in great need of corpora and well-agreed annotation schemes. |
| Approach: | They propose a model that generalizes a narrative structure in the form of world building elements (characters, time and space) and text worlds themselves and switches between them. |
| Outcome: | The proposed model can be used for annotating narratives in corpora of literary texts, criminal evidence, teaching materials, quests, etc. |
Decision Conversations Decoded (N18-5)
Copied to clipboard
| Challenge: | a system that tracks the decision process is aimed at group decision making facilitation . the system tracks the options being considered, why they are proposed, by whom and with whose support. |
| Approach: | They propose a system that tracks the decision process and organizes collective thoughts into a summary . the system is based on the scientific field of Decision Analysis . |
| Outcome: | The proposed system can help identify agreement and dissent or recommend an alternative based on this information. |
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution (2023.findings-eacl)
Copied to clipboard
Ankita Gupta, Marzena Karpinska, Wenlong Zhao, Kalpesh Krishna, Jack Merullo, Luke Yeh, Mohit Iyyer, Brendan O’Connor
| Challenge: | Existing datasets vary in definition of coreferences and are curated for linguistic experts. |
| Approach: | They propose to use ezCoref to create a crowdsourcing-friendly coreference annotation methodology that teaches annotators only cases that are treated similarly across existing datasets. |
| Outcome: | The proposed method reannotates 240 passages from seven existing english coreference datasets while teaching annotators only cases that are treated similarly across them. |
Three Dimensions of Reproducibility in Natural Language Processing (L18-1)
Copied to clipboard
K. Bretonnel Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany Callahan, Orin Hargraves, Foster Goss, Nancy Ide, Aurélie Névéol, Cyril Grouin, Lawrence E. Hunter
| Challenge: | a recent editorial on reproducibility in language processing defined three dimensions of reproducibility . authors had already submitted a correction, but there is no consensus on the definitions . |
| Approach: | They propose an ontology of reproducibility in natural language processing to address these problems . they propose to analyze three dimensions of reproducible in natural languages papers . authors propose to use a 'replicability' term to describe the reproducibility of a conclusion, finding, value . |
| Outcome: | The proposed ontology aims to enhance future research and communication about the topic and retrospective meta-analyses. |
A Benchmark and Evaluation of Automated Language of Study Extraction from Computational Linguistics Publications (2026.eacl-srw)
Copied to clipboard
| Challenge: | Language of study extraction is an aspect of computational linguistics papers that is useful for analyses of trends and diversity in computational linguists. |
| Approach: | They propose to benchmark and evaluate automated language of study extraction from computational linguistics papers. |
| Outcome: | The proposed language extraction benchmarks show that they can extract languages from papers with accuracy without high computational costs. |
Quality Estimation for Partially Subjective Classification Tasks via Crowdsourcing (2020.lrec-1)
Copied to clipboard
| Challenge: | a common approach to quality estimation is to ask multiple reviewers to evaluate the same artifacts. |
| Approach: | They propose a probabilistic model for subjective classification tasks that incorporates the qualities of artifacts as well as the abilities and biases of creators and reviewers as latent variables to be jointly inferred. |
| Outcome: | The proposed model estimates the quality of speech more effectively than a vote aggregation, measured by correlation with a fine-grained classification by experts. |
UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu (2026.findings-acl)
Copied to clipboard
| Challenge: | Evaluating how large language models capture grammatical structure of low-resource languages remains underexplored. |
| Approach: | They evaluate a set of 5,696 minimal pairs that contrast grammatical acceptability across ten core syntactic and morpho-syntactical phenomena in Urdu. |
| Outcome: | The proposed framework compares multilingual models with the proprietary model . the proposed framework achieves the highest average accuracy on regular phenomena . |
Evaluating Style Transfer for Text (N19-1)
Copied to clipboard
| Challenge: | Existing studies on style transfer for text are lacking a standard set of evaluation practices. |
| Approach: | They propose a set of metrics for automated evaluation that are more strongly correlated with human judgment and show tradeoffs between aspects of interest. |
| Outcome: | The proposed models exhibit tradeoffs between aspects of interest and human judgment, demonstrating the importance of evaluating them at specific points of their tradeoff plots. |
CaBSALLM: Efficient Context-Aware Batch Annotation of Conversational Streams with Large Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Large-scale annotations of subjective, discourse-dependent social interactions remain a critical bottleneck in computational social science. |
| Approach: | They propose a pipeline that incorporates lightweight conversational context and a dynamic batching method to improve throughput and scalability. |
| Outcome: | The proposed pipeline improves throughput and scalability while preserving interpretive depth essential to complex social annotations. |
Language Models (Mostly) Do Not Consider Emotion Triggers When Predicting Emotion (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing work has sought to identify what triggers or causes a particular emotion, but the relationship between those triggers and the prediction of emotion detection models is little understood. |
| Approach: | They propose a dataset to evaluate the ability of large language models to identify emotion triggers . they compare features considered important for emotion prediction models to those considered less salient . |
| Outcome: | The proposed dataset compares large language models and fine-tuned models on social media posts . it shows that emotion triggers are not considered salient features for emotion prediction models . |
Persona Prompting as a Lens on LLM Social Reasoning (2026.eacl-long)
Copied to clipboard
Jing Yang, Moritz Hechtbauer, Elisabeth Khalilov, Evelyn Luise Brinkmann, Vera Schmitt, Nils Feldhus
| Challenge: | Persona prompting (PP) is increasingly used to steer large language models towards user-specific generation, but its effect on rationales remains underexplored. |
| Approach: | They examine how LLM-generated rationales vary when conditioned on different demographic personas . they use word-level rationale annotations to measure agreement with human annotations based on PP . |
| Outcome: | The proposed model improves classification on the most subjective task, but fails to align with real-world demographic counterparts. |
PhyVer: Physics-Grounded Material Claim Verification with Multi-Fidelity Physical Evidence (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing claim verification pipelines operate over text, producing ungrounded judgments. |
| Approach: | They propose a physics-grounded material claim verification system that can be used to verify claims with physical evidence. |
| Outcome: | The proposed system reduces MAE and sign-offs with experts over text-only GPT-5.1. |
Revisiting Evaluation of Question Answering Systems in Low-Resource Indic Languages: Bridging Human and Metric Alignment (2026.acl-short)
Copied to clipboard
| Challenge: | Evaluating Question Answering systems in low-resource Indic languages remains challenging due to the scarcity of annotated data and the lack of reliable evaluation metrics. |
| Approach: | They propose a language-based multi-aspect evaluation framework for question answering systems . the framework integrates semantic similarity, factual completeness, numerical accuracy and contextual relevance . |
| Outcome: | The proposed metric is evaluated across eight Indic-language QA tasks using multiple LLMs . Across all settings, it shows stronger agreement with human evaluation . |
Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement (2026.acl-industry)
Copied to clipboard
Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma, Kosuke Takahashi, Takahiro Omi, Kosuke Arima, Takuto Asakura, Chung-Chi Chen, Tatsuya Ishigaki
| Challenge: | Large language models (LLMs) make it easy to generate large numbers of product ideas. |
| Approach: | They propose to use a dataset of 3,000 individual scores across 300 patent-grounded product ideas to assess whether an automatic judge approximates an aggregate consensus. |
| Outcome: | The proposed model evaluators disagree on fine-grained ordinal scores, suggesting structured heterogeneity rather than random noise. |
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress (2025.acl-short)
Copied to clipboard
| Challenge: | In machine translation evaluation, metric performance is assessed based on agreement with human judgments. |
| Approach: | They incorporate human baselines into the MT meta-evaluation to gain a clearer understanding of metric performance and establish an upper bound. |
| Outcome: | The results suggest human parity, but there are several reasons to caution . |
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are prone to hallucination and rely on static, pre-annotated references for evaluation. |
| Approach: | They propose a framework to assess large language models without fixed ground-truth answers by iteratively generating web queries and synthesizing external evidence. |
| Outcome: | The proposed framework achieves substantial to perfect agreement with human evaluations on multiple free-form QA benchmarks. |
Estimating User Communication Styles for Spoken Dialogue Systems (2020.lrec-1)
Copied to clipboard
| Challenge: | a neural network estimation system for spoken dialogues can be used to estimate the communication style of a user's interaction, but this is rarely implemented in a live system. |
| Approach: | They propose a neural network approach to estimate the communication style of spoken interaction, namely elaborateness and directness. |
| Outcome: | The proposed method can estimate the elaborateness and directness of spoken interaction and improve the results with additional linguistic features. |
Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often produce inconsistent answers when faced with different phrasings of the same prompt. |
| Approach: | They propose an unsupervised method that flips the phrasings of prompts into a hard pseudo-label . they use Consensus Cross-Entropy to create a consensus, and representation alignment loss to pull lower-confidence predictors toward consensus . |
| Outcome: | The proposed method raises observed agreement by 11.62% and improves mean F1 by 8.94% on 11 datasets spanning four NLP tasks . |
Criteria for the Annotation of Implicit Stereotypes (2022.lrec-1)
Copied to clipboard
| Challenge: | social media has brought with it a massive channel for spreading and reinforcing stereotypes . most stereotypes are expressed implicitly and identifying them automatically remains a challenge . |
| Approach: | They propose criteria to facilitate the subjective task of identifying the presence of stereotypes . they propose a newsCom-Implicitness corpus of 1,911 sentences, of which 426 are explicit and implicit racial stereotypes. |
| Outcome: | The proposed criteria show that they obtain different inter-annotator agreement values . the criteria are applied to a corpus of 1,911 sentences, of which 426 are explicit and implicit racial stereotypes . |
Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning? (2025.acl-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | Large Vision-Language Models (LVLMs) are expensive and time-consuming to evaluate . however, they are limited in their use in industrial settings due to their limited availability and limited resources. |
| Approach: | They evaluate 13 open-source LVLMs as judges for diverse chart comprehension and reasoning tasks. |
| Outcome: | The proposed models can be used to assess chart comprehension and reasoning tasks, but they are expensive and time-consuming. |
Beyond Accuracy: Alignment and Error Detection across Languages in the Bi-GSM8K Math-Teaching Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in LLMs have significantly improved mathematical problem-solving, with models like GPT-4 achieving human-level performance. |
| Approach: | They propose a bilingual English-Korean dataset enriched with teacher solutions, student solutions, and annotations marking students’ initial errors. |
| Outcome: | The proposed model achieves high agreement with human judgments and lower latency and resource usage than commercial APIs, demonstrating strong computational efficiency. |
LLM-Based Zero-Shot Soft Labeling for Anticipating Disagreement in Negotiation Dialogues (2026.acl-srw)
Copied to clipboard
| Challenge: | Negotiation involves complex emotional and strategic dynamics that pose challenges for AI agents in negotiation dialogues. |
| Approach: | They propose a zero-shot soft-labeling method using large language models . they also examine the performance of model training on rule-based annotated hard and soft labels . |
| Outcome: | The proposed method shows a maximum HIT@3 score of 0.87 against rule-based annotated hard labels . failure cases also demonstrated the limitations of rule--based annotation . |
Gaining and Losing Influence in Online Conversation (L18-1)
Copied to clipboard
| Challenge: | a study aimed to determine if people who are influential in online discussions retain influence when placed in a topic that is less familiar or perhaps not as interesting. |
| Approach: | They conducted a study to determine if people who are highly influential retain influence when moving to a topic that is less familiar or perhaps not as interesting. |
| Outcome: | The results show that people who are highly influential in group discussions lose influence when placed in a topic that is less familiar or perhaps not as interesting. |
Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation (2026.acl-industry)
Copied to clipboard
| Challenge: | In practice, LLMs are largely diagnostic, with the signals rarely translating into direct quality improvements under real production constraints. |
| Approach: | They propose a two-stage, evaluator-guided automatic post-editing framework that turns MQM-style evaluation into targeted repairs. |
| Outcome: | The proposed framework improves both COMET and CometKiwi scores over one-stage evaluation methods while severities and error spans show strong agreement with human annotations and human editor preferences. |
CMA-R: Causal Mediation Analysis for Explaining Rumour Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on explainable fake news or rumour detection by and large use attention weights as explanation, but the use of attention weighted explanations is problematic. |
| Approach: | They propose a causal mediation analysis approach to explain the decision-making process of neural models for rumour detection on Twitter by identifying salient tweets that explain model predictions and highlighting causally impactful words in the tweets. |
| Outcome: | The proposed approach shows strong agreement with human judgements for critical tweets determining the truthfulness of stories. |
Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback (2026.eacl-long)
Copied to clipboard
| Challenge: | Novelty assessment is a central yet understudied aspect of peer review . manuscript submissions double roughly every 15 years, and individual reviewers now complete an average of 14 reviews per year. |
| Approach: | They propose a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages: content extraction, retrieval and synthesis of related work, and structured comparison for evidence-based assessment. |
| Outcome: | The proposed approach outperforms existing LLM-based baselines on 182 ICLR 2025 submissions with human-annotated reviewer novelty assessments. |
Evaluating Text Generation Quality Using Spectral Distances of Surprisal (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metric fails to capture text surprisal, but FACE-2 produces stronger agreement with human preferences. |
| Approach: | They propose a new automatic evaluation metric for open-ended text generation . they propose metric that extracts the dynamic patterns (spectrum) of text surprisal . |
| Outcome: | The proposed metric outperforms existing methods in revealing the model scaling effect . it produces stronger agreement with human preferences from a large human-annotated dataset . |
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (P18-1)
Copied to clipboard
| Challenge: | a recent study found that language models fail to learn long-range syntax sensitive dependencies. |
| Approach: | They propose to use a subject-verb agreement diagnostic to determine whether language models can learn long-range syntax sensitive dependencies. |
| Outcome: | The proposed model outperforms left-corner and bottom-up variants in learning non-local dependencies. |
What Speakers really Mean when they Ask Questions: Classification of Intentions with a Supervised Approach (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing work on hidden intentions of speakers in questions during meals is based on written or oral data, which are less easy to interpret. |
| Approach: | They propose a typology of hidden intentions in questions asked during meals . they implement an automatic classification model based on annotated data and selected linguistic features. |
| Outcome: | The proposed model is based on annotated data and features and evaluates its performance. |
Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Model (LLM) judges are limited to textual content, resulting in expensive and opaque evaluation methods. |
| Approach: | They propose a framework that enables large language model judges to reason over audio cues . they introduce a human chain-of-thought annotation protocol to improve judge diagnostic capability . |
| Outcome: | The proposed framework achieves higher agreement with human raters than ALMs and transcript-only LLM judges while being significantly more cost-effective. |
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to condition models on a concise rationale are less accurate than models that can use the entire context. |
| Approach: | They propose a method to optimize a bound on the Information Bottleneck objective to extract concise rationales from a binary mask and an end-task predictor that uses only the residual sentences. |
| Outcome: | The proposed model outperforms existing norm-minimization techniques in task performance and agreement with human rationales in the ERASER benchmark. |
Measuring Uncertainty in Translation Quality Evaluation (TQE) (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing automated tools are not good enough to evaluate translation quality . existing tools are often accused of having low reliability and agreement . |
| Approach: | They propose to use a method to accurately estimate the confidence intervals depending on the sample size of the translated text. |
| Outcome: | The proposed method aims to estimate the confidence intervals (CITATION) depending on the sample size of the translated text, e.g. the amount of words or sentences, that needs to be processed on TQE workflow step for confident and reliable evaluation of overall translation quality. |
Generating Spatial Knowledge Graphs from Automotive Diagrams for Question Answering (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Useful answers require obvious landmarks as a reference point . a decomposed pipeline is the most effective strategy for generating a high-quality SKG . |
| Approach: | They propose to generate a spatial knowledge graph from a vehicle dashboard diagram . they use large vision-language models to generate the graph using a decomposed pipeline . |
| Outcome: | The proposed method identifies landmarks with 71.3% agreement with human annotators on a new vehicle dataset. |
How Does Response Length Affect Long-Form Factuality (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite growing attention to LLM factuality, the effect of response length on factual accuracy remains underexplored. |
| Approach: | They propose an automatic and bi-level long-form factuality evaluation framework which achieves high agreement with human annotations while being cost-effective. |
| Outcome: | The proposed framework achieves high agreement with human annotations while being cost-effective. |
Learning Multilingual Agentic Policy to Control Sycophancy (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are effective at adapting to users’ styles, preferences, and contextual signals, but can manifest as sycophancy, i.e., alignment with user-implied beliefs or assumptions even when these contradict factual correctness, uncertainty, or proper logical reasoning. |
| Approach: | They propose to use large language models to model sycophancy as a decision-making problem by learning agentic policies that are trained to optimise a multi-objective reward that balances task success, scophancies resistance and behavioural consistency. |
| Outcome: | The proposed model equips a model with an explicit action space that includes answering directly, countering misleading signals, or asking for clarification. |
Crossing the Reward Bridge: Expanding Reinforcement Learning with Verifiable Rewards Across Diverse Domains (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) has been effective on structured tasks, but its reliance on simple, rule-based verifiers creates a bottleneck. |
| Approach: | They propose a framework that uses a generative verifier to provide soft, probabilistic rewards. |
| Outcome: | The proposed framework outperforms existing models up to 10x their size and can be scalable and effective. |
Can Unconfident LLM Annotations Be Used for Confident Conclusions? (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. |
| Approach: | They propose a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations to use. |
| Outcome: | The proposed method produces accurate estimates and valid confidence intervals while reducing the number of human annotations by over 25%. |
Work Hard, Play Hard: Collecting Acceptability Annotations through a 3D Game (2022.lrec-1)
Copied to clipboard
| Challenge: | Corpus-based studies on acceptability judgements have always been popular thanks to the release of the CoLA corpus, a large-scale corpus of sentences extracted from linguistic handbooks as examples of acceptable/non acceptable phenomena in English. |
| Approach: | They present a 3D video game that was used to collect acceptability judgments on italian sentences and compare them with experts’ acceptability judgements. |
| Outcome: | The proposed game compares the annotations of Italian sentences with those of experts and shows that they are more reliable than crowd-sourced annotations. |
A Comparison Of Emotion Annotation Schemes And A New Annotated Data Set (L18-1)
Copied to clipboard
| Challenge: | a series of study on positive/negative sentiments has been conducted on tweets, but recognition of more nuanced affect has received little attention . valence, arousal, dominance and surprise are the most commonly used emotion representation schemes . |
| Approach: | They propose to annotate tweets with scores on four emotion dimensions . they compare annotator agreement with relative annotation schemes over categorical ones . |
| Outcome: | The proposed model improves agreement with relative annotation schemes over categorical ones on Ekman's six basic emotions. |
Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for Sequence Labeling require high-quality annotations, but imperfect annotations are relatively easy to obtain from crowdsourcing (noisy labels) Existing approaches to learn a model without knowing the underlying ground truth label sequences in the target domain are expensive and time-consuming. |
| Approach: | They propose a framework Consensus Network that can be trained on annotations from multiple sources. |
| Outcome: | The proposed framework improves on learning with crowd annotations and unsupervised cross-domain model adaptation in two practical settings. |
HateXScore: A Metric Suite for Evaluating Reasoning Quality in Hate Speech Explanations (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks do not assess why a text is deemed hateful . authors present a new metric to evaluate the reasoning quality of model explanations . |
| Approach: | They propose a metric suite to evaluate the reasoning quality of model explanations. |
| Outcome: | The proposed metric validates it as a practical tool for trustworthy and transparent moderation on six diverse hate speech datasets. |
A Multi-Task Learning Framework for Multi-Target Stance Detection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models fail to learn target-specific representations and are prone to overfitting. |
| Approach: | They propose a multi-task learning network to train one model on all target pairs . their results show that their proposed model outperforms the best-performing baseline by 12.39% . |
| Outcome: | The proposed model outperforms the best-performing baseline model by 12.39% in macro-averaged F1-score. |
Quantifying Qualitative Data for Understanding Controversial Issues (L18-1)
Copied to clipboard
| Challenge: | 'Controversy' is a state of sustained public debate on a topic or issue that evokes conflicting opinions, beliefs, claims, arguments, and points of view. |
| Approach: | They propose a crowdsourced approach to quantifying qualitative information on controversial issues by analyzing crowdsourced assertions in social media. |
| Outcome: | The proposed dataset consists of over 2,000 assertions on 16 controversial issues. |
FLEX: Expert-level False-Less EXecution Metric for Text-to-SQL Benchmark (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for text-to-SQL systems show many false positives and negatives . however, the Execution Accuracy (EX) metric is flawed and can diverge from human experts. |
| Approach: | They propose a method to evaluate text-to-SQL systems using large language models to emulate human expert-level evaluation of SQL queries. |
| Outcome: | The proposed metric improves agreement with human experts with comprehensive context and sophisticated criteria. |
A Reassessment of Reference-Based Grammatical Error Correction Metrics (C18-1)
Copied to clipboard
| Challenge: | Existing studies on the correlation of GEC metrics with human judgments were inconclusive . a recent study found that GLEU produces counter-intuitive scores in common test examples . |
| Approach: | They propose to use GLEU to evaluate grammatical error correction (GEC) systems . they also use statistical significance tests to assess their agreement with human judgments . |
| Outcome: | The proposed metrics show no significant advantage over MaxMatch (GLEU) the results contradict previous studies that claim GLEU superior . |
Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent methods have obtained promising results by extracting relation labels from participants . obtaining linguistic annotations from novice crowdworkers is difficult . crowdsourcing allows for fast and cost-effective collection of labelled data, but because tasks need to be intuitive, crowdworker cannot be asked to perform them. |
| Approach: | They propose to use a selection-only approach to obtain linguistic annotations from novices . current study shows that the method is cost- and time-intensive . |
| Outcome: | The current study shows that selection and training improves the agreement between workers and gold labels, but the method is cost- and time-intensive. |
Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language teachers need to be accessible and have the necessary resources to create effective content for their students. |
| Approach: | They propose to extract grammar descriptions from a natural text corpus that answer questions about morphosyntax and semantics from lexical corpus. |
| Outcome: | The proposed method is applied to two Indian languages, Kannada and Marathi, which, unlike English, do not have well-developed resources for second language learning. |
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024.emnlp-main)
Copied to clipboard
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | Existing open-source evaluation paradigms lack flexibility and performance . language model-based evaluation is cheap and scalable, but it is difficult to evaluate . |
| Approach: | They propose a language model-based evaluation paradigm that uses a scalar indicator of quality to assess LM outputs. |
| Outcome: | The proposed language model-based evaluation model is more powerful than its predecessor. |
DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning (2026.eacl-long)
Copied to clipboard
Nithin Sivakumaran, Justin Chen, David Wan, Yue Zhang, Jaehong Yoon, Elias Stengel-Eskin, Mohit Bansal
| Challenge: | a key strength of human intelligence is the ability to debate and discuss reasoning with others. |
| Approach: | They propose a multi-agent framework that uses disagreements between visual agents to identify useful visual tools that can resolve inter-agency disagreement. |
| Outcome: | The proposed framework beats the strongest baseline on A-OKVQA and MMMU, respectively. |
Gold Standard Annotations for Preposition and Verb Sense with Semantic Role Labels in Adult-Child Interactions (C18-1)
Copied to clipboard
| Challenge: | Existing corpus of child-directed speech augments existing corpus for semantic role labels . sense and number of arguments were open to multiple interpretations due to rapidly changing discourse . |
| Approach: | They propose to augment an existing corpus of child-directed speech to provide supervised learning of semantic role labels. |
| Outcome: | The resulting corpus is a gold standard for supervised learning of semantic role labels in child-directed speech. |
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
ROSE: An Intent-Centered Evaluation Metric for NL2SQL (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions are becoming unreliable due to its sensitiveness to syntactic variation and inconsistent consistency with ground-truth SQL. |
| Approach: | They propose an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL. |
| Outcome: | The proposed metric outperforms the next-best metric by nearly 24% on the expert-aligned validation set **ROSE-VEC**. |
Charting the Linguistic Landscape of Developing Writers: An Annotation Scheme for Enhancing Native Language Proficiency (2024.lrec-main)
Copied to clipboard
| Challenge: | An annotation task was designed to capture orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education (DevEd) courses. |
| Approach: | They propose an annotation task to capture orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education courses. |
| Outcome: | The proposed annotation task captures orthographic, grammatical, lexical, semantic, and discursive patterns exhibited by college native English speakers participating in developmental education courses. |
Is this Sentence Difficult? Do you Agree? (D18-1)
Copied to clipboard
| Challenge: | a crowdsourcing-based approach to model sentence complexity is proposed . word-level predictors shown to correlate with greater processing difficulties are e.g. word frequency, age of acquisition, root frequency effect, orthographic neighbourhood frequency . |
| Approach: | They propose a crowdsourcing-based approach to model human perception of sentence complexity using a corpus of sentences rated with judgments of complexity for two typologically-different languages. |
| Outcome: | The proposed model predicts agreement among annotators independently from the assigned judgment and the perception of sentence complexity in Italian and English. |
End-to-End Lexically Constrained Machine Translation for Morphologically Rich Languages (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to enforce word forms in translations struggle to make them agree with the rest of the output. |
| Approach: | They propose to train neural machine translation models with lemmatized constraints to infer correct word inflection. |
| Outcome: | The proposed model reduces errors in translation of constrained terms in automatic and manual evaluations on English-Czech language pairs. |
Validity, Agreement, Consensuality and Annotated Data Quality (2022.lrec-1)
Copied to clipboard
| Challenge: | a wide consensus is rife regarding the need for reference annotated datasets . however, the creation of such datasets is accompanied by theorectical and practical issues . |
| Approach: | They propose to use agreement among annotators as an indicator of consensus . they argue that it is difficult to produce gold-standard annotated datasets . |
| Outcome: | The proposed model focuses on the complex relations between agreement and reference and the emergence of consensus. |
Establishing Annotation Quality in Multi-label Annotations (2022.coling-1)
Copied to clipboard
| Challenge: | Multi-label annotations allow multiple interpretations of a single item, but they also affect the chance that two coders agree with each other. |
| Approach: | They propose a bootstrapped method to obtain chance agreement for each measure and a method to get an adjusted agreement coefficient that is more interpretable. |
| Outcome: | The proposed method allows for an adjusted agreement coefficient that is more interpretable on simulated datasets. |
Identification of Multiple Logical Interpretations in Counter-Arguments (2025.emnlp-main)
Copied to clipboard
Wenzhi Wang, Paul Reisert, Shoichi Naito, Naoya Inoue, Machi Shimmei, Surawat Pothong, Jungmin Choi, Kentaro Inui
| Challenge: | Counter-arguments (CAs) are a good way to improve learners' critical thinking skills . however, it is difficult to provide every learner tailored feedback due to limited human resources and heavy workloads. |
| Approach: | They propose to annotate a dataset of 134 CAs annotated with 13 logical predicate questions and train a model with Reinforcement Learning with Verifiable Rewards to identify multiple logical interpretations. |
| Outcome: | The proposed model performs on par with larger proprietary models. |
Reproducibility and Automation of the Appraisal Taxonomy (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for Appraisal annotation are descriptive and lack of data hinders progress . |
| Approach: | They propose to use annotated data to measure the performance of automated Appraisal annotations in a publicly available dataset. |
| Outcome: | The proposed methods show poor agreement at more detailed categories and fair agreement at coarse-level categories. |
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used as evaluators for natural language generation . human rubrics are often static and misaligned with how models internally represent language quality. |
| Approach: | They propose to use large language models to generate interpretable and task-aware evaluation dimensions and apply them within models. |
| Outcome: | The proposed model improves the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria. |
Beyond Understanding: Evaluating the Pragmatic Gap in LLMs’ Cultural Processing of Figurative Language (2026.eacl-long)
Copied to clipboard
| Challenge: | Using figurative language as a proxy for cultural nuance and local knowledge, large language models struggle with connotative meaning. |
| Approach: | They evaluate large language models' ability to process culturally grounded language . they use figurative language as a proxy for cultural nuance and local knowledge . |
| Outcome: | The proposed models can understand and use figurative expressions that encode local knowledge and social nuance. |
Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has shown that self-citing large language models (LLMs) fail to faithfully reflect their context usage throughout the generation process. |
| Approach: | They propose a plug-and-play approach using model internals for faithful answer attribution in RAG applications that detects context-sensitive answer tokens and pairs them with retrieved documents contributing to their prediction. |
| Outcome: | The proposed approach achieves citation quality and efficiency comparable to self-citation while allowing for a finer-grained control of attribution parameters. |
LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (2026.eacl-long)
Copied to clipboard
| Challenge: | a single prompt can inspire countless valid stories, making objective verification impossible. |
| Approach: | They propose a large-scale benchmark for creative writing evaluation using a reddit corpus and a 2,480-pair test set. |
| Outcome: | The proposed model outperforms existing OTS judges and generative reward models in the evaluation of creative writing. |
TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TableQA benchmarks focus on simple flat tables and suffer from data leakage . current benchmarks are monolingual and fail to capture cross-lingual variability . |
| Approach: | They propose a table-based TableQA benchmark to evaluate LLMs on real-world tasks. |
| Outcome: | The proposed benchmarks show that they achieve high agreement with human judgment . the proposed framework improves on the alignment between model responses and reference answers . |
FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Current content moderation filters focus on general safety and ignore cultural context . authors: FanarGuard improves accuracy and provides a practical step toward context-sensitive safeguards. |
| Approach: | They propose a bilingual moderation filter that evaluates both safety and cultural alignment in Arabic and English. |
| Outcome: | The proposed moderation filter performs better with human annotations than state-of-the-art filters on safety benchmarks. |
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing web agents only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. |
| Approach: | They propose a large multimodal model-powered web agent that can complete user instructions end-to-end by interacting with real-world websites. |
| Outcome: | The proposed agent achieves 59.1% task success rate, surpassing both GPT-4 and WebVoyager setups. |
Aligning Black-box Language Models with Human Judgments (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as automated judges to evaluate recommendation systems, search engines, and other subjective tasks. |
| Approach: | They propose a framework to align LLM judgments with individual human evaluators or their aggregated judgments without retraining or fine-tuning the LLM. |
| Outcome: | The proposed framework achieves 142% improvement in agreement across 29 tasks and exceeds inter-human agreement on four out of six tasks. |
Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods (2023.emnlp-main)
Copied to clipboard
| Challenge: | Feature attribution scores are used to explain the prediction of a text classifier to users by highlighting a k number of tokens. |
| Approach: | They propose to determine the number of optimal k tokens that should be displayed from sequential properties of attribution scores. |
| Outcome: | The proposed method is dynamic across sentences, method-agnostic, and deals with sentence length bias. |
SLIDE - a Sentiment Lexicon of Common Idioms (L18-1)
Copied to clipboard
| Challenge: | Compositional solutions for phrase sentiment are not able to handle idioms because their sentiment is not derived from the sentiment of the individual words. |
| Approach: | They propose a crowdsourcing approach for collecting sentiment annotations of idiomatic expressions using crowdsourcing. |
| Outcome: | The proposed approach is able to capture sentiment strength and ambiguity in idiomatic expressions using crowdsourcing. |
Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT (2022.lrec-1)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is integral to many NLP applications such as chatbots and question answering. |
| Approach: | They propose to annotate Arabic nested entities instead of flat annotations by manually annotating 550K tokens with 21 entity types including person, organization, location, event and date. |
| Outcome: | The proposed model achieved an overall micro F1-score of 0.884 and the annotation guidelines and source code are publicly available. |
Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluation (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluation tools for Large Language Models (LLMs) are inconsistency, bias, and lack of transparent decision criteria. |
| Approach: | They propose a cost-aware, adversarial multi-agent framework that orchestrates structured debate among role-specialized agents to produce reliable and interpretable evaluations. |
| Outcome: | The proposed framework orchestrates structured debate among role-specialized agents to produce reliable and interpretable evaluations. |
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts (2021.emnlp-main)
Copied to clipboard
| Challenge: | despite progress toward data-driven conversational agents, dialogue models still suffer from issues surrounding safety and offensive language. |
| Approach: | They analyze reddit threads and reddits to determine the stance of offensive dialogue models . they find 42% of human responses agree with toxic comments, compared to 13% with safe comments . |
| Outcome: | The proposed model produces 29% fewer offensive replies than the baseline model. |
Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement (2020.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation methods rely on agreement between annotators, which implies a single correct interpretation. |
| Approach: | They propose an agreement-independent quality metric based on answer-coherence to evaluate on expected disagreement. |
| Outcome: | The proposed model shows that agreement is the most important indicator of quality in semantic annotation tasks. |
Developing a Rhetorical Structure Theory Treebank for Czech (2024.lrec-main)
Copied to clipboard
| Challenge: | a paper on the Czech RST Discourse Treebank is the first version of a textual annotation system based on the Rhetorical Structure Theory . document is annotated using the RST, a global coherence model proposed by Mann and Thompson . |
| Approach: | They introduce the first version of the Czech RST Discourse Treebank . paper presents an annotation process and provides corpus statistics and evaluation . |
| Outcome: | The paper presents the first version of the Czech RST Discourse Treebank . the treebank includes two gold annotations representing divergent interpretations . |
PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation of multilingual Large Language Models is challenging due to a variety of factors including the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and lack of local, cultural nuances in translated benchmarks. |
| Approach: | They evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations. |
| Outcome: | The proposed models perform best in most Indic languages, while the agreement drops for direct assessment especially for Bengali and Odia. |
Improving Accessibility of SCOTUS Opinions: A Benchmark Study and a New Dataset for Generic Heading Prediction and Specific Heading Generation (2025.coling-main)
Copied to clipboard
| Challenge: | SCOTUS opinions are notoriously long and use specialised language, making them laborious to read and understand. |
| Approach: | They propose generic and specific headings for each section to be trained automatically . they compare the performance of different systems trained for each subtask . |
| Outcome: | The proposed system can achieve a score of 0.90% in predicting generic headings . the proposed system also achieves similar scores in generating specific headings. |
Recognising Agreement and Disagreement between Stances with Reason Comparing Networks (P19-1)
Copied to clipboard
| Challenge: | Existing methods for (dis)agreement detection focus on conversational settings . however, non-dialogic stance-bearing utterances are common in real-world scenarios . |
| Approach: | They propose a reason comparing network to leverage reason information for stance comparison. |
| Outcome: | The proposed method outperforms baselines on a well-known stance corpus. |
A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space (2021.emnlp-main)
Copied to clipboard
| Challenge: | Cross-lingual language models house representations for many different languages in the same space. |
| Approach: | They investigate linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. |
| Outcome: | The results show that word order agreement and agreement in morphological complexity are strongest predictors of cross-linguality. |
The Search for Agreement on Logical Fallacy Annotation of an Infodemic (2022.lrec-1)
Copied to clipboard
Claire Bonial, Austin Blodgett, Taylor Hudson, Stephanie M. Lukin, Jeffrey Micher, Douglas Summers-Stay, Peter Sutor, Clare Voss
| Challenge: | a parallel "infodemic" has emerged with the COVID-19 pandemic . logical fallacies can be subtly encoded in the structure of a document across multiple sentences . |
| Approach: | They evaluate an annotation schema for labeling logical fallacy types using linguist annotations . they propose to use a machine learning algorithm to train annotators for fallacy detection . |
| Outcome: | The proposed annotation schema is clear and non-overlapping for manual and system assignment. |
Reasoning-Based Refinement of Unsupervised Text Clusters with LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Unsupervised methods are used to induce latent semantic structure from large text collections, yet their outputs often contain incoherent, redundant, or poorly grounded clusters. |
| Approach: | They propose a framework that leverages large language models as semantic judges to validate and restructure unsupervised clustering algorithms. |
| Outcome: | The proposed framework improves cluster coherence and human-aligned labeling quality over traditional models and representation-based baselines. |
Why Don’t Prompt-Based Fairness Metrics Correlate? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to assess fairness using prompts have low correlations between fairness metrics. |
| Approach: | They propose a method to enhance the correlation between fairness metrics by using pre-trained language models. |
| Outcome: | The proposed method improves the correlation between fairness metrics by using pre-trained language models. |
An Evaluation Framework for Legal Document Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing metrics for summarizing legal documents fail to evaluate intent in the original text. |
| Approach: | They propose an automated intent-based summarization metric which shows a better agreement with human evaluation as compared to other automated metrics like BLEU, ROUGE-L etc. |
| Outcome: | The proposed method shows that human evaluation is more accurate than other metrics. |
Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods (2023.findings-acl)
Copied to clipboard
| Challenge: | A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. |
| Approach: | They propose to use saliency methods to evaluate whether an explanation is faithful and argue that Pearson-r is a better-suited alternative to rank correlation. |
| Outcome: | The proposed methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. |
CitaLaw: Enhancing LLM with Citations in Legal Domain (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks have focused on enabling large language models (LLMs) to generate citationsupported outputs. |
| Approach: | They propose to use a citation-based framework to evaluate LLMs' ability to produce legally sound responses with appropriate citations. |
| Outcome: | The proposed framework enables LLMs to retrieve supporting citations from the reference corpus and align these citation with the corresponding sentences in their responses. |
Can LLM be a Personalized Judge? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new study examines the reliability of large language models (LLMs) for personalization and role-playing evaluation without examining its validity. |
| Approach: | They investigate the reliability of LLM-as-a-Personalized-Judge for personalization . they find that personas provided to LLMs have limited predictive power . |
| Outcome: | The proposed model is less reliable than previously thought, the authors show . human annotation reveals that third-person crowd worker evaluations of personalized preferences are even worse than LLM predictions. |
Integrating Argumentation and Hate-Speech-based Techniques for Countering Misinformation (2024.emnlp-main)
Copied to clipboard
| Challenge: | scalable strategies to combat online misinformation are short-term and insufficient, authors say . current reactive approaches, like content flagging and banning, do little to change perception of misinformants . human evaluations show that our framework generates expert-like responses . |
| Approach: | They propose a framework that generates persuasive responses from hate-speech counter-responses . human evaluations show that the framework generates expert-like responses . |
| Outcome: | The proposed framework generates expert-like responses and is 14% more engaging, 21% more natural, and 18% more factual than the best available alternatives. |
Increasing Argument Annotation Reproducibility by Using Inter-annotator Agreement to Improve Guidelines (L18-1)
Copied to clipboard
| Challenge: | Argument Mining systems require large amounts of data to characterize phenomena and find patterns that can be exploited by an automatic analyzer. |
| Approach: | They propose to exploit inter-annotator agreement measures to improve Argument annotation guidelines. |
| Outcome: | The proposed method improves Argument annotation guidelines by exploiting inter-annotator agreement measures. |
Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models enable them to follow freeform instructions, including imitating generic or specific demographic personas in conversations. |
| Approach: | They propose to investigate persona biases by experimenting with UNIVERSALPERSONA, a model that incorporates both generic and specific personas. |
| Outcome: | The proposed model systematically measures persona biases in harmful expression and harmful agreement. |
Advancing Oversight Reasoning across Languages for Audit Sycophantic Behaviour via X-Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have demonstrated capabilities that are satisfactory to a wide range of users by adapting to their culture and wisdom. |
| Approach: | They propose an Oversight Reasoning framework that audits human–LLM dialogues, reasons about them, captures sycophancy and corrects the final outputs. |
| Outcome: | The proposed framework detects sycophancy, reduces unwarranted agreement and improves cross-turn consistency across different scenarios and languages. |
Object Naming in Language and Vision: A Survey and a New Dataset (2020.lrec-1)
Copied to clipboard
| Challenge: | Object naming has been studied in Psycholinguistics, but has received little attention in Computational Linguistics. |
| Approach: | They propose a dataset that provides 36 name annotations for each of 25K objects in images selected from VisualGenome. |
| Outcome: | The proposed dataset shows that people choose certain names for objects, on average. |
What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite the subjective nature of many NLU evaluations, little attention has been paid to the distribution of human opinions. |
| Approach: | They use a dataset with 464,500 annotations to study Collective HumAn OpinionS . they argue that models lack the ability to recover the distribution over human labels . |
| Outcome: | The proposed dataset examines the distribution of human opinions in NLU evaluation datasets. |
Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study examines the recognition of irony by humans and automatic systems . a fine-grained annotation scheme allows for improved modeling of ironity in automatic systems. |
| Approach: | They propose a fine-grained annotation scheme that allows for better recognition of irony by humans and automatic systems. |
| Outcome: | The proposed model improves on tweets annotated with high confidence and agreement . it also performs better on high-confidence and highagreement samples compared to automated systems . |
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving the correctness of output from large language models generate a constant number of samples per question, but Adaptive-Consistency reduces sample budget by up to 7.9 times with an average accuracy drop of less than 0.1%. |
| Approach: | They propose a model-agnostic technique that dynamically adjusts the number of samples per question using a lightweight stopping criterion. |
| Outcome: | The proposed technique reduces sample budget by 7.9 times with an average accuracy drop of less than 0.1%. |
Quantifying Metric and Model Agreement in Bias Evaluation of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | a systematic way to measure agreement across bias metrics and models is lacking . a lack of agreement between metrics and model results may be a problem . |
| Approach: | They introduce Metric Agreement Score and Model Agreement Score to measure agreement across bias metrics and models. |
| Outcome: | The proposed measures show that metrics within the same category behave independently of each other. |
Evaluating the Creativity of LLMs in Persian Literary Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research has focused primarily on English, with limited exploration of non-English literary traditions and without standardized methods for assessing creativity. |
| Approach: | They build a dataset of user-generated Persian literary spanning 20 diverse topics and assess model outputs along four creativity dimensions . |
| Outcome: | The proposed models generate Persian literary text enriched with culturally relevant expressions. |
Investigating the Saliency of Sentiment Expressions in Aspect-Based Sentiment Analysis (2023.findings-acl)
Copied to clipboard
| Challenge: | In a set of masking experiments, we examine the extent to which the tokens identified as salient by LIME and a gradient-based method are being used by the classifier. |
| Approach: | They use a BERT BASE model to mask the sentiments of an English dataset and find that both methods produce faithful rationales. |
| Outcome: | The proposed classifier outperforms both the gradient-based and black-box saliency methods on the SemEval 2016 english dataset. |
Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Continual pre-training has been used for a multitude of domains and tasks . a continually pre-trained model can show a non-decreasing performance on unseen domains . |
| Approach: | They propose a method that generates domain-specific prompts by agreement and disagreement losses. |
| Outcome: | The proposed method achieves improvements of 3.57% and 3.4% on two real-world datasets. |
An LLM-based Temporal-spatial Data Generation and Fusion Approach for Early Detection of Late Onset Alzheimer’s Disease (LOAD) Stagings Especially in Chinese and English-speaking Populations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches struggle with temporal-spatial challenges in capturing subtle linguistic shifts across different disease stages. |
| Approach: | They propose a large language model-driven T-S fusion framework that integrates multilingual LLMs, contrastive learning and interpretable marker discovery to revolutionize late onset AD detection. |
| Outcome: | The proposed framework achieves state-of-the-art performance in late onset AD detection while enabling cross-linguistic diagnostics. |
Weak Reward Model Transforms Generative Models into Robust Causal Event Extraction Systems (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics that reflect the performance of causal event extraction tasks are poorly reflecting the inherent ambiguity of cause and effect boundaries. |
| Approach: | They propose to use a weak-to-strong supervision method to train an evaluation model while still achieving high performance in training an RL model. |
| Outcome: | The proposed method achieves high agreement with human-annotated data while still achieving high performance in training an RL model. |
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | supervised learning is a key component of offensive language detection, but there is little attention given to the quality of annotated data. |
| Approach: | They propose to examine the level of agreement among annotators while selecting data to create offensive language datasets, a task involving a high level of subjectivity. |
| Outcome: | The proposed datasets show that annotators' agreement has a strong effect on classifiers performance and robustness. |
ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games (2023.emnlp-main)
Copied to clipboard
| Challenge: | We show that language models can generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. |
| Approach: | They propose a corpus of 32 reasoning-focused text games expressed as hundreds of lines of Python code to facilitate this task. |
| Outcome: | The proposed games can generate runnable games on unseen topics in 28% of cases. |
CPT-Agent: A Cognitive Process Theory-driven Framework for Student Simulation in Writing Development (2026.acl-long)
Copied to clipboard
| Challenge: | Existing LLMs model overly capable learners who over-apply feedback, resulting in pedagogically implausible behavior. |
| Approach: | They propose a framework that decouples cognitive ability from writing proficiency and models their interaction during writing and revision. |
| Outcome: | The proposed model produces distinguishable proficiency levels and is consistent with instructional theories. |
Large Language Models for Propaganda Span Annotation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using propagandistic techniques to manipulate online audiences is increasing in recent years. |
| Approach: | They investigate whether Large Language Models (LLMs) such as GPT-4 can extract propagandistic spans and the potential of employing them to collect more cost-effective annotations. |
| Outcome: | The proposed model provides labels that have higher agreement with expert annotators and lead to specialized models that achieve state-of-the-art over an unseen Arabic testing set. |
Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation. |
| Approach: | They propose to integrate actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. |
| Outcome: | The proposed scoring-aware multi-reward reinforcement learning integrates actual evaluation schemes into the training process. |
LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models (2026.acl-long)
Copied to clipboard
Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing evaluation of Large Language Models on static benchmarks is vulnerable to data contamination and leaderboard overfitting. |
| Approach: | LLMEval-Fair framework provides a framework for dynamic evaluation of Large Language Models . evaluators use a proprietary bank of 220k graduate-level questions to analyze model data . |
| Outcome: | LLMEval-Fair provides robust and credible evaluation framework for Large Language Models . it provides a strong empirical validation for the dynamic evaluation paradigm . |
When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews (2023.emnlp-main)
Copied to clipboard
| Challenge: | despite being widely accepted standard for validating scholarly research, peer-review process has faced criticism. |
| Approach: | They propose a task of automatically identifying contradictions among reviewers on a given article. |
| Outcome: | The proposed model detects contradictory statements from the review pairs and makes it publicly available for further investigations. |
OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large vision-language models struggle to generate long and factual captions . traditional measures for hallucination and factuality are not well suited for longer captions. |
| Approach: | They propose a method for measuring caption factuality of long captions that leverages open-vocabulary visual grounding and tool-based verification without relying on human annotations. |
| Outcome: | The proposed method improves agreement with human judgements and captures both caption descriptiveness and factual precision in the same metric. |
PopAut: An Annotated Corpus for Populism Detection in Austrian News Comments (2024.lrec-main)
Copied to clipboard
| Challenge: | Populism is a phenomenon that is noticeably present in political landscapes worldwide . prior work on populism analysis focused on analyzing populist content expressed by politicians . |
| Approach: | They present a corpus of news comments annotated for populism in the german language . they use machine learning to detect populist comments in text . |
| Outcome: | The proposed corpus outperforms existing dictionaries for populism detection in text . it features 1,200 comments collected between 2019-2021 . |
Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. |
| Approach: | They propose to use human error span annotations to evaluate translations of six translation systems across one seen news domain and two unseen technical domains to address these biases. |
| Outcome: | The proposed model improves on the human annotations in two unseen domains and on the news domains. |
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics have been developed and validated for English and other languages . this narrow focus leaves Indian languages largely overlooked, casting doubt on universality of current evaluation practices. |
| Approach: | They propose a large-scale benchmark that compares 26 automatic metrics with human judgments across six major Indian languages. |
| Outcome: | ITEM evaluates alignment of 26 automatic metrics with human judgments across six languages . authors: outliers exert significant impact on metric-human agreement, improve fidelity . they say the results offer critical guidance for advancing metric design and evaluation in Indian languages - a global market for machine translation and text summarization systems. |
IntrEx: A Dataset for Modeling Engagement in Educational Conversations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | IntrEx is the first large dataset annotated for interestingness and expected interestingness in teacher-student interactions. |
| Approach: | They propose a large dataset annotated for interestingness and expected interestingness in teacher-student interactions. |
| Outcome: | The proposed dataset is the first large dataset annotated for interestingness and expected interestingness in teacher-student interactions. |
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts (2025.emnlp-main)
Copied to clipboard
Yuho Lee, Jiaqi Deng, Nicole Hee-Yeon Kim, Hyangsuk Min, Taewon Yun, Minjeong Ban, Kim Yul, Hwanjun Song
| Challenge: | HAMLET is a framework for evaluating the long-context comprehension of large language models. |
| Approach: | They propose a framework for evaluating the long-context comprehension of large language models . HAMLET structures key information of source texts into a three-level hierarchy . |
| Outcome: | HAMLET achieves 90% agreement with expert judgments while reducing evaluation cost by up to 25. |
Russian Learner Corpus: Towards Error-Cause Annotation for L2 Russian (2024.lrec-main)
Copied to clipboard
Daniil Kosakin, Sergei Obiedkov, Ivan Smirnov, Ekaterina Rakhilina, Anastasia Vyrenkova, Ekaterina Zalivina
| Challenge: | Russian Learner Corpus (RLC) is a large collection of learner texts written by native speakers of over forty languages. |
| Approach: | They propose an automatic error annotation tool that locates and labels errors according to a simplified version of the RLC error-type system. |
| Outcome: | The proposed tool locates and labels errors according to a simplified version of the RLC error-type system. |
RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension (2026.acl-long)
Copied to clipboard
Yelin Chen, Fanjin Zhang, Suping Sun, Yunhe Pang, Yuanchun Wang, Jian Song, XiaoYan Li, Lei Hou, Shu Zhao, Jie Tang, Juanzi Li
| Challenge: | Existing benchmarks for understanding research papers offer limited fine-grained evaluation at scale. |
| Approach: | They propose a large-scale question-answering benchmark built from review–rebuttal exchanges of high-quality computer science papers. |
| Outcome: | The proposed model is based on human-verified QA pairs and contains 15K questions. |
Sparse Logistic Regression with High-order Features for Automatic Grammar Rule Extraction from Treebanks (2024.lrec-main)
Copied to clipboard
| Challenge: | Descriptive grammars are valuable, but they lack quantitative data. |
| Approach: | They propose to extract and explore significant fine-grained grammar patterns and potential syntactic grammar rules from treebanks to create an easy-to-understand corpus-based grammar. |
| Outcome: | The proposed model captures well-known and less well- known significant grammar rules in Spanish, French, and Wolof. |
MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference (2026.findings-acl)
Copied to clipboard
Jeonghyun Park, Ingeol Baek, Seunghyun Yoon, Haeun Jang, Aparna Garimella, Akriti Jain, Nedim Lipka, Hwanhee Lee
| Challenge: | Existing benchmarks on multi-hop QA focus on single-hop and layered ambiguity, but they focus on ambiguous questions . ambiguities can arise at any stage, complicating the reasoning process . |
| Approach: | They propose a benchmark to evaluate ambiguity in multi-hop question answering . they propose MARCH, which uses 2,209 carefully annotated questions . |
| Outcome: | The proposed framework outperforms existing approaches and significantly outperfies existing frameworks. |
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. |
| Approach: | They propose to use an ensemble of large language models to flag mislabeled examples by using an LLM-as-a-judge approach to detect label errors in existing datasets. |
| Outcome: | The proposed method improves label accuracy and consistency in large language models. |
The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreement (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for post-hoc explanations for transformer models disagree with each other . disagreement is often overlooked and the reasons for disagreement are not investigated . |
| Approach: | They propose to use a dynamic *k* approach to estimate syntactic spans to improve agreement between different methods. |
| Outcome: | The proposed method better agrees on syntactic span level, especially for the methods that agree the least with other methods. |
EducationQ: Evaluating LLMs’ Teaching Capabilities Through Multi-Agent Dialogue Framework (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used as educational tools, yet evaluating their teaching capabilities remains challenging due to the resource-intensive nature of teacher-student interactions. |
| Approach: | They propose a multi-agent dialogue framework that efficiently assesses teaching capabilities through simulated dynamic educational scenarios. |
| Outcome: | The proposed framework outperforms open-source models on 1,498 questions across 13 disciplines and 10 difficulty levels on 1,400 questions. |
User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are rapidly being adopted for tasks like drafting emails, summarizing meetings, and answering health questions. |
| Approach: | They conducted a scenario-based evaluation of Large language models (LLMs) using 90 PrivacyLens scenarios. |
| Outcome: | The proposed models can leak private information in complex scenarios, but they do not measure user perceptions directly. |
ConSensus: Multi-Agent Collaboration for Multimodal Sensing (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. |
| Approach: | They propose a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents. |
| Outcome: | The proposed framework matches or exceeds debate methods on multimodal sensing benchmarks while achieving 12.7 times reduction in token cost. |
Decomposing Unitization and Typing for Efficient and Consistent Span-Bound Concept Annotation (2026.findings-acl)
Copied to clipboard
| Challenge: | Substantial resources are typically spent on unitizing, the task of identifying precise span boundaries for entity mentions. |
| Approach: | They propose a method that focuses manual efforts on typed position annotations instead of full concept annotation. |
| Outcome: | The proposed procedure reduces the cost of concept annotations by focusing on typed positions instead of full concept annotation. |
Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards (2026.acl-long)
Copied to clipboard
Tianhao Niu, Ziyu Han, Qiguang Chen, Shiqi Zhou, Baocai Shan, Hengjie Fang, Qingfu Zhu, Wanxiang Che
| Challenge: | Existing efforts to generate static visualizations focus on static charts and interactive dashboards. |
| Approach: | They propose a dashboard2code task that requires a model to explore an interactive dashboard, acquire feedback from its own interactions and generate code that reproduces the target dashboard. |
| Outcome: | The proposed task is based on 180 carefully designed and manually verified dashboard–code pairs spanning three difficulty levels and covering eight common real-world interaction patterns. |
TounsiBench: Benchmarking Large Language Models for Tunisian Arabic (2025.emnlp-main)
Copied to clipboard
| Challenge: | a dataset of Tunisian Arabic instructions and prompts is used to evaluate LLMs' ability to understand and generate responses in Tunisia . we assess the quality, correctness, relevance, and dialectal adherence of LLM responses . |
| Approach: | They propose a benchmark for evaluating the capabilities of large language models in Tunisian Arabic . they use a dataset of Tunisia Arabic instructions and prompts to evaluate their models . |
| Outcome: | The proposed model can judge quality, correctness, relevance, and dialectal adherence . the model can also generate a leaderboard for the Tunisian Arabic language . |
All Prompts Are Created Equal? Evaluating Robustness of LLM Judges Against Non-Adversarial Prompt Variations (2026.findings-acl)
Copied to clipboard
| Challenge: | Our work establishes new standards for assessing LLM judge reliability beyond simple accuracy metrics. |
| Approach: | They propose a diagnostic framework grounded in attribute verifiability that enables principled decisions about evaluation automation. |
| Outcome: | The proposed framework establishes new standards for assessing LLM judge reliability beyond simple accuracy metrics. |
Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing behavior trees are not suitable for high-dimensional perceptual inputs such as images or language. |
| Approach: | They propose a framework that leverages expert-regularized reinforcement learning to preserve semantic faithfulness while employing a factorized policy that aggregates sequential condition-node decisions into a single decision unit. |
| Outcome: | The proposed framework outperforms imitation learning and reinforcement learning but risks misalignment of condition nodes with intended semantics and poor credit assignment. |
Feeling Right vs. Being Right: How AI Sycophancy Affects Value-Laden Deliberation (2026.acl-long)
Copied to clipboard
| Challenge: | Unlike human flattery, AI sycophancy is intentional and self-interested . scophancies are a byproduct of RLHF's user-preference alignment process . |
| Approach: | They propose to operationalize AI sycophancy as excessive face-saving, either active (preserving positive face through agreement) or passive (preserving negative face by withholding challenge). |
| Outcome: | The findings show that sycophancy is a byproduct of RLHF's user-preference alignment process and that it is not a human trait. |
Rolling Out Data Quality Overnight, without losing the plot: A Multi-Agent System for Speech Data Quality Management (2026.findings-acl)
Copied to clipboard
Rishabh Kumar, Abhinav Painuli, Chriss Philip Saji, Devesh Soni, Amrith Krishna, Ganesh Ramakrishnan
| Challenge: | Using automation to improve quality management is expensive and resource-intensive for speech datasets. |
| Approach: | They propose a natural language-driven agentic framework that compiles user requirements into dependency-aware DAG workflows over modular tools for audio, transcript, and metadata verification. |
| Outcome: | The proposed framework achieves 80-90% agreement with expert verification while requiring less than 20% of the cost and time of manual QC. |
Beyond Evidence: Belief-Chain Conditioning for Persuasive Misinformation Debunking Explanation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to misinformation correction focus on relying on audience beliefs to generate factually accurate responses and to engage with users' mental states. |
| Approach: | They construct large language models with cognitive chains and use them to model their outputs on beliefs that engage with users' mental states. |
| Outcome: | The proposed model improves explanation quality for audiences with misinformation-aligned beliefs by incorporating believers’ chains into the model. |