Papers with validation
An Industry Evaluation of Embedding-based Entity Alignment (2020.coling-industry)
Copied to clipboard
| Challenge: | Knowledge graphs (KGs) are increasingly important in various applications such as question answering and search engines. |
| Approach: | They propose to use a supervised learning environment with unbiased seed mappings for training and validation to evaluate alignment methods in an industrial context. |
| Outcome: | The proposed methods are evaluated in an industrial context and are compared with DBpedia and Wikidata benchmarks. |
Reasoning Over Paths via Knowledge Base Completion (D19-53)
Copied to clipboard
| Challenge: | Existing methods to predict missing links in knowledge graphs are lacking. |
| Approach: | They propose a method to automatically rank paths between a source and target entity pair using a knowledge base completion model. |
| Outcome: | The proposed method can rank and rank paths in biomedical knowledge graphs with a KBC model. |
Angry or Sad ? Emotion Annotation for Extremist Content Characterisation (2022.lrec-1)
Copied to clipboard
| Challenge: | Social platforms play an increasingly important role in the propagation of extremist ideas. |
| Approach: | They propose to use a linguistic annotation scheme to characterize extremist content in French . they validate the scheme and test its ability to capture various aspects of emotions . |
| Outcome: | The proposed method combines sociological and linguistic knowledge to characterize extremist content in French. |
TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability (2025.emnlp-demos)
Copied to clipboard
Mohammad Aflah Khan, Ameya Godbole, Johnny Wei, Ryan Yixiang Wang, James Flemings, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
| Challenge: | Existing workflows for pretraining large language models are cumbersome, fragmented and inaccessible. |
| Approach: | They propose an open-source library for editing, inspection, and analysis of large language model datasets. |
| Outcome: | TokenSmith is an open-source library for editing, inspection, and analysis of large language model datasets. |
DASR: Distributed Adaptive Scene Recognition - A Multi-Agent Cloud-Edge Framework for Language-Guided Scene Detection (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Current approaches to analyzing driving scenarios rely on massive data collection followed by manual filtering. |
| Approach: | They propose a cloud-based framework for language-guided scene detection in connected vehicles . the framework leverages cloud- and edge-deployed large language models to identify relevant driving scenarios while optimizing on-vehicle buffer storage. |
| Outcome: | The proposed framework performs better on complex driving tasks and reduces storage requirements. |
Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens (2026.findings-eacl)
Copied to clipboard
| Challenge: | anthropological accounts of culture often focus on static facts or homogeneous values . large language models are being implemented in translation systems, educational tools and search engines . |
| Approach: | They propose to categorize how benchmarks frame culture such as knowledge, preference, performance, or bias. |
| Outcome: | The proposed framework categorizes how benchmarks frame culture, such as knowledge, preference, performance, or bias. |
MOROCO: The Moldavian and Romanian Dialectal Corpus (P19-1)
Copied to clipboard
| Challenge: | Using the MOldavian and ROmanian Dialectal COrpus, we perform empirical studies on dialect identification tasks. |
| Approach: | They introduce the MOldavian and ROmanian Dialectal COrpus corpus which contains 33564 samples of text collected from the news domain. |
| Outcome: | The proposed model is based on a shallow and deep approach to discriminate between two different languages. |
DialSQL: Dialogue Based Structured Query Generation (P18-1)
Copied to clipboard
| Challenge: | Recent advances in deep learning and semantic parsing have improved the translation accuracy of natural language questions to structured queries. |
| Approach: | They propose a dialogue-based structured query generation framework that leverages human intelligence to boost performance of existing algorithms via user interaction. |
| Outcome: | The proposed framework improves on a WikiSQL dataset from 61.3% to 69.0% using only 2.4 validation questions per dialogue. |
STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions (2023.acl-short)
Copied to clipboard
Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Christian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, Mark Cieliebak
| Challenge: | We present a corpus of Swiss German speech annotated with Standard German text at the sentence level. |
| Approach: | They present a corpus of Swiss German speech annotated with Standard German sentences . they use a web app to show the speakers standard German sentences and record them . |
| Outcome: | The corpus contains 343 hours of speech from all Swiss German dialect regions . it is the largest public speech corpus for Swiss German to date . |
Show Your Work with Confidence: Confidence Bands for Tuning Curves (2024.naacl-long)
Copied to clipboard
| Challenge: | a rush to scale up has left us with large, costly language models and little understanding of how different designs compare. |
| Approach: | They propose a method to construct valid confidence bands for tuning curves . they validated their method with ablations and analyze the effect of sample size . |
| Outcome: | The proposed method shows that bootstrap confidence bands do not approximate their target confidence. |
Sanitizing Large Language Models in Bug Detection with Data-Flow (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have potential in code reasoning tasks but the hallucination effect can compromise the reliability of bug reports. |
| Approach: | They propose a new schema of bug detection that enforces LLMs to emit data-flow paths in few-shot chain-of-thought prompting and validates them via the program-property decomposition. |
| Outcome: | The proposed approach achieves 91.03% precision and 74.00% recall upon synthetic benchmarks and boosts precision by 21.99% with the sanitization. |
Bridging the Training-Inference Gap for Dense Phrase Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for building dense retrievers are often misaligned and do not reflect retrieval scenario at inference time. |
| Approach: | They propose a way to validate dense retrievers using a small subset of the entire corpus. |
| Outcome: | The proposed model improves top-1 phrase retrieval accuracy by 2 3 points and top-20 passage retrieval by 2 4 points for open-domain question answering. |
Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging (2023.acl-long)
Copied to clipboard
| Challenge: | Massively multilingual language models have shown strong performance in zero-shot (ZS-XLT) and few-shot cross-lingual transfer setups where models are fine-tuned on task data in a source language are transferred without any or with only a few annotated instances to the target language(s). |
| Approach: | They propose a method that averages different checkpoints during task fine-tuning to improve model robustness. |
| Outcome: | The proposed method overestimates model performance in cross-lingual transfer setups where models are evaluated at checkpoints that generalize best to validation instances in the target languages. |
Impact of Evaluation Methodologies on Code Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methodologies for code summarization tasks do not consider timestamps of code and comments. |
| Approach: | They propose a time-segmented evaluation methodology for code summarization that considers timestamps of code and comments during evaluation. |
| Outcome: | The proposed evaluation methodology compares with other evaluation methodologies that have been widely used. |
An empirical study of validating synthetic data for formula generation (2025.findings-naacl)
Copied to clipboard
Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, Gust Verbruggen
| Challenge: | Large language models (LLMs) can be leveraged to help write formulas in spreadsheets, but formula data resources are scarce, limiting the ability to fine-tune them. |
| Approach: | They validate a corpus of formulas with a model to generate synthetic natural language utterances for fine-tuning. |
| Outcome: | The proposed model generates synthetic natural language utterances with a model that is accurate enough to fine-tune them. |
Translating away Translationese without Parallel Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Translated texts exhibit systematic linguistic differences compared to original texts in the same language, referred to as translationese . studies show translationeses have effects on various cross-lingual natural language processing tasks . |
| Approach: | They propose a translation-based style transfer approach that learns from monolingual data . they combine a self-supervised approach with an unsupervised approach . |
| Outcome: | The proposed method reduces translationese to a level of a random classifier after style transfer while preserving the content and fluency in the target original style. |
Evaluation of HTR models without Ground Truth Material (2022.lrec-1)
Copied to clipboard
Phillip Benjamin Ströbel, Martin Volk, Simon Clematide, Raphael Schwitter, Tobias Hodel, David Schoch
| Challenge: | Optical Character Recognition (OCR) is a well-established technique for digitising historical printed collections in libraries and archives. |
| Approach: | They propose to use masked language models to evaluate handwritten text recognition models . they propose to introduce GT-free metrics to evaluate models to ensure best results . |
| Outcome: | The proposed model evaluations are based on lexicon-based and masked language models. |
Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows (2026.findings-acl)
Copied to clipboard
Haoyu Dong, Pengkun Zhang, Yan Gao, Xuanyu Dong, Yilin Cheng, Mingzhe Lu, Adina Yakefu, Shuxin Zheng
| Challenge: | FinWorkBench evaluates real-world enterprise-grade finance and accounting workflows . a human evaluation of GPT 5.1 Pro passes only 38.4% of workflows, a study finds . |
| Approach: | They propose a workflow construction process that combines LLM-assisted mining and expert annotation to build 172 composite workflows. |
| Outcome: | The proposed process combines expert annotation with LLM-assisted mining of workflows from authentic enterprise environments. |
ArzEn: A Speech Corpus for Code-switched Egyptian Arabic-English (2020.lrec-1)
Copied to clipboard
| Challenge: | a corpus of Arabic-English code-switching (CS) spontaneous speech is collected in an Egyptian university soundproof room . the language in Egypt is rather complex and poses many challenges to natural language processing (NLP) |
| Approach: | They present an Egyptian Arabic-English code-switching (CS) spontaneous speech corpus. |
| Outcome: | The proposed corpus is designed to be used in automatic speech recognition systems . it provides a useful resource for analyzing the CS phenomenon from linguistic, sociological, and psychological perspectives. |
Self-play through Computational Runtimes improves Chart Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models (VLMs) achieve impressive zero-shot performance on multimodal reasoning tasks. |
| Approach: | They propose a self-play programming interface which leverages visual reasoning models to generate code to decompose a complex visual reasoning task in sub-tasks and use it as a tool to solve decomposed tasks. |
| Outcome: | The proposed approach outperforms baselines on difficult chart reasoning benchmarks. |
Natural Language Deduction with Incomplete Information (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for reasoning given incomplete information are inadequate . current approaches to reasoning are based on latent reasoning by large language models . |
| Approach: | They propose a system that generates a natural language "proof" by abductively inferring a premise from another premise and a conclusion. |
| Outcome: | The proposed system can handle the underspecified setting where not all premises are stated at the outset; additional assumptions need to be materialized to prove a claim. |
“Where Does This Strange Smell Come from?”: Enabling Conversational Interfaces for Artificial Olfaction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Artificial Olfaction (AO) systems are not compatible with smart home scenarios due to diverse obstacles and the need for natural interaction. |
| Approach: | They propose to use large language models to train a CIAO system for Odor Classification and Odor Source Localization in smart home scenarios. |
| Outcome: | The proposed system outperforms existing systems in indoor event detection scenarios. |
Examining Temporalities on Stance Detection towards COVID-19 Vaccination (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have highlighted the importance of vaccination as an effective strategy to control the transmission of the COVID-19 virus. |
| Approach: | They evaluate a range of transformer-based models using chronological and random splits of social media data to examine the impact of temporal concept drift on stance detection towards COVID-19 vaccination. |
| Outcome: | The proposed models show that the models performed better with chronological and random splits than with random split models. |
MultiClaimNet: A Massively Multilingual Dataset of Fact-Checked Claim Clusters (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a growing number of unverified claims and expanding size of fact-checked databases require alternative, more efficient solutions. |
| Approach: | They propose to group fact-checked claims into multilingual clusters to improve claim retrieval and validation. |
| Outcome: | The proposed approach reduces redundancy by grouping claims into clusters . the proposed dataset contains 85.3K fact-checked claims written in 78 languages . |
EMO-KNOW: A Large Scale Dataset on Emotion-Cause (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets focus on extracting parts of the document that contain the emotion cause and fail to provide more abstractive, generalizable root cause. |
| Approach: | They propose to use 9.8 million cleaned tweets to create a large-scale dataset of emotion causes, derived from 9.8 millions tweets over 15 years. |
| Outcome: | The proposed dataset comprises over 700,000 tweets with corresponding emotion-cause pairs spanning 48 emotion classes, validated by human evaluators. |
Reproducibility in NLP: What Have We Learned from the Checklist? (2023.findings-acl)
Copied to clipboard
| Challenge: | Scientific progress in NLP rests on the reproducibility of researchers’ claims. |
| Approach: | They examine 10,405 anonymous responses to the NLP Reproducibility Checklist . they find evidence of an increase in reporting of information after the Checklist's introduction . |
| Outcome: | The authors find that 44% of submissions that gather new data are 5% less likely to be accepted than those that did not. |
Machine Theory of Mind Needs Machine Validation (2025.findings-acl)
Copied to clipboard
| Challenge: | In recent years there has been an explosion of interest in studying the extent to which language models (LMs) display a theory of mind (ToM) despite the growth of evaluation tools, the extent of evidence for ToM remains unclear. |
| Approach: | They conduct a survey of 16 recent studies aimed at measuring ToM in language models and found that only half do so for patterns only a machine might exploit. |
| Outcome: | The results show that the datasets that show high LM performance on ToM tasks are easier than their peers, likely due to the presence of spurious patterns in the data. |
DORM: Preference Data Weights Optimization for Reward Modeling in LLM Alignment (2025.findings-emnlp)
Copied to clipboard
Rongzhi Zhang, Chenwei Zhang, Xinyang Zhang, Liang Qiu, Haoming Jiang, Yuchen Zhuang, Qingru Zhang, Hyokun Yun, Xian Li, Bing Yin, Tuo Zhao, Chao Zhang
| Challenge: | Existing approaches to align large language models with human preferences are noisy and varying in importance of preference samples. |
| Approach: | a new method enhances reward modeling by learning to dynamically weigh preference data. |
| Outcome: | a new method improves the performance of large language models with human preferences . it initializes data importance and iteratively refines them to maximize validation performance. |
PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing solutions for large language models rely on manual rewriting or labeled data for validation . Existing approaches rely only on comprehensive tool documentation and in-context demonstrations . |
| Approach: | They propose a framework that "plays" with each tool to explore its input-output behaviors. |
| Outcome: | Experiments show that PLAY2PROMPT improves zero-shot tool performance across open and closed models. |
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (LVLMs) learn visual capacity through visual instruction tuning. |
| Approach: | They propose a method for LVLMs to be trained by selective layers tuning . they propose removing non-critical layers outside the visual region . |
| Outcome: | The proposed approach preserves nearly 99% of visual performance and improves textual task results while reducing training time. |
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored methods to enhance self-correction in large language models, but little attention has been given to understanding the models’ internal mechanisms underlying error detection. |
| Approach: | They propose to use a large language model to analyze arithmetic errors in four smaller-sized LLMs and identify their internal mechanisms. |
| Outcome: | The proposed models heavily rely on consistency headstextemdashattention heads that assess surface-level alignment of numerical values in arithmetic solutions. |
PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing natural language generation (NLG) metrics fail to capture domain-specific nuances . patent claims require precise assessment of structural elements such as antecedent consistency and claim dependency. |
| Approach: | They propose a multi-dimensional evaluation framework specifically designed for patent claims . PatentScore integrates hierarchical decomposition of claim elements, validation patterns and scoring across structural, semantic, and legal dimensions. |
| Outcome: | The proposed evaluation framework outperforms existing evaluation frameworks on patent claims . patentScore achieved highest correlation with expert annotations on 400 patent claims dataset . |
Let the Comments Speak: A Multi-Agent Framework based on Large Language Model for Comment-Guided Code Refactoring (2026.findings-acl)
Copied to clipboard
| Challenge: | Current Large Language Models focus on syntax and ignore the vital semantic signals in code comments. |
| Approach: | They propose a Multi-Agent framework for COmment-guided code Refactoring that populates original code with precise comments to provide necessary semantic guidance for subsequent refactoring. |
| Outcome: | The proposed framework significantly improves code quality and achieves higher developer acceptance compared to baselines. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
TAIGR: Towards Modeling Influencer Content on Social Media via Structured, Pragmatic Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Health influencers are often conveyed through conversational narratives and rhetorical strategies rather than explicit factual claims. |
| Approach: | They propose a framework to analyze influencer discourse using takeaway argumentation inference with Grounded References. |
| Outcome: | The proposed framework is based on a content validation task over influencer video transcripts on health, showing that accurate validation requires modeling the discourse’s pragmatic and argumentative structure rather than treating transcripts as flat collections of claims. |