Papers with GitHub
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track (2025.emnlp-industry)
Copied to clipboard
| Challenge: | EMNLP 2025 Industry Track highlights key insights, novel research trends and challenges encountered in practical language technology applications. |
| Approach: | Kai Chen will present the technical advances behind the open-source Intern-series large models . he will highlight how models acquire expert-level skills in specialized domains . |
| Outcome: | This talk will highlight the technical advances behind the open-source Intern-series models . it will highlight how models acquire expert-level skills in specialized domains while retaining broad generalization ability. |
CogKTR: A Knowledge-Enhanced Text Representation Toolkit for Natural Language Understanding (2022.emnlp-demos)
Copied to clipboard
Zhuoran Jin, Tianyi Men, Hongbang Yuan, Yuyang Zhou, Pengfei Cao, Yubo Chen, Zhipeng Xue, Kang Liu, Jun Zhao
| Challenge: | Existing knowledge-enhanced methods are limited to knowledge-intensive tasks. |
| Approach: | They propose a knowledge-enhanced text representation toolkit for natural language understanding . it combines knowledge acquisition, knowledge representation, knowledge injection and knowledge application . |
| Outcome: | The proposed toolkit supports knowledge acquisition, knowledge representation, knowledge injection, and knowledge application. |
Web-based Annotation Interface for Derivational Morphology (2022.naacl-demo)
Copied to clipboard
| Challenge: | a visual interface for manual annotation of language resources for derivational morphology is created using relatively simple programming techniques. |
| Approach: | They propose a web-based visual interface for manual annotation of language resources for derivational morphology. |
| Outcome: | The proposed interface can be used for manual annotation of derivational morphology resources. |
BME-UW at SRST-2019: Surface realization with Interpreted Regular Tree Grammars (D19-63)
Copied to clipboard
| Challenge: | adaamko's system restores word order and inflection from a graph of typed, directed dependencies between lemmas. |
| Approach: | They propose a method that restores word order and inflection from a graph of typed, directed dependencies between lemmas. |
| Outcome: | The proposed system restores word order and inflection from a graph of typed, directed dependencies between lemmas. |
Loki: An Open-Source Tool for Fact Verification (2025.coling-demos)
Copied to clipboard
Haonan Li, Xudong Han, Hao Wang, Yuxia Wang, Minghan Wang, Rui Xing, Yilin Geng, Zenan Zhai, Preslav Nakov, Timothy Baldwin
| Challenge: | Loki is an open-source fact-checking tool designed to address the growing problem of misinformation. |
| Approach: | They propose a tool that breaks down the fact-checking task into five steps . they propose LOKI, which offers a semiautomated, human-in-the-loop approach . |
| Outcome: | a new open-source tool is designed to address the growing problem of misinformation . the tool breaks down the fact-checking task into five steps to assist human judgment . |
SymbolicThought: Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing methods for constructing character relationships from plain text are time-consuming and low in coverage. |
| Approach: | They propose a human-in-the-loop framework that combines LLM-based extraction with symbolic reasoning. |
| Outcome: | The proposed framework improves annotation accuracy and consistency while significantly reducing time cost. |
Toward Zero-Shot Instruction Following (2024.eacl-srw)
Copied to clipboard
| Challenge: | a novel approach to zero-shot cross-task generalization is proposed . prior work relied on demonstrations, but this approach could be overestimated . |
| Approach: | They propose a "demonstration-driven instruction following" setting for zero-shot cross-task generalization . they propose to automatically find out critical sentences in a paragraph-style task definition . |
| Outcome: | The proposed approach yields state-of-the-art performance on the Super-NaturalInstructions. |
ELEVANT: A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool (2022.emnlp-demos)
Copied to clipboard
| Challenge: | a tool for fine-grained evaluation of entity linkers on benchmarks is presented . a typical evaluation shows that overall precision and recall are poor . particular benchmarks often require very particular skills from an entity linker . |
| Approach: | They propose a tool for fine-grained evaluation of entity linkers on benchmarks . they use a graph-based tool to analyze performance of a set of entity links . |
| Outcome: | The proposed tool provides an automatic breakdown of the performance by error categories and by entity type. |
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models (2024.acl-demos)
Copied to clipboard
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, Huajun Chen
| Challenge: | Large Language Models (LLMs) suffer from knowledge cutoff or fallacy issues, which means they are unaware of unseen events or generate text with incorrect facts owing to outdated/noisy data. |
| Approach: | They propose an easy-to-use knowledge editing framework for Large Language Models that allows users to easily edit updated knowledge and adjust undesired behavior while minimizing the impact on unrelated inputs. |
| Outcome: | The proposed framework surpasses traditional fine-tuning in terms of reliability and generalization. |
Entity Resolution and Location Disambiguation in the Ancient Hindu Temples Domain using Web Data (N18-5)
Copied to clipboard
| Challenge: | Existing systems for resolving entities and disambiguating locations based on publicly available web data are challenging because of the limited information available on the Web. |
| Approach: | They propose a system for resolving entities and disambiguating locations based on publicly available web data in the domain of ancient Hindu Temples. |
| Outcome: | The proposed system resolves entities and disambiguates locations with high confidence using grammar rules and clustering algorithms. |
FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP (N19-4)
Copied to clipboard
| Challenge: | Existing approaches combine word embeddings with character-level features to model additional features such as subword structures and meaning ambiguity. |
| Approach: | They present FLAIR, an NLP framework that enables embeddings of word and document data . they propose a hierarchical learning architecture that concatenates output states of a character-level CNN or RNN with the output states from a task data. |
| Outcome: | The proposed framework hides embedding-specific engineering complexity and allows researchers to "mix and match" various embeddables with little effort. |
DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population (2022.emnlp-demos)
Copied to clipboard
Ningyu Zhang, Xin Xu, Liankuan Tao, Haiyang Yu, Hongbin Ye, Shuofei Qiao, Xin Xie, Xiang Chen, Zhoubo Li, Lei Li
| Challenge: | Existing knowledge extraction tools are not complete due to emerging entities and relations in real-world applications. |
| Approach: | They propose an open-source knowledge extraction toolkit DeepKE that supports low-resource, document-level and multimodal scenarios in the knowledge base population. |
| Outcome: | The proposed toolkit supports low-resource, document-level and multimodal scenarios in the knowledge base population. |
CogIE: An Information Extraction Toolkit for Bridging Texts and CogNet (2021.acl-demo)
Copied to clipboard
| Challenge: | CogNet is a knowledge base that integrates three types of knowledge: linguistic knowledge, world knowledge and commonsense knowledge. |
| Approach: | They propose an information extraction toolkit called CogIE that is a bridge connecting raw texts and CogNet. |
| Outcome: | The proposed toolkit can ground raw texts to CogNet and leverage different types of knowledge to enrich extracted results. |
GAIA: A Fine-grained Multimedia Knowledge Extraction System (2020.acl-demos)
Copied to clipboard
Manling Li, Alireza Zareian, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare Voss, Daniel Napierski, Marjorie Freedman
| Challenge: | Open source knowledge extraction tools are used for many real-world applications, but there is no comprehensive system for KE. |
| Approach: | They propose a multimedia knowledge extraction system that takes multimedia data from various sources and languages as input and creates a coherent, structured knowledge base. |
| Outcome: | The system achieves top performance at the recent NIST TAC SM-KBP2019 evaluation. |
Identifying Emotional and Polar Concepts via Synset Translation (2024.starsem-1)
Copied to clipboard
| Challenge: | Emotion identification and polarity classification seek to determine sentiment expressed by a writer. |
| Approach: | They propose a translation-based method for labeling each individual word sense and lexical concept into 20 different languages and translate them into multilingual sentiment lexicons. |
| Outcome: | The proposed method outperforms existing methods and is available on GitHub . it contains 12,429 emotional synsets and 15,567 polar synset. |
COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System (2022.acl-demo)
Copied to clipboard
Manling Li, Revanth Gangi Reddy, Ziqi Wang, Yi-shyuan Chiang, Tuan Lai, Pengfei Yu, Zixuan Zhang, Heng Ji
| Challenge: | a new system extracts supporting and refuting claims from COVID-19 related news . the system is publicly available at GitHub and DockerHub, with complete documentation. |
| Approach: | They propose a COVID-19 Claim Radar system that extracts supporting and refuting claims . the system leverages Wikidata as the hub to consolidate coreferential knowledge elements . |
| Outcome: | The system extracts supporting and refuting claims from COVID-19 pandemic information . it leverages Wikidata as the hub to merge coreferential knowledge elements . |
PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries (2025.naacl-long)
Copied to clipboard
Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, Patrick Ng, Zhiguo Wang
| Challenge: | Existing text-to-SQL systems focus on user questions with clear intentions that can be answered, but real user questions can be ambiguous with multiple interpretations or unanswerable due to a lack of relevant data. |
| Approach: | They construct a conversational text-to-SQL dataset called PRACTIQ, consisting of ambiguous and unanswerable questions inspired by real-world user questions. |
| Outcome: | The proposed system generates conversations with four turns, generating the user’s question, an assistant response seeking clarification, and the user's clarified SQL response with the natural language explanation of the execution results. |
No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size (2025.coling-industry)
Copied to clipboard
Ashok Urlana, Charaka Vinayak Kumar, Bala Mallikarjunarao Garlapati, Ajeet Kumar Singh, Rahul Mishra
| Challenge: | Large language models are playing a pivotal role in deploying strategic use cases across organizations . lack of studies examining potential challenges and risks associated with LLMs . |
| Approach: | They propose a case study and a practical guide for industries to utilize LLMs more efficiently. |
| Outcome: | The proposed study examines the literature on large language models with industry practitioners and examines industrial publications to address these questions. |
CogKGE: A Knowledge Graph Embedding Toolkit and Benchmark for Representing Multi-source and Heterogeneous Knowledge (2022.acl-demo)
Copied to clipboard
Zhuoran Jin, Tianyi Men, Hongbang Yuan, Zhitao He, Dianbo Sui, Chenhao Wang, Zhipeng Xue, Yubo Chen, Jun Zhao
| Challenge: | Existing methods focus on entity-centric knowledge, but CogKGE supports heterogeneous knowledge. |
| Approach: | They propose a knowledge graph embedding toolkit to represent multi-source and heterogeneous knowledge. |
| Outcome: | The proposed toolkit provides a unified programming framework for KGE tasks and a series of knowledge representations for downstream tasks. |
RESIN: A Dockerized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System (2021.naacl-demos)
Copied to clipboard
Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan Windisch Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Heng Ji
| Challenge: | We present a new information extraction system that can construct temporal event graphs from news documents. |
| Approach: | They propose a temporal event graph extraction system that can extract news documents . they extend the system from sentence-level event extraction to cross-document cross-media event extraction . |
| Outcome: | The proposed system can extract temporal event graphs from news documents in multiple languages and multiple data modalities. |
TP-Detector: Detecting Turning Points in the Engineering Process of Large-scale Projects (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Extensive experiments demonstrate the effectiveness of our proposed method on a constructed dataset compared to baseline methods. |
| Approach: | They propose a novel task of detecting turning points in the engineering process of large-scale projects by treating news streams as a window with multiple instances. |
| Outcome: | The proposed mode is able to detect transitions in news streams with multiple instances. |
Zuo Zhuan Ancient Chinese Dataset for Word Sense Disambiguation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a core task in natural language processing . ancient Chinese has rarely been used in WSD tasks due to lack of a dataset . |
| Approach: | They annotate ancient Chinese text Zuo Zhuan using a copyright-free dictionary . they apply a method to find the most appropriate sense in a context using k-NN . |
| Outcome: | The proposed dataset will be available on GitHub. |
FastFit: Fast and Effective Few-Shot Text Classification with a Multitude of Classes (2024.naacl-demo)
Copied to clipboard
| Challenge: | Few-shot prompting of large language models (LLMs) via API calls presents a unique challenge when dealing with a multitude of classes that share similar semantic meanings. |
| Approach: | They present a Python package that integrates batch contrastive learning and token-level similarity score to provide fast few-shot classification. |
| Outcome: | The proposed method significantly improves multi-class classification speed and accuracy across English and Multilingual datasets. |
PromptLab: A Collaborative Platform for Prompt Engineering and Dataset Curation (2026.eacl-demo)
Copied to clipboard
Maged S. Al-shaibani, Zaid Alyafeai, Dania Refai, Nawaf Alomari, Ahmed Ashraf, Mais Alheraki, Mustafa Alturki, Hamzah Luqman, Irfan Ahmad
| Challenge: | PromptLab is a web-based prompt engineering platform for collaborative prompt development across diverse natural language processing tasks and datasets. |
| Approach: | They propose to integrate prompt generation via OpenRouter and provide real-time validation with multiple Large Language Models. |
| Outcome: | The platform addresses primary challenges in prompt development, including template creation, collaborative review, and quality assurance through a comprehensive workflow that supports both individual researchers and team-based projects. |
AGRank: Augmented Graph-based Unsupervised Keyphrase Extraction (2022.aacl-main)
Copied to clipboard
| Challenge: | Unsupervised keyphrase extraction (UKE) is highly anticipated because no labeled data is needed to train a model. |
| Approach: | They propose an augmented graph-based unsupervised model to identify keyphrases from a document by integrating graph and deep learning methods. |
| Outcome: | The proposed model is effective and robust for long and short documents. |
MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models (2023.emnlp-demo)
Copied to clipboard
| Challenge: | MusicAgent integrates numerous music-related tools and an autonomous workflow to address user requirements. |
| Approach: | a new system is built to integrate music-related tools and an autonomous workflow . the system is based on large language models (LLMs) that can be used to organize and decompose requests . |
| Outcome: | the proposed system integrates numerous music-related tools and an autonomous workflow to address user requirements. |
A weakly supervised textual entailment approach to zero-shot text classification (2023.eacl-main)
Copied to clipboard
Marc Pàmies, Joan Llop, Francesco Multari, Nicolau Duran-Silva, César Parra-Rojas, Aitor Gonzalez-Agirre, Francesco Alessandro Massucci, Marta Villegas
| Challenge: | Existing methods to train on weakly supervised datasets are expensive due to the computational cost of pre-training. |
| Approach: | They propose a method that trains on a weakly supervised dataset that is used as a proxy for a textual entailment problem and a target zero-shot text classification task. |
| Outcome: | The proposed model achieves state-of-the-art performance in the scientific domain and competitive results in other areas. |
VehicleWorld: A Highly Integrated Multi-Device Environment for Intelligent Vehicle Interaction (2025.findings-emnlp)
Copied to clipboard
Jie Yang, Jiajun Chen, Zhangyue Yin, Shuo Chen, Yuxin Wang, Yiran Guo, Yuan Li, Yining Zheng, Xuanjing Huang, Xipeng Qiu
| Challenge: | Traditional Function Calling (FC) approaches operate statelessly, requiring multiple exploratory calls to build environmental awareness before execution, leading to inefficiency and limited error recovery. |
| Approach: | They propose a state-based function call approach that maintains explicit system state awareness and implements direct state transitions to achieve target conditions. |
| Outcome: | The proposed approach outperforms traditional function calling approaches, achieving superior execution accuracy and reduced latency. |
LLM as Entity Disambiguator for Biomedical Entity-Linking (2025.acl-short)
Copied to clipboard
| Challenge: | Entity linking involves normalizing a mention in medical text to a unique identifier in a knowledge base, such as UMLS or MeSH. |
| Approach: | They propose to use a large language model as an entity disambiguator to enhance the accuracy of alias-matching entity linking methods. |
| Outcome: | The proposed method surpasses existing methods on biomedical datasets by up to 16 points in accuracy. |
BabelDOC: Better Layout-Preserving PDF Translation via Intermediate Representation (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing document translation pipelines face a tension between linguistic processing and layout preservation. |
| Approach: | They propose a framework for layout-preserving PDF translation that decouples visual layout metadata from semantic content. |
| Outcome: | The proposed framework improves layout fidelity, visual aesthetics, and terminology consistency over representative baselines while maintaining competitive translation precision. |
LogiTorch: A PyTorch-based library for logical reasoning on natural language (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Logical reasoning on natural language is one of the most challenging tasks for deep learning models. |
| Approach: | They propose a library that includes logical reasoning benchmarks and models . they use a dataset to run the library or fine tune the models with different hyperparameters . |
| Outcome: | This library includes logical reasoning benchmarks, models, utility functions and co-reference resolution. |
string2string: A Modern Python Library for String-to-String Algorithms (2024.acl-demos)
Copied to clipboard
| Challenge: | Notable algorithms include the Smith-Waterman algorithm for pairwise local alignment, the Hirschberg algorithm for global alignment, and the Wagner-Fischer algorithm for edit distance. |
| Approach: | **string2string** is an open-source library that offers efficient algorithms for string-to-string problems. |
| Outcome: | **string2string** is an open-source library that offers efficient algorithms for string-to-string problems. |
Corpus Annotation Graph Builder (CAG): An Architectural Framework to Create and Annotate a Multi-source Graph (2023.eacl-demo)
Copied to clipboard
| Challenge: | Graphs are a natural representation of complex data as their structure allows users to discover (often implicit) relations among the nodes intuitively. |
| Approach: | They propose a corpus annotation graph framework that extends graphs with automatically extracted annotations. |
| Outcome: | The proposed framework can be used for further analyses across multiple downstream tasks. |
Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions (2024.acl-demos)
Copied to clipboard
| Challenge: | Fundus is a news scraper that extracts news articles from the web with just a few lines of code. |
| Approach: | They introduce Fundus, a news scraper that enables users to obtain news articles with just a few lines of code. |
| Outcome: | The proposed news scraper optimizes for quality and provides a unified interface for newspapers. |
EVALIGN: Visual Evaluation of Translation Alignment Models (2023.eacl-demo)
Copied to clipboard
| Challenge: | EvAlign is a visual analytics framework for quantitative and qualitative evaluation of automatic translation alignment models. |
| Approach: | They propose to use EvAlign to analyze automatic translation alignment models and compare their performance with other baseline and state-of-the-art models. |
| Outcome: | The framework hosts nine gold standard datasets and the predictions of multiple alignment models. |
Benchmarking Deep Search over Heterogeneous Enterprise Data (2025.emnlp-industry)
Copied to clipboard
Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Kung-Hsiang Huang, Caiming Xiong, Chien-Sheng Wu
| Challenge: | Existing methods struggle to conduct deep searches and retrieve all necessary evidence. |
| Approach: | They propose a benchmark for evaluating deep search, a retrieval-augmented generation that requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources. |
| Outcome: | The proposed benchmarks show that even the best-performing agentic RAG methods achieve an average performance score of 32.96 on the benchmark. |
mbrs: A Library for Minimum Bayes Risk Decoding (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Minimum Bayes risk (MBRS) decoding is a decision rule of text generation tasks that outperforms conventional maximum a posteriori (MAP) decoders by selecting high-quality outputs based on quality or preference rather than probability. |
| Approach: | They propose to use minimum bayes risk (MBRS) decoding to determine outputs based on quality rather than probability. |
| Outcome: | MBRS is an MIT-licensed open-source project with a focus on speed, reproducibility, and extensibility. |
mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models (2026.acl-demo)
Copied to clipboard
| Challenge: | mllm-shap is an open-source Python platform for researchers and ML practitioners that extends Shapley value (SV) explainability from text-only large languagemodels to multimodal LLMs that process both text and audio. |
| Approach: | They present a framework that extends Shapley value (SV) explainability from text-only large languagemodels to multimodal LLMs thatjointly process text and audio. |
| Outcome: | mllm-shap is the first publicly available framework for complete, reproducible SV-based explainability of text-audioMLLMs. |
SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are expensive and easy to conduct but ineffective to reflect dialogue quality. |
| Approach: | They propose a self-supervised fine-grained dialogue evaluation framework which can automatically assign fine-granular scores for arbitrarily dialogue data. |
| Outcome: | The proposed framework is highly consistent with human evaluations and better than the state-of-the-art models. |
GreaterPrompt: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimization (2025.acl-demo)
Copied to clipboard
| Challenge: | Recent advances in prompt optimization have introduced diverse techniques that automatically enhance prompts to better align model outputs with user expectations. |
| Approach: | They propose a framework that unifies different methods under a unified, customizable API while delivering highly effective prompts for different tasks. |
| Outcome: | The proposed framework unifies multiple methods under a unified, customizable API while delivering highly effective prompts for different tasks. |
Evaluating Large Language Models with Enterprise Benchmarks (2025.naacl-industry)
Copied to clipboard
Bing Zhang, Mikio Takeuchi, Ryo Kawahara, Shubhi Asthana, Maruf Hossain, Guang-Jie Ren, Kate Soule, Yifan Mai, Yada Zhu
| Challenge: | Existing benchmarks lack domain-specific datasets for evaluating large language models . existing benchmarks often lack domain specific datasets, which can be difficult to convert to standardized metrics or regulatory issues. |
| Approach: | They propose to use 25 publicly available domain-specific English benchmarks from diverse domains . they propose to combine a wide range of natural language processing tasks for holistic evaluation . |
| Outcome: | The proposed framework includes 25 publicly available domain-specific English benchmarks from diverse enterprise domains like financial services, legal, climate, cyber security, and 2 public Japanese finance benchmarks. |
TokCollate: A Comprehensive Tool for Tokenizer Evaluation and Visualization across Languages (2026.acl-demo)
Copied to clipboard
| Challenge: | Tokenization quality varies significantly across languages, leading to disparities in LLM performance and cost for speakers of less-resourced languages. |
| Approach: | They propose a Python-based evaluation framework and a JavaScript visualization interface that evaluates tokenizers in a variety of languages. |
| Outcome: | TokCollate is an evaluation framework for tokenizers with a JavaScript visualization interface. |
entity-linkings: A Unified Library for Entity Linking (2026.eacl-demo)
Copied to clipboard
| Challenge: | Entity linking (EL) is the task of mapping named entities in text to canonical entries in a knowledge base. |
| Approach: | They propose a unified library for using and developing entity linking systems . a strong emphasis is placed on usability, making it highly extensible . |
| Outcome: | a new library aims to disambiguate named entities in text by mapping them to canonical entries in a knowledge base. |
PaperMage: A Unified Toolkit for Processing, Representing, and Manipulating Visually-Rich Scientific Documents (2023.emnlp-demo)
Copied to clipboard
Kyle Lo, Zejiang Shen, Benjamin Newman, Joseph Chang, Russell Authur, Erin Bransom, Stefan Candra, Yoganand Chandrasekhar, Regan Huff, Bailey Kuehl, Amanpreet Singh, Chris Wilhelm, Angele Zamarron, Marti A. Hearst, Daniel Weld, Doug Downey, Luca Soldaini
| Challenge: | Existing tools for working with scientific documents are limited and documents are often in difficult-to-use PDF formats. |
| Approach: | They propose an open-source Python toolkit for analyzing and processing visually-rich scientific documents. |
| Outcome: | PaperMage provides turn-key recipes for common scientific document processing use-cases. |
Rank-Aware Negative Training for Semi-Supervised Text Classification (2023.tacl-1)
Copied to clipboard
| Challenge: | Semi-supervised text classification-based paradigms employ the spirit of self-training, but the accuracy of pseudo-labels can be a problem in real-world scenarios. |
| Approach: | They propose a Rank-aware Negative Training framework to address SSTC in noisy label learning . they rank unlabeled texts based on evidential support from the labeled texts. |
| Outcome: | The proposed framework overcomes state-of-the-art alternatives and achieves competitive performance in other scenarios. |
Structured Discourse Representation for Factual Consistency Verification (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to verify factual consistency of text capture a performance gap compared with sentence-level entailment. |
| Approach: | They propose a method that combines structured discourse information extraction with a classifier, FDSpotter, for factual consistency verification. |
| Outcome: | The proposed method achieves competitive performance on two tasks: data-to-text generation and text summarisation. |
ViHOS: Hate Speech Spans Detection for Vietnamese (2023.eacl-main)
Copied to clipboard
| Challenge: | Increasing use of social networking sites can cause problems for human moderators to review tagged comments. |
| Approach: | They present a dataset that contains 26k spans on 11k comments and detailed annotation guidelines . they also provide definitions of hateful and offensive spans in Vietnamese comments . |
| Outcome: | The proposed dataset shows that it is difficult to detect specific types of spans in the dataset . the dataset is the first human-annotated corpus containing 26k spans on 11k comments . |
CocoSciSum: A Scientific Summarization Toolkit with Compositional Controllability (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Using a controlled summarization system, we provide a toolkit for scientific summarizing documents. |
| Approach: | They propose a toolkit for controlled summarization of scientific documents that adjusts key attributes specifically of length and keyword inclusion. |
| Outcome: | The proposed system generates high-quality summaries based on user preferences, adjusting key attributes specifically of length and keyword inclusion. |
ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses (2025.emnlp-demos)
Copied to clipboard
Michael Galarnyk, Rutwik Routu, Vidhyakshaya Kannan, Kosha Bheda, Prasun Banerjee, Agam Shah, Sudheer Chava
| Challenge: | ARR Responsible NLP Research checklist is designed to encourage best practices for responsible research . previous research has shown that self-reported checklist responses don't always accurately represent papers . |
| Approach: | They propose a retrieval-augmented generation application that can be used to assist authors with conference checklists. |
| Outcome: | The proposed application can be used to help authors with conference checklists and review their work. |
TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretability (2025.emnlp-demos)
Copied to clipboard
Mohammad Aflah Khan, Ameya Godbole, Johnny Wei, Ryan Yixiang Wang, James Flemings, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
| Challenge: | Existing workflows for pretraining large language models are cumbersome, fragmented and inaccessible. |
| Approach: | They propose an open-source library for editing, inspection, and analysis of large language model datasets. |
| Outcome: | TokenSmith is an open-source library for editing, inspection, and analysis of large language model datasets. |
UKP-SQuARE v3: A Platform for Multi-Agent QA Research (2023.acl-demo)
Copied to clipboard
Haritz Puerto, Tim Baumgärtner, Rachneet Sachdeva, Haishuo Fang, Hao Zhang, Sewin Tariverdian, Kexin Wang, Iryna Gurevych
| Challenge: | Current approaches to QA models are multi-dataset models, but combining expert agents can yield large performance gains over multi-agent models. |
| Approach: | They extend an online platform for QA research to support three families of multi-agent systems: agent selection, early-fusion of agents, and late-fusion. |
| Outcome: | The proposed model can be compared with multi-dataset models and achieve high inference speed and performance. |
FC-KBQA: A Fine-to-Coarse Composition Framework for Knowledge Base Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for question answering over knowledge bases (KBQA) suffer from generalization issues due to coarse-grained modeling of the logical expression. |
| Approach: | They propose a fine-to- coarse-grained framework for KBQA to ensure generalization and executability of the logical expression. |
| Outcome: | The proposed framework derives new state-of-the-art performance on GrailQA and WebQSP, and runs 4 times faster than baseline. |
GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration (2023.acl-demo)
Copied to clipboard
Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hailey Schoelkopf, Stella Biderman, Martin Potthast, Jimmy Lin
| Challenge: | Using the mature and well-tested methods from the domain of Information Retrieval (IR) we propose to integrate Pyserini with Hugging Face to provide qualitative analysis tools for NLP researchers. |
| Approach: | They propose to integrate Pyserini with Hugging Face to provide qualitative analysis tools for NLP researchers. |
| Outcome: | The proposed tools can be integrated with the Hugging Face ecosystem of open-source AI libraries and artifacts. |
SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in code generation benchmarks, but these benchmarks focus on single-file scenarios with constrained context scope. |
| Approach: | They propose an open-source framework to effectively resolve GitHub issues using a code file retrieval module and a model-based code editing module. |
| Outcome: | The proposed approach achieves state-of-the-art performance on two GitHub benchmarks. |
EZ-STANCE: A Large Dataset for Zero-Shot Stance Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | EZ-STANCE is a large dataset for zero-shot stance detection in english . it includes both noun-phrase targets and claim targets covering a wide range of domains. |
| Approach: | They present a large English ZSSD dataset with 30,606 annotated text-target pairs . they propose to transform EZ-STANCE into the NLI task by applying two simple yet effective prompts to noun-phrase targets. |
| Outcome: | The proposed dataset includes noun-phrase targets and claim targets covering a wide range of domains. |
GraphLSS: Integrating Lexical, Structural, and Semantic Features for Long Document Extractive Summarization (2025.naacl-short)
Copied to clipboard
| Challenge: | Graph-based methods for extracting documents have been popular, but they often require external tools or additional machine learning models to define graph components. |
| Approach: | They propose a heterogeneous graph construction for extractive summarization that defines two levels of information and four types of edges without any need for auxiliary learning models. |
| Outcome: | The proposed graph construction outperforms previous graph-based models on two datasets and is available on GitHub. |
Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains (2026.findings-acl)
Copied to clipboard
Md. Faiyaz Abdullah Sayeedi, Subhey Sadi Rahman, Md. Mahbub Alam, Md. Adnanul Islam, Jannatul Ferdous Deepti, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Large Language Models (LLMs) have redefined Machine Translation, enabling context-aware and fluent translations across hundreds of languages and textual domains. |
| Approach: | They propose a framework and dataset to evaluate the translation quality and fairness of open-source LLMs. |
| Outcome: | The proposed framework and dataset evaluates translation quality and fairness of open-source LLMs. |
Pisets: A Robust Speech Recognition System for Lectures and Interviews (2025.naacl-industry)
Copied to clipboard
Ivan Bondarenko, Daniil Grebenkin, Oleg Sedukhin, Mikhail Klementev, Derunets Roman, Lyudmila Budneva
| Challenge: | Sustainable speech recognition systems are essential for scientists, journalists, and anyone processing audio recordings of interviews and meetings. |
| Approach: | They propose a speech-to-text system "Pisets" which is based on a three-component architecture aimed at improving speech recognition accuracy while minimizing errors and hallucinations associated with the Whisper model. |
| Outcome: | The proposed system ensures robust transcribing of long audio data across various acoustic conditions compared to WhisperX and the usual Whisper model. |
A Survey of Data Augmentation Approaches for NLP (2021.findings-acl)
Copied to clipboard
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, Eduard Hovy
| Challenge: | Data augmentation is a field of research that has been underexplored due to the discrete nature of language data. |
| Approach: | They present a comprehensive survey of data augmentation for NLP by summarizing the literature in a structured manner. |
| Outcome: | The proposed methods are used for popular NLP applications and tasks and highlight current challenges and directions for future research. |
Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities, but still suffer from inconsistency issues. |
| Approach: | They develop a ConsisEval benchmark to evaluate LLMs' inconsistency . they find that LLM models can paradoxically fail at easier problems . |
| Outcome: | The proposed model achieves highest consistency score but inconsistent to specific questions due to distraction by redundant information, misinterpretation of questions, etc. |
BioEL: A Comprehensive Python Package for Biomedical Entity Linking (2025.findings-naacl)
Copied to clipboard
Prasanth Bathala, Christophe Ye, Batuhan Nursal, Shubham Lohiya, David Kartchner, Cassie S. Mitchell
| Challenge: | Entity Linking in biomedical literature is a critical task that enhances the extraction and integration of information from diverse scientific literature. |
| Approach: | They propose a Python package that allows for better Entity Linking in biomedical literature . the package includes four components: Ontology Object, Dataset Object and Evaluation Framework . |
| Outcome: | The proposed open-source package enables the implementation and comparison of biomedical entity linking tasks. |
Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis (C18-1)
Copied to clipboard
| Challenge: | a lack of reproducibility and generalisability is a major threat to scientific development in Natural Language Processing. |
| Approach: | They propose to use a model zoo to document and release language models and published code . they recommend that future replication experiments should consider a variety of datasets . |
| Outcome: | The proposed methods are compared on six English datasets and are based on the results. |
Fast and Accurate Factual Inconsistency Detection Over Long Documents (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generative AI models exhibit remarkable potential, however, hallucinations across various tasks present a significant challenge, particularly for longer inputs. |
| Approach: | They propose a task-agnostic model that uses large text chunks to condition over long texts and employ a novel algorithm to explain its decisions through relevant source sentence retrieval. |
| Outcome: | The proposed model outperforms existing methods on benchmarks and a new long-form dialogue dataset and surpasses competitive systems in efficiency and model explanation evaluations. |
Multi-Constraint State Tracking with Negation: A Diagnostic Benchmark for LLM World Modeling (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluations of Large Language Models (LLMs) lack the ability to maintain and update internal representations of an evolving world despite their ubiquity in real-world language. |
| Approach: | They propose a diagnostic benchmark for multi-constraint state tracking that evaluates an LLM's ability to maintain consistent world models across sequences of actions involving inventory changes, spatial movement, temporal ordering, and systematic negation. |
| Outcome: | The proposed model is based on 100,847 questions spanning 12 real-world domains and includes culturally diverse entity names to enable analysis of cross-cultural robustness. |
Stance Detection in COVID-19 Tweets (2021.acl-long)
Copied to clipboard
| Challenge: | a global pandemic of COVID-19 has forced major changes in our daily lives . a new stance detection dataset is being used to track the stances of Twitter users . |
| Approach: | They use Twitter stance data to collect stances on topics related to the pandemic . they train models to take advantage of large amounts of unlabeled data . |
| Outcome: | The proposed model improves on existing stance detection datasets and unlabeled data. |
COMPKE: Complex Question Answering under Knowledge Editing (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for knowledge editing do not accurately evaluate how well models apply knowledge in real-life situations. |
| Approach: | They propose a benchmark to evaluate how well updated models apply new knowledge in real-life situations. |
| Outcome: | The proposed method achieves 39.47 accuracy on GPT-4o-mini but drops significantly to 3.83 on Qwen2.5-3B. |
MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning (2026.findings-eacl)
Copied to clipboard
Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Existing benchmarks primarily focus on English or a narrow subset of high-resource languages, leaving significant gaps in assessing multilingual and cross-lingual mathematical reasoning. |
| Approach: | They propose a parallel multilingual benchmark for mathematical problem solving and reasoning that encompasses 2,890 parallel Bangla-English gold standard artifacts. |
| Outcome: | The proposed model encompasses 2,890 parallel Bangla-English gold standard artifacts, totaling 30K aligned question–answer pairs across thirteen languages, representing high-, medium-, and low-resource linguistic settings. |
Enhancing Reliability in Community Question Answering with an Expert-Oriented RAG System (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing LLMs require substantial computational resources and are prone to generating hallucinated or unreliable content. |
| Approach: | They propose an expert-oriented Retrieval-Augmented Generation framework which leverages user modeling to identify archived questions with answers that fully or partially address the user’s new query. |
| Outcome: | The proposed framework synthesizes expert-written answers from similar questions to generate unified answers. |
uMedSum: A Unified Framework for Clinical Abstractive Summarization (2025.acl-long)
Copied to clipboard
Aishik Nagar, Yutong Liu, Andy T. Liu, Viktor Schlegel, Vijay Prakash Dwivedi, Arun-Kumar Kaliya-Perumal, Guna Pratheep Kalanchiam, Yili Tang, Robby T. Tan
| Challenge: | Clinical abstractive summarization struggles to balance faithfulness and informativeness, sacrificing key information or introducing confabulations. |
| Approach: | They develop a modular hybrid framework that integrates confabulation removal and key information addition into abstractive summarization methods. |
| Outcome: | The proposed framework outperforms state-of-the-art abstractive summarization methods in both quantitative metrics and expert evaluations. |
Repo4QA: Answering Coding Questions via Dense Retrieval on GitHub Repositories (2022.coling-1)
Copied to clipboard
| Challenge: | Stack Overflow and GitHub are open source communities that are gaining popularity . developers need to raise programming questions in coding forums and navigate to GitHub repositories . |
| Approach: | They propose a questionrepository matching task that bridges the gap between repositories and real-world coding questions. |
| Outcome: | The proposed model outperforms state-of-the-art methods on coding questions and repositories . it can find suitable coding repositoriels and bridge the gap between them . |
A Self-verified Method for Exploring Simile Knowledge from Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have succeeded in natural language processing because they learn generic knowledge from a large corpus. |
| Approach: | They propose a method that allows pre-trained language models to explore simile knowledge from PLMs . they enhance PLM models with a multi-level simile recognition task that evaluates similes aplenty . |
| Outcome: | The proposed method can explore more accurate simile knowledge for PLMs. |
MultiMatch: Multihead Consistency Regularization Matching for Semi-Supervised Text Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | **MultiMatch** is a semi-supervised learning (SSL) algorithm that combines co-training and consistency regularization with pseudo-labeling. |
| Approach: | They propose a semi-supervised learning algorithm that integrates co-training and consistency regularization with pseudo-labeling. |
| Outcome: | The proposed algorithm outperforms the second-best approach on 8 out of 10 setups from 5 natural language processing datasets and outperformed the second best by 3.26%. |
Dynamic and Efficient Inference for Text Generation via BERT Family (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve performance of pre-trained language models are limited due to large-scale parameters and the universal autoregressive decoding paradigm. |
| Approach: | They propose a novel fine-tuning method which can make a single pre-trained model support Dynamic and Efficient infERence and achieve an adaptive trade-off between model performance and latency. |
| Outcome: | The proposed method achieves higher BLEU scores than the strong autoregressive Transformer model on translation tasks with 3 12 times speedup and faster inference speed compared with the BART model on four GLGE benchmark tasks. |
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021.acl-long)
Copied to clipboard
| Challenge: | Creating effective visualizations is a challenge given the myriad of parameters that users need to provide. |
| Approach: | They propose to synthesize visualization programs from natural language utterances and programmatic context using PlotCoder. |
| Outcome: | The proposed architecture models both the code context and the input utterance. |
CodeRAG-Bench: Can Retrieval Augment Code Generation? (2025.findings-naacl)
Copied to clipboard
Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, Daniel Fried
| Challenge: | Language models excel at generating code, but many programs are difficult to generate using only parametric knowledge. |
| Approach: | They propose a retrieval-augmented code generation benchmark that provides reproducible evaluations on retrieval and end-to-end code generation performance. |
| Outcome: | The proposed benchmark covers programming, open-domain, and repository-level tasks and provides reproducible evaluations on retrieval and end-to-end code generation performance. |
Exploring Question Guidance and Answer Calibration for Visually Grounded Video Question Answering (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for videoQA lack temporal localization labels, leading to inaccurate localization. |
| Approach: | They propose a Question-Guided and Answer-Calibrated TRansformer which guides and calibrates localization using question and option texts without localization labels. |
| Outcome: | The proposed model achieves comparable accuracy to large-scale pretrained models and leads in localization aspects. |
UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have enabled the development of coding agents for real-world code generation. |
| Approach: | They propose a novel LLM-driven test case generator that analyzes codebases and dependencies to generate test cases for real-world Python projects. |
| Outcome: | The proposed framework improves the performance of SWE-Bench by analyzing codebases and dependencies. |
The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks to evaluate LLMs' capabilities are inadequate for assessing their musical capabilities. |
| Approach: | They propose to use a large-scale music benchmark specifically designed to evaluate the music-related capabilities of large language models (LLMs). |
| Outcome: | The proposed framework evaluates 16 large language models in the domain of music. |
CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Modern software development demands code that is maintainable, testable, and scalable by organizing the implementation into modular components with iterative reuse of existing codes. |
| Approach: | They propose a benchmark to evaluate LLMs' ability to perform codeflow by reusing existing functions over multiple turns. |
| Outcome: | The proposed benchmarks show that LLMs perform significantly worse in multi-turn codeflow scenarios and that their performance inversely correlates with dependency complexity. |
Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES) (2026.findings-acl)
Copied to clipboard
Guido Ivetta, Pietro Palombini, Sofía Martinelli, Marcos J Gomez, M Emilia Echeveste, Sunipa Dev, Vinodkumar Prabhakaran, Luciana Benotti
| Challenge: | a geo-cultural gap in NLP evaluation hinders evaluation of societal biases . authors propose a new method to collect stereotypes from large language models . |
| Approach: | They propose a new method that integrates sourcing and validation of existing data into a single workflow. |
| Outcome: | The proposed method improves LACES by integrating new stereotype entries and validation of existing data. |
A Multi-Task Learning Framework for Multi-Target Stance Detection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models fail to learn target-specific representations and are prone to overfitting. |
| Approach: | They propose a multi-task learning network to train one model on all target pairs . their results show that their proposed model outperforms the best-performing baseline by 12.39% . |
| Outcome: | The proposed model outperforms the best-performing baseline model by 12.39% in macro-averaged F1-score. |
Measuring Faithful and Plausible Visual Grounding in VQA (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lack of visual grounding (VG) in VQA systems can manifest in over-reliance on irrelevant image parts or a disregard for the visual modality entirely. |
| Approach: | They propose a new metric that measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
| Outcome: | The proposed metric measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
A UIMA Database Interface for Managing NLP-related Text Annotations (L18-1)
Copied to clipboard
| Challenge: | despite the use of UIMA as a document-based schema, it does not provide native database support. |
| Approach: | They develop a database interface to allow generic use of UIMA documents in database systems. |
| Outcome: | The framework is evaluated in relation to file system-based storage and provides data protection. |
In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models fail to adapt to unfamiliar speakers and language varieties . however, there are significant gaps in the adaptation of certain varieties based on the test speaker, variety, or recording conditions . |
| Approach: | They propose a framework that allows for in-context learning in Phi-4 Multimodal . they find that as few as 12 example utterances reduce word error rates by 19.7% . |
| Outcome: | The proposed framework reduces word error rates by 19.7% across diverse English corpora. |
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent generative language models have shown promise in abstractive summarization tasks. |
| Approach: | They propose to use Fr echet embedding distance and angular embeddable similarity to evaluate the performance of generative language models in abstractive summarization tasks. |
| Outcome: | The proposed metric shows close relation with human judgments and has overall better correlations with them. |
WIKIR: A Python Toolkit for Building a Large-scale Wikipedia-based English Information Retrieval Dataset (2020.lrec-1)
Copied to clipboard
| Challenge: | ad-hoc information retrieval methods usually require large amounts of annotated data to be effective. |
| Approach: | They propose an open-source toolkit to automatically build large-scale English information retrieval datasets based on Wikipedia. |
| Outcome: | The proposed toolkit builds large-scale English information retrieval datasets based on Wikipedia with 59,252 queries and 2,617,003 pairs. |
SynFix: Dependency-Aware Program Repair via RelationGraph Analysis (2025.findings-acl)
Copied to clipboard
Xunzhu Tang, Jiechao Gao, Jin Xu, Tiezhu Sun, Yewei Song, Saad Ezzini, Wendkûuni C. Ouédraogo, Jacques Klein, Tegawendé F. Bissyandé
| Challenge: | Existing methods for resolving repository-level debugging are limited by their interdependencies. |
| Approach: | They propose a RelationGraph-based approach that integrates large language models with structural search and synchronization techniques for coordinated program repair across codebases. |
| Outcome: | SynFix resolves 52.33% of issues in SWE-bench-lite, 55.8% in Swe-bech-verified and 29.86% in S WE-beach-full. |
Connecting a French Dictionary from the Beginning of the 20th Century to Wikidata (2022.lrec-1)
Copied to clipboard
| Challenge: | The Petit Larousse illustré is a French dictionary first published in 1905 . its value remains intact, but some descriptions are more historical than contemporary . wikidata identifiers can be used to identify, compare, and verify historically-situated representations . |
| Approach: | They propose a wikidata-based annotation of the Petit Larousse illustré entries from 1905 . they link the entries to current data sources and use them to extract, complement, and process knowledge. |
| Outcome: | The proposed lexical resource connects dictionary entries from 1905 to current data sources. |
How Inclusively do LMs Perceive Social and Moral Norms? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) are used in decision-making systems and as interactive assistants. |
| Approach: | They propose to prompt 11 LMs on rules-of-thumb and compare their outputs with 100 human annotators. |
| Outcome: | The proposed model is compared with 100 human annotators to find out if they are inclusive of diverse human values. |
ChatEL: Entity Linking with Chatbots (2024.lrec-main)
Copied to clipboard
| Challenge: | Entity Linking (EL) is a challenging task in natural language processing . existing approaches focus on creating elaborate contextual models that are unwieldy and difficult to train . |
| Approach: | They propose a framework to prompt LLMs to return accurate results for Entity Linking . they use a three-step framework to generate a set of EL models that can be open-source . |
| Outcome: | The proposed framework improves the average F1 performance across 10 datasets by more than 2%. |
Uncovering Visual-Semantic Psycholinguistic Properties from the Distributional Structure of Text Embedding Space (2025.acl-long)
Copied to clipboard
| Challenge: | Imageability and concreteness are psycholinguistic properties that link visual and semantic spaces. |
| Approach: | They propose an unsupervised measure that quantifies sharpness of peaks in an image-caption dataset. |
| Outcome: | The proposed method is more robust than existing methods and predicts these properties for classification. |
Instance-Guided Prompt Learning for Few-Shot Text Matching (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Few-shot text matching is a more practical technique to determine whether two texts are semantically identical. |
| Approach: | They propose a pluggable prompt learning method for few-shot text matching . they use the semantics of instances to regulate the effects of the gate on the prompt tokens . |
| Outcome: | The proposed method outperforms baselines on MRPC and QQP. |
Leveraging Capsule Routing to Associate Knowledge with Medical Literature Hierarchically (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to integrate knowledge into text can confuse the representation and import unexpected noises. |
| Approach: | They propose to leverage capsule routing to associate knowledge with medical literature hierarchically . they extract two fragments from medical literature and encode them into fragment representations . |
| Outcome: | The proposed method can more accurately associate knowledge with medical literature than mainstream methods. |
code-transformed: The Influence of Large Language Models on Code (2026.findings-eacl)
Copied to clipboard
| Challenge: | Using Large Language Models, code generation capabilities have transformed programming practices. |
| Approach: | They analyze 20,000 GitHub repositories linked to arXiv papers published between 2020 and 2025 . they identify measurable trends in the evolution of coding style that align with LLM-generated code . |
| Outcome: | The proposed study examines 20,000 GitHub repositories linked to arXiv papers . it finds that LLMs influence code style, and that they can be observed in real-world code . |
StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Domain-specific datasets of harmful prompts are scarce and often rely on manual construction. Existing efforts to improve domain knowledge and reduce harmful prompt generation are lacking. |
| Approach: | They propose a framework that transforms domain knowledge into actionable constraints and increases the implicitness of generated harmful prompts. |
| Outcome: | The proposed framework yields high-quality datasets combining strong domain relevance with implicitness, enabling more realistic red-teaming and advancing LLM safety research. |
RoD-TAL: A Benchmark for Answering Questions in Romanian Driving License Exams (2026.findings-eacl)
Copied to clipboard
Andrei Vlad Man, Răzvan-Alexandru Smădu, Cristian-George Craciun, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel
| Challenge: | a growing need for tools that support legal education, especially in under-resourced languages such as Romanian . we evaluate the capabilities of large language models and vision-language models in legal education . |
| Approach: | They evaluate the capabilities of Large Language Models and Vision-Language Models in Romanian driving law through textual and visual question-answering tasks. |
| Outcome: | The proposed model improves retrieval performance and QA accuracy in Romanian driving tests. |
uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work on distilling Whisper’s knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. |
| Approach: | They propose a framework that distills Whisper’s knowledge into small models using pseudo-labels and reduces the size by up to 50%. |
| Outcome: | The proposed model outperforms the teacher model by 5-7 WER points and is 25-50% more efficient when scaling the data. |
Almost Free Semantic Draft for Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Empirical experiments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. |
| Approach: | They propose a method to sample and consider a semantic draft as global information from semantic space for decoding with almost free of cost. |
| Outcome: | Empirical results show that the proposed method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. |
NIPS Conversational Intelligence Challenge 2017 Winner System: Skill-based Conversational Agent with Supervised Dialog Manager (C18-1)
Copied to clipboard
| Challenge: | a dialog system developed for the 1st NIPS Conversational Intelligence Challenge 2017 (ConvAI) aims to train a bot capable of conversing with humans based on a given passage of text. |
| Approach: | They propose to implement a bot capable of conversing with humans based on a given passage of text. |
| Outcome: | The proposed bot won the 1st NIPS Conversational Intelligence Challenge 2017 (ConvAI) the system has been trained in a supervised setting using a dialogue manager . |
Jejueo Datasets for Machine Translation and Speech Synthesis (2020.lrec-1)
Copied to clipboard
| Challenge: | Jejueo, or the Jeju language, is a minority language used on Jeju Island . there have been many efforts to revitalize the language, but few computational approaches have been used to solve its problems. |
| Approach: | They construct two new Jejueo datasets using interviews and transcripts . they build machine translation and speech synthesis using these datasets based on their results . |
| Outcome: | The proposed datasets will attract interest of both language and machine learning communities. |
Query Decomposition for RAG: Balancing Exploration-Exploitation (2026.eacl-long)
Copied to clipboard
Roxana Petcu, Kenton Murray, Daniel Khashabi, Evangelos Kanoulas, Maarten de Rijke, Dawn Lawrie, Kevin Duh
| Challenge: | Complex user queries often involve the exclusion of information, negation, or missing entities. |
| Approach: | They propose to decompose user requests into subqueries, retrieve potentially relevant documents for each and then aggregate them to generate an answer. |
| Outcome: | The proposed method achieves 35% gain in document-level precision and 15% increase in -nDCG . it also improves the downstream task of long-form generation. |
Document Classification for COVID-19 Literature (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a global pandemic has made it more important than ever to quickly and accurately retrieve relevant scientific literature for effective consumption by researchers in a wide variety of fields. |
| Approach: | They analyze a LitCovid dataset to find out how classification models can help organize COVID-19 research papers. |
| Outcome: | The proposed model outperforms all baseline models on the LitCovid dataset . it also outperformed BioBERT and other models with micro-F1 and accuracy scores of 86% and 75% . |
An Analysis of Dialogue Act Sequence Similarity Across Multiple Domains (2022.lrec-1)
Copied to clipboard
| Challenge: | a recent study shows that many machine learning models perform poorly when exposed to domain shifts due to contextual differences. |
| Approach: | They analyze dialogue act sequences from related domains to predict performance degradation . they find that when dialogue acts sequences are dissimilar they lie further away in embedding space . |
| Outcome: | The proposed model can be trained even when the datasets are corrupted with noise. |
Active Few-Shot Learning for Text Classification (2025.naacl-long)
Copied to clipboard
Saeed Ahmadnia, Arash Yousefi Jordehi, Mahsa Hosseini Khasheh Heyran, Seyed Abolghasem Mirroshandel, Owen Rambow, Cornelia Caragea
| Challenge: | Recent advances in Large Language Models (LLMs) have boosted the use of Few-Shot Learning (FSL) methods in natural language processing. |
| Approach: | They propose a method that identifies effective support instances from the unlabeled pool and can work with different LLMs. |
| Outcome: | The proposed method improves on five tasks on which it is tested on five LLMs. |
DeltaScore: Fine-Grained Story Evaluation with Perturbations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics for stories are limited in assessing intricate aspects of storytelling, such as fluency and interestingness. |
| Approach: | They propose a novel method that uses perturbation techniques to evaluate story aspects . they compare fluency, coherence, relatedness, logicality, interestingness and interestingness to existing metrics . |
| Outcome: | The proposed method shows that one specific perturbation is highly effective in capturing multiple aspects. |
ViHateT5: Enhancing Hate Speech Detection in Vietnamese With a Unified Text-to-Text Transformer Model (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for HSD in Vietnamese focus on fine-tuning general pre-trained models, primarily trained on formal textual datasets like Wikipedia. |
| Approach: | They propose a T5-based HSD model pre-trained on a domain-specific dataset . their results highlight the significance of label distribution in pre-training data on model efficacy. |
| Outcome: | The proposed model can tackle multiple tasks using a unified model and achieve state-of-the-art performance across all standard HSD benchmarks in Vietnamese. |
Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | a study of FOMC pronouncements shows how important the FOMC communications are . hawkish-dovish classification is difficult because of the negative connotations of words . |
| Approach: | They propose to use a dataset to classify FOMC monetary policy stances . they construct a measure of monetary stance for the FOMC document release days . |
| Outcome: | The proposed model is based on a best-performing model and is available on Huggingface and GitHub under CC BY-NC 4.0 license. |
Systematic Inequalities in Language Technology Performance across the World’s Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have revealed that NLP is limited to a subset of the world’s 6,500 languages. |
| Approach: | They propose a framework for estimating the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP. |
| Outcome: | The proposed framework estimates the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP. |
TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks (2026.eacl-long)
Copied to clipboard
| Challenge: | Automated software engineering is a critical task of software engineers. |
| Approach: | They propose a benchmark to evaluate software migration in real-world Python projects. |
| Outcome: | The proposed benchmark consists of GitHub repositories whose tests fail in response to dependency updates. |
SWE-QA: Can Language Models Answer Repository-level Code Questions? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for understanding and reasoning about entire soft-ware repositories focus on small, self-contained code snippets. |
| Approach: | They propose a repository-level code question answering benchmark to facilitate research on automated QA systems in real-world repositories. |
| Outcome: | The proposed benchmarks are designed to facilitate research on automated QA systems in real-world repositories. |
Implicit Deep Latent Variable Models for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Variational auto-encoders have been used for text generation but their representation power is limited due to two reasons. |
| Approach: | They advocate sample-based representations of variational distributions for natural language . they further develop an LVM to directly match the aggregated posterior to the prior . |
| Outcome: | The proposed model can be viewed as a natural extension of VAEs with a regularization of maximizing mutual information, mitigating the "posterior collapse" issue. |
The Effect of Sampling Temperature on Problem Solving in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Despite anecdotal reports, changes in temperature do not have a statistically significant impact on LLM performance for problem-solving tasks. |
| Approach: | They use a multiple-choice question-and-answer exam to investigate the effect of sampling temperature on LLM performance. |
| Outcome: | The results show that temperature changes do not have a statistically significant impact on LLM performance on problem-solving tasks. |
CDS: Data Synthesis Method Guided by Cognitive Diagnosis Theory (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, but the complexity of emerging tasks and higher performance demands highlight the need for continuous improvement. |
| Approach: | They propose a method that refines evaluation results and characterizes model profiles at the knowledge component level. |
| Outcome: | The proposed method improves performance across multiple benchmarks and academic exams. |
Towards Robust Few-Shot Relation Classification: Incorporating Relation Description with Agreement (2025.findings-emnlp)
Copied to clipboard
Mengting Hu, Jianfeng Wu, Ming Jiang, Yalan Xie, Zhunheng Wang, Rui Ying, Xiaoyi Liu, Ruixuan Xu, Hang Gao, Renhong Cheng
| Challenge: | Existing approaches to recognize relational relationships with a few support samples are limited for unlimited queries. |
| Approach: | They propose a simple but effective framework that uses relation descriptions as external knowledge to enhance the model’s comprehension of the relation semantics. |
| Outcome: | The proposed framework outperforms strong baselines while being robust against various NOTA rates. |
Multi-Vector Attention Models for Deep Re-ranking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Document retrieval systems often use two styles of neural network models . dual encoder models are used for retrieval and deep re-ranking, while cross-attention models are typically used for shallow reranking. |
| Approach: | They propose a dual encoder and cross-attention neural network architectures that combine query and document representations to optimize retrieval accuracy. |
| Outcome: | The proposed architecture trades off retrieval accuracy with joint computation and offline document storage cost. |
MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) can generate coherent text, but they struggle to recognise user intent behind queries. |
| Approach: | They propose a novel approach leveraging multi-level intent, domain, and slot knowledge distillation for multi-turn NLU. |
| Outcome: | The proposed model improves multi-turn conversation understanding by integrating teacher teachers into a student model. |
DISRPT: A Multilingual, Multi-domain, Cross-framework Benchmark for Discourse Processing (2024.lrec-main)
Copied to clipboard
Chloé Braud, Amir Zeldes, Laura Rivière, Yang Janet Liu, Philippe Muller, Damien Sileo, Tatsuya Aoyama
| Challenge: | DISRPT is a multilingual, multi-domain, and cross-framework benchmark dataset for discourse processing. |
| Approach: | They present a multilingual, multi-domain, and cross-framework benchmark dataset for discourse processing that includes 13 languages and 24 corpora covering about 4 millions tokens and around 250,000 discourse relation instances from 4 discourse frameworks. |
| Outcome: | The DISRPT dataset includes data from 24 corpora covering about 4 millions tokens and around 250,000 discourse relation instances from 4 discourse frameworks. |
I run as fast as a rabbit, can you? A Multilingual Simile Dialogues Datasets (2023.findings-acl)
Copied to clipboard
| Challenge: | A simile is a figure of speech that compares two different things via shared properties. |
| Approach: | They propose a multilingual simile dialogue dataset that can be used to study similes in real-life scenarios. |
| Outcome: | The proposed dataset is the largest manually annotated simile dataset and contains both English and Chinese data. |
Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Current PP methods face severe bottlenecks, including pipeline bubbles and memory footprint. |
| Approach: | They propose a sequence-level one-forward-one-backward (1F1B) PP method for training LLMs on long sequences with high throughput and memory efficiency. |
| Outcome: | The proposed method achieves 1.14X training throughput with half memory footprint compared to baseline methods . it trains an LLM with 30B parameters on sequences up to 64k tokens using 64X NVIDIA A100 GPUs . |
Machine Unlearning of Pre-trained Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using curated datasets, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining. |
| Approach: | They propose a framework for machine unlearning in pre-trained LLMs and integrate gradient ascent with gradient descent on in-distribution data to achieve robustness. |
| Outcome: | The proposed framework is over 105 times more efficient than retraining on in-distribution data and provides detailed guidelines for efficient hyperparameter tuning in the unlearning process. |
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |
EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria through Retrieval-Augmented Fine-Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Eligibility criteria (EC) are critical components of clinical trial design, specifying parameters for participant inclusion and exclusion. |
| Approach: | They propose a method that utilizes Retrieval-Augmented Fine-Tuning to generate structured and cohesive EC directly from clinical trial titles and descriptions. |
| Outcome: | The proposed method outperforms Llama-3.1-8B-Instruct and Llm-as-a-Judge models in BERTScore and EC score. |
Grammar Control in Dialogue Response Generation for Language Learning Chatbots (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing language learning chatbots and research on second language acquisition benefit from these affordances. |
| Approach: | They ground a dialogue response generation model in a pedagogical repository of grammar skills and evaluate prompting, fine-tuning, and decoding strategies for grammar-controlled dialogue response generators. |
| Outcome: | The proposed model outperforms GPT-3.5 when tolerating minor response quality losses and predicts grammar-controlled responses to support grammar acquisition adapted to learner proficiency. |
TWBias: A Benchmark for Assessing Social Bias in Traditional Chinese Large Language Models through a Taiwan Cultural Lens (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capabilities in natural language processing, but concerns about social bias amplification remain. |
| Approach: | They propose a social bias evaluation benchmark for Traditional Chinese LLMs that integrates chat templates and diverse prompts for comprehensive bias assessment. |
| Outcome: | The proposed model incorporates chat templates and diverse prompts for comprehensive bias assessment focusing on Taiwan's cultural context and prioritizing gender and ethnicity bias evaluation. |
Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation (2025.emnlp-main)
Copied to clipboard
Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, Baishakhi Ray
| Challenge: | In the era of evaluating large language models, data contamination is an increasingly prominent concern . static benchmarking has been used for evaluation, but there are limitations of *dynamic* benchmarks . |
| Approach: | They propose a series of optimal design principles for *dynamic* benchmarking and analyze the limitations of existing *static* benchmarks. |
| Outcome: | The proposed benchmarks highlight a critical gap in the evaluation of LLMs. |
VEEF-Multi-LLM: Effective Vocabulary Expansion and Parameter Efficient Finetuning Towards Multilingual Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a significant disadvantage for low-resource languages . VEEF-Multi-LLM-8B excels in multilingual instruction-following tasks . |
| Approach: | They propose a low-resource multilingual large language model that expands the vocabulary for multilingual support. |
| Outcome: | The proposed model outperforms existing models in multilingual instruction-following tasks, but lags behind English-centric models in some tasks. |
Factual Knowledge Assessment of Language Models Using Distractors (2025.coling-main)
Copied to clipboard
Hichem Ammar Khodja, Abderrahmane Ait gueni ssaid, Frederic Bechet, Quentin Brabant, Alexis Nasr, Gwénolé Lecorvé
| Challenge: | Language models encode extensive factual knowledge within their parameters. |
| Approach: | They propose a new interpretable knowledge assessment method that leverages distractors to provide incorrect alternatives to the correct answer. |
| Outcome: | The proposed method shows that it is aligned with human judgment and stronger robustness to verbalization artifacts. |
A Learning-Based Dependency to Constituency Conversion Algorithm for the Turkish Language (2022.lrec-1)
Copied to clipboard
Büşra Marşan, Oğuz K. Yıldız, Aslı Kuzgun, Neslihan Cesur, Arife B. Yenice, Ezgi Sanıyar, Oğuzhan Kuyrukçu, Bilge N. Arıcan, Olcay Taner Yıldız
| Challenge: | a team of linguists manually annotated a set of constituency trees. |
| Approach: | They propose to create the first Turkish-based dependency-to-constituency conversion algorithm using a morphologic analyser and feature-based machine learning model. |
| Outcome: | The proposed algorithm can be used to generate new constituency treebanks and training data for NLP resources like constituency parsers. |
Kanbun-LM: Reading and Translating Classical Chinese in Japanese Methods by Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Classical Chinese was introduced to Japan approximately 2,000 years ago . it was gradually adapted to a Japanese form called Kanbun-Kundoku (Kanbun) in Japanese reading and translating methods . |
| Approach: | They construct a dataset that compares Classical Chinese and Kanbun in Japan using character reordering and machine translation tasks. |
| Outcome: | The proposed dataset compares the current language models with human scores and compared them with human-level models. |
Understanding LLMs’ summarization capabilities: an analysis of biomedical abstract and lay summary generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Abstracts use technical language for academic audiences, while lay summaries aim to make findings accessible to non-specialists. |
| Approach: | They evaluate the performance of lightweight LLMs in generating biomedical abstracts and lay summaries in a zero-shot setting. |
| Outcome: | The proposed models perform well in generating biomedical abstracts and lay summaries in a zero-shot setting. |
Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede’s Cultural Dimensions (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are deployed in many countries, but they fail to account for cultural variances among their potential users. |
| Approach: | They propose to use Hofstede’s cultural dimension framework to quantify cultural alignment using latent variable analysis to evaluate large language models against cultural dimensions of regions like the United States, China, and Arab countries. |
| Outcome: | The proposed model is compared against LLMs in the United States, China, and Arab countries and demonstrates that all models struggle to grasp cultural values, while GPT-4 shows a unique capability to adapt to cultural nuances, particularly in Chinese settings. |
RNSum: A Large-Scale Dataset for Automatic Release Note Generation via Commit Logs Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | a release note is a technical document that describes the latest changes to a software product. |
| Approach: | They propose to extract and then abstract release notes from GitHub repositories using a transformer-based network like BART. |
| Outcome: | The proposed methods generate less noisy release notes at higher coverage than baselines. |
LongTail-Swap: benchmarking language models’ abilities on rare words (2025.findings-emnlp)
Copied to clipboard
Robin Algayres, Charles-Éric Saint-James, Mahi Luthra, Jiayi Shen, Youssef Benchekroun, Dongyan Lin, Rashel Moritz, Juan Pino, Emmanuel Dupoux
| Challenge: | LongTail-Swap is a benchmark that focuses on the tail of the word distribution, i.e., measures the ability of LMs to learn new words with very little exposure, like infants do. |
| Approach: | They introduce LongTail-Swap, a benchmark that measures the ability of language models to learn new words with very little exposure, like infants do. |
| Outcome: | The proposed benchmark measures the ability of language models to learn new words with very little exposure, like infants do. |
Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Personalized Large Language Models are increasingly used in diverse applications . prior research examined how well LLMs adhere to predefined personas in writing style . inconsistent responses are influenced by multiple factors, including the assigned persona, stereotypes, and model design choices. |
| Approach: | They propose a standardized framework to analyze consistency in persona-assigned LLMs. |
| Outcome: | The proposed framework evaluates personas across multiple tasks and runs. |
RADS: Reinforcement Learning-Based Sample Selection Improves Transfer Learning in Low-resource and Imbalanced Clinical Settings (2026.findings-acl)
Copied to clipboard
| Challenge: | Experimental evaluations show that RL methods favor outliers rather than truly informative samples under low-resource and class-imbalanced conditions. |
| Approach: | They propose a robust sample selection strategy using reinforcement learning to identify the most informative samples using a class imbalance approach. |
| Outcome: | The proposed strategy improves model transferability while maintaining robust performance under extreme class imbalance compared to traditional methods. |
MATCH: Task-Driven Code Evaluation through Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GitHub Copilot generates 46% of the code on GitHub. |
| Approach: | They propose a reference-free metric that uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions. |
| Outcome: | This paper compares the performance of a new similarity score with existing metrics. |
HeLa-Mem: Hebbian Learning and Associative Memory for LLM Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Existing memory systems represent conversation history as unstructured embedding vectors, retrieving information through semantic similarity. |
| Approach: | They propose a bio-inspired memory architecture that models memory as a dynamic graph with Hebbian learning dynamics. |
| Outcome: | The proposed architecture leverages both semantic similarity and learned associations . it can be used to build a bio-inspired memory graph with Hebbian learning dynamics . |
Tonguescape: Exploring Language Models Understanding of Vowel Articulation (2025.naacl-long)
Copied to clipboard
| Challenge: | a study shows that language models can explain vowel pronunciation based on tongue positions . a visual LM can explain the relationship between vowels and tongue positions, but it is unclear whether they align textual information with visual information. |
| Approach: | They created video and image datasets from MRI data to examine if LMs associate real tongue positions with vowel articulation. |
| Outcome: | The proposed model can explain vowel pronunciation and the correlation between vowels and tongue positions as textual knowledge. |
CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated significant advances in computer science research . current agent-related applications include code writing, code base generation, code correction and more. |
| Approach: | They propose a benchmark to assess the effectiveness of Large Language Models (LLMs) they propose GitHub agent framework that automates deployment of GitHub repositories . |
| Outcome: | The proposed framework improves the deployment of GitHub code repositories and thereby boosts developer productivity. |
Self-Training with Pseudo-Label Scorer for Aspect Sentiment Quad Prediction (2024.acl-long)
Copied to clipboard
| Challenge: | Aspect Sentiment Quad Prediction (ASQP) aims to predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review. |
| Approach: | They propose a self-training framework with a pseudo-label scorer to assess the match between reviews and their pseudo-labels and train a generative model on it. |
| Outcome: | The proposed framework can predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review, and it can significantly improve self-training. |
Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm (2024.findings-acl)
Copied to clipboard
Yuanzhen Xie, Xinzhou Jin, Tao Xie, Matrixmxlin Matrixmxlin, Liang Chen, Chenyun Yu, Cheng Lei, Chengxiang Zhuo, Bo Hu, Zang Li
| Challenge: | In-context learning of large-language models has achieved remarkable success in the field of natural language processing . however, the single-step chain-of-thought prompting approach faces challenges such as attention diffusion and inadequate performance in complex tasks like text-to-SQL. |
| Approach: | They propose a workflow paradigm method to enhance the attention and problem-solving scope of large-language models through decomposition. |
| Outcome: | The proposed method outperforms existing methods on three datasets and improves the upper limit of LLM-based approaches. |
TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments (2025.findings-acl)
Copied to clipboard
Yuheng Lu, Qian Yu, Hongru Wang, Zeming Liu, Wei Su, Yanping Liu, Yuhang Guo, Maocheng Liang, Yunhong Wang, Haifeng Wang
| Challenge: | Existing GUI agents struggle to adapt to dynamic and interconnected nature of real-world digital environments, authors show . |
| Approach: | They propose a benchmark to evaluate the transferability of GUI agents across three key dimensions . transBench includes 15 app categories with diverse functionalities . |
| Outcome: | The proposed benchmark shows that existing GUI agents struggle to adapt to dynamic, interconnected environments. |
FRASIMED: A Clinical French Annotated Resource Produced through Crosslingual BERT-Based Annotation Projection (2024.lrec-main)
Copied to clipboard
Jamil Zaghir, Mina Bjelogrlic, Jean-Philippe Goldman, Soukaïna Aananou, Christophe Gaudet-Blavignac, Christian Lovis
| Challenge: | Existing methods for generating annotations for large datasets are time-consuming and resource-intensive. |
| Approach: | They propose a method for generating translated versions of annotated datasets through crosslingual annotation projection. |
| Outcome: | The proposed method shows that it is efficient and high-quality in the resulting dataset. |
Korean-Specific Dataset for Table Question Answering (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing question answering systems mainly focus on text data, but few Korean datasets exist . a dataset for table question answering is written in English, but it lacks Korean-specific datasets . |
| Approach: | They construct Korean-specific datasets for table question answering using crowd-sourced workers . they then fine-tune the model with these datasets and report the evaluation results . |
| Outcome: | The proposed model is based on Korean datasets and is publicly available . the model is evaluated against other datasets from Korean question answering systems . |
TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts (2024.emnlp-main)
Copied to clipboard
| Challenge: | a framework for formal proof writing using formal languages like Lean4 is needed to prove mathematical theorems using formal language. |
| Approach: | They propose a framework that trains a general-purpose LLM to be a Lean4 expert. |
| Outcome: | The proposed framework achieves cumulative accuracies of 36.48% and 33.61% on MiniF2F-Valid and Test datasets. |
HealthAlignSumm : Utilizing Alignment for Multimodal Summarization of Code-Mixed Healthcare Dialogues (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Collaboration between doctors and AI scientists is leading to personalized models to stream-line healthcare tasks and improve productivity. |
| Approach: | They propose to use alignment techniques to combine a doctor-patient dialogue with a visual component of the BART model. |
| Outcome: | The proposed model in-tegrates visual components with the BART ar-chitecture. |
To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models for multilingual automatic speech recognition (ASR) are computationallyintensive and lack proper comprehensive evaluations. |
| Approach: | They propose to distill knowledge from large teacher models into smaller student variants that are more efficient. |
| Outcome: | The proposed model outperforms existing models on standard benchmarks and dialectal data. |
Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks (2024.acl-long)
Copied to clipboard
Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed
| Challenge: | MLLMs have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension, but they are limited to English-based settings. |
| Approach: | They propose a family of Arabic multimodal large language models with strong vision and language capabilities. |
| Outcome: | The proposed models show strong performance on visual reasoning tasks and language capabilities. |
AxCell: Automatic Extraction of Results from Machine Learning Papers (2020.emnlp-main)
Copied to clipboard
Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, Robert Stojnic
| Challenge: | AxCell is an automatic machine learning pipeline for extracting results from papers . it uses a table segmentation subtask to learn relevant structural knowledge that aids extraction. |
| Approach: | They propose to use a table segmentation subtask to learn relevant structural knowledge that aids extraction. |
| Outcome: | The proposed approach improves state of the art for results extraction and can be used for semi-automated results extraction in production. |
VisualRWKV: Exploring Recurrent Neural Networks for Visual Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have been gaining popularity with large language models, but few attempts have been made to incorporate efficient linear Recurrent Neural Networks (RNNs) into VLMs. |
| Approach: | They propose a linear RNN model with a data-dependent recurrence and sandwich prompts to enhance modeling capabilities and a 2D image scanning mechanism to enrich the processing of visual sequences. |
| Outcome: | The proposed model achieves competitive performance compared to Transformer-based models on various benchmarks. |
When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents? (2026.acl-long)
Copied to clipboard
Yibo Peng, James Song, Lei Li, Xinyu Yang, Mihai Christodorescu, Ravi Mangal, Corina S. Pasareanu, Haizhong Zheng, Beidi Chen
| Challenge: | Code agents are increasingly trusted to autonomously fix bugs on platforms such as GitHub, yet their security evaluation focuses on functional correctness. |
| Approach: | They propose to attack functionally correct yet vulnerable (FCV) patches by combining multi-turn reasoning with tool invocation and environment interaction. |
| Outcome: | The proposed FCV-Attack achieves an attack success rate of 40.7% on GPT-5 Mini + OpenHands. |
Aligning the Romanian Reference Treebank and the Valence Lexicon of Romanian Verbs (2022.lrec-1)
Copied to clipboard
| Challenge: | Among the language resources for Romanian, there are ones that describe the syntactic and semantic aspects of the language. |
| Approach: | They propose to align two language resources for Romanian: the Romanian Reference Treebank and the Valence Lexicon of Romanian Verbs. |
| Outcome: | The proposed alignments identify morpho-syntactic annotation mistakes, incomplete valence frames or missing ones. |
Two Intermediate Translations Are Better Than One: Fine-tuning LLMs for Document-level Translation Refinement (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research has shown that large language models (LLMs) can enhance translation quality through self-refinement. |
| Approach: | They propose to extend translation refinement from sentence-level to document-level by using document-to-document (Doc2Doc) translations. |
| Outcome: | The proposed method improves translation quality across ten translation tasks with LLaMA-3-8B-Instruct and Mistral-Nemo-Instru. |
Humanitarian Corpora for English, French and Spanish (2024.lrec-main)
Copied to clipboard
| Challenge: | et al., a leading database of humanitarian documents, compiled with ReliefWeb reports . documents selected with language identification and noise reduction techniques . authors present corpora of English, French and Spanish humanitarian documents . |
| Approach: | They present three corpora of English, French and Spanish humanitarian documents compiled with ReliefWeb reports . documents were tokenized, lemmatized, tagged by part of speech, and enriched with metadata . authors propose a project to develop a humanitarian dictionary with a focus on conceptual variation . |
| Outcome: | The corpora were compiled to satisfy the research needs of the Humanitarian Encyclopedia project with a focus on conceptual variation. |
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing scientific benchmarks lack human-annotated difficulty levels and structured taxonomies of scientific concepts. |
| Approach: | They propose a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats with human-annotated difficulty levels and detailed explanations. |
| Outcome: | The proposed model achieves only 63.77% accuracy and struggles with visual reasoning tasks. |
A Corpus of German Reddit Exchanges (GeRedE) (2020.lrec-1)
Copied to clipboard
| Challenge: | Reddit is a popular online platform combining social news aggregation, discussion and microblogging. |
| Approach: | They propose a method to filter out German data and further pre-processing steps to find out what is linguistically peculiar in the German data. |
| Outcome: | The proposed method filters out German data and includes metadata and annotation layers. |
Python Code Generation by Asking Clarification Questions (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work addresses text-to-code generation using pretrained language models (PLMs) for large-scale NLD: Logistic Regression. |
| Approach: | They propose a dataset containing pairs of natural language descriptions and code with created synthetic clarification questions and answers to solve the under-specified nature of a natural language description. |
| Outcome: | The proposed model improves on previous models, while introducing new challenges to the community, including when and what clarification questions should be asked. |
FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual models have been released, but many of the world's languages are not covered. |
| Approach: | They propose a method that initializes the embedding matrix for a new tokenizer based on information in the source model's embeddable matrix. |
| Outcome: | The proposed method outperforms random initialization and previous work on language modeling and on a range of downstream tasks (NLI, QA, and NER). |
GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors (2020.lrec-1)
Copied to clipboard
| Challenge: | Lack of large-scale datasets has been a major hindrance to the development of NLP tasks such as spelling correction and grammatical error correction. |
| Approach: | They propose to make GitHub Typo Corpus a multilingual dataset of misspellings and grammatical errors available for use in NLP. |
| Outcome: | The proposed dataset contains more than 350k edits and 65M characters in more than 15 languages. |
A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new taxonomy of negation is proposed to improve neural information retrieval models . negation types are covered in existing datasets, allowing for faster convergence . |
| Approach: | They propose a taxonomy of negation that derives from philosophical, linguistic, and logical definitions . they also propose analyzing the performance of retrieval models on existing datasets using a logic-based classification mechanism. |
| Outcome: | The proposed taxonomy produces a balanced data distribution over negation types . it also provides a better training setup that leads to faster convergence on the NevIR dataset . |
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation (2025.acl-long)
Copied to clipboard
Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, Scarlett Li
| Challenge: | Existing benchmarks focus on standalone programming problems, such as HumanEval, MBPP, and LiveCodeBench. |
| Approach: | They propose to use large language models to evaluate their ability to perform incremental development within code repositories by collecting pull requests from 83 GitHub repositorias and using rule-based and intent-based filtering to construct task instances focused on new feature development. |
| Outcome: | The proposed benchmarks show that large language models perform significantly worse in the FEA-Bench, highlighting considerable challenges in repository-level incremental code development. |
KazParC: Kazakh Parallel Corpus for Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Statistical machine translation gained ground over rule-based machine translation in the late 1990s thanks to its ability to learn from large bilingual corpora. |
| Approach: | They propose to develop a parallel corpus for machine translation across Kazakh, English, Russian, and Turkish. |
| Outcome: | The proposed model outperforms Google Translate and Yandex Translate in terms of performance and evaluation metrics. |
KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, sentiment analysis is a widely employed text classification task that involves extracting the sentiment expressed by individuals towards a variety of entities. |
| Approach: | They propose to use KazSAnDRA to automate Kazakh sentiment analysis by developing and evaluating four machine learning models for polarity and score classification. |
| Outcome: | The proposed dataset is the first and largest publicly available dataset of its kind. |
Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub (2025.acl-long)
Copied to clipboard
Bohan Lyu, Xin Cong, Heyang Yu, Pan Yang, Cheng Qian, Zihe Wang, Yujia Qin, Yining Ye, Yaxi Lu, Chen Qian, Zhong Zhang, Yukun Yan, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing approaches lack flexibility to address diverse and ever-evolving user queries in open domains. |
| Approach: | They propose to evaluate LLMs on open-domain knowledge that requires tools to solve diverse and ever-evolving user queries. |
| Outcome: | The proposed system outperforms baselines in the open domain task-solving benchmark. |
LEGOBench: Scientific Leaderboard Generation Benchmark (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a growing number of papers make it difficult to stay informed about the latest state-of-the-art research. |
| Approach: | They propose a benchmark to evaluate systems that generate scientific leaderboards . they use 22 years of submission data on arXiv and 11k machine learning leaderboard data on paperswithcode . |
| Outcome: | The proposed model shows significant performance gaps in the LEGOBench model . the model is based on a language model and four graph-based leaderboard generation task configuration . |
ReviewRL: Towards Automated Scientific Review with RL (2025.emnlp-main)
Copied to clipboard
Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru Wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, Biqing Qi, Bowen Zhou
| Challenge: | Existing automated review systems struggle with factual accuracy, rating consistency, and analytical depth. |
| Approach: | They propose a framework for generating comprehensive and factually grounded scientific paper reviews using supervised fine-tuning and reinforcement learning. |
| Outcome: | The proposed framework outperforms existing methods on ICLR 2025 papers. |
Debate4MATH: Multi-Agent Debate for Fine-Grained Reasoning in Math (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing data annotation methods suffer from high annotation cost and lack of effective automatic validation. |
| Approach: | They propose a Fine-grained Multi-Agent Debate framework and a dataset that prompts multiple agents to debate and then a Multi-agent Debates Reward Model (MRM) to improve its mathematical reasoning capabilities. |
| Outcome: | The proposed model outperforms the state-of-the-art methods by 1.2% and 3.5% on a GSM8K dataset and 45.1% on the MATH dataset. |
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have shown strong abilities to understand natural language, but how these factors affect the models’ language perception is unclear. |
| Approach: | They compare the self-attention of several existing large language models in different sizes to assess the effect of scaling and instruction tuning on language perception. |
| Outcome: | The proposed models are closer to non-native speakers than native speakers in attention, suggesting a sub-optimal language perception of all models. |
A Comprehensive Evaluation of Biomedical Entity Linking Models (2023.emnlp-main)
Copied to clipboard
David Kartchner, Jennifer Deng, Shubham Lohiya, Tejasri Kopparthi, Prasanth Bathala, Daniel Domingo-Fernández, Cassie Mitchell
| Challenge: | Current methods struggle to correctly link genes and proteins and often have difficulty incorporating context into linking decisions. |
| Approach: | They evaluate nine recent state-of-the-art biomedical entity linking models under a unified framework. |
| Outcome: | The proposed models are compared along axes of accuracy, speed, ease of use, generalization, adaptability and adaptability to new ontologies and datasets. |
R.R.: Unveiling LLM Training Privacy through Recollection and Ranking (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing privacy attacks focus on membership inference or data extraction, but reconstructing specific personally identifiable information (PII) in training data remains challenging. |
| Approach: | They propose a two-step privacy stealing attack that enables attackers to reconstruct PII entities from scrubbed training data where the PI I entities have been masked. |
| Outcome: | The proposed attack can reconstruct PII entities from scrubbed training data where the PI I entities have been masked. |
Region-Grounded Report Generation for 3D Medical Imaging: A Fine-Grained Dataset and Graph-Enhanced Framework (2026.acl-long)
Copied to clipboard
Cong Huy Nguyen, Son Dinh Nguyen, Guanlin Li, Tuan Dung Nguyen, Aditya Narayan Sankaran, Mai Huy Thong, Thanh Trung Nguyen, Mai Hong Son, Reza Farahbakhsh, Phi Le Nguyen, Noel Crespi
| Challenge: | Current methods map whole volumes to reports, ignoring the clinical workflow of analyzing localized Regions of Interest (RoIs) Current models exhibit suboptimal accuracy and are prone to significant hallucinations. |
| Approach: | They propose a framework that mimics the professional radiologist diagnostic workflow by employing graph-based relational modules to capture dependencies between RoI attributes. |
| Outcome: | The proposed framework surpasses existing models by 19.7% in BLEU and 4.7% in ROUGE-L while achieving a 45.8% improvement in clinical metrics. |
BenNumEval: A Benchmark to Assess LLMs’ Numerical Reasoning Capabilities in Bengali (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in general-purpose tasks but struggle with numerical reasoning, especially in low-resource languages like Bengali. |
| Approach: | They propose a benchmark to assess LLMs on numerical reasoning tasks in Bengali. |
| Outcome: | The proposed benchmark assesses LLMs on numerical reasoning tasks in Bengali. |
Exploring the Impact of Model Scaling on Parameter-Efficient Tuning (2023.emnlp-main)
Copied to clipboard
Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang, Ning Ding, Xingzhi Sun, Guotong Xie, Zhiyuan Liu, Maosong Sun
| Challenge: | Parameter-efficient tuning (PET) methods can drive large pre-trained language models by training only minimal parameters. |
| Approach: | They propose a parameter-efficient tuning method that is compatible with a tunable module and uses a random number generator to optimize fewer table parameters. |
| Outcome: | The proposed method is compatible with a tunable module and tested on 11 NLP tasks. |
VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraft (2024.findings-acl)
Copied to clipboard
| Challenge: | Multi-agent collaboration using LLMs is a challenging research topic that aims to enable multiple autonomous agents to coordinate their actions and achieve a common goal. |
| Approach: | They propose a benchmark for multi-agent collaboration in the Minecraft environment and introduce a Directed Acyclic Graph Multi-Agent Framework to resolve complex inter-ag dependencies. |
| Outcome: | The proposed framework outperforms existing ModelVerse, reducing hallucinations and improving task decomposition efficacy. |
The Lies Characters Tell: Utilizing Large Language Models to Normalize Adversarial Unicode Perturbations (2025.findings-acl)
Copied to clipboard
| Challenge: | Homoglyphs are visually homogeneous to Latin letters and are used to mask offensive content. |
| Approach: | They propose two methods to normalize homoglyphs by replacing non-Latin characters with a delimiter and using large language models to determine which characters should be replaced with Latin letters. |
| Outcome: | The proposed methods normalize homoglyphs by replacing non-Latin characters with a delimiter and prompting large language models to "fill in the blanks" the authors found that the proposed methods produced normalized text with an average cosine similarity score of 0.91 to the original tweets and 0.96 to the tweets using the direct method. |
LinguaGame: A Linguistically Grounded Game-Theoretic Paradigm for Multi-Agent Dialogue Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have enabled Multi-Agent Systems (MASs) where agents interact through natural language to solve complex tasks or simulate multi-party dialogues. |
| Approach: | They propose a linguistically-grounded game-theoretic paradigm for multi-agent dialogue generation that uses a training-free equilibrium approximation algorithm to model dialogue over communicative intents and strategies. |
| Outcome: | The proposed framework improves agents’ communication efficiency by helping them convey their intended meaning more effectively through language. |
NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports (2023.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trial reports (CTRs) are indispensable for the development of personalized medicine. |
| Approach: | They propose a resource to help researchers interpret clinical trial reports . they use natural language inference to compute textual entailment . |
| Outcome: | The proposed resource is the first to cover interpretation of full clinical trial reports . it includes tasks to determine inference relation between natural language statements and CTRs . |
A MISMATCHED Benchmark for Scientific Natural Language Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for scientific NLI are derived from various computer science domains, whereas non-CS domains are completely ignored. |
| Approach: | They propose a scientific natural language inference benchmark called MisMatched that incorporates sentence pairs having an implicit scientific NLI relation into model training. |
| Outcome: | The proposed benchmark covers three non-CS domains and contains 2,700 human annotated sentence pairs. |
READoc: A Unified Benchmark for Realistic Document Structured Extraction (2025.findings-acl)
Copied to clipboard
Zichao Li, Aizier Abulaiti, Yaojie Lu, Xuanang Chen, Jia Zheng, Hongyu Lin, Xianpei Han, Shanshan Jiang, Bin Dong, Le Sun
| Challenge: | Document Structured Extraction (DSE) is a field of document structure analysis that aims to extract structured content from raw documents. |
| Approach: | They propose a benchmark to evaluate document structured extraction systems by converting unstructured PDFs into semantically rich Markdown. |
| Outcome: | The proposed benchmark is based on 3,576 diverse and real-world documents from arXiv, GitHub, and Zenodo. |
Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising performance in temporal reasoning tasks such as temporal question answering. |
| Approach: | They propose to use large language models to detect temporal relations between events with in-context learning and lightweight fine-tuning approaches to assess their performance. |
| Outcome: | The proposed models significantly underperform smaller encoder-only models based on RoBERTa in the Temporal Relation Classification task. |
LORE: Continual Logit Rewriting Fosters Faithful Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using aspect-oriented summarization as a case study, we propose **LOgit REwriting**, a new controlled generation paradigm which can be faithful to external knowledge and to the LLM’s intentions. |
| Approach: | They propose a controlled generation paradigm which can be faithful to external knowledge and to the LLM's intentions. |
| Outcome: | The proposed paradigm can be faithful to external knowledge and to the LLM's intentions while balancing that with accuracy. |
TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators (2025.findings-acl)
Copied to clipboard
Jianling Li, ShangZhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie WangHaojie, Jianrong Wang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Triton is a high-level Python-like programming language for building efficient GPU kernels. |
| Approach: | They propose a TritonBench benchmark that provides a comprehensive evaluation of Tritonic operators on widely deployed GPUs. |
| Outcome: | The proposed benchmarks show that current LLMs struggle to generate efficient Triton operators on widely deployed GPUs aligned with industry applications. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Sparse Rewards Can Self-Train Dialogue Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have been driven by supervised fine-tuning and high-quality human feedback. however, acquiring meaningful human feedback has become increasingly challenging and costly. |
| Approach: | They propose a method that empowers LLM agents to enhance their performance without external feedback. |
| Outcome: | The proposed method improves tool-based interactions while preserving general model capabilities across diverse benchmarks. |
Chinese Inertial GAN for Handwriting Signal Generation and Recognition (2025.acl-long)
Copied to clipboard
| Challenge: | Inertial sensors can measure the acceleration and angular velocity of moving objects and are widely used in electronic devices such as smartphones, smartwatches, and fitness bands. |
| Approach: | They propose to use Chinese glyph encoding, forced optimal transport, and semantic relevance alignment to acquire unlimited training samples for Chinese inertial writing recognition. |
| Outcome: | The proposed system improves the performance of six widely used classifiers from 6.7% to 98.4%. |
Investigating Human and LLMs’ Decisions in Unverifiable Environments: A Case Study with GitHub Activity Overview (2026.findings-acl)
Copied to clipboard
| Challenge: | examining the behaviors of Large Language Models as artificial social actors is underexplored, especially in unverifiable scenarios where conventional benchmarking has little to help improve their abilities. |
| Approach: | They propose a method to collect, compare, and reason about human and LLMs' decisions in an unverifiable scenario and use it to examine their behaviors. |
| Outcome: | The proposed method compared human and LLM decisions in an unverifiable scenario on GitHub and found that proprietary LLMs behave more like humans than open-source LLM systems. |
Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to managing non-linear dialogue flow are misaligned with the intrinsically hierarchical and branching structure of natural discourse. |
| Approach: | They propose a framework that models multi-turn dialogue history as a dynamic tree structure. |
| Outcome: | The proposed framework enhances task completion rates and improves token efficiency across various LLMs. |
CPC-GRPO: Answer-Free Reinforcement Learning with Cross-Prompt Consensus Rewards (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards is a popular post-training tool for large language models, but relies on a ground-truth answer or external verifier, which limits applicability and increases cost. |
| Approach: | They propose an answer-free training objective that derives rewards solely from the model’s own probabilities by exploiting prompt paraphrases as multiple semantic views of the same intent. |
| Outcome: | The proposed objective derives rewards solely from the model’s own probabilities by exploiting prompt paraphrases as multiple semantic views of the same intent. |
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces (2026.findings-acl)
Copied to clipboard
Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Shenglin Zhang, Kaipeng Zhang
| Challenge: | Existing benchmarks for agentic programming in long-horizon command-line interface tasks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics. |
| Approach: | They propose a benchmark to evaluate agentic capabilities across long-horizon command-line interface tasks. |
| Outcome: | The proposed benchmarks cover four engineering categories: from scratch, feature addition, bug fixing, and refactoring. |
MoRI: Learning Motivation-Grounded Reasoning for Scientific Ideation in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing LLMs emulate human research workflows but lack scientific grounding . empirical results show that MoRI outperforms strong commercial LLM models . |
| Approach: | They propose a framework that explicitly learns scientific reasoning from research motivations to methodologies. |
| Outcome: | The proposed framework outperforms commercial LLMs and agentic baselines in novelty, technical rigor, and feasibility. |
Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to training large language models fail to capture deep semantic properties and edge cases. |
| Approach: | They propose a framework that leverages Liquid Haskell proofs for validating equivalence and execution-based counterexamples for inequivalent training. |
| Outcome: | The proposed framework achieves 13.3pp accuracy gain on EquiBench and consistent gains on PySecDB. |
Idiosyncratic Versus Normative Modeling of Atypical Speech Recognition: Dysarthric Case Studies (2025.emnlp-main)
Copied to clipboard
| Challenge: | Past studies have focused on fully personalized (or idiosyncratic) models for atypical speech . past studies focused on idiotic models, but current approaches focus on generalizing and handling idiomatic patterns . |
| Approach: | They compare four models that generalize and handle idiosyncrasy to find atypical speech . they find the dysarthric-idios-ync model performs better than the idioconic approach . |
| Outcome: | The proposed model generalizes and handles idiosyncrasy better than the idiocy model . the model requires less personalized data and reduces word error rate from 71% to 32% . |
TA-GRPO-d: Trajectory-Aware GRPO for Optimizing Denoising Trajectories in Diffusion LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing dLLMs rely on fixed denoising schedules and cannot learn efficient unmasking orders. |
| Approach: | They propose a framework that transforms dLLM decoding into a trajectory-aware policy . it uses a confidence-gated denoising strategy that decides which tokens to unmask . |
| Outcome: | The proposed model can learn which tokens to unmask and how many to unmak per step . it can learn the output quality and efficiency of the decoding path itself . |
SciCoQA: Quality Assurance for Scientific Paper–Code Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Discrepancies between scientific papers and their code undermine reproducibility, a concern that grows as automated research agents scale scientific output beyond human review capacity. |
| Approach: | They propose a synthetic generation pipeline to scale beyond AI to Physics, Quantitative Biology, and other computational sciences. |
| Outcome: | The proposed pipeline scales beyond AI to Physics, Quantitative Biology, and other computational sciences. |
Modeling Human Perspectives with Socio-Demographic Representations (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that human disagreement is widespread across many annotation tasks. |
| Approach: | They propose a method that jointly models annotator perspectives while learning socio-demographic representations. |
| Outcome: | The proposed method outperforms concatenation-based methods in predicting annotator perspectives . it learns socio-demographic representations and analyzes how demographic factors relate to variation . |
Adversarial Metric Learning for Fine-Grained Emotion Classification (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in fine-grained emotion classification relied on contrastive learning with hard-pair mining. |
| Approach: | They propose an adversarial metric learning framework that replaces fixed similarity metrics with a learnable metric family and trains representations to remain discriminative under worst-case similarity distortions. |
| Outcome: | The proposed framework trains a pairwise discriminator to maximally confuse two hard pair types while training the encoder to remain discriminative under worst-case similarity distortions. |