Papers with real-world
Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction (2026.findings-eacl)
Copied to clipboard
Chong Zhang, Yixi Zhao, Yulu Xie, Chenshu Yuan, Yi Tu, Ya Guo, Mingxu Chai, Ziyu Shen, Yue Zhang, Qi Zhang
| Challenge: | PTLMs have shown remarkable success in multiple information extraction tasks . however, their performance in real-world scenarios falls short of expectations . |
| Approach: | They propose to use an entity-centric dataset to evaluate PTLMs' performance . they find that inadequate annotations in benchmark datasets lead to spurious correlations . |
| Outcome: | The proposed dataset disentangles the falsely-coupled segment and entity annotations that arises from the block-level annotation of FUNSD. |
Deep Adversarial Learning for NLP (N19-5)
Copied to clipboard
| Challenge: | Adversarial learning is a game-theoretic learning paradigm that has achieved huge successes in the field of Computer Vision recently. |
| Approach: | This tutorial introduces the foundations of deep adversarial learning and some practical problems and solutions in NLP. |
| Outcome: | This tutorial introduces the foundations of deep adversarial learning and some practical problems and solutions in NLP. |
Language Agents: Foundations, Prospects, and Risks (2024.emnlp-tutorials)
Copied to clipboard
| Challenge: | Language agents are autonomous agents that can follow language instructions to perform diverse tasks in real-world or simulated environments. |
| Approach: | They propose to provide a conceptual framework for language agents and a comprehensive discussion on key topics. |
| Outcome: | The proposed tutorial provides a conceptual framework of language agents and comprehensive discussion on important topic areas. |
Thesis Proposal: Uncertainty in Knowledge Graph Embeddings (2025.naacl-srw)
Copied to clipboard
| Challenge: | Knowledge graph embedding (KGE) methods map entities and relations from knowledge graphs into numerical vector spaces. |
| Approach: | They propose to investigate various types of uncertainty in knowledge graph embedding methods and explore strategies to quantify, mitigate, and reason under uncertainty effectively. |
| Outcome: | The proposed methods have shown to be reliable in high-stakes domains and provide greater confidence in their use beyond benchmark datasets. |
Learning with Limited Text Data (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Natural Language Processing (NLP) relies on labeled data to perform state-of-the-art performance . labeles are often required to label large amounts of textual data . this tutorial will provide an overview of labeleing in NLP . |
| Approach: | This tutorial will provide a systematic overview of methods for learning from limited labeled data. |
| Outcome: | This tutorial will provide a systematic and up-to-date overview of the proposed methods . it will highlight current challenges and future directions . |
Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent self-learning methods based on user satisfaction metrics and contextual bandits have shown promising results to enable consistent improvements in conversational AI systems. |
| Approach: | They propose a meta-gradient learning approach that adjusts constraint violation penalty terms adaptively through a user-defined meta objective that encourages balanced constraint satisfaction across domains. |
| Outcome: | The proposed framework supports fine-grained exploration targets for individual domains via user-defined constraints. |
Robustness and Adversarial Examples in Natural Language Processing (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial aims to raise awareness of practical concerns about NLP robustness . it aims at addressing the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift . |
| Approach: | This tutorial aims to bring awareness of practical concerns about NLP robustness . it reviews recent studies on analyzing the weakness of NLP systems when facing adversarial inputs . |
| Outcome: | This tutorial aims to bring awareness of practical concerns about NLP robustness . it will examine the weaknesses of NLP systems when faced with adversarial inputs and data with a distribution shift . |
Deep Reinforcement Learning for NLP (P18-5)
Copied to clipboard
| Challenge: | Many natural language processing tasks can be formulated as deep reinforcement learning (DRL) problems. |
| Approach: | This tutorial provides an introduction to the foundations of deep reinforcement learning . it describes recent advances in designing deep reinforcement for NLP . |
| Outcome: | This tutorial provides an introduction to the foundations of deep reinforcement learning and some practical solutions for NLP tasks. |
SELENE: Selective and Evidence-Weighted LLM Debating for Efficient and Reliable Reasoning (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing multi-agent debate frameworks are computationally expensive and prone to degradation under pro-longed debates due to redundant exchanges and unstable judging. |
| Approach: | They propose a framework that unifies Selective Debate Initiation (SDI) with Evidence Weighted Self-Consistency (EWSC) for adaptive, debate-on-demand reasoning. |
| Outcome: | Evaluated on BoolQ, CosmosQA, and an internal QnA benchmark, the proposed framework achieves higher factual robustness and efficiency. |
Tulun: Transparent and Adaptable Low-resource Machine Translation (2025.acl-demo)
Copied to clipboard
| Challenge: | a low-resource language that is the lingua franca in Timor-Leste lacks available corpora in the health domain. |
| Approach: | They propose a solution that combines neural MT with large language model-based post-editing guided by existing glossaries and translation memories. |
| Outcome: | The proposed system outperforms both standalone MT and LLM approaches across six low-resource languages on the FLORES dataset. |
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) with web search capabilities show significant potential for deep research. |
| Approach: | They introduce a framework for end-to-end training of LLM-based deep research agents . they implement a specialized multi-agent architecture where browsing agents extract relevant information from various webpage structures. |
| Outcome: | The proposed framework improves on open-domain research tasks by 28.9 points over prompt engineering and 7.2 points over RAG-based RL agents. |
LEAF: Learning and Evaluation Augmented by Fact-Checking to Improve Factualness in Large Language Models (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with factual accuracy in knowledge-intensive domains like healthcare. |
| Approach: | They propose a framework for improving LLM factuality in medical question answering . RAFE, Fact-Check-then-RAG and Learning from Fact Check are components . |
| Outcome: | Experimental results show that LEAF outperforms Factcheck-GPT in detecting inaccuracies and corrects errors without labeling . the framework provides a scalable solution for industrial applications requiring high factuality scores. |
MoFE: Mixture of Frozen Experts Architecture (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are characterized by their immense size, often consisting of at least one billion parameters. |
| Approach: | They propose a mixture of Frozen Experts architecture that integrates PEFT and MoE to enhance both training efficiency and model scalability. |
| Outcome: | The proposed architecture outperforms other methods while achieving the highest efficiency. |
Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Differential privacy (DP) is a method that is increasingly popular in NLP but the choice of granularity at which it is applied is often neglected. |
| Approach: | They propose to apply differential privacy (DP) to a neural machine translation dataset to protect individual data points. |
| Outcome: | The proposed method is more resistant to membership inference attacks than the document-level NMT system. |
Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders (2024.acl-long)
Copied to clipboard
| Challenge: | Conversational systems often rely on embedding models for intent classification and intent clustering tasks. |
| Approach: | They propose a toolkit that gives a more holistic view of intent embedding models by considering three tasks– (1) intent classification, (2) intent clustering, and (3) a novel triplet task. |
| Outcome: | The proposed model improves on the linguistic dimensions while affecting performance on downstream task metrics. |
RedactOR: An LLM-Powered Framework for Automatic Clinical Data De-Identification (2025.acl-industry)
Copied to clipboard
Praphul Singh, Charlotte Dzialo, Jangwon Kim, Sumana Srivatsa, Irfan Bulu, Sri Gadde, Krishnaram Kenthapadi
| Challenge: | Existing de-identification methods suffer from recall errors, limited generalization, and inefficiencies, limiting their real-world applicability. |
| Approach: | They propose a multi-modal framework for de-identifying electronic health records using a retrieval-based entity relexicalization approach. |
| Outcome: | The proposed framework achieves competitive performance while optimizing token usage to reduce LLM costs. |
Simultaneous Speech-to-Text Translation Web Application for Estonian (2026.eacl-demo)
Copied to clipboard
| Challenge: | a new open-source web application for simultaneous speech-to-text translation is developed for Estonian . the system translates live Estonian speech into English, Russian, and Ukrainian text, and also supports English-to Estonian translation. |
| Approach: | They propose a web application that combines streaming speech recognition with a simultaneous translation model. |
| Outcome: | The proposed system outperforms existing streaming speech recognition systems in Estonian-to-English translation. |
OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking (2025.emnlp-main)
Copied to clipboard
Zekun Xi, Wenbiao Yin, Jizhan Fang, Jialong Wu, Runnan Fang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated remarkable progress in machine writing such as open domain long-form generation. |
| Approach: | They propose a slow-thinking machine writing framework that emulates the human-like process of iterative expansion and reflection to improve the knowledge density of generated articles. |
| Outcome: | The proposed framework improves the knowledge density of generated articles without compromising metrics such as coherence and depth. |
Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Domain-specific question answering (QA) requires a comprehensive understanding of a specific domain to answer specialized questions. |
| Approach: | They propose a new alignment objective to align the LLM preference with different human preferences uniformly to optimize LLM performance in real-world, domain-specific QA settings. |
| Outcome: | The proposed pipeline is superior for real-scenario domain-specific question answering with LLMs. |
Implanting LLM’s Knowledge via Reading Comprehension Tree for Toxicity Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for toxic content detection are small language model (SLM) based and large language model(LLM) -based. |
| Approach: | They propose to implant LLM's knowledge into SLM based methods to stick to both types of models' strengths by constructing a reading comprehension tree to transfer knowledge between two models. |
| Outcome: | The proposed method can stick to both types of models' strengths . it is compared with existing methods on real-world and machine-generated datasets. |
A Large-Scale Real-World Evaluation of an LLM-Based Virtual Teaching Assistant (2025.acl-industry)
Copied to clipboard
| Challenge: | Empirical studies on their effectiveness and acceptance in real-world classrooms are limited, leaving their practical impact uncertain. |
| Approach: | They develop an LLM-based virtual teaching assistant and deploy it in an introductory AI programming course with 477 graduate students. |
| Outcome: | The proposed system is tested in an introductory AI programming course with 477 graduate students. |
Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for EEG2Text have neglected EEG instability, a problem that has confounded inference and sparked debate. |
| Approach: | They propose to use a 128-channel high-density EEG cap to evaluate EEG2Text models . they find existing benchmarks have neglected EEG instability, a flaw that has confounded inferences and sparked debate . |
| Outcome: | The proposed benchmarks provide key evidence for teacher-forcing-free decoding of EEG2Text models. |
Patient-Similarity Cohort Reasoning in Clinical Text-to-SQL (2026.eacl-long)
Copied to clipboard
| Challenge: | Despite recent advances, performance remains far from clinically reliable . specialized medical terminology and fine-grained temporal reasoning are key to executing clinical data analysis. |
| Approach: | They propose a benchmark for clinical text-to-SQL that demands multi-table joins, clinically meaningful filters, and executable SQL. |
| Outcome: | The proposed benchmark performs well on a set of 20 proprietary and open-source models . it scores 74.7% execution, while DeepSeek-R1 leads open-sourced at 69.2% . |
Chatbot Arena Estimate: towards a generalized performance benchmark for LLM capabilities (2025.naacl-industry)
Copied to clipboard
Lucas Spangher, Tianle Li, William F. Arnold, Nick Masiewicki, Xerxes Dotiwalla, Rama Kumar Pasumarthi, Peter Grabowski, Eugene Ie, Daniel Gruhl
| Challenge: | Existing benchmark aggregation methods, such as Elo-based systems, can be resource-intensive, public facing, and time-consuming. |
| Approach: | They propose a framework for aggregating performance across diverse benchmarks that generates a “Goodness” and a ‘Fastness” score. |
| Outcome: | The proposed framework achieves higher Pearson correlation with Chatbot Arena Elo scores than MMLU’s correlation with chatbot Arena scores, validating its reliability for real-world LLM evaluation. |
Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Despite recent advances in multilingual information retrieval, a significant gap remains between research efforts and real-world deployment. |
| Approach: | They propose to use Quranic multilingual corpus to develop an ad-hoc IR system that can satisfy users’ information needs in multiple languages. |
| Outcome: | The proposed model achieves promising results across diverse retrieval scenarios. |
Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing LLMs require a new call to the inference endpoint/API for each new query . repeated calls to the endpoints/AP Is expensive and impractical for many real-world use cases. |
| Approach: | They compare the performance of various LLMs for query-based meeting summarization . they find that combining queries for the same context in a single prompt can be used to minimize repeated calls. |
| Outcome: | The proposed approach reduces the number of calls to the inference endpoints/APIs in meeting summarization tasks. |
L4: Mutual Learning Helps Lifelong Language Learning (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing distillation methods rely on domain-specific teachers, limiting their ability to update in real-time and adapt to dynamic environments. |
| Approach: | They propose a framework that enables continuous mutual learning from task streams without relying on domain-specific teachers. |
| Outcome: | The proposed framework reduces catastrophic forgetting while improving performance on various benchmark datasets making it suitable for real-world, dynamic natural language processing (NLP) applications. |
A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition (2023.findings-acl)
Copied to clipboard
Limao Xiong, Jie Zhou, Qunxi Zhu, Xiao Wang, Yuanbin Wu, Qi Zhang, Tao Gui, Xuanjing Huang, Jin Ma, Ying Shan
| Challenge: | Existing models for named entity recognition (NER) are based on large-scale labeled datasets, which always obtain using crowdsourcing. |
| Approach: | They propose a CONfidence-based partial Label Learning method to integrate prior and posterior confidences for crowd-annotated named entity recognition models. |
| Outcome: | The proposed model improves on real-world and synthetic datasets compared with baselines. |
TEN: Table Explicitization, Neurosymbolically (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for extracting tabular data from semistructured text are error-prone and costly. |
| Approach: | They propose a neurosymbolic approach to extract tabular data from semistructured text . TEN is a triadic feedback loop that iteratively refines table hypotheses . |
| Outcome: | The proposed approach outperforms neural baselines in exact match accuracy and lower hallucination rates. |
Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. |
| Approach: | They use large language models (LLMs) for automatic evaluation to evaluate a sample . they propose several recommendations for integrating LLMs into future classroom evaluations . |
| Outcome: | The proposed model is able to output high scores without meeting the evaluation instructions, the authors note . their model is not able for students to manipulate the model to output specific strings, they say . |
The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions (2023.emnlp-main)
Copied to clipboard
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Recent advances in large language models (LLMs) have produced models that exhibit remarkable performance across a variety of NLP tasks. |
| Approach: | They analyze a large-scale collection of user-GPT conversations to identify a significant gap between academic research in NLP and the needs of real-world NLP applications. |
| Outcome: | The proposed model outperforms existing models in a large-scale collection of user-GPT conversations and identifies a significant gap between the tasks that users frequently request from LLMs and the tasks commonly studied in academic research. |
Beyond Surface-Level Pattern Trap: LLM Agents for Faster and Smarter Cross-Architecture Code Migration (2026.findings-acl)
Copied to clipboard
| Challenge: | cross-architecture code migration is a resource-intensive and errorprone task. |
| Approach: | a framework for cross-architecture code migration is proposed to decouple implementation details through functional mining and code refactoring. |
| Outcome: | a new framework improves performance and correctness over state-of-the-art frameworks on OpenCV migration tasks. |
Semantic Agreement Enables Efficient Open-Ended LLM Cascades (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have enabled impressive progress across a range of language tasks, but they are steep computational cost. |
| Approach: | They propose a semantic consensus mechanism for reliable deferral by combining model outputs with semantic consensus. |
| Outcome: | Evaluated from 500M to 70B-parameter models, semantic cascades improve deferral accuracy, match or surpass target-model quality at 40% of the cost and reduce latency by up to 60%. |
HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Assistant Scenarios (2025.findings-acl)
Copied to clipboard
Jun Wang, Jiamu Zhou, Xihuai Wang, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Jincheng Jincheng, Muning Wen, Weinan Zhang, Qiuying Peng, Jun Wang
| Challenge: | Evaluating the performance of LLMs in multi-turn interactions presents significant challenges due to the complexity and variability of user behavior. |
| Approach: | They propose a benchmark framework for assessing LLMs’ function-calling capabilities in multi-turn dialogues. |
| Outcome: | The proposed framework is based on a dataset derived from popular mobile apps and anonymized user logs. |
WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai (2025.emnlp-main)
Copied to clipboard
Peerat Limkonchotiwat, Pume Tuchinda, Lalita Lowphansirikul, Surapon Nonesung, Panuthep Tasawong, Alham Fikri Aji, Can Udomcharoenchaikit, Sarana Nutanong
| Challenge: | Existing benchmarks for large language models rely on translations, missing cultural and domain specificity. |
| Approach: | They present a human-authored dataset for evaluation and instruction tuning in Thai . findings highlight need for culturally and professionally grounded instruction data . |
| Outcome: | a human-authored dataset for evaluation and instruction tuning in Thai outperforms translation-based models . findings highlight need for culturally and professionally grounded instruction data . |
Chain-of-Interactions: Multi-step Iterative ICL Framework for Abstractive Task-Oriented Dialogue Summarization of Conversational AI Interactions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have introduced paradigm-shifting approaches in natural language processing, yet their transformative in-context learning (ICL) capabilities remain underutilized, especially in customer service dialogue summarization. |
| Approach: | They propose a single-instance, multi-step framework that orchestrates information extraction, self-correction, and evaluation through sequential interactive generation chains. |
| Outcome: | The proposed framework outperforms existing models and prompts in the customer service dialogue summarization domain. |
PD3F: A Pluggable and Dynamic DoS-Defense Framework against resource consumption attacks targeting Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work lacks mitigation strategies against resource consumption attacks . existing work does not provide mitigation strategies for real-world LLM deployments . |
| Approach: | They propose a pluggable and dynamic doS-Defense framework which employs a two-stage approach to defend against resource consumption attacks from both the input and output sides. |
| Outcome: | The proposed framework significantly mitigates resource consumption attacks, improving users’ access capacity by up to 500% during adversarial load. |
Complex Claim Verification with Evidence Retrieved in the Wild (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work makes simplifying assumptions in retrieval that depart from real-world use cases: no access to evidence, access to curated evidence, or access to published evidence after a claim was made. |
| Approach: | They propose a pipeline to check claims using raw evidence from the web . they restrict their retriever to only search documents available prior to the claim's making . |
| Outcome: | The proposed method is based on a political claim dataset and shows that the evidence summary produced by the system is reliable and relevant to answering key questions. |
Towards Open-Domain Twitter User Profile Inference (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to user profile inference focus on limited attributes and can reveal users' private information. |
| Approach: | They propose a prompt-based generation method which can infer values implicitly mentioned in Twitter user profiles. |
| Outcome: | The proposed method can infer more comprehensive user profiles than baseline extraction-based methods, but limitations remain to be applied for real-world use. |
GPT-NER: Named Entity Recognition via Large Language Models (2025.findings-naacl)
Copied to clipboard
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, Chen Guo
| Challenge: | Large-scale language models (LLMs) have shown impressive ability for in-context learning with limited training data. |
| Approach: | They propose a novel sequence labeling task that transforms a sequence labeled as a text-generation task into a self-verification task that LLMs can adapt to. |
| Outcome: | The proposed model performs better on NER than supervised models on a variety of tasks . the proposed model can be easily adapted by LLMs to generate a text sequence . |
Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing white-box jailbreak methods require full model accessibility and require computational costs. |
| Approach: | They propose a black-box jailbreak attack using Zeroth-Order optimization using ZO-SPSA. |
| Outcome: | The proposed method achieves highest jailbreak success rate on three LVLMs, including InstructBLIP, LLaVA and MiniGPT-4. |
PrAd: Prompt Adaptive Tuning for Decoder-only Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based methods suffer from increased input lengths and sensitivity to weight initialization . adapter-based approaches can substantially increase inference time . |
| Approach: | a new paradigm for prompt-based tuning addresses the problem of fine tuning pretrained models . prompt--based methods suffer from increased input lengths and sensitivity to weight initialization . a prompt-oriented approach employs adapters for flexible input transformation . |
| Outcome: | a proposed framework can achieve comparable or better performance and higher inference efficiency even in multi-task scenarios. |
Learning to Update Knowledge Graphs by Reading News (D19-1)
Copied to clipboard
| Challenge: | Existing methods to update knowledge graphs rely on elaborately designed IE systems and domain-specific rules. |
| Approach: | They propose a novel neural network method to update knowledge graphs (KGs) they use a text-based attention mechanism to guide updating messages through KGs . |
| Outcome: | The proposed method can effectively broadcast news information to KG structures and perform necessary link-adding or link-deleting operations to ensure the KG up-to-date according to news snippets. |
DIRAS: Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | RAG systems leave out important relevant information (low recall) and excessively related but irrelevant information (high precision) authors propose a manual annotation-free schema that can be used for RAGs with limited performance. |
| Approach: | They propose a manual annotation-free schema that annotates unseen queries with calibrated relevance scores. |
| Outcome: | Evaluators show that DIRAS can achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs. |
HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to exit pre-trained language models suffer from the limitation that they have to sequentially traverse through all layers prior to the selected exit layer, which degrades their performance. |
| Approach: | They propose a homotopic and adaptive layer skipping fine-tuning method that adaptively selects the layers to skip based on a predefined budget. |
| Outcome: | The proposed method outperforms all state-of-the-art baselines on the GLUE benchmark and shows that it is highly efficient. |
Interpreting the Robustness of Neural NLP Models to Textual Perturbations (2022.findings-acl)
Copied to clipboard
| Challenge: | Modern Natural Language Processing models are sensitive to input perturbations and their performance can decrease when applied to noisy data. |
| Approach: | They propose to explain the extent to which a model is affected by an unseen textual perturbation by the learnability of the perturbation. |
| Outcome: | The proposed model is better at identifying a perturbation (higher learnability) but worse at ignoring it (lower robustness). |
Analyzing Key Factors Influencing Emotion Prediction Performance of VLLMs in Conversational Contexts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that large language models and vision large language model (VLLMs) possess EI and the ability to understand emotional stimuli in the form of text and images. |
| Approach: | They analyze the key elements affecting the emotion prediction performance of VLLMs in conversational contexts. |
| Outcome: | The proposed model performance was compared with other models in a conversational context. |
Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification (2020.emnlp-main)
Copied to clipboard
Prithviraj Sen, Marina Danilevsky, Yunyao Li, Siddhartha Brahma, Matthias Boehm, Laura Chiticariu, Rajasekar Krishnamurthy
| Challenge: | Existing approaches to explain models are difficult to interpret and have undesirable biases. |
| Approach: | They propose a neural network architecture for learning transparent sentences . they use linguistic expressions built on top of predicates extracted using shallow natural language understanding . |
| Outcome: | The proposed model outperforms statistical relational learning and other neuro-symbolic methods and performs better than black-box recurrent neural networks. |
SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams (2023.acl-long)
Copied to clipboard
Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, Seungwhan Moon
| Challenge: | Existing models lack a large-scale benchmark to capture user–assistant interactions . et al., 2022: 145-160. |
| Approach: | They propose a video-grounded task-oriented dialog dataset that captures real-world AI-assisted user scenarios in VR. |
| Outcome: | The proposed dataset captures real-world AI-assisted user scenarios in VR. |
CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fingerprinting model ownership traces are vulnerable to illegal plagiarism and are not reliable. |
| Approach: | They propose a rule-driven fingerprinting framework that encodes contextual correlations across multiple dialogue turns. |
| Outcome: | The proposed framework achieves stronger stealth and robustness than previous work. |
SupChain-Bench: Benchmarking Large Language Models for Real-World Supply Chain Management (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models have shown promise in complex reasoning and tool-based decision making, but supply chain workflows require reliable long-horizon, multi-step orchestration grounded in domain-specific procedures. |
| Approach: | They propose a unified real-world benchmark that assesses supply chain domain knowledge and long-horizon tool-based orchestration grounded in standard operating procedures. |
| Outcome: | The proposed framework achieves the strongest and consistent tool-calling performance in real-world operational settings. |
On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on the non-deterministic properties of language models, but these properties remain under-explored in machine translation. |
| Approach: | They propose a method that evaluates MT systems and identifies temperature-constrained non-deterministic MT as a distinct phenomenon. |
| Outcome: | The proposed framework provides higher-quality candidates than Deterministic MT under temperature constraints. |
Pulling Out All The Full Stops: Punctuation Sensitivity in Neural Machine Translation and Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Several studies have documented the sensitivity of machine translation models to noise, including spelling errors and dialects. |
| Approach: | They propose to test machine translation systems for punctuation sensitivity by performing sentence-final perturbation tests with full stops, exclamation and questions marks. |
| Outcome: | The proposed tests show that commercial translation systems are vulnerable to single punctuation insertion or deletion, resulting in unreliable translations. |
How to Learn in a Noisy World? Self-Correcting the Real-World Data Noise in Machine Translation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Semantic misalignment, as the primary source of the noise, poses a challenge for training machine translation systems. |
| Approach: | They propose a process for simulating misalignment controlled by semantic similarity which closely resembles misaligned sentences in real-world web-crawled corpora. |
| Outcome: | The proposed model significantly improves translation performance in the presence of misalignment noise and when applied to real-world, noisy web-mined datasets, across a range of translation tasks. |
CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing adversarial text generation approaches can lead to generation lacking diversity or fluency, whereas perturbing in the intermediate representation space can lead a model to generate generations that are not related to the input. |
| Approach: | They propose to generate adversarial texts through controllable attributes that are known to be invariant to task labels. |
| Outcome: | The proposed model generates more diverse and fluent adversarial examples, compared to existing approaches, and is more robust against model re-training and different model architectures. |
"Newspaper Eat" Means "Not Tasty": A Taxonomy and Benchmark for Coded Language in Real-World Chinese Online Reviews (2026.acl-long)
Copied to clipboard
| Challenge: | Current language models handle coded language poorly, with limited real-world datasets and clear taxonomies. |
| Approach: | They propose a taxonomy that captures common encoding strategies including phonetic, orthographic, and cross-lingual substitutions. |
| Outcome: | The proposed model fails to detect or understand coded language in Chinese reviews . negative reviews can expose users to social pressure, retaliation, or reduced visibility . |
How to Talk to Language Models: Serialization Strategies for Structured Entity Matching (2025.findings-naacl)
Copied to clipboard
| Challenge: | Entity matching (EM) identifies whether two data records refer to the same entity . however, its performance heavily depends on how structured entities are “talked” through serialized text. |
| Approach: | They propose a novel serialization scheme for entities with complex relations in knowledge graphs based on random walks and use open-source LLMs to encode sampled semantic walks for matching. |
| Outcome: | The proposed scheme achieves leading performance on EM in canonical and heterogeneous KGs. |
A Web Service for Pre-segmenting Very Long Transcribed Speech Recordings (L18-1)
Copied to clipboard
| Challenge: | a new algorithm that pre-segments long speech recordings into manageable chunks is proposed . the run time of classical text-to-speech alignment algorithms is quadratically growing with the length of the input . |
| Approach: | They propose two algorithms that pre-segment long speech recordings into manageable chunks . first algorithm is fast but cannot guarantee short chunks on noisy recordings or erroneous transcriptions a second algorithm delivers short chunk but is less effective in terms of run time and chunk boundary accuracy . |
| Outcome: | The proposed algorithms reduce the run time of the speech segmentation system to under real-time even on recordings that could not previously be processed. |
Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for generating explanations for neural networks ignore feature interactions between words and phrases. |
| Approach: | They propose to build hierarchical explanations by detecting feature interactions by combining words and phrases at different levels of the hierarchy. |
| Outcome: | The proposed method is evaluated on two benchmark datasets, via automatic and human evaluations. |
Autoregressive Knowledge Distillation through Imitation Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive models are ubiquitous in natural language processing due to the sequential nature of text generation. |
| Approach: | They propose a compression technique for autoregressive models driven by an imitation learning perspective on knowledge distillation. |
| Outcome: | The proposed method outperforms other distillation algorithms on translation and summarization tasks while increasing inference speed 14 times. |
Ranking Manipulation for Conversational Search Engines (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research demonstrates that Large Language Models are highly vulnerable to jailbreaking and prompt injection attacks. |
| Approach: | They propose a tree-of-attacks-based jailbreaking technique which promotes low-ranked products . they propose enabling LLMs to be jailed and prompt injections to disrupt safety . |
| Outcome: | The proposed technique promotes low-ranked products in conversational search engines. |
FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant advances in every natural language processing task, but they are vulnerable to small perturbations in the inputs, raising concerns about their robustness in the real world. |
| Approach: | They propose a large-scale benchmark for extensive evaluation of LLMs in the fact verification domain covering 17 realistic word-level and character-level perturbations and 4 types of subpopulations. |
| Outcome: | The proposed model is brittle to small input changes and exhibits performance variations across different subpopulations. |
“It’s Not Just Hate”: A Multi-Dimensional Perspective on Detecting Harmful Speech Online (2022.emnlp-main)
Copied to clipboard
| Challenge: | Detecting offensive content is becoming a critical task in natural language processing . but most datasets use a single binary label for hate or incivility, even though each concept is multi-faceted . a more fine-grained multi-label approach addresses conceptual and performance issues . |
| Approach: | They propose to use a dataset to annotate offensive online speech with six labels . they propose to apply a more fine-grained approach to predicting incivility and hateful content . |
| Outcome: | The proposed approach outperforms or matches benchmark datasets on the annotated tweets. |
Peering Behind the Shield: Guardrail Identification in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Identifying guardrails in conversational AI agents is critical for identifying malicious content . identifying guardrail components in black-box AI agents poses security challenges . |
| Approach: | They propose a method that leverages guard-specific adversarial prompts to detect guardrails in black-box AI agents. |
| Outcome: | The proposed method achieves perfect classification accuracy in multiple scenarios. |
We Need to Talk About Reproducibility in NLP Model Comparison (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that standard splits produce low reproducible and unreliable conclusions . reproducibility of empirical experimental conclusions is a problem in NLP domain . |
| Approach: | They propose to transform the reproducibility of a model comparison into a probabilistic function . they propose to use a regularized corpus splitting strategy to estimate the model's performance . |
| Outcome: | The proposed estimator achieves a high SNR and significantly increases reproducibility. |
Belief Revision: The Adaptability of Large Language Models Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations assume language models operate with consistent information. |
| Approach: | They propose a dataset to test LMs' belief revision ability when presented with new evidence. |
| Outcome: | The proposed framework improves language models’ adaptiveness to changing information, highlighting a critical trade-off. |
CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts in natural language processing (NLP) commonsense reasoning research have produced a number of new datasets and benchmarks. |
| Approach: | They propose a manually-curated, multi-task benchmark that evaluates models' ability to apply commonsense reasoning in the context of six real-world NLP tasks. |
| Outcome: | The proposed benchmark evaluates the ability of models to apply commonsense reasoning in the context of six real-world NLP tasks. |
Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification (2026.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that LENS outperforms GRPO in delivering higher performance and faster convergence. |
| Approach: | They propose a framework that purifies prompts by identifying and removing interference tokens and then transfers successful rollouts to supervise policy optimization on original noisy prompts. |
| Outcome: | The proposed framework outperforms GRPO in the real-world, with a 3.88% gain and speedup. |
PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs (2023.emnlp-main)
Copied to clipboard
Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang, HyunJeong Choe, David Greene, Chuan He, Rattima Nitisaroj, Anna Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, Zhou Yu
| Challenge: | PRESTO dataset contains 550K contextual multilingual conversations between humans and virtual assistants. |
| Approach: | They propose to use a dataset of 550K contextual multilingual conversations between humans and virtual assistants to study some of the more challenging aspects of parsing realistic conversations. |
| Outcome: | The dataset contains 550K contextual conversations between humans and virtual assistants. |
Protecting Bystander Privacy via Selective Hearing in Audio LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Audio Large language models capture speech from unintended bystanders, raising privacy risks that existing benchmarks and defences did not consider. |
| Approach: | They propose to evaluate selective hearing by evaluating a model’s ability to attend to an intended main speaker while refusing to process or reveal information about incidental bystander speech. |
| Outcome: | The proposed model can attend to an intended main speaker while refusing to process or reveal information about incidental bystander speech. |
SUMIE: A Synthetic Benchmark for Incremental Entity Summarization (2025.coling-main)
Copied to clipboard
| Challenge: | Existing datasets that test incrementally update entity summaries are lacking. |
| Approach: | They propose a fully synthetic dataset that exposes real-world IES challenges by generating diverse attributes, summaries, and unstructured paragraphs with 99% alignment accuracy. |
| Outcome: | The proposed dataset shows that state-of-the-art LLMs struggle to update summaries with an F1 higher than 80.4%. |
The Mirage of Model Editing: Revisiting Evaluation in the Wild (2025.acl-long)
Copied to clipboard
| Challenge: | despite near-perfect results, effectiveness of model editing in real-world applications remains unclear. |
| Approach: | They propose QAEdit and WILD to better reflect real-world use of model editing . they propose a benchmark aligned with widely used question answering datasets and a task-agnostic evaluation framework . |
| Outcome: | The proposed QAEdit benchmark and WILD evaluation framework show that current models perform worse than previously reported. |
Multi-Domain Named Entity Recognition with Genre-Aware and Agnostic Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a key component of many text processing pipelines. |
| Approach: | They propose a new architecture tailored to the task of identifying named entities with data from multiple genres. |
| Outcome: | The proposed architecture outperforms baseline and competitive methods on all three setups with differences ranging between +1.95 to +3.11 average F1 across multiple genres when compared to standard approaches. |
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang
| Challenge: | Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks. |
| Approach: | They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona. |
| Outcome: | The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents . |
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models generate huge amounts of text, making it impractical to manually distinguish whether a text is machine-generated. |
| Approach: | They propose two methods to detect machine-generated text by leveraging Log-Rank information and propose a faster method that uses less perturbations to achieve the same level of performance. |
| Outcome: | The proposed methods improve over the state of the art by 3.9 and 1.75 AUROC points absolute and require less perturbations to achieve the same level of performance. |
Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in multimodal pre-trained models have significantly improved information extraction from visually-rich documents (VrDs). |
| Approach: | They propose a method to predict token sequences within visually-rich documents by a simple prediction head. |
| Outcome: | The proposed method can be used to predict token mentions as token sequences within documents. |
WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across various vision-language tasks. |
| Approach: | They propose a systematic taxonomy to evaluate MLLMs' ability to interpret real-world music scores and answer complex musicological queries. |
| Outcome: | The proposed model is based on real-world music scores and user-generated questions and discussions, and is scalable and controlled. |
Hallucination Detection in Structured Query Generation via LLM Self-Debating (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Hallucination remains a key challenge in applying large language models to structured query generation . we propose the Self-Debating framework to enhance detection performance . |
| Approach: | They propose a framework that prompts an LLM to generate contrastive explanations from opposing perspectives . they also propose 'self-debating' framework to enhance detection performance . |
| Outcome: | The proposed framework outperforms LLM-as-a-Judge baselines in hallucination detection . the framework generates contrastive explanations from opposing perspectives . |
GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown consistent improvements across many tasks requiring natural language understanding, coding, mathematical or logical reasoning . |
| Approach: | They propose to use GaRAGe to evaluate whether LLMs can identify relevant grounding when generating RAG answers. |
| Outcome: | The proposed model over-summarises rather than ground answers strictly on annotated relevant passages, or deflects when no relevant grounding is available. |
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models have driven major progress in NLP, but memory and compute requirements hinder practical deployment. |
| Approach: | They propose a framework that preserves high accuracy while achieving 1-bit weight quantization . the orthogonal-kronecker transformation learns an orthogonale mapping via EM minimization - a new approach to quantization is proposed . |
| Outcome: | The proposed framework achieves 1-bit weight quantization with low activations with low-bit activations. |
Making Large Language Models Better Data Creators (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced the field of NLP significantly, but deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security. |
| Approach: | They propose a unified data creation pipeline that requires only a single formatting example. |
| Outcome: | The proposed pipeline can generate data with a single formatting example. |
Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on citation generation has focused on unambiguous settings with single answers, failing to address the complexity of real-world scenarios. |
| Approach: | They propose a task of QA with source citation in ambiguous settings where multiple valid answers exist, where multiple sources exist. |
| Outcome: | The proposed framework generates multiple answers and cites their sources, allowing users to verify the factuality of each answer and make informed decisions. |
Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing Multimodal Large Language Models (MLLMs) are predominantly trained on consistent visual-textual inputs, leaving open the question of whether they can handle semantic mismatches in layout-rich content. |
| Approach: | They propose to use multimodal inconsistency reasoning to assess MLLMs' ability to reason about semantic mismatches in webpages, presentation slides, and posters. |
| Outcome: | The proposed model outperforms open-source models in detecting inconsistencies in webpages, presentation slides, and posters while remaining vulnerable to inconsistent errors. |
A Reality Check on Context Utilisation for Retrieval-Augmented Generation (2025.acl-long)
Copied to clipboard
Lovisa Hagström, Sara Vera Marjanovic, Haeun Yu, Arnav Arora, Christina Lioma, Maria Maistro, Pepa Atanasova, Isabelle Augenstein
| Challenge: | Existing studies on LM context utilisation of retrieved information have focused on synthetic text. |
| Approach: | They propose a dataset of unreliable, insufficient and difficult-to-understand contexts with real-world queries and contexts manually annotated for stance to compare them to synthetic datasets. |
| Outcome: | The proposed model outperforms synthetic datasets and exaggerates rare context characteristics, leading to inflated context utilisation results. |
LegalChainReasoner: Grounding Criminal Judicial Opinion Generation via Structured Legal Chains (2026.acl-long)
Copied to clipboard
| Challenge: | Current legalAI tasks divide sentencing and legal reasoning into two separate tasks, resulting in inconsistency between the reasoning and predictions. |
| Approach: | They propose a new task that generates both legal reasoning and sentencing decisions using a framework that applies structured legal chains to guide the model through comprehensive case assessments. |
| Outcome: | The proposed model outperforms baseline models on real-world, open-source Chinese legal case datasets. |
CUB: Benchmarking Context Utilisation Techniques for Language Models (2026.acl-long)
Copied to clipboard
Lovisa Hagström, Youna Kim, Haeun Yu, Sang-goo Lee, Richard Johansson, Hyunsoo Cho, Isabelle Augenstein
| Challenge: | Existing language models (LMs) can be distracted by irrelevant contexts or ignore relevant information that contradicts outdated parametric memory. |
| Approach: | They develop a benchmark to help diagnose CMTs under diverse noisy context conditions within retrieval-augmented generation (RAG) they find that most existing CMT struggle to handle the full spectrum of context types encountered in real-world RAG scenarios. |
| Outcome: | The proposed benchmark compares seven state-of-the-art methods across three datasets and tasks, and shows that many lack the robustness needed to handle the full spectrum of context types encountered in real-world RAG scenarios. |
EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records (2024.emnlp-main)
Copied to clipboard
Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang, May Dongmei Wang
| Challenge: | EHRAgent enables clinicians to interact with EHRs using natural language . reliance on rule-based conversion systems often necessitates additional training or effort from data engineers. |
| Approach: | They propose a large language model agent that generates and executes code in natural language to facilitate clinicians in directly interacting with EHRs. |
| Outcome: | The proposed agent outperforms the strongest baseline by up to 29.6% in success rate on three real-world EHR datasets. |
CompTab: A Comprehensive Benchmark for Real-World TableQA with Complex Reasoning and Irregular Tables (2026.acl-long)
Copied to clipboard
Zhen Yang, Wei Du, Jie Wang, Wenze Zhou, Xiangfeng Meng, Zhengyang Wang, Suping Sun, Ziwei Du, Haodong Zou, Jie Chen, Yongbin Liu, Shicheng Tan, Jiahao Ying, Shu Zhao
| Challenge: | Existing benchmarks focus on well-structured tables and fail to reflect irregular structures and complex reasoning commonly encountered in real-world scenarios. |
| Approach: | They propose a benchmark to evaluate TableQA under complex reasoning and irregular table conditions. |
| Outcome: | The proposed framework improves generalization and realism of large language models under complex and irregular table conditions. |
Who Writes What: Unveiling the Impact of Author Roles on AI-generated Text Detection (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require accurate text detection, but authors' characteristics are neglected. |
| Approach: | They investigate how author characteristics impact AI-generated text detection . they use corpus of human-authored texts and parallel AI-generated texts . |
| Outcome: | The results show that gender, CEFR proficiency, academic field and language environment influence detector accuracy. |
Evaluating Compound AI Systems through Behaviors, Not Benchmarks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Compound AI (CAI) systems combine LLMs with retrievers and tools to enable information-seeking applications in the real-world. |
| Approach: | They propose a behavior-driven evaluation framework that generates test specifications aligned with real usage contexts. |
| Outcome: | The proposed framework generates test specifications aligned with real usage contexts and transforms them into concrete test cases. |
Guaranteed Guess: A Language Modeling Approach for CISC-to-RISC Transpilation with Testing Guarantees (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ISA-centric transpilation pipelines are used to translate low-level programs between ISAs . GG provides high code coverage across unit tests and better energy efficiency . |
| Approach: | They propose a ISA-centric transpilation pipeline that embeds large language models into software testing frameworks to ensure accuracy. |
| Outcome: | The proposed method achieves high code coverage across unit tests and functional/semantic correctness of 99% on HumanEval and 49% on BringupBench programs. |
DDO: Dual-Decision Optimization for LLM-Based Medical Consultation via Multi-Agent Collaboration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs fail to capture the dual nature of medical consultation (MC) this mismatch often results in ineffective symptom inquiry and unreliable disease diagnosis. |
| Approach: | They propose a novel LLM-based framework that performs Dual-Decision Optimization by decoupling the two sub-tasks and optimizing them with distinct objectives through a collaborative multi-agent workflow. |
| Outcome: | The proposed framework outperforms existing LLM-based approaches on three real-world MC datasets and achieves competitive performance with state-of-the-art generation-based methods. |
Towards Equitable Natural Language Understanding Systems for Dialectal Cohorts: Debiasing Training Data (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior research has shown that biases exist in these models against certain languages or dialects. |
| Approach: | They propose to use a dialect identification model to obtain targeted training data augmentation for under-represented dialects to debias NLU model for dialectal cohorts in NLU systems. |
| Outcome: | The proposed framework can provide insights on dialect disparity in real-world NLU systems and targeted data argumentation can help narrow the model’s performance gap between standard language speakers and dialect speakers. |
Basic Reading Distillation (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models require high computational resources which limits their deployment in real-world applications. |
| Approach: | They propose to distill large language models into smaller language models by either knowledge distillation or task distillation. |
| Outcome: | The proposed model outperforms or performs comparable to over 20x bigger LLMs on language inference benchmarks and BIG-bench tasks. |
TSVer: A Benchmark for Fact Verification Against Time-Series Evidence (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for fact-checking lack structured evidence, provide insufficient justifications for verdicts, or rely on synthetic claims. |
| Approach: | They propose a temporal and numerical reasoning dataset based on time-series evidence that is annotated with time frames and a verdict and justifications reflecting how the evidence is used to reach the verdict. |
| Outcome: | The proposed dataset improves the quality of the annotations and achieves an inter-annotator agreement of = 0.745 on verdicts. |
What Happens to a Dataset Transformed by a Projection-based Concept Removal Method? (2024.lrec-main)
Copied to clipboard
| Challenge: | A number of recent methods have used linear projections to remove information about a concept from a language representation. |
| Approach: | They propose to use linear projections to remove a concept from a language representation to create a transformed set of word embeddings. |
| Outcome: | The proposed methods inject strong statistical dependencies into the transformed datasets. |
RedCoder: Automated Multi-Turn Red Teaming for Code LLMs (2026.acl-long)
Copied to clipboard
Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Dongwon Jung, Hadi Askari, Wenxuan Zhou, Zhe Zhao, Muhao Chen
| Challenge: | Existing red-teaming approaches for code generation rely on extensive human effort and are prone to generating malicious code under adversarial environments. |
| Approach: | They propose a red-teaming agent that engages victim models in multi-turn conversations to elicit vulnerable code. |
| Outcome: | Experiments show that RedCoder outperforms red-teaming methods in inducing vulnerabilities in code generation. |
A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code (2026.findings-acl)
Copied to clipboard
Keke Lian, Wang Bin, Lei Zhang, Libo Chen, Junjie Wang, Ziming Zhao, Yujiu Yang, Miaoqian Lin, Haotong Duan, Haoran Zhao, Shuang Liao, Mingda Guo, Quan Jiazheng, Yilu Zhong, Chenhao He, Chen Zichuan, Jie Wu, Haoling Li, Zhaoxuan Li, Jiongchi Yu, Hui LI, Dong Zhang
| Challenge: | Existing security evaluation benchmarks lack relevance to real-world AI programming tasks . current LLMs struggle with secure coding, research shows . |
| Approach: | They propose a repository-level evaluation benchmark to assess security of AI-generated code. |
| Outcome: | The proposed framework mirrors real-world AI programming tasks and offers valuable insights into the state of AI code generation. |
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. |
| Approach: | They propose a benchmark to evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
| Outcome: | The proposed benchmarks evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |