Papers with complexity
Unsupervised Sentence-embeddings by Manifold Approximation and Projection (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to generate sentence-embeddings are task-agnostic and often lose information because of word-order. |
| Approach: | They propose to generate sentence-embeddings by projecting sentences onto a fixed-dimensional manifold with the objective of preserving local neighbourhoods in the original space. |
| Outcome: | The proposed method consistently performs similar to or better than several state-of-the-art approaches on six publicly available text-classification datasets. |
The Second Multilingual Surface Realisation Shared Task (SR’19): Overview and Evaluation Results (D19-63)
Copied to clipboard
| Challenge: | EMNLP’19 Workshop on Multilingual Surface Realisation aims to stimulate the exploration of advanced neural networks for multilingual sentence generation from Universal Dependency (UD) structures. |
| Approach: | They present results from the SR'19 Shared Task, a multilingual surface realisation task organised as part of the EMNLP'19 Workshop on Multilingual Surface Realisation. |
| Outcome: | The SR'19 shared task was organised as part of the EMNLP'19 Workshop on Multilingual Surface Realisation . it consisted of two tracks with different levels of complexity . the shallow track was offered in eleven, and the deep track in three languages . |
RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoning (2025.emnlp-industry)
Copied to clipboard
Deyi Ji, Yuekui Yang, Liqun Liu, Peng Shu, Haiyang Wu, Shaogang Tang, Xudong Chen, Shaoping Ma, Tianrun Chen, Lanyun Zhu
| Challenge: | Recent advances in large language models have improved the detection of non-compliant content, but critical gaps persist in fine-grained understanding, explainability, and generalization. |
| Approach: | They propose a framework that combines active reinforcement learning, fine-grained violation understanding and progressive multi-stage training. |
| Outcome: | The proposed framework outperforms general-purpose LLMs and specialized models in fine-grained violation understanding, explainability, and generalization. |
MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents (2026.tacl-1)
Copied to clipboard
Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, Reut Tsarfaty
| Challenge: | Automated agents powered by large language models are becoming more ingrained into how people seek information . but evaluation benchmarks for LLMs rarely feature natural questions that are time-consuming . a new benchmark, MoNaCo, aims to address this gap by eliciting and manually answering time-wasting questions . |
| Approach: | They propose a benchmark of 1,315 natural and time-consuming questions that require dozens of intermediate steps to solve. |
| Outcome: | MoNaCo benchmarks achieve at least 61.2% F1 in real-world time-consuming questions hampered by low recall and hallucinations . Frontier LLMs evaluated on MoN achieving at least 61% F1, harmed by low memory and halluzinations. |
Neural Network based Extreme Classification and Similarity Models for Product Matching (N18-3)
Copied to clipboard
| Challenge: | Matching a seller listed item to an appropriate product has become a fundamental step for e-commerce platforms. |
| Approach: | They propose to use a shallow neural network to match a seller's item to an appropriate product . they also propose a similarity approach based on deep siamese network to train and infer product information. |
| Outcome: | The proposed models outperform the baseline models by more than 5% in terms of accuracy and are capable of efficient training and inference. |
CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools (2023.emnlp-demo)
Copied to clipboard
Jingwei Ni, Julia Bingler, Chiara Colesanti-Senni, Mathias Kraus, Glen Gostlow, Tobias Schimanski, Dominik Stammbach, Saeid Ashraf Vaghefi, Qian Wang, Nicolas Webersinke, Tobias Wekhof, Tingyu Yu, Markus Leippold
| Challenge: | a lack of transparency in sustainability reporting is a key challenge due to the sheer volume and complexity of sustainability reports . only a few entities worldwide have the resources to analyze these reports at scale . a novel LLM-based system to automate the analysis of corporate sustainability reports is needed . |
| Approach: | They propose a novel LLM-based system to automate the analysis of corporate sustainability reports. |
| Outcome: | The proposed system automates the analysis of corporate sustainability reports. |
Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing datasets limited in size con- sidering complexity of dialogues . current trends lean towards end-to-end models while modular systems tend to be preferred in industrial applications. |
| Approach: | They propose a graph-based representation of dialogues that can be exploited for data augmentation, multi- reference training and evaluation of non-deterministic agents. |
| Outcome: | The proposed graph-based representation of dialogues can be exploited for data augmentation, multi- reference training and evaluation of non-deterministic agents. |
LingConv: An Interactive Toolkit for Controlled Paraphrase Generation with Linguistic Attribute Control (2025.emnlp-demos)
Copied to clipboard
| Challenge: | LINGCONV is an interactive toolkit for controllable text generation . it allows fine-grained control over 40 specific linguistic attributes spanning lexical, syntactic, and discourse dimensions. |
| Approach: | They propose a toolkit for paraphrase generation that allows finegrained control over 40 specific linguistic attributes. |
| Outcome: | The toolkit is available at https://mohdelgaar-lingconv.hf.space, with a demo video at https:youtu.be/wRBJEJ6EALQ. |
GAP-Gen: Guided Automatic Python Code Generation (2023.eacl-srw)
Copied to clipboard
| Challenge: | Several previous approaches convert a sentence into a formal statement by mapping verbs to functions in the formal language. |
| Approach: | They propose a Guided Automatic Python Code Generation method based on Python syntactic constraints and semantic constraints. |
| Outcome: | The proposed method achieves better results on automatic Python code generation task than previous methods. |
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models are difficult to reverse engineer because of their internal operations. |
| Approach: | They train small attention-only transformers on a symbolic version of the Indirect Object Identification task. |
| Outcome: | The proposed model with only two attention heads achieves perfect IOI accuracy despite lacking MLPs and normalization layers . |
Spatial and Temporal Language Understanding: Representation, Reasoning, and Grounding (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
| Approach: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
| Outcome: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
Holistic Inter-Annotator Agreement and Corpus Coherence Estimation in a Large-scale Multilingual Annotation Campaign (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this paper we examine the complexity of persuasion technique annotation in a multilingual annotation campaign involving 6 languages and approximately 40 annotators. |
| Approach: | They propose a word embedding-based annotator agreement metric and propose 'holistic IAA' metric to measure the coherence of the entire dataset. |
| Outcome: | The proposed method is compared with the existing IAA metrics and its correlation with the results. |
Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing text-to-SQL datasets that capture complex reasoning are limited by their execution accuracy. |
| Approach: | They present a bilingual text-to-SQL dataset specific to complex reasoning . their evaluation shows that Archer challenges the capabilities of current models . |
| Outcome: | The proposed dataset challenges state-of-the-art models with 6.73% execution accuracy . the dataset contains 1,042 English and 1,042, Chinese questions and 521 unique SQL queries . |
Simplifying Outcomes of Language Model Component Analyses with ELIA (2026.eacl-demo)
Copied to clipboard
| Challenge: | ELIA is an interactive web application that simplifies the outputs of various language model component analyses for a broader audience. |
| Approach: | They propose to use a vision-language model to automatically generate natural language explanations for the complex visualizations produced by these methods. |
| Outcome: | The proposed system integrates three key techniques and generates natural language explanations for complex visualizations. |
LLM-Enhanced Self-Evolving Reinforcement Learning for Multi-Step E-Commerce Payment Fraud Risk Detection (2025.acl-industry)
Copied to clipboard
| Challenge: | e-commerce payment fraud detection is a new area for reinforcement learning (RL) and Large Language Models (LLMs). |
| Approach: | They propose to integrate reinforcement learning (RL) with Large Language Models (LLMs) by framing transaction risk as a multi-step Markov Decision Process (MDP), RL optimizes risk detection across multiple payment stages. |
| Outcome: | The proposed approach improves fraud detection accuracy and demonstrates zero-shot capability. |
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn dialog policy require elaborate design and user goals. |
| Approach: | They propose an algorithm that estimates the reward signal and infers the user goal in dialog sessions. |
| Outcome: | The proposed algorithm achieves higher task success than state-of-the-art models on a multi-domain task-oriented dialog dataset. |
Rad-Flamingo: A Multimodal Prompt driven Radiology Report Generation Framework with Patient-Centric Explanations (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing reports are labor-intensive and expert-intensive, resulting in inconsistencies and a lack of patient-centered insight. |
| Approach: | They propose a multimodal prompt-driven report generation framework that integrates diverse data modalities to produce comprehensive and context-aware radiology reports. |
| Outcome: | The proposed framework improves report quality, improves understandability and could foster better patient-doctor communication. |
BotEval: Facilitating Interactive Human Evaluation (2024.acl-demos)
Copied to clipboard
| Challenge: | Using language models to perform complex interactive tasks is becoming more common with the rapid progress in natural language processing (NLP) models. |
| Approach: | They develop an evaluation toolkit that enables human-bot interactions as part of the evaluation process. |
| Outcome: | The evaluation toolkit enables human-bot interactions as part of the evaluation process, rather than making judgements for a static input. |
Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa (2022.acl-demo)
Copied to clipboard
| Challenge: | Developing better methods for a task is a common feature of the computational linguistics literature. |
| Approach: | They propose to use bootstrap to compute significance levels with the BOOtSTrap SAmpling procedure to evaluate models that predict hard labels and soft labels as well. |
| Outcome: | The proposed method can be used to evaluate models that predict hard labels and soft labels on benchmark data sets. |
ReasonGraph: Visualization of Reasoning Methods and Extended Inference Paths (2025.acl-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) reasoning processes are complex and lack of organized visualization tools creates barriers to understanding, evaluation, and improvement. |
| Approach: | They propose a web-based platform for visualizing and analyzing LLM reasoning processes. |
| Outcome: | The proposed platform shows high parsing reliability, efficient processing, and excellent usability across various downstream applications. |
LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs (2026.acl-short)
Copied to clipboard
| Challenge: | Relation extraction is a core NLP task which involves extracting [head, relation, dependent] RDF triples from text. |
| Approach: | They evaluate four large language models against a graph-based parser on six relation extraction datasets with sentence graphs of varying sizes and complexities. |
| Outcome: | The graph-based parser outperforms the LLMs on six relation extraction datasets with sentence graphs of varying sizes and complexities. |
Evaluating Question Answering Evaluation (D19-58)
Copied to clipboard
| Challenge: | Existing n-gram based QA metrics have a number of drawbacks and are not suitable for all extractive tasks. |
| Approach: | They propose to use BERTScore to evaluate translation for question answering (QA) they also explore whether existing n-gram based metrics are suitable for generative QA . |
| Outcome: | The proposed BERTScore metric fails to provide stronger correlation with human judgements . |
Modeling Referential Gaze in Task-oriented Settings of Varying Referential Complexity (2022.findings-aacl)
Copied to clipboard
| Challenge: | Referential gaze is a fundamental phenomenon for psycholinguistics and human-human communication. |
| Approach: | They propose a multimodal NLP task to predict when the gaze is referential . they train a sequential attention-based LSTM model and a transformer encoder architecture to model referential gaze and transfer gaze features to unseen situated settings . |
| Outcome: | The proposed model can be applied to situations with different referential complexities . the proposed model is based on an attention-based LSTM model and a multivariate transformer encoder architecture . |
Unveiling Divergent Inductive Biases of LLMs on Temporal Data (2024.naacl-short)
Copied to clipboard
| Challenge: | Temporal relations play a crucial role across diverse applications, including event summarization, predicting future events and medical information processing. |
| Approach: | They evaluate the performance of large language models in the analysis of temporal data using two prompt types, Question Answering and Textual Entailment. |
| Outcome: | The proposed models show that they are biased towards specific temporal relationships, while GPT-3.5 prefers “AFTER” for implicit and explicit events, while TE models lean towards “BEFORE”. |
A Computational Model of Latvian Morphology (2024.lrec-main)
Copied to clipboard
| Challenge: | a computational model of Latvian morphology provides a formal structure for Latvian word form inflection . the model explicitly enumerates and handles the many exceptions to the general Latvian inflation principles . |
| Approach: | They propose a computational model of Latvian morphology that provides a formal structure for Latvian word form inflection. |
| Outcome: | The proposed model provides a good coverage for modern Latvian literary language and potential to extend to Latgalian language. |
ViClaim: A Multilingual Multilabel Dataset for Automatic Claim Detection in Videos (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing efforts in misinformation detection focus on written text, leaving a significant gap in addressing the complexity of spoken text in video transcripts. |
| Approach: | They propose to annotate video transcripts in three languages and six topics using a custom annotation tool. |
| Outcome: | The proposed tool shows strong cross-validation performance but challenges for generalization to unseen topics. |
Neural Gibbs Sampling for Joint Event Argument Extraction (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for event argument extraction cannot adequately model the correlation between event arguments and their roles. |
| Approach: | They propose a Bayesian model to jointly extract event arguments using Gibbs sampling . they train two neural networks to model prior distribution and conditional distribution over event arguments . |
| Outcome: | The proposed model can achieve comparable results to existing methods on two widely-used datasets. |
Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts (2022.aacl-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a longstanding NLP task that consists of identifying an entity in a sentence or document. |
| Approach: | They construct a dataset of seven entity types annotated over 11,382 tweets . they provide a set of language model baselines and analyze the performance of the model . |
| Outcome: | The proposed dataset contains seven entity types annotated over 11,382 tweets . the authors focus on short-term degradation of NER models over time and strategies to fine-tune a language model over different periods . |
Text Extraction and Script Completion in Images of Arabic Script-Based Calligraphy: A Thesis Proposal (2025.naacl-srw)
Copied to clipboard
| Challenge: | despite its artistic elements, Arabic calligraphy is difficult to read, even for those fluent in Arabic. |
| Approach: | They analyze the variability in calligraphic styles and the influence of artistic distortions to improve text extraction and script completion. |
| Outcome: | The proposed methods improve text extraction and script completion in Arabic calligraphy . the authors show that the proposed techniques are more efficient than traditional methods . |
SpatialWebAgent: Leveraging Large Language Models for Automated Spatial Information Extraction and Map Grounding (2025.acl-demo)
Copied to clipboard
| Challenge: | Understanding and extracting spatial information from text is vital for a wide range of applications, says nielsen . inherent complexity of geographic expressions in natural language presents significant hurdles for traditional extraction methods. |
| Approach: | They propose a system that leverages large language models to extract spatial information from natural language. |
| Outcome: | SpatialWebAgent is designed to extract, standardize, and ground spatial information from natural language text directly onto maps. |
Fast and Scalable Dialogue State Tracking with Explicit Modular Decomposition (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches for dialogue state tracking are mainly based on classification-based and extraction-based methods. |
| Approach: | They propose a model which incorporates both classification-based and extraction-based methods and integrates four modules to jointly extract dialogue states. |
| Outcome: | The proposed model outperforms the state-of-the-art models in multi-domain dialogues with many turns of utterances. |
ADEPT-SQL: A High-performance Text-to-SQL Application for Real-World Enterprise-Level Databases (2025.acl-demo)
Copied to clipboard
Yongnan Chen, Zhuo Chang, Shijia Gu, Yuanhang Zong, Zhang Mei, Shiyu Wang, Hezixiang Hezixiang, Hongzhi Chen, Jin Wei, Bin Cui
| Challenge: | et al., 2017) address domain-specific knowledge barriers, schemas complexity, and computational costs of large LLMs. |
| Approach: | They propose a domain-adapted Text2SQL system that addresses critical deployment challenges in professional fields. |
| Outcome: | The proposed system achieves 97% execution accuracy on real-world databases . it is faster than existing systems and has a higher performance than existing ones. |
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal large language models fail to capture complexity and traceability of reasoning processes . SciVQR includes domain-specific visuals and challenges models to combine visual comprehension with reasoning. |
| Approach: | They propose a multimodal benchmark for scientific reasoning covering 54 subfields . SciVQR includes domain-specific visuals and challenges models to combine visual comprehension with reasoning . |
| Outcome: | SciVQR evaluates 54 subfields in mathematics, physics, chemistry, geography, astronomy, and biology . the results highlight the need for improved multi-step reasoning and integration of interdisciplinary knowledge . |
ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis (2026.findings-acl)
Copied to clipboard
| Challenge: | Text-to-image (T2I) models are trained on literal, object-centric prompts designed to reflect the visible contents of an image. |
| Approach: | They propose a method to extract key subjects and enhance their representation at embedding-level using Large Language Models. |
| Outcome: | The proposed model significantly improves image-caption consistency and human preference alignment. |
The Parallelism Tradeoff: Limitations of Log-Precision Transformers (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing work on transformers established their Turing completeness, albeit with assumptions like infinite precision and arbitrarily powerful feedforward subnets. |
| Approach: | They propose to simulate transformers whose arithmetic precision is logarithmic in the number of input tokens by constant-depth logspace-uniform threshold circuits. |
| Outcome: | The proposed model emerges from the transformer architecture’s high parallelizability, suggesting a potential inherent weakness of the scaling paradigm. |
CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation remains challenging due to pretraining on predominantly English-centric datasets. |
| Approach: | They propose a method that combines reward scores with model confidence to improve model selection for fine-tuning. |
| Outcome: | The proposed method outperforms existing methods in translation accuracy and data efficiency. |
Citation: A Key to Building Responsible and Accountable Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) bring transformative benefits alongside unique challenges, including intellectual property (IP) and ethical concerns. |
| Approach: | They propose a new approach to mitigate intellectual property and ethical risks associated with large language models. |
| Outcome: | The proposed approach could enhance content transparency and verifiability . it should account for both non-parametric and parametric content . |
EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluation (2026.eacl-demo)
Copied to clipboard
| Challenge: | EvalSense is a flexible framework for constructing domain-specific evaluation suites for large language models . it provides out-of-the-box support for a broad range of model providers and evaluation strategies . |
| Approach: | They propose a framework for constructing domain-specific evaluation suites for large language models. |
| Outcome: | The proposed framework provides out-of-the-box support for a broad range of model providers and evaluation strategies. |
Penman: An Open-Source Library and Tool for AMR Graphs (2020.acl-demos)
Copied to clipboard
| Challenge: | Abstract Meaning Representation encodes acyclic graphs in PENMAN notation format . the open-source Python library Penman provides a robust parser and functions for graph inspection and manipulation . |
| Approach: | They propose a framework for encoding acyclic graphs in PENMAN notation . the open-source Python library Penman provides a robust parser and functions for graph inspection and manipulation . |
| Outcome: | The open-source Python library Penman provides a robust parser, functions for graph inspection and manipulation, and functions for formatting graphs into PENMAN notation. |
CRWIZ: A Framework for Crowdsourcing Real-Time Wizard-of-Oz Dialogues (2020.lrec-1)
Copied to clipboard
| Challenge: | Crowdsourcing platforms such as Amazon Mechanical Turk have been effective for collecting large corpora of task-based and open-domain conversational dialogues, but difficulties arise when task- based dialogues require expert domain knowledge or rapid access to domain-relevant information. |
| Approach: | They propose a framework for collecting real-time Wizard of Oz dialogues through crowdsourcing for collaborative, complex tasks. |
| Outcome: | The proposed framework avoids interactions that breach procedures only known to experts while enabling the capture of a wide variety of interactions. |
TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents (2025.emnlp-demos)
Copied to clipboard
Haofei Yu, Keyang Xuan, Fenghai Li, Kunlun Zhu, Zijie Lei, Jiaxun Zhang, Ziheng Qi, Kyle Richardson, Jiaxuan You
| Challenge: | Existing research systems often design and use agentic workflows to perform research tasks such as ideation, scientific coding, review writing, and tree-based search. |
| Approach: | They propose an open-source codebase, an interactive web demonstration, and a PyPI Python package to make state-of-the-art auto-research pipelines broadly accessible to every researcher and developer. |
| Outcome: | The proposed framework adapts easily to new tools and supports iterative growth. |
A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for fine-grained entity typing have a hierarchical structure . prior work has integrated only explicit hierarchic information by formulating a hierarchy-aware loss or by representing instances and labels in a joint Euclidean embedding space. |
| Approach: | They propose a fully hyperbolic model for multi-class multi-label classification that performs all operations in hyperbolical space. |
| Outcome: | The proposed model performs all operations in hyperbolic space on two challenging datasets and shows it is comparable to state-of-the-art methods on fine-grained classification with remarkable reduction of parameter size. |
Towards Singable Lyrics Translation Using Large Language Models (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing studies on lyrics translation have relied on fine-tuning open-source language models. |
| Approach: | They examine a multilingual lyrics translation dataset and apply prompting methods to large language models to evaluate singability. |
| Outcome: | The proposed methods improve singability and naturalness, compared to naive translation, the authors show . human evaluations using songs created from translated lyrics show that complex prompting strategies improve singable naturalness . |
A Unified Framework for Modeling Heterogeneous Financial Data via Dual-Granularity Prompting (2026.acl-industry)
Copied to clipboard
| Challenge: | Recent industrial credit scoring models rely heavily on manually tuned statistical learning methods due to the complexity of heterogeneous financial data and the challenge of modeling evolving creditworthiness. |
| Approach: | They propose a framework that reformulates credit scoring as a multi-scale sequential learning problem. |
| Outcome: | FinLangNet improves KS and bad debt rate by 6.3 pp in real world deployments. |
RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced in recent years, scaling up in both parameter count and context length. |
| Approach: | They propose a method to compute attention over a subset of context tokens and to implement token selection in a blockwise manner. |
| Outcome: | The proposed method reduces end-to-end inference latency by up to 2.55x with minimal accuracy loss compared to full attention in long-context scenarios for very large models. |
Adaptive Multi-pass Decoder for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | End-to-end neural machine translation (NMT) has attracted increasing attention in recent years. |
| Approach: | They propose an adaptive multi-pass decoder which introduces a flexible multi- pass polishing mechanism to extend the capacity of NMT via reinforcement learning. |
| Outcome: | The proposed architecture improves Chinese-English translation with 1.55 BLEU . the proposed architecture adopts a flexible multi-pass polishing mechanism . |
QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive results on a variety of text understanding tasks. |
| Approach: | They propose a two-stage distillation approach that allows retrieval augmentation to be carried over without the increased compute associated with it. |
| Outcome: | The proposed approach can carry over the gains of retrieval augmentation without suffering the increased compute typically associated with it. |
Investigating the Role and Impact of Disfluency on Summarization (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing studies have focused on disfluency detection and removal, with limited studies into its impact on downstream tasks. |
| Approach: | They propose to incorporate disfluency in summarization models to reduce the impact of replacement disfluencies on natural language processing tasks. |
| Outcome: | The proposed model improves on both public and real-life datasets and shows that it can handle disfluent data with up to 6.99-point degradation in Rouge-L score and replacement disfluencies have the highest negative impact. |
Relevance-guided Supervision for OpenQA with ColBERT (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent work has focused on learning to retrieve passages for open-domain question answering . if notions of relevance are not tailored to questions, the MRC model will not reliably see the best passages . |
| Approach: | They propose a retrieval model that uses coarse-grained vector representations of questions and passages to adapt it to OpenQA. |
| Outcome: | The proposed system improves OpenQA retrieval on Natural Questions, SQuAD, and TriviaQA. |
Exploring Conditional Variational Mechanism to Pinyin Input Method for Addressing One-to-Many Mappings in Low-Resource Scenarios (2024.acl-short)
Copied to clipboard
| Challenge: | Experimental results demonstrate the superior performance of our method. |
| Approach: | They propose to leverage conditional variational mechanism to simplify pinyin IME . they employ a strategy that facilitates interaction between pinyan and Chinese character information . |
| Outcome: | The proposed method improves the performance of pinyin input method engine (IME) under low-resource conditions. |
NP-Hard Lower Bound Complexity for Semantic Self-Verification (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing work on semantic self-verification has been focused on mechanistic interpretation, but the computational complexity of semantic self verification has remained undercharacterized. |
| Approach: | They propose to formalize Semantic Self-Verification as problem of determining whether a statement accurately characterizes its own semantic properties within a given interpretive framework. |
| Outcome: | The proposed model is NP-complete and has implications for safety and fairness approaches . a 3-SAT formula is used to model an instance of SSV involving ambiguous terms . |
ComfyUI-Copilot: An Intelligent Assistant for Automated Workflow Development (2025.acl-demo)
Copied to clipboard
Zhenran Xu, Yangxue Yangxue, Yiyu Wang, Qingli Hu, Zijiao Wu, Baotian Hu, Longyue Wang, Weihua Luo, Kaifu Zhang
| Challenge: | ComfyUI-Copilot is a large language model-powered plugin for AI-driven art creation. |
| Approach: | They propose a large language model-powered plugin to enhance the usability of ComfyUI. |
| Outcome: | The new plugin improves the usability and efficiency of ComfyUI . it offers intelligent node and model recommendations and automated one-click workflow construction. |
Enhancing multi-modal Relation Extraction with Reinforcement Learning Guided Graph Diffusion Framework (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for cross-modal relation extraction focus on single-modal data, which limits their use in real-world situations. |
| Approach: | They propose a framework that leverages pre-trained models to encode multi-modal data into scene graphs and combine them into a cross-modal graph. |
| Outcome: | The proposed model outperforms existing methods on multi-modal relation extraction tasks. |
TimeRes: A Turkish Benchmark For Evaluating Temporal Understanding of Large Language Models (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing benchmarks focus on English and underexplore how linguistic structure contributes to temporal meaning. |
| Approach: | They propose a Turkish benchmark to evaluate temporal understanding of Large Language Models (LLMs) their benchmark examines Reichenbach’s temporal points and reported speech through date arithmetic . |
| Outcome: | The proposed model fails to resolve reported speech and fails to generalize across word order variations. |
An Efficient Framework for Whole-Page Reranking via Single-Modal Supervision (2026.acl-industry)
Copied to clipboard
Zishuai Zhang, Sihao Yu, null Xiewenyi, Ying Nie, Junfeng Wang, Zhiming Zheng, Dawei Yin, Hainan Zhang
| Challenge: | Existing whole-page reranking methods require large-scale expert annotations to achieve high-quality results. |
| Approach: | They propose a whole-page reranking framework that converts single-modal rankers into page-level guidance by constructing budget-aware candidates for cross-modal annotations and distilling intra-modality preferences to align relevance scales across modalities. |
| Outcome: | The proposed framework reduces annotation costs by 70-90% while outperforming fully-annotated reranking baselines. |
Low-Complexity Probing via Finding Subnetworks (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to probing neural networks for linguistic properties are to train a shallow multi-layer perceptron (MLP) on top of the model's internal representations. |
| Approach: | They propose a subtractive pruning-based probe where they find an existing subnetwork that performs the linguistic task of interest. |
| Outcome: | The proposed probe achieves higher accuracy on pre-trained models and lower accuracy on random models, and better learning on its own. |
Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models implicitly embed reasoning trees, but their internal mechanisms remain largely opaque due to the complexity of non-linear interactions and high-dimensional operations. |
| Approach: | They propose to use circuit analysis and self-influence functions to map the reasoning process of large models. |
| Outcome: | The proposed model is able to map human-interpretable reasoning paths and a model's underlying circuits reveal human-mediated reasoning processes. |
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts (2024.findings-acl)
Copied to clipboard
Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, Dan Roth
| Challenge: | Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. |
| Approach: | They propose to use flowcharts as visual contexts to assess the capabilities of visual question-answering multimodal language models in reasoning. |
| Outcome: | The proposed benchmarks evaluate models' ability to follow visual information without pre-existing knowledge on a suite of open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. |
MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Dataset (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in LLMs have demonstrated superior performance in a variety of reasoning tasks (Liu et al., 2023b; Chan e t al, 2024; Qin eetal., 2023) However, to truly achieve conscious processing, the integration of System II reasoning ability is essential. |
| Approach: | They propose a three-step process for reasoning with distributional changes, termed as a metaphysical resoning, and propose 'MARS' task to assess LLMs' reasoning abilities. |
| Outcome: | The proposed task is based on a three-step discriminative process and is compared with a standard model with 20 LLMs of varying sizes and methods. |
CrowdSelect: SyntheticInstruction Data Selection with Multi-LLM Wisdom (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for capturing instruction-following complexity rely on single-dimensional signals, but they fail to capture complexity across diverse fields. |
| Approach: | They propose three foundational metrics that leverage Multi-LLMs wisdom to capture instruction-response pair characteristics and propose CrowdSelect, an integrated metric incorporating a clustering-based approach to maintain response diversity. |
| Outcome: | The proposed metrics outperform existing models on MT-bench and Arena-hard and show improvements of 4.81% on full and LoRA fine-tuning. |
Scientia Potentia Est—On the Role of Knowledge in Computational Argumentation (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing research on argumentation models does not provide a systematic overview of the types of knowledge required in CA tasks. |
| Approach: | They propose a taxonomy of the types of knowledge required in CA tasks . authors propose exploitation of these knowledge types for four main research areas . |
| Outcome: | The proposed taxonomy proposes a systematic overview of the types of knowledge required in CA tasks. |
Common Phone: A Multilingual Dataset for Robust Acoustic Modelling (2022.lrec-1)
Copied to clipboard
| Challenge: | Current state-of-the-art acoustic models can easily comprise more than 100 million parameters. |
| Approach: | They propose to train a gender-balanced, multilingual corpus from 76.000 contributors via Mozilla’s Common Voice project to perform phonetic symbol recognition and validate the quality of the generated phonetic annotation. |
| Outcome: | The proposed model can perform phonetic symbol recognition and validate the quality of the generated phonetic annotation. |
On the Impacts of Contexts on Repository-Level Code Generation (2025.findings-naacl)
Copied to clipboard
| Challenge: | CodeLLMs are widely used for code generation, but their ability to handle repository-level dependencies remains underexplored. |
| Approach: | They propose a benchmark for evaluating repository-level code generation based on dependency contexts. |
| Outcome: | The proposed model improves dependency handling and introduces a new metric, Dependency Invocation Rate (DIR), to measure context utilization. |
Reusable workflows for gender prediction (L18-1)
Copied to clipboard
| Challenge: | Existing systems for author profiling (AP) modeling require extensive feature engineering and testing. |
| Approach: | They propose to implement a system for author profiling (AP) modeling that reduces the complexity and time of building a sophisticated model for a number of different AP tasks. |
| Outcome: | The proposed model achieves comparable results to state of the art models for cross-genre gender prediction, but lags when genre of test set is different from genre of train set. |
LARA: Linguistic-Adaptive Retrieval-Augmentation for Multi-Turn Intent Classification (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Multi-turn intent classification is challenging due to the complexity and evolving nature of conversational contexts . lack of data on multi-turn datasets makes it difficult to collect multi-turned datasets a challenge . |
| Approach: | They propose a framework for multi-turn intent classification that integrates a retrieval-augmented mechanism with a fine-tuned smaller model. |
| Outcome: | The proposed framework improves accuracy on multi-turn intent classification tasks across six languages. |
Analyzing Homonymy Disambiguation Capabilities of Pretrained Language Models (2024.lrec-main)
Copied to clipboard
Lorenzo Proietti, Stefano Perrella, Simone Tedeschi, Giulia Vulpis, Leonardo Lavalle, Andrea Sanchietti, Andrea Ferrari, Roberto Navigli
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Processing (NLP) but current pretrained language models lack the granularity to perform disambiguation . |
| Approach: | They propose a large-scale resource that leverages homonymy relations to cluster WordNet senses and train Homonymy Disambiguation systems. |
| Outcome: | The proposed model can distinguish homonyms with up to 95% accuracy even without fine-tuning the underlying PLM. |
A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation (2020.acl-main)
Copied to clipboard
Jan Deriu, Katsiaryna Mlynchyk, Philippe Schläpfer, Alvaro Rodrigo, Dirk von Grünigen, Nicolas Kaiser, Kurt Stockinger, Eneko Agirre, Mark Cieliebak
| Challenge: | Existing methods to efficiently construct corpus for question answering over structured data are time-consuming and cost-intensive. |
| Approach: | They propose a method to efficiently construct a corpus for question answering over structured data. |
| Outcome: | The proposed method triples the annotation speed while maintaining complexity of queries. |
The Computational Complexity of Distinctive Feature Minimization in Phonology (N18-2)
Copied to clipboard
| Challenge: | a standard assumption in phonology is that finding a minimal feature specification is an automatic part of acquisition and generalization. |
| Approach: | They analyze the problem of determining whether a set of phonemes forms a natural class and find the minimal feature specification for the class. |
| Outcome: | The proposed model is based on a greedy algorithm that fails to find minimal features . the proposed model can be used to find features that are universal across languages . |
A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition (2023.findings-acl)
Copied to clipboard
Limao Xiong, Jie Zhou, Qunxi Zhu, Xiao Wang, Yuanbin Wu, Qi Zhang, Tao Gui, Xuanjing Huang, Jin Ma, Ying Shan
| Challenge: | Existing models for named entity recognition (NER) are based on large-scale labeled datasets, which always obtain using crowdsourcing. |
| Approach: | They propose a CONfidence-based partial Label Learning method to integrate prior and posterior confidences for crowd-annotated named entity recognition models. |
| Outcome: | The proposed model improves on real-world and synthetic datasets compared with baselines. |
LLM-Based Zero-Shot Soft Labeling for Anticipating Disagreement in Negotiation Dialogues (2026.acl-srw)
Copied to clipboard
| Challenge: | Negotiation involves complex emotional and strategic dynamics that pose challenges for AI agents in negotiation dialogues. |
| Approach: | They propose a zero-shot soft-labeling method using large language models . they also examine the performance of model training on rule-based annotated hard and soft labels . |
| Outcome: | The proposed method shows a maximum HIT@3 score of 0.87 against rule-based annotated hard labels . failure cases also demonstrated the limitations of rule--based annotation . |
How to Train a Real-World Silicon Concierge? Internalizing Complex Business Workflow to Only OneModel (2026.acl-industry)
Copied to clipboard
| Challenge: | Traditional industrial agents rely on modular workflows that fracture into a labyrinth of ad-hoc patches, leading to cascading errors and high latency. |
| Approach: | They propose a paradigm shift from external workflows to internalized knowledge representation that consolidates complex business logic and SOPs directly into the model’s parameters. |
| Outcome: | The proposed model breaks the impossible triangle of latency, accuracy, and complexity. |
Query Graph Generation for Answering Multi-hop Complex Questions from Knowledge Bases (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on complex knowledge base question answering addresses two types of complexity at the same time. |
| Approach: | They propose a modified staged query graph generation method that handles both types of complexity at the same time. |
| Outcome: | The proposed method achieves state-of-the-art on three benchmark KBQA datasets. |
WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging (2025.acl-short)
Copied to clipboard
| Challenge: | Multiple choice question (MCQ) benchmarks are widely used to evaluate Large Language Models (LLMs). |
| Approach: | They propose a method to increase the complexity of existing multiple-choice benchmarks by randomly replacing a choice with “None of the above”. |
| Outcome: | The proposed method can be applied to 6 popular benchmarks and evaluate 18 open-weight LLMs. |
Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models (2025.findings-emnlp)
Copied to clipboard
Jiyue Jiang, Alfred Kar Yin Truong, Yanyu Chen, Qinghang Bao, Sheng Wang, Pengan Chen, Jiuming Wang, Lingpeng Kong, Yu Li, Chuan Wu
| Challenge: | Cantonese is considered a low-resource language due to the dominance of Mandarin . rich colloquial vocabulary of Cantone, English loanwords, and code-switching characteristics add to the complexity of corpus collection and processing. |
| Approach: | We collect Cantonese texts from open source corpora, Hong Kong-specific forums, Wikipedia . we refine the model through supervised fine-tuning on curated Cantonesian tasks . |
| Outcome: | The model achieves state-of-the-art (SOTA) performance on four Cantonese benchmarks. |
Combining (Second-Order) Graph-Based and Headed-Span-Based Projective Dependency Parsing (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing graph-based methods that score dependency trees do not score dependency arcs at all. |
| Approach: | They propose a headed-span-based method that decomposes the score of a dependency tree into scores of headed spans. |
| Outcome: | The proposed method improves over first-order graph-based methods, but does not score dependency arcs at all. |
Generalization in Text-based Games via Hierarchical Reinforcement Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement Learning (RL) based agents are promising for text-based games, but their generalization remains a challenge. |
| Approach: | They propose a hierarchical framework for reinforcement learning based on knowledge graphs . they propose to decompose the game into subtasks and execute a sub-policy in the low level to conduct goal-conditioned reinforcement learning. |
| Outcome: | The proposed framework enjoys favorable generalizability on a set of difficulty levels and is able to handle complex training tasks. |
CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on narrow tasks such as multiple-choice cloze tests, isolated translation, or simple paraphrasing. |
| Approach: | They propose a benchmark to measure Chinese idioms' cultural and contextual nuances . they evaluate 2,937 human-verified examples covering 1,765 common idiomes . |
| Outcome: | The proposed benchmarks achieve 95% accuracy on Evaluative Connotation, but only 85% on Appropriateness and 40% top-1 accuracy in Open Cloze. |
ProMISe: A Proactive Multi-turn Dialogue Dataset for Information-seeking Intent Resolution (2024.findings-eacl)
Copied to clipboard
| Challenge: | Work done during internship at Amazon Alexa AI. |
| Approach: | They propose to use iterative suggested question-answering conversation to improve the trade-off between satisfaction of the user’s intent and keeping the information exchange natural. |
| Outcome: | The proposed proposed question-answering conversation improves the satisfaction of the user’s intent while keeping the information exchange natural and cognitive load of the interaction minimal on the users. |
Investigating Transferability in Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on deep NLP models has centered on probing, a method that involves training classifiers for different tasks on model representations. |
| Approach: | They propose a method for determining the impact of each pretrained layer on transfer task performance by ablation. |
| Outcome: | The proposed method shows that pretraining models improve performance on downstream tasks . the results highlight the limitations of methods that operate on frozen models or single data samples. |
Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods of logical reasoning focus on entity-aware information but ignore hierarchical relations that may even have mutual effects. |
| Approach: | They propose a holistic graph network that deals with context at both discourse-level and word-level as the basis for logical reasoning. |
| Outcome: | The proposed method improves on logical reasoning QA datasets and natural language inference datasets. |
CIAug: Equipping Interpolative Augmentation with Curriculum Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Current methods for interpolative data augmentation select samples at random, which might make it difficult for the model to generalize better and converge faster. |
| Approach: | They propose a curriculum-based learning method that leverages the relative position of samples in hyperbolic embedding space as a complexity measure to gradually mix up increasingly difficult and diverse samples along training. |
| Outcome: | The proposed method achieves state-of-the-art results over existing methods on 10 benchmark datasets across 4 languages in text classification and named-entity recognition tasks. |
No Simple Answer to Data Complexity: An Examination of Instance-Level Complexity Metrics for Classification Tasks (2025.naacl-long)
Copied to clipboard
| Challenge: | Understanding data complexity at the instance level has become increasingly important in Natural Language Processing (NLP) and machine learning (ML). |
| Approach: | They empirically examine the relationship between instance-level complexity scores and metric selection for classification tasks. |
| Outcome: | The results show that storing training loss provides similar complexity rankings to other methods, but not demographic fairness, even in downstream predictions. |
Mind the Query: A Benchmark Dataset towards Text2Cypher Task (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Graph databases store data in nodes and relationships, enabling more natural modeling of complex, interconnected data. |
| Approach: | They present a high-quality dataset for the Text2Cypher task . it is enabling the translation of natural language (NL) questions into executable Cypher queries over graph databases. |
| Outcome: | The proposed dataset includes 27,529 NL queries and corresponding Cyphers spanning across 11 real-world graph datasets. |
How Good Are LLMs at Processing Tool Outputs? (2026.eacl-long)
Copied to clipboard
Kiran Kate, Yara Rizk, Poulami Ghosh, Ashu Gulati, Tathagata Chakraborti, Zidane Wright, Mayank Agarwal
| Challenge: | Real-world task automation tasks require large language models to call tools, which often return complex JSON responses. |
| Approach: | They evaluated 15 open and closed weight models using multiple prompting approaches to evaluate their tool response processing task and their ability to process structured (JSON) responses. |
| Outcome: | The proposed model can process structured (JSON) responses with 3% to 50% performance differences. |
DETQUS: Decomposition-Enhanced Transformers for QUery-focused Summarization (2025.naacl-long)
Copied to clipboard
Yasir Khan, Xinlei Wu, Sangpil Youm, Justin Ho, Aryaan Mehboob Shaikh, Jairo Garciga, Rohan Sharma, Bonnie J Dorr
| Challenge: | Query-focused tabular summarization is an emerging task in table-to-text generation . traditional transformer-based approaches face challenges due to token limitations and the complexity of reasoning over large tables. |
| Approach: | They propose a system that leverages tabular decomposition alongside a fine-tuned encoder-decoder model to improve summarization accuracy. |
| Outcome: | a new system outperforms the state-of-the-art REFACTOR model in a Query-focused tabular summarization task . the proposed system achieves a ROUGE-L score of 0.4437, outperforming the previous state- of-the art model . |
Enhancing Healthcare LLM Trust with Atypical Presentations Recalibration (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for eliciting and calibrating large language models have focused on general reasoning datasets, yielding only modest improvements. |
| Approach: | They propose a method which leverages atypical presentations to adjust model confidence estimates. |
| Outcome: | The proposed method reduces calibration errors by approximately 60% on three medical question answering datasets and outperforms existing methods such as vanilla verbalized confidence, CoT verbalised confidence and others. |
Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models lack this active understanding capacity, limiting their applicability in real-world scenarios. |
| Approach: | They propose a benchmark to assess the impact of multimodal inputs on lexical ambiguities. |
| Outcome: | The proposed benchmark assesses the impact of multimodal inputs on lexical ambiguities. |
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work analyzes, quantifies, and mitigates language model biases such as gender, race or religion-related stereotypes in static word embeddings and contextual representations. |
| Approach: | They explain the complexity of gender and language around it and examine how current representations perpetuate harms associated with binary gender. |
| Outcome: | The proposed model and dataset biases perpetuate harms associated with the treatment of gender as binary in English language technologies. |
Generating Commonsense Reasoning Questions with Controllable Complexity through Multi-step Structural Composition (2025.coling-main)
Copied to clipboard
| Challenge: | Existing work mainly learns to map text into questions, lacking a mechanism to control results with desired complexity. |
| Approach: | They propose a novel controllable framework to generate QGs with desired complexity using contextual and commonsense clues from text. |
| Outcome: | The proposed framework can generate complex questions with desired complexity levels. |
X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment (2024.findings-naacl)
Copied to clipboard
DongJae Shin, HyeonSeok Lim, Inho Won, ChangSu Choi, Minjun Kim, SeungWoo Song, HanGyeol Yoo, SangMin Kim, KyungTae Lim
| Challenge: | constructing multilingual data for large multimodal models presents its own set of challenges due to language diversity and complexity. |
| Approach: | They propose to use GPT4-V to construct multimodal training datasets using a text-only version of GPT4. |
| Outcome: | The proposed method performs well in Korean and English, surpassing existing methods. |
Automatically Estimating Textual and Phonemic Complexity for Cued Speech: How to See the Sounds from French Texts (2024.lrec-main)
Copied to clipboard
| Challenge: | Cued Speech (CS) is a visual communication system developed for people with hearing loss to complement speech reading at the phonetic level. |
| Approach: | They propose a method to phonemize written corpora so that each word is aligned with the corresponding CS key(s) this method is part of a wider project aimed at creating an augmented reality system displaying a virtual coding hand where the user will be able to choose a text upon its complexity for cueing. |
| Outcome: | The proposed method is part of a wider project aimed at creating an augmented reality system displaying a virtual coding hand where the user can choose a text upon its complexity for cueing. |
Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-document event coreference resolution are prone to learning simple co-occurrences due to the complexity of contexts. |
| Approach: | They propose a collaborative approach to cross-document event coreference resolution that leverages both a universally capable LLM and a task-specific SLM. |
| Outcome: | The proposed approach surpasses the performance of both large and small language models individually, underscoring its effectiveness in diverse scenarios. |
Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that text-generation models can be overconfident when there are multiple correct options. |
| Approach: | They propose a QE approach called BoostedProb which boosts the model’s confidence in cases where there are multiple viable output options. |
| Outcome: | The proposed approach achieves on average +0.194 improvement in Pearson correlation to ground-truth quality and outperforms more costly approaches like supervised or ensemble-based QE in certain settings. |
WRIME: A New Dataset for Emotional Intensity Estimation with Subjective and Objective Annotations (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on emotion analysis use subjective emotional intensity labels by the writers and objective ones by the readers. |
| Approach: | They annotate 17,000 SNS posts with both the writer's subjective emotional intensity and the reader's objective emotional intensity to construct a Japanese emotion analysis dataset. |
| Outcome: | The results show that the reader cannot fully detect the emotions of the writer, especially anger and trust. |
Mitigating the Inconsistency Between Word Saliency and Model Confidence with Pathological Contrastive Training (2022.findings-acl)
Copied to clipboard
| Challenge: | Neural networks are used for various NLP tasks, but their complexity makes them difficult to interpret. |
| Approach: | They propose a framework to mitigate the model pathology and obtain more interpretable models by using contrastive learning and saliency-based samples augmentation to calibrate the sentences representation. |
| Outcome: | The proposed framework can mitigate the model pathology and generate more interpretable models while keeping the model performance. |
Generating Scientific Claims for Zero-Shot Scientific Fact Checking (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for scientific fact checking require domain expertise and time consuming. |
| Approach: | They propose a new supervised method for generating claims from scientific sentences and a novel method for negating claims. |
| Outcome: | The proposed method improves on existing methods on biomedical claims and negations. |
Untangling Hate Speech Definitions: A Semantic Componential Analysis Across Cultures and Domains (2025.findings-naacl)
Copied to clipboard
| Challenge: | a new framework for analyzing hate speech definitions is proposed to address cultural differences in interpretations . a dataset of 493 definitions from more than 100 cultures is used to analyze hate speech . |
| Approach: | They propose a framework for a cross-cultural and cross-domain analysis of hate speech definitions . they use open-source LLMs to analyze the impact of different definitions on hate speech detection . |
| Outcome: | The proposed framework enables cross-cultural and cross-domain analysis of hate speech definitions . it reveals that many domains borrow definitions from one another without taking into account target culture . |
HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Assistant Scenarios (2025.findings-acl)
Copied to clipboard
Jun Wang, Jiamu Zhou, Xihuai Wang, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Jincheng Jincheng, Muning Wen, Weinan Zhang, Qiuying Peng, Jun Wang
| Challenge: | Evaluating the performance of LLMs in multi-turn interactions presents significant challenges due to the complexity and variability of user behavior. |
| Approach: | They propose a benchmark framework for assessing LLMs’ function-calling capabilities in multi-turn dialogues. |
| Outcome: | The proposed framework is based on a dataset derived from popular mobile apps and anonymized user logs. |
When Shallow is Good Enough: Automatic Assessment of Conceptual Text Complexity using Shallow Semantic Features (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches to automatic assessment of text complexity focus on syntactic and lexical complexity. |
| Approach: | They propose to use graph-based deep semantic features to automatically assess conceptual text complexity by using DBpedia as a proxy to human knowledge. |
| Outcome: | The proposed features outperform the state-of-the-art features on pairwise comparison of two versions of the same text and five-level classification task. |
UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities (2026.acl-long)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. |
| Approach: | They propose a framework to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. |
| Outcome: | The proposed framework shows superiority over existing methods on 10 benchmarks of multiple modalities. |
GENDEX: Generative Data Augmentation Strategy Leveraging External Data for Abstractive Dialogue Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to summarize text data are limited by the lack of data. |
| Approach: | They propose a method that uses external data to generate synthetic dialogues from short texts containing people and their interpersonal interactions. |
| Outcome: | The proposed method shows robust performance, generalizability, and scalability regardless of complexity of dialogues. |
Auxiliary tasks to boost Biaffine Semantic Dependency Parsing (2022.findings-acl)
Copied to clipboard
| Challenge: | Semantic dependency parsing (SDP) is a task of producing a dependency graph for a sentence. |
| Approach: | They propose to use simple auxiliary tasks that introduce some form of interdependence between arcs to circumvent such an independence of decision. |
| Outcome: | The proposed method shows modest but systematic performance gains on a near-state-of-the-art baseline using transformer-based contextualized representations. |
Syntax-guided Contrastive Learning for Pre-trained Language Model (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies rely on additional syntax-driven attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. |
| Approach: | They propose a syntax-guided contrastive learning method which does not change the transformer architecture and does not alter the transformer structure. |
| Outcome: | The proposed method achieves consistent improvements in a variety of tasks including grammatical error detection, entity tasks, structural probing and GLUE. |
Unsupervised Human Preference Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for in-context learning and parameter-efficient fine-tuning fail to capture the complexity of human preferences, especially given the small, personal datasets individuals possess. |
| Approach: | They propose a method that uses small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. |
| Outcome: | The proposed method outperforms baseline personalization methods on email and article datasets and significantly outperformed existing methods. |
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on curriculum learning focus on selecting the best distribution of data to train a system. |
| Approach: | They propose a self-supervised neural machine translation model that self-selects data without being told to do so. |
| Outcome: | The proposed model self-selects samples of increasing complexity and task relevance without being told to do so, and performs a denoising curriculum. |
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (N19-1)
Copied to clipboard
| Challenge: | Existing transfer learning methods employ language models pretrained on large generic corpora, but results come at a high computational cost and require task-specific architectures. |
| Approach: | They propose a transfer learning approach that combine a task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. |
| Outcome: | The proposed method surpasses well established transfer learning methods with greater level of complexity on a variety of affective and text classification tasks surpassing well established methods with higher level of difficulty. |
LanguageFlow: Advancing Diffusion Language Generation with Probabilistic Flows (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work has demonstrated success in controlling sentence attributes and structure based on diffusion language models. |
| Approach: | They propose a language-rectified flow method that reformulates standard probabilistic flow models to learn ordinary differential equations to transport between the source and target distributions. |
| Outcome: | The proposed method outperforms baselines on three fine-grained control tasks and multiple high-quality text editing tasks. |
MBA-RAG: a Bandit Approach for Adaptive Retrieval-Augmented Generation through Question Complexity (2025.coling-main)
Copied to clipboard
| Challenge: | Existing RAG frameworks either indiscriminately perform retrieval or rely on rigid single-label classifiers to select retrieval methods. |
| Approach: | They propose a framework that dynamically selects the most suitable retrieval strategy based on query complexity. |
| Outcome: | The proposed framework achieves state-of-the-art results on multiple single-hop and multi-hop datasets while reducing retrieval costs. |
Span-based discontinuous constituency parsing: a family of exact chart-based algorithms with time complexities from O(nˆ6) down to O(nˆ3) (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel chart-based parser for discontinuous constituency trees is proposed for span-based span parsing . it can process discontinuous constituent trees of block degree two, including ill-nested structures . |
| Approach: | They propose a chart-based algorithm for span-based parsing of discontinuous constituency trees . they build variants with smaller search spaces and time complexities ranging from O(n6) down to O(N3) . |
| Outcome: | The proposed algorithm can process 98% of constituents in linguistic treebanks while having the same complexity as continuous constituency parsers. |
On the Computational Modelling of Michif Verbal Morphology (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing computational models of the verbal morphology of the Métis language are insufficient to model the language's unique phonological interactions. |
| Approach: | They propose a finite-state computational model of the verbal morphology of Michif . they use composed finite state transducers to model concatenative morphologies . |
| Outcome: | The proposed model is based on a series of finite-state transducers. |
Consistent Autoformalization for Constructing Mathematical Libraries (2024.emnlp-main)
Copied to clipboard
| Challenge: | Autoformalization is the task of automatically translating mathematical content written in natural language to a formal language expression. |
| Approach: | They propose to use three mechanisms to improve autoformalization quality . they propose to combine most-similar retrieval augmented generation, denoising steps and auto-correction with syntax error feedback to improve syntactic, terminological and semantic control. |
| Outcome: | The proposed mechanisms can deliver syntactically, terminologically and semantically more consistent results across different models. |
Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on multilingual models have focused on their cross-lingual transfer behavior . a recent study examined multilingual model learning from the multilingual pretraining signal . |
| Approach: | They analyze checkpoints during multilingual pretraining to identify when models acquire in-language and cross-lingual abilities. |
| Outcome: | The proposed model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. |
Scalable Data Synthesis through Human-like Cognitive Imitation and Data Recombination (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) rely on massive amounts of training data, however, the quantity of empirically observed data is limited. |
| Approach: | They propose a data synthesis framework that mimics human cognitive behaviors by recombining and interconnecting heterogeneous data from diverse sources. |
| Outcome: | The proposed framework mimics human cognitive behaviors by recombining and interconnecting heterogeneous data from diverse sources thereby enhancing advanced reasoning capabilities in large language models. |
MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs (2026.acl-long)
Copied to clipboard
Xiangyu Zhao, Wanghan Xu, Bo Liu, Yuhao Zhou, Fenghua Ling, Ben Fei, Xiaoyu Yue, Lei Bai, Wenlong Zhang, Xiao-Ming Wu
| Challenge: | Existing datasets often rely on synthetic data or figure-caption pairs, failing to capture the depth and complexity of geoscientific reasoning. |
| Approach: | They propose a multimodal scientific dataset and benchmark curated from open-access publications. |
| Outcome: | MSEarth features over 289K figures with captions enriched by contextual discussions and reasoning from original papers. |
CAMAL: A Novel Dataset for Multi-label Conversational Argument Move Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that combine CNN and LSTM structures with speaker ID graphs improve the F1-score of our baseline models to detect speakers’ intents by a large margin. |
| Approach: | They propose a conversational multi-label corpus of teaching transcripts for Conversational Argument Move AnaLysis (CAMAL) the dataset includes 165 discussion transcripts facilitated by pre-service teachers and students . |
| Outcome: | The proposed model improves the F1-score of the baseline model to detect speakers’ intents by a large margin. |
When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using sports data, an LLM can analyze sports narratives to infer points from actions, identify related entities, attribute points accurately to players and teams, and draw conclusions. |
| Approach: | They propose a method to synthesize NBA basketball game narratives using real NBA basketball data and propose 'SportsGen' they find that most models fail to accurately aggregate basketball scores due to frequent scoring patterns and open-source models suffer from significant score hallucinations. |
| Outcome: | The proposed method can evaluate LLMs’ reasoning capabilities under complex scenarios with varying narrative lengths and density of information. |
VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems (2022.naacl-main)
Copied to clipboard
| Challenge: | Neural module networks (NMN) have been used in image-grounded tasks such as Visual Question Answering (VQA) however, very limited work on NMN has been studied in the video-ground dialogue tasks. |
| Approach: | They propose to use video as the grounding feature in video-grounded dialogues to model the information retrieval process in videogrounded language tasks as a pipeline of neural modules. |
| Outcome: | The proposed model can achieve promising performance on video-grounded dialogue and QA benchmarks. |
Low-Resource Comparative Opinion Quintuple Extraction by Data Augmentation with Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Comparative Opinion Quintuple Extraction (COQE) aims to predict comparative opinion quintuples from comparative sentences. |
| Approach: | They propose a low-resource approach to extract comparative opinion quintuples from comparative sentences . they propose augmentation using ChatGPT and a data-centric approach . |
| Outcome: | The proposed approach improves the existing pipeline-based method and achieves state-of-the-art results. |
Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts (2024.findings-acl)
Copied to clipboard
| Challenge: | Prediction with Explanation is the largest expert-annotated dataset for legal judgment prediction and explanation in the Indian context . |
| Approach: | They propose to use an annotated legal judgment prediction corpus to improve models' accuracy . they employ transformer-based models tailored for both general and Indian legal contexts . |
| Outcome: | The proposed system improves the accuracy and explanatory depth of models for legal judgments. |
Flames: Benchmarking Value Alignment of LLMs in Chinese (2024.naacl-long)
Copied to clipboard
Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, Yingchun Wang, Dahua Lin
| Challenge: | Existing benchmarks for large language models (LLMs) do not accurately uncover safety vulnerabilities in LLMs. |
| Approach: | They propose a value alignment benchmark called Flames that encompasses both harmlessness principles and a unique morality dimension that integrates specific Chinese values such as harmony. |
| Outcome: | The proposed model performs poorly on Flames, particularly in safety and fairness dimensions. |
AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios (2025.naacl-long)
Copied to clipboard
Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, Zhongyu Wei
| Challenge: | Large language models are increasingly employed to empower autonomous agents to simulate human behavior. |
| Approach: | They propose to evaluate LLM-driven agents through multi-turn interactions using a bottom-up approach to create diverse social scenarios constructed from extensive scripts. |
| Outcome: | The proposed model evaluates LLM-driven agents through multi-turn interactions emphasizing goal completion and implicit reasoning. |
Rumor Detection on Social Media with Temporal Propagation Structure Optimization (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for detecting rumors on social media neglect the temporal aspect of rumor propagation. |
| Approach: | They propose a method that incorporates temporal information by building a weighted propagation tree and a coding tree. |
| Outcome: | The proposed approach preserves essential structure of rumor propagation while reducing noise. |
G-Transformer for Document-Level Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work extends translation unit from single sentence to multiple sentences. |
| Approach: | They propose to introduce locality assumption as an inductive bias into Transformer and reduce the hypothesis space of attention from target to source. |
| Outcome: | The proposed model achieves state-of-the-art BLEU scores on three benchmark datasets. |
Bandits Don’t Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Training data for machine translation (MT) is often sourced from multiple large corpora that are multi-faceted in nature. |
| Approach: | They propose to optimize the balance between translationese and natural training data to relieve system developers from manual schedule design. |
| Outcome: | The proposed model relieves system developers from manual schedule design. |
What does it take to bake a cake? The RecipeRef corpus and anaphora resolution in procedural text (2022.findings-acl)
Copied to clipboard
| Challenge: | Current research on anaphora resolution is mostly based on declarative text, such as chemical patents or instruction manuals. |
| Approach: | They propose a framework for anaphora annotation for the chemical domain for modeling anamorphic phenomena in recipes and chemical patents. |
| Outcome: | The proposed framework improves resolution of anaphora in recipes, suggesting transferability of general procedural knowledge. |
M3: Multi-level dataset for Multi-document summarisation of Medical studies (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing summarisation systems are not up to such complex tasks, yet limited tools exist to determine where and why they are failing. |
| Approach: | They propose to use a dataset to evaluate the quality of summarisation systems in the biomedical domain. |
| Outcome: | The proposed model can be used to evaluate the quality of summarisation systems in the biomedical domain. |
code-transformed: The Influence of Large Language Models on Code (2026.findings-eacl)
Copied to clipboard
| Challenge: | Using Large Language Models, code generation capabilities have transformed programming practices. |
| Approach: | They analyze 20,000 GitHub repositories linked to arXiv papers published between 2020 and 2025 . they identify measurable trends in the evolution of coding style that align with LLM-generated code . |
| Outcome: | The proposed study examines 20,000 GitHub repositories linked to arXiv papers . it finds that LLMs influence code style, and that they can be observed in real-world code . |
MPID: A Modality-Preserving and Interaction-Driven Fusion Network for Multimodal Sentiment Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Current methods for multimodal sensing analysis overlook nuanced differences and similarities across modalities, leading to potential biases. |
| Approach: | They propose a Modal-Preserving and Interaction-Driven Fusion Network to address these challenges by integrating text with audio and a separate Adaptive Graded Fusion Module for text and visual data. |
| Outcome: | The proposed model achieves state-of-the-art on CMU-MOSI, CMU -MOSEI, and CH-SIMS datasets. |
Depth-bounding is effective: Improvements and evaluation of unsupervised PCFG induction (D18-1)
Copied to clipboard
| Challenge: | Recent attempts to improve grammar induction systems by bounding complexity of the model have not been compared against unbounded induction models. |
| Approach: | They propose to use a Bayesian induction inducer to limit the search space of the model and then sample trees with or without bounding. |
| Outcome: | The proposed model produces trees more accurately than or competitively with state-of-the-art constituency grammar induction models. |
Temporal Relations Annotation and Extrapolation Based on Semi-intervals and Boundig Relations (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for temporal relations annotation and management are not widely used because they are too complex from the computational perspective. |
| Approach: | They propose a system for the annotation and management of temporal relations that combines the richness and expressiveness of Freksa’s approach with the simplicity of Allen’s notation. |
| Outcome: | The proposed system achieves more agreeable representations of temporal relations without increasing the complexity of the labeling process. |
CLHA: A Simple Yet Effective Contrastive Learning Framework for Human Alignment (2024.lrec-main)
Copied to clipboard
Feiteng Fang, Liang Zhu, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang
| Challenge: | Large language models (LLMs) have attracted considerable attention from academic and industrial communities due to their outstanding performance in various natural language processing tasks. |
| Approach: | They propose a Contrastive Learning Framework for Human Alignment to evaluate the noise within the data and dynamically adjust the training process. |
| Outcome: | The proposed framework surpasses other algorithms in terms of reward model scores, automatic evaluations, and human assessments on the widely used dataset "Helpful and Harmless" |
MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing (2024.findings-acl)
Copied to clipboard
Jiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen, Nan Hu, Guilin Qi, Haiyun Jiang, Siyuan Cheng, Bozhong Tian
| Challenge: | Current benchmarks focus on coarse-grained knowledge, leaving the intricacies of fine-grounded knowledge unexplored. |
| Approach: | They propose a benchmark and dataset specifically designed for FG multimodal entity knowledge editing. |
| Outcome: | The proposed benchmark underscoring the complexity of FG knowledge editing in MLLMs. |
Large Language Models are good multi-lingual learners : When LLMs meet cross-lingual prompts (2025.coling-main)
Copied to clipboard
| Challenge: | Experimental results show that Large Language Models can generate rule-based data in long contexts without following all specified rules. |
| Approach: | They propose a novel prompting strategy Multi-Lingual Prompt which automatically translates the error-prone rule that an LLM struggles to follow into another language, thus drawing greater attention to it. |
| Outcome: | The proposed framework outperforms state-of-the-art prompting methods on public datasets across various tasks, with a specific case study in text-to-MIP instances. |
Benchmarking Data Science Agents (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as promising data science aids, assisting humans in data analysis and processing. |
| Approach: | They propose an evaluation paradigm and benchmarks that assess the performance of data science agents throughout the entire data science lifecycle. |
| Outcome: | The proposed evaluation paradigm streamlines dataset preparation, improves coverage, and expands benchmarking comprehensiveness. |
SDA: Semantic Discrepancy Alignment for Text-conditioned Image Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for textconditioned image retrieval are limited due to the scale of training and the complexity of attributes in modification texts. |
| Approach: | They propose a general boosting framework to address these issues by employing semantic discrepancy alignment by formulating distance consistency and neighbor consistency between the image and text domains. |
| Outcome: | The proposed framework improves retrieval performance on three prominent datasets with state-of-the-art results. |
Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing variational dialog models have pre-trained, restricting diversity of responses . a diffusion model increases complexity of prior distribution and its compatibility with PLMs . |
| Approach: | They propose a hierarchical conditional variational autoencoder with diffusion priors to address these challenges. |
| Outcome: | The proposed method generates more diverse responses without dialog pre-training. |
Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards (2022.findings-emnlp)
Copied to clipboard
Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, Curtis Langlotz
| Challenge: | Neural image-to-text radiology report generation systems have been successful on NLG metrics, but they are not factually complete or consistent due to inadequate training and evaluation. |
| Approach: | They propose a method to improve the factual completeness and correctness of generated radiology reports by using a dataset containing annotated chest X-ray images. |
| Outcome: | The proposed method significantly improves factual completeness and correctness of generated radiology reports on two open radiology report datasets. |
LongWanjuan: Towards Systematic Measurement for Long Text Quality (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing efforts to improve data quality have focused on deduplication and the evaluation of data diversity and difficulty. |
| Approach: | They propose a set of metrics to evaluate the quality of long texts by evaluating three fundamental linguistic dimensions: coherence, cohesion, and complexity. |
| Outcome: | The proposed model improves on long-text tasks with over 160B tokens and categorizes long texts into holistic, aggregated, and chaotic types. |
Firewall Routing: Blocking Leads to Better Hybrid Inference for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have significantly enhanced performance across various NLP tasks . high computational costs and latency associated with deploying such models pose bottlenecks . |
| Approach: | They propose a dynamic hybrid inference framework that efficiently selects between a strong and a weak LLM based on the complexity of the query. |
| Outcome: | The proposed method outperforms existing routing strategies by up to 5.29% in APGR . large models often introduce higher latency, making them less suitable for real-time or resource-constrained applications. |
Learning to Search Effective Example Sequences for In-Context Learning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods address these factors in isolation, overlooking their interdependencies. Existing approaches focus on sequence selection, while focusing on the sequence of examples. |
| Approach: | They propose a method that considers key factors involved in sequence selection and incrementally builds the sequence. |
| Outcome: | Experiments across various datasets and language models show that the proposed method significantly reduces the search space and improves performance. |
ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch (2026.acl-long)
Copied to clipboard
Zheng Liu, Honglin Lin, Xiaoyang Wang, Xin Gao, Yu Li, Mengzhang Cai, Yun Zhu, Zhanping Zhong, Qizhi Pei, Zhuoshi Pan, Xiaoran Shang, Conghui He, Bin Cui, Wentao Zhang, Lijun Wu
| Challenge: | Existing open-source vision language models lack high-quality training data for chart reasoning . current models are simplistic and repetitive, while associated QA pairs are prone to hallucinations . |
| Approach: | They propose a framework to synthesize complex charts and reliable reasoning data from scratch. |
| Outcome: | Experimental results show that ChartVerse-8B surpasses existing models in QA and difficulty . lack of high-quality training data hampers development of open-source models . |
Tracking Life’s Ups and Downs: Mining Life Events from Social Media Posts for Mental Health Analysis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have indicated that major life events can greatly impact individuals’ mental health, but shedding its light on social media data is challenging due to the complexity and ambiguity nature of life events. |
| Approach: | They propose to extract life events mentioned in posts on social media to uncover a social media event dataset which includes 12 major life event categories that are likely to occur in everyday life. |
| Outcome: | The proposed dataset includes 12 life event categories that are likely to occur in everyday life and is human-annotated under iterative procedure and boasts a high level of quality. |
Cluster-Former: Clustering-based Sparse Transformer for Question Answering (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models for encoding long sequences in deep learning suffer from high latency and memory demands. |
| Approach: | They propose a clustering-based sparse Transformer framework to perform attention across chunked sequences. |
| Outcome: | The proposed framework achieves state-of-the-art on several major QA benchmarks. |
LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation (2025.naacl-long)
Copied to clipboard
Sachit Kuhar, Wasi Uddin Ahmad, Zijian Wang, Nihal Jain, Haifeng Qian, Baishakhi Ray, Murali Krishna Ramanathan, Xiaofei Ma, Anoop Deoras
| Challenge: | Recent code completion models focus on local file contexts, but do not fully capture the complexities of real-world software development. |
| Approach: | They propose a version-specific code-completion task across eight libraries as they evolve over the years and an in-depth analysis of two widely used public libraries: PyTorch and Matplotlib. |
| Outcome: | The proposed model improves performance with public libraries, compared with existing models. |
SimulSpeech: End-to-End Simultaneous Speech to Text Translation (2020.acl-main)
Copied to clipboard
| Challenge: | SimulSpeech is an end-to-end simultaneous speech to text translation system . conventional approaches to simultaneous speech translation divide the translation process into two stages . |
| Approach: | They develop an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. |
| Outcome: | The proposed system achieves reasonable BLEU scores and lower delay compared to full-sentence translation model. |
How (Non-)Optimal is the Lexicon? (2021.naacl-main)
Copied to clipboard
| Challenge: | lexical meanings are mapped to wordforms by usage pressures and constraints on sequences of symbols. |
| Approach: | They propose a coding-theoretic view of the lexicon and a novel generative statistical model to quantify its compressibility under various constraints. |
| Outcome: | The proposed model shows that (compositional) morphology and graphotactics can account for most of the complexity of natural codes—as measured by code length. |
RAFFLES: Reasoning-based Attribution of Faults for LLM Systems (2026.eacl-long)
Copied to clipboard
Chenyang Zhu, Spencer Hong, Jingyu Wu, Kushal Chawla, Yuhui Tang, Youbing Yin, Nathan Wolfe, Erin Babinsky, Daben Liu
| Challenge: | Existing evaluation frameworks focus on simple metrics and end-to-end outcomes, but they struggle with longer contexts. |
| Approach: | They propose an offline evaluation architecture that incorporates iterative reasoning to evaluate the quality of the candidate faults and rationales of the Judge. |
| Outcome: | The proposed architecture outperforms baseline evaluation frameworks with two datasets to identify step-level faults in multi-agent systems and ReasonEval datasets. |
Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to generalize compositionally are inadequate, but there is no evidence for this. |
| Approach: | They propose a model-agnostic algorithm for subsampling instances with diverse structures from a labeled instance pool with structured outputs. |
| Outcome: | The proposed algorithm leads to comparable or better generalization than prior algorithms in 9 out of 10 dataset-split type pairs. |
From Word to World: Can Large Language Models be Implicit Text-based World Models? (2026.acl-long)
Copied to clipboard
Yixia Li, Hongru Wang, Jiahao Qiu, Zhenfei Yin, Dongdong Zhang, Cheng Qian, Zeping Li, Xiaoteng Ma, Guanhua Chen, Heng Ji
| Challenge: | Agentic learning increasingly hinges on interaction, yet real-world experience is expensive, limited, and often irreversible at inference time. |
| Approach: | They propose a framework that reframes language modeling as next-state prediction under interaction. |
| Outcome: | The proposed framework evaluates world models in text-based environments . it shows that sufficiently trained models capture coherent environment dynamics . |
Spoiler Detection as Semantic Text Matching (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on spoiler detection shows promising results in safeguarding viewers from general spoilers, but it fails to address the issue of users abstaining from show-related content during their watch. |
| Approach: | They propose to use semantic text matching to assign an episode number to a spoiler given a specific TV show and a dataset to evaluate its performance. |
| Outcome: | The proposed dataset can be used to evaluate the performance of the proposed model and to compare it with other datasets. |
The Bulgarian Event Corpus: Overview and Initial NER Experiments (2022.lrec-1)
Copied to clipboard
| Challenge: | Initial experiments on standard NER task due to complexity of dataset and rich NE annotation scheme are promising with respect to some labels and give insights on handling better other ones. |
| Approach: | They describe a Bulgarian Event Corpus (BEC) that includes named entities and events with their roles. |
| Outcome: | The proposed corpus is multi-domain and oriented towards Social Sciences and Humanities (SSH) it includes named entities and events with their roles. |
Efficient Sentence Embedding using Discrete Cosine Transform (D19-1)
Copied to clipboard
| Challenge: | Modern NLP systems rely on word embeddings as input units to encode statistical semantic and syntactic properties of words. |
| Approach: | They propose to use discrete cosine transform to compress word sequences in order-preserving manner. |
| Outcome: | The proposed model preserves syntactic information in semantic probing tasks . it is comparable to vector averaging but mediocre in performance. |
SQLWOZ: A Realistic Task-Oriented Dialogue Dataset with SQL-Based Dialogue State Representation for Complex User Requirements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TOD datasets present simplified interactions with simple slot-value style constraints and preferences. |
| Approach: | They propose a novel TOD dataset that captures complex user requirements using SQL statements. |
| Outcome: | The proposed dataset captures complex, real-world user requirements. |
Generating Multi-Aspect Queries for Conversational Search (2026.eacl-long)
Copied to clipboard
| Challenge: | Conversational information seeking (CIS) systems aim to model the user’s information need within the conversational context and retrieve the relevant information. |
| Approach: | They propose a multi-aspect query generation and retrieval framework which uses Large Language Models to break the user utterance into multiple queries. |
| Outcome: | The proposed framework outperforms state-of-the-art query rewriting methods on six widely used CIS datasets and fine-tunes the model on MASQ yields significant improvements. |
Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current methods for multimodal representation learning for electrocardiograms often result in suboptimal alignment of ECG signals with their corresponding text reports. |
| Approach: | They propose a framework to learn ECG representations by aligning ECG signals with paired free-text reports. |
| Outcome: | The proposed framework outperforms existing methods in zero-shot classification and linear probing tasks using 12 leads. |
An empirical study of validating synthetic data for formula generation (2025.findings-naacl)
Copied to clipboard
Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, Gust Verbruggen
| Challenge: | Large language models (LLMs) can be leveraged to help write formulas in spreadsheets, but formula data resources are scarce, limiting the ability to fine-tune them. |
| Approach: | They validate a corpus of formulas with a model to generate synthetic natural language utterances for fine-tuning. |
| Outcome: | The proposed model generates synthetic natural language utterances with a model that is accurate enough to fine-tune them. |
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues (2026.findings-acl)
Copied to clipboard
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Moore Wang, Yongqian Wen, Yuanxing Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, Yonghong Lin, An Ping, Shihao Li, Yanghai Wang, Tianhao Peng, Jiaheng Liu
| Challenge: | Existing evaluation benchmarks for Multimodal Large Language Models (MLLMs) focus on single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. |
| Approach: | They propose a video understanding benchmark for MLLMs in multi-turn dialogues that assesses six core competencies that focus on perceptivity and interactivity. |
| Outcome: | The MT-Video-Bench evaluates 1,000 multi-turn dialogues from diverse domains and reveals significant performance discrepancies and limitations in handling multi-turned video dialogues. |
A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for speech-based dialogs have found the inadequacy of relying on simple classification techniques to accomplish the automation task. |
| Approach: | They propose a Label-Aware BERT Attention Network (LABAN) for zero-shot multi-intent detection by encoding input utterances with BERT and building a label embedded space by considering embedded semantics in intent labels. |
| Outcome: | The proposed approach can detect many unseen intent labels correctly on a few/zero-shot setting, and achieves state-of-the-art performance on five multi-intent datasets in normal cases. |
ConvTextTM: An Explainable Convolutional Tsetlin Machine Framework for Text Classification (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) have reshaped the industry . complexity of such models makes them a “black box” and can cause ethical concerns . |
| Approach: | They propose a convolutional TM architecture that breaks down text into a sequence of fragments . they propose to use a tokenization scheme to bind the tokens to the text fragments. |
| Outcome: | The proposed architecture improves on a set of text fragments and eliminates the need for a corpus-specific vocabulary. |
DirectProbe: Studying Representations without Classifiers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches for probing opaque representations often use training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation’s goodness. |
| Approach: | They propose a heuristic that directly studies the geometry of a representation by building upon the notion of 'version space' they argue that doing so can be unreliable because different representations may need different classifiers . |
| Outcome: | Experiments with linguistic tasks and contextualized embeddings show that even without training classifiers, DirectProbe can shine lights on how an embeddable space represents labels and anticipate the classifier performance for the representation. |
Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | logical puzzles that involve determining identity of characters require a variety of reasoning skills. |
| Approach: | They propose a benchmark for suppositional reasoning based on knights and knaves puzzles . they show lower-performing models exhibit a diverse range of reasoning errors . |
| Outcome: | The proposed benchmark demonstrates that models struggle with suppositional reasoning . lower performing models struggle to grasp the concept of truth and lies, the study finds . |
CKnowEdit: A New Chinese Knowledge Editing Dataset for Linguistics, Facts, and Logic Error Correction in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | CKnowEdit is the first-ever knowledge editing dataset designed to correct linguistic, factual, and logical errors in Large Language Models. |
| Approach: | They propose a Chinese knowledge editing dataset to correct linguistic, factual, and logical errors in Large Language Models. |
| Outcome: | The proposed dataset highlights the challenges that LLMs face in mastering Chinese . CKnowEdit can correct linguistic, factual, and logical errors in Chinese, the authors show . |
Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification (D19-1)
Copied to clipboard
| Challenge: | Existing models for multi-label classification ignore complexity and dependencies among labels . Experimental results show that our method can obtain more accurate multi-lab classification results. |
| Approach: | They propose a meta-learning method to capture complex label dependencies . they use a Meta-learner to jointly learn the training policies and prediction policies for different labels. |
| Outcome: | The proposed method can capture complex label dependencies on fine-grained entity typing and text classification tasks. |
Vision-Language Models Can Self-Improve Reasoning via Reflection (2025.naacl-long)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) has been shown to improve the reasoning capability of large language models (LLMs). |
| Approach: | They propose a framework which iteratively enhances the model’s Vision-language Reasoning by Reflecting on CoT Rationales. |
| Outcome: | The proposed framework improves multimodal reasoning on vision-language tasks by 23% to 60% over baselines. |
Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for narrative understanding fail to represent complexity and uncertainty of relationships in real-life social scenarios. |
| Approach: | They propose a benchmark for extracting and analysing intricate character relation graphs from detective narratives using large-scale large-language models. |
| Outcome: | The proposed dataset extracts and analyses character relation graphs from detective narratives using advanced Large Language Models like GPT-3.5, GPT-4, and Llama2 . |
Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Enterprise systems are crucial for enhancing productivity and strategic growth, but data is fragmented across multiple sources and access controls are complex. |
| Approach: | They propose a benchmark that simulates enterprise settings with 500 diverse tasks . they show that even the most capable models achieve only 41.8% task completion . |
| Outcome: | The proposed benchmark shows that even the most capable models achieve only 41.8% task completion. |
DRBO: Mitigating Short Board Effect via Dynamic Reward Balancing in Multi-reward LLM Optimization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new framework to optimize large language models (LLMs) for evaluation metrics is needed to balance weaker metrics. |
| Approach: | They propose a Dynamic Reward Balancing Optimization framework to mitigate the "short-board effect" they apply it to single-task and multi-type task scenarios . |
| Outcome: | The proposed framework improves performance and balances performance across multiple metrics. |
FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Improving training efficiency remains a challenge in large-scale Reinforcement Learning (RL). |
| Approach: | They propose a curriculum RL framework with stage-wise context scaling to improve RL training efficiency. |
| Outcome: | The proposed framework outperforms state-of-the-art reasoning models on five benchmarks and achieves 49.6% accuracy on AIME 2024. |
HYDRA: A Multi-Head Encoder-only Architecture for Hierarchical Text Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art approaches rely on complex components like graph encoders, label semantics, and autoregressive decoders. |
| Approach: | They propose a multi-head encoder-only architecture for hierarchical text classification that treats each level as a separate classification task with its own label space. |
| Outcome: | The proposed architecture matches or exceeds state-of-the-art methods on four benchmarks. |
HellaSwag: Can a Machine Really Finish Your Sentence? (P19-1)
Copied to clipboard
| Challenge: | Existing commonsense models struggle to perform inferences that are trivial for humans, but are often misclassified by state-of-the-art models. |
| Approach: | They propose a dataset that is adversarial to state-of-the-art commonsense reasoning and use it to build a model that is surprisingly robust. |
| Outcome: | The proposed dataset is compared with existing models and scaled up towards a critical 'Goldilocks zone' wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. |
Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | ReasonFormer is a unified reasoning framework for complex decision-making . it is based on the dual-process theory of cognitive science, where two cognitive systems interact to form a whole reasoning process. |
| Approach: | They propose a unified reasoning framework that mirrors the modular reasoning process of humans . they decouple the representation module and the reasoning modules to capture different levels of cognition . |
| Outcome: | The proposed framework shows that humans can perform better in complex decision-making tasks. |
HTML: Hierarchical Topology Multi-task Learning for Semantic Parsing in Knowledge Base Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches struggle with mapping questions to precise logical forms . Existing frameworks struggle with complex mapping of questions to logical form . |
| Approach: | They propose a framework that leverages a hierarchical multi-task learning paradigm to enhance the performance of logical form generation. |
| Outcome: | The proposed framework outperforms supervised fine-tuning methods and training-free ones on large language models. |
CodeArena: Evaluating and Aligning CodeLLMs on Human Preference (2025.emnlp-main)
Copied to clipboard
Jian Yang, Jiaxi Yang, Wei Zhang, Jin Ke, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Zhoujun Li, Binyuan Hui, Junyang Lin
| Challenge: | Code large language models (codeLLMs) focus on synthesizing the correct code snippet, ignoring the alignment with human preferences. |
| Approach: | They propose a benchmark code-based on 40 categories and 44 programming languages to emulate real-world coding tasks. |
| Outcome: | The proposed benchmarks show that open-source code LLMs perform better than open-sourced ones. |
What Kind of Language Is Hard to Language-Model? (P19-1)
Copied to clipboard
| Challenge: | a recent study suggests that language models perform poorly across languages. |
| Approach: | They propose a model that fits a paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at least-pairwise parallel corpora. |
| Outcome: | The proposed model is able to handle missing data and is aware of inter-sentence variation. |
VeriFastScore: Speeding up long-form factuality evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | VeriFastScore model can be used to evaluate long-form factuality but requires multiple LLM calls and can take up to 100s to evaluate a single response. |
| Approach: | They propose a model that leverages synthetic data to fine-tune Llama3.1 8B for extracting and verifying all verifiable claims within a given text based on evidence from Google Search. |
| Outcome: | The proposed model achieves strong correlation with the original VeriScore pipeline at both the example level and system level while achieving an overall speedup of 6.6 over Veriscore. |
Dual Graph Convolutional Networks for Aspect-based Sentiment Analysis (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to model relationships between aspects and opinion words are inefficient due to informal expressions and complexity of online reviews. |
| Approach: | They propose a dual graph convolutional networks model that considers complementarity of syntax structures and semantic correlations simultaneously. |
| Outcome: | The proposed model outperforms state-of-the-art methods on three public datasets and validates it. |
Few-shot Transfer Learning for Knowledge Base Question Answering: Fusing Supervised Models with In-Context Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing Knowledge Base Question Answering (KBQA) architectures are expensive and time-consuming to deploy. |
| Approach: | They propose a KBQA architecture that performs KB-retrieval using multiple source-trained retrievers and re-ranks using an LLM. |
| Outcome: | The proposed architecture outperforms adaptations of SoTA KBQA models when training data is limited. |
Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge (2025.naacl-long)
Copied to clipboard
Li Zhou, Taelin Karidi, Wanlong Liu, Nicolas Garneau, Yong Cao, Wenyu Chen, Haizhou Li, Daniel Hershcovich
| Challenge: | Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet lack a robust methodology to dissect these phenomena comprehensively. |
| Approach: | They propose a multilingual dataset centered on food-related cultural facts and variations in food practices. |
| Outcome: | The proposed model incorporates cultural context significantly and improves its ability to access cultural knowledge. |
Exploring the Capability of Multimodal LLMs with Yonkoma Manga: The YManga Dataset and Its Challenging Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | YManga dataset is the first specifically designed for yonkoma manga understanding . |
| Approach: | They propose to use a dataset of 1,015 yonkoma strips with 10,150 human annotations to define three tasks for panel sequence detection, intent generation and description generation for masked panels. |
| Outcome: | The proposed dataset contains 1,015 high-quality yonkoma strips with 10,150 human annotations. |
Sequential Randomized Smoothing for Adversarially Robust Speech Recognition (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing, naive defenses against adversarial attacks are lagging . a new paper aims to break these defenses with adaptive noise ensembling . |
| Approach: | They propose a randomized smoothing paradigm that can be used to break adversarial attacks . they use speech enhancement methods and a novel use for ASR output ensembling methods . |
| Outcome: | The proposed model is robust to all attacks that use inaudible noise and can only be broken with very high distortion. |
KHANQ: A Dataset for Generating Deep Questions in Education (2022.coling-1)
Copied to clipboard
| Challenge: | Existing QG datasets are not suitable for educational question generation because the questions are not real questions asked by humans during learning. |
| Approach: | They propose a dataset for question generation that contains 1,034 high-quality learner-generated questions seeking an in-depth understanding of the taught online courses in Khan Academy. |
| Outcome: | The proposed dataset contains 1,034 high-quality learner-generated questions seeking an in-depth understanding of the taught online courses in Khan Academy. |
An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models (2022.acl-long)
Copied to clipboard
| Challenge: | et al. (2017) show that imitation learning algorithms for machine translation introduce mismatches between training and inference that lead to undertraining and poor generalization in editing scenarios. |
| Approach: | They propose a framework for training non-autoregressive sequence-to-sequence models for editing tasks where the original input sequence is iteratively edited to produce the output. |
| Outcome: | The proposed framework significantly improves output quality and controls complexity better on the simplification task. |
Exploiting Tree Structure for Credit Assignment in Reinforcement Learning with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning has shown strong promise for strengthening reasoning ability of large language models, but sparse, delayed rewards make token-level credit assignment a central challenge. |
| Approach: | They propose a critic-free algorithm that rewards tokens that change the solution. |
| Outcome: | The proposed algorithm improves on in-distribution benchmarks and out-of-disttribution settings. |
SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate remarkable performance across diverse tasks, yet their effectiveness often depends on costly commercial APIs or cloud services. |
| Approach: | They propose a dual-mode compatible approach that fine-tunes models through shortest-response preference optimization and a confidence-aware rejection mechanism. |
| Outcome: | The proposed approach reduces redundant outputs and response times while reducing computational costs by over 50% and cascade latency by over 80%. |
GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus primarily on pure graph understanding, lacking a comprehensive evaluation across all graph types and detailed capability definitions. |
| Approach: | They propose a benchmark to evaluate LLMs' graph comprehension and reasoning abilities using a three-tier hierarchical taxonomy and a granular taxonomies. |
| Outcome: | The proposed model includes 11 datasets with 5,140 graphs of varying complexity. |
Factual Knowledge Assessment of Language Models Using Distractors (2025.coling-main)
Copied to clipboard
Hichem Ammar Khodja, Abderrahmane Ait gueni ssaid, Frederic Bechet, Quentin Brabant, Alexis Nasr, Gwénolé Lecorvé
| Challenge: | Language models encode extensive factual knowledge within their parameters. |
| Approach: | They propose a new interpretable knowledge assessment method that leverages distractors to provide incorrect alternatives to the correct answer. |
| Outcome: | The proposed method shows that it is aligned with human judgment and stronger robustness to verbalization artifacts. |
Detecting Multiword Expression Type Helps Lexical Complexity Assessment (2020.lrec-1)
Copied to clipboard
| Challenge: | Multiword expressions (MWEs) represent lexemes that should be treated as single lexical units due to their idiosyncratic nature. |
| Approach: | They re-annotate a complex word identification shared task 2018 dataset . they find that a lexical complexity assessment system benefits from the information . |
| Outcome: | The proposed dataset provides valuable information for the text simplification community. |
Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for unimodal large language models are inadequate for MLLMs due to multimodal data complexity and multi-phase training. |
| Approach: | MM-DETECT analyzes data contamination using a framework that defines two contamination categories - unimodal and cross-modal . |
| Outcome: | The proposed framework quantifies contamination severity across multiple-choice and caption-based Visual Question Answering tasks. |
Are we Estimating or Guesstimating Translation Quality? (2020.acl-main)
Copied to clipboard
| Challenge: | A carefully engineered ensemble of pre-trained multilingual language models won the QE shared task at WMT19. |
| Approach: | They propose to use pre-trained multilingual language models to train quality estimation for machine translation. |
| Outcome: | A carefully engineered ensemble of pre-trained language models wins the QE shared task at WMT19. |
Circuit Complexity Bounds for RoPE-based Transformer Architecture (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies provide the circuit complexity bounds to Transformer-like architectures. position embedding has emerged as a crucial technique in modern large language models. |
| Approach: | They propose to use position embedding to improve Transformer-like architectures by analyzing their circuits and analyzing the results. |
| Outcome: | The proposed model is able to solve canonical tasks without embedding positional information. |
Revisiting Over-Smoothness in Text to Speech (2022.acl-long)
Copied to clipboard
| Challenge: | Non-autoregressive text to speech models ignore correlation in time and frequency domains, causing blurry results. |
| Approach: | They revisit the problem of over-smoothness in non-autoregressive text to speech models . they use methods that reduce complexity of data distributions and improve modeling methods . |
| Outcome: | The proposed models achieve better voice quality and faster inference speed than autoregressive models. |
SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation (2025.acl-long)
Copied to clipboard
Wenyu Zhang, Wei En Ng, Lixin Ma, Yuwen Wang, Junqi Zhao, Allison Koenecke, Boyang Li, Wanglu Wanglu
| Challenge: | Current vision-language models lack multi-dimensional spatial reasoning capabilities for human-like understanding and applications. |
| Approach: | They propose a hierarchical evaluation framework that probes models across increasing levels of complexity and integrates spatial, visual, and logical understanding. |
| Outcome: | The proposed framework probes models across increasing levels of complexity, from basic skills to multi-skill integration and high-level reasoning that combines spatial, visual, and logical understanding. |
Generating Scientific Definitions with Controllable Complexity (2022.acl-long)
Copied to clipboard
| Challenge: | Unfamiliar terminology and complex language can make understanding science difficult for readers. |
| Approach: | They propose a task and dataset for defining scientific terms and controlling the complexity of generated definitions by a sequence-to-sequence approach. |
| Outcome: | The proposed system is based on a sequence-to-sequence approach and human evaluations show it offers superior fluency while controlling complexity. |
FastGAS: Fast Graph-based Annotation Selection for In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to select unlabeled examples for annotation require a long time due to their complexity, hindering their practical viability. |
| Approach: | They propose a graph-based selection method to efficiently identify high-quality instances while minimizing computational overhead. |
| Outcome: | The proposed method significantly reduces selection time and improves performance on different tasks. |
Generating Biographies on Wikipedia: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies (2022.acl-long)
Copied to clipboard
| Challenge: | Existing efforts to encourage article creation focus on reducing the gender gap in Wikipedia articles. |
| Approach: | They propose a model that retrieves web evidence and generates biographies section by section . they analyze available web evidence to determine the accuracy of the generated text . |
| Outcome: | The proposed model can generate biographies section by section, including citation information, using retrieval mechanisms and a cache-based pre-trained encoder-decoder. |
Bootstrapping Techniques for Polysynthetic Morphological Analysis (2020.acl-main)
Copied to clipboard
| Challenge: | Polysynthetic languages have exceptionally large and sparse vocabularies due to the number of morpheme slots and combinations in a word. |
| Approach: | They propose linguistically-informed approaches for bootstrapping a neural morphological analyzer . they use a finite state transducer to train an encoder-decoder model . |
| Outcome: | The proposed method improves on a polysynthetic language's model by "hallucinating" missing linguistic structure and resampling from a Zipf distribution to simulate a more natural distribution of morphemes. |
Explainable Multi-hop Question Generation: An End-to-End Approach without Intermediate Question Labeling (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that generate complex questions do not explain reasoning process behind generated multi-hop questions. |
| Approach: | They propose an end-to-end question rewriting model that increases question complexity through sequential rewrite. |
| Outcome: | The proposed model generates complex questions that require multi-step reasoning over multiple documents. |
On the Formal Standardization of Terminology Resources: The Case Study of TriMED (2020.lrec-1)
Copied to clipboard
| Challenge: | Terminology standardization plays an important role in the management of terminological resources. |
| Approach: | They propose to re-model an existing multilingual terminological database for the medical domain, TriMED, and propose a method to make it compliant to the latest ISO/TC 37 standards. |
| Outcome: | The proposed model should be compliant with the three most recent ISO/TC 37 standards and has a new data category repository and a Web application that can be used to access the multilingual terminological records. |
GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization (2024.emnlp-main)
Copied to clipboard
Yangfan Ye, Xiachong Feng, Xiaocheng Feng, Weitao Ma, Libo Qin, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin
| Challenge: | Current studies focus on single-language or single-document tasks for news summarization . lack of a benchmark inhibits researchers from adequately studying this invaluable problem. |
| Approach: | They propose a novel task that unifies Multi-lingual, Cross-lingual and Multi-document Summarization into one task. |
| Outcome: | The proposed task encapsulates the real-world requirements all-in-one and is validated by extensive analysis. |
Rhetorical Device-Aware Sarcasm Detection with Counterfactual Data Augmentation (2025.findings-acl)
Copied to clipboard
| Challenge: | Sarcasm is a complex form of sentiment expression widely used in human daily life. |
| Approach: | They propose a device-aware sarcasm dataset with counterfactually augmented data to capture its complexity. |
| Outcome: | The proposed dataset shows that it is more balanced than zero-shot models. |
Automatically Discovering How Misogyny is Framed on Social Media (2025.naacl-long)
Copied to clipboard
| Challenge: | a study on misogyny on social media reveals that there are many forms of misogamy, each addressing another MisogYny Problem (MP) the detection of the way misogony is framed is important for identifying misos . |
| Approach: | This paper considers the automatic discovery of misogyny problems and their frames through the Dis-MP&F method . it proposes a data-driven, rich Taxonomy of MisogYny (ToM) method that can be used to generate a misomy benchmark dataset. |
| Outcome: | The proposed method can generate a data-driven, rich Taxonomy of misogyny (ToM) and produces promising results on a misomyne benchmark dataset. |
Detecting Conversational Mental Manipulation with Intent-Aware Prompting (2025.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to detect mental manipulations are limited due to complexity of detecting subtle, covert tactics in conversations. |
| Approach: | They propose an approach to detect mental manipulations using large language models using intent-aware prompting by capturing the intents of participants. |
| Outcome: | The proposed approach significantly reduces false negatives, helping detect more instances of mental manipulation with minimal misjudgment of positive cases. |
FalAI: A Dataset for End-to-end Spoken Language Understanding in a Low-Resource Scenario (2024.lrec-main)
Copied to clipboard
Andres Pineiro-Martin, Carmen Garcia-Mateo, Laura Docio-Fernandez, Maria del Carmen Lopez-Perez, Jose Gandarela-Rodriguez
| Challenge: | End-to-end (E2E) Spoken Language Understanding systems extract structured information from speech signals using a single model. |
| Approach: | They propose to use a dataset to extract structured information from speech signals . they define splits for noisy audio, hesitant audio and audio where sentence has changed . |
| Outcome: | The proposed model exploits acoustic information and avoids cascading errors . falAI dataset is the largest public SLU dataset in Galician and first to be obtained in low-resource scenario. |
medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs (2025.coling-main)
Copied to clipboard
| Challenge: | Electronic Medical Records (EMRs) are the digitized record of a patient's medical and health information and are integral to modern healthcare. |
| Approach: | They propose a framework that combines Large Language Models (LLMs) with knowledge graphs (KGs) to enhance diagnostic capabilities. |
| Outcome: | The proposed framework assigns weighted importance to entities in medical records based on their type, enabling precise localization of candidate diseases within KGs. |
Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work on sequence labelling has explored different strategies to mitigate the lack of manually annotated data for the large majority of the world languages. |
| Approach: | They propose to use the mask objective to exploit the few-shot capabilities of pre-trained language models to improve their performance. |
| Outcome: | The proposed model-transfer outperforms data-transference and fine-tuning outperformed few-shot methods for Argument Mining task. |
From Answers to Arguments: Toward Trustworthy Clinical Diagnostic Reasoning with Toulmin-Guided Curriculum Goal-Conditioned Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are obstructed by their opaque and often unreliable reasoning. |
| Approach: | They propose a framework for trustworthy clinical argumentation by adapting the Toulmin model to the diagnostic process. |
| Outcome: | The proposed method achieves diagnostic accuracy comparable to resource-intensive RL methods while offering a more stable and efficient training pipeline. |
FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for unlearning undesirable knowledge have overlooked complexity and interconnectedness of knowledge, authors say . previous studies have neglected the complex nature of knowledge and neglected its internal dependencies. |
| Approach: | They propose a new concept called superficial unlearning to evaluate faithfulness of unlearning in knowledge QA settings. |
| Outcome: | The proposed method shows significant effectiveness in real-world knowledge QA settings. |
Multi-class Hierarchical Question Classification for Multiple Choice Science Exams (2020.lrec-1)
Copied to clipboard
Dongfang Xu, Peter Jansen, Jaycie Martin, Zhengnan Xie, Vikas Yadav, Harish Tayyar Madabushi, Oyvind Tafjord, Peter Clark
| Challenge: | Prior work has demonstrated that question classification (QC) can help answer a question more accurately. |
| Approach: | They propose to use a large dataset for question classification (QC) that contains 7,787 science exam questions paired with detailed classification labels from a fine-grained hierarchical taxonomy of 406 problem domains to train a BERT-based model. |
| Outcome: | The proposed model achieves a large (+0.12 MAP) gain while also achieving state-of-the-art performance on benchmark open-domain and biomedical QC datasets. |
HSDreport: Heart Sound Diagnosis with Echocardiography Reports (2024.findings-emnlp)
Copied to clipboard
Zihan Zhao, Pingjie Wang, Liudan Zhao, Yuchen Yang, Ya Zhang, Kun Sun, Xin Sun, Xin Zhou, Yu Wang, Yanfeng Wang
| Challenge: | Existing methods for heart sound diagnosis are limited to a few fixed categories and do not utilize echocardiography reports, the gold standard in the diagnosis of related diseases. |
| Approach: | They propose a benchmark that mandates the direct utilization of heart sounds obtained from auscultation to predict echocardiography reports. |
| Outcome: | The proposed method outperforms existing methods and existing multimodal LLMs in detecting key abnormalities in heart sounds. |
TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition (2025.acl-long)
Copied to clipboard
Tianwei Lin, Jiang Liu, Wenqiao Zhang, Yang Dai, Haoyuan Li, Zhelun Yu, Wanggui He, Juncheng Li, Jiannan Guo, Hao Jiang, Siliang Tang, Yueting Zhuang
| Challenge: | Existing methods for fine-tuning are resource-efficient, but performance often falls short . a new approach, TeamLoRA, integrates collaborative and competitive modules to improve performance. |
| Approach: | They propose to introduce task-specific LoRA as domain experts to improve learning efficiency . teamLoRA integrates collaborative and competition modules to improve model learning . |
| Outcome: | Experiments show that TeamLoRA improves performance in multi-task learning . teamLorea integrates collaborative and competitive modules to improve performance . |
How does the task complexity of masked pretraining objectives affect downstream performance? (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is a widely used self-supervised pretraining objective. |
| Approach: | They propose to use a mask-based objective to predict a token that is replaced with a masked token given its context. |
| Outcome: | The proposed objectives show that they should have half the complexity needed to perform comparably to MLM. |
Generating Clarification Questions for Disambiguating Contracts (2024.lrec-main)
Copied to clipboard
| Challenge: | Contractual clauses are obligatory and can detail downstream implementation activities . however, contract ambiguities can be difficult to comprehend and can lead to errors . |
| Approach: | They propose a legal NLP task that generates clarification questions for contracts . they propose generating questions that identify contract ambiguities on a document level . |
| Outcome: | The proposed task generates clarification questions for contracts that detect ambiguities on a document level and can generate an F2 score of 0.87. |
Cross-Policy Compliance Detection via Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Policy compliance detection is the task of ensuring that a scenario conforms to a policy. |
| Approach: | They propose to decompose policy compliance detection into question answering . they propose to use an existing dataset to augment expert annotations . |
| Outcome: | The proposed approach improves accuracy in cross-policy setups, especially when policies are unseen in training. |
Divide-Verify-Refine: Can LLMs Self-align with Complex Instructions? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing research shows LLMs struggle with complex instructions involving multiple constraints. |
| Approach: | They propose a framework to divide complex instructions into single constraints and prepare appropriate tools to verify responses. |
| Outcome: | The proposed framework doubles Llama3.1-8B’s constraint adherence and triples Mistral-7B’ s performance. |
HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to answer multi-hop questions are query-agnostic and the extracted facts are ambiguous as they lack context. |
| Approach: | They propose to use a knowledge graph to extract query-relevant information from unstructured text. |
| Outcome: | The proposed method achieves performance improvements on two popular datasets. |
A Benchmark Dataset for Multi-Level Complexity-Controllable Machine Translation (2022.lrec-1)
Copied to clipboard
Kazuki Tani, Ryoya Yuasa, Kazuki Takikawa, Akihiro Tamura, Tomoyuki Kajiwara, Takashi Ninomiya, Tsuneo Kato
| Challenge: | Existing test datasets for MLCC-MT have three problems: A source language sentence and its simplified target language sentence are not necessarily exactly parallel. |
| Approach: | They propose to use a test dataset to evaluate multi-level complexity-controllable machine translation (MLCC-MT) their results are compared to a standard test dataset constructed from the Newsela corpus . |
| Outcome: | The proposed test dataset is based on the Newsela corpus and is released . it includes automatic filtering, manual check for parallel target language sentences . |
SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data. |
| Approach: | They propose to use a scientific entity and relation extraction dataset to capture interactions between entities in full texts. |
| Outcome: | The proposed dataset captures the intricate use and interactions among entities in full texts and provides an out-of-distribution test set to offer a more realistic evaluation. |
Evolving Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Current models are static entities incapable of compressing complexity of real world into generalisable concepts . authors: lack of endogenous mechanism for representation updating renders models vulnerable to domain mismatch and catastrophic forgetting . |
| Approach: | a meta-control system distils on-the-fly abstract representations of states, actions, goals . authors propose a paradigm for autonomous learning driven by pseudo-symbolic abstraction . |
| Outcome: | a meta-control system distils on-the-fly abstract representations of states, actions, goals . a novel approach resolves the domain mismatch problem and lays the groundwork for truly autonomous AI models . |
Ideological Knowledge Representation: Framing Climate Change in EcoLexicon (2024.lrec-main)
Copied to clipboard
| Challenge: | a method to extract ideological knowledge from corpora is proposed to represent environmental concepts and terms in terminological resources . political discourse is not considered specialized language, but politicians use scientific terms to soften the message . |
| Approach: | They propose to use terminological knowledge bases to represent political discourse on environmental concepts and terms. |
| Outcome: | The proposed method shows how climate change discourse changes across ideological spectrum . it uses spanish and english parliamentary debates to extract ideological knowledge from corpora. |
TabularMath: Understanding Math Reasoning over Tables with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Mathematical reasoning has long been a key benchmark for evaluating large language models. |
| Approach: | They propose a framework that transforms math word problems into scalable tabular reasoning tasks. |
| Outcome: | The proposed framework transforms math word problems into scalable and verified tabular reasoning tasks. |
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
DAICT: A Dialectal Arabic Irony Corpus Extracted from Twitter (2020.lrec-1)
Copied to clipboard
| Challenge: | Current scholarship is yet to reach an agreement on a universal definition of the concept of irony. |
| Approach: | They propose to query Twitter using irony-related hashtags to collect ironic messages which are then manually annotated by two linguists according to their working definition of irony. |
| Outcome: | The proposed corpus will be a valuable resource for developing open domain systems for automatic irony recognition in Arabic and its dialects in social media text. |
Instance Regularization for Discriminative Language Model Pre-training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have optimized independent strategies of ennoising or denosing . Existing methods treat training instances equally throughout the training process . |
| Approach: | They propose to use ennoising and denoising to train discriminative pre-trained language models . they propose to model the complexity of restoring the original sentences from corrupted ones . |
| Outcome: | Experimental results show that the proposed method improves pre-training efficiency, effectiveness, and robustness. |
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning (2025.findings-acl)
Copied to clipboard
Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, Xiaoyu Shen
| Challenge: | Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought prompting. |
| Approach: | They examine the factors influencing CoT distillation including granularity, format and teacher model. |
| Outcome: | The proposed model is based on four teacher models and seven student models across seven mathematical and commonsense reasoning datasets. |
Automated Knowledge Graph Construction using Large Language Models and Sentence Complexity Modelling (2025.emnlp-main)
Copied to clipboard
Sydney Anuyah, Mehedi Mahmud Kaushik, Sri Rama Krishna Reddy Dwarampudi, Rakesh Shiradkar, Arjan Durresi, Sunandan Chakraborty
| Challenge: | integrating coreference and decomposition increases recall on rare relations by over 20%. |
| Approach: | They propose an open-source pipeline for extracting sentence-level knowledge graphs by combining robust coreference resolution with syntactic sentence decomposition. |
| Outcome: | The proposed pipeline achieves a 99.8% exact-match accuracy on sentence simplification. |
QUARTZ: QA-based Unsupervised Abstractive Refinement for Task-oriented Dialogue Summarization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a framework for task-oriented utility-based dialogue summarization is proposed . QUARTZ is a tool for task summarizing dialogues, but its outputs lack task-specific focus. |
| Approach: | They propose a framework for task-oriented utility-based dialogue summarization . QUARTZ generates summaries and question-answer pairs from a dialogue in a zero-shot manner . |
| Outcome: | The proposed framework achieves competitive results in zero-shot settings, rivaling fully-supervised State-of-the-Art methods. |
Uniform Complexity for Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not capture factors that contribute to producing consistent text. |
| Approach: | They propose a benchmark test to evaluate text complexity in generative models by observing linguistic properties of input prompts. |
| Outcome: | The proposed model fails to preserve complexity of input prompts even if finetuned with professionally written texts. |
PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning (2025.acl-long)
Copied to clipboard
Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, Jun Liu
| Challenge: | Large language models demonstrate remarkable capabilities across various domains, including mathematics and logic reasoning. |
| Approach: | They propose a physics-based reasoning benchmark that includes physics theorems and constraints and a Physics Solution Auto Scoring Framework to evaluate physics based reasoning in large language models. |
| Outcome: | The proposed framework enables models to achieve less than 60% on answer-level evaluation, with performance dropping from knowledge questions (75.11%) to hard problems (31.99%). |
Is It Possible to Modify Text to a Target Readability Level? An Initial Investigation Using Zero-Shot Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Text simplification and elaboration tasks are limited to only relatively altering the readability of texts to cater to a diverse audience. |
| Approach: | They propose to generate 8 versions of a text at different readability levels using ChatGPT and Llama-2 and introduce a two-step process to generate paraphrases. |
| Outcome: | The proposed task requires the generation of 8 versions at various target readability levels for each input text. |
Theory of Mind in Freely-Told Children’s Narratives: A Classification Approach (2023.findings-acl)
Copied to clipboard
| Challenge: | Language and Theory of Mind (ToM) competences are often studied with younger children and standardized tests, but as both are social competences, data and methods with higher ecological validity are critical. |
| Approach: | They leveraged a corpus of 442 freely-told stories by Dutch children aged 4-12 to study language and ToM with NLP-tools. |
| Outcome: | The proposed method is robust relative to the complexity of the task for humans and is consistent with previous studies. |
Global Eye: Breaking the “Fixed Thinking Pattern” during the Instruction Expansion Process (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods focus on constructing multi-perspective prompts to expand instructions, overlooking the “Fixed Thinking Pattern” issue of Large Language Models. |
| Approach: | They propose a method that analyzes the statistical characteristics of newly generated instructions and updates the prompts after a fixed number of instruction expansions. |
| Outcome: | The proposed method surpasses open-source LLMs and GPT3.5 in several metrics. |
Control Large Language Models via Divide and Conquer (2024.emnlp-main)
Copied to clipboard
| Challenge: | Lexically Constrained Generation (LCG) is a crucial task of text generation. |
| Approach: | They propose a Divide and Conquer Generation strategy to enhance LLMs' performance in Lexically Constrained Generation with prompt-based controlling. |
| Outcome: | The proposed strategy shows 90% improvement on the most challenging LCG task. |
Hierarchical Attention Generates Better Proofs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in formal theorem proving, but their token-level processing often fails to capture the inherent hierarchical nature of mathematical proofs. |
| Approach: | They propose a regularization method that aligns LLMs’ attention mechanisms with mathematical reasoning structures and establishes a five-level hierarchy from foundational elements to high-level concepts. |
| Outcome: | The proposed method improves proof success rates by 2.05% on miniF2F and 1.69% on ProofNet while reducing proof complexity by 23.81% and 16.50% respectively. |
Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Symbolic logical reasoning is a critical yet underexplored capability of large language models (LLMs). |
| Approach: | They propose a framework that aligns natural language with logical symbols to establish a shared representation and reshapes training dynamics around phase-transition boundaries to progressively strengthen reasoning at increasing logical depths. |
| Outcome: | The proposed framework mitigates logical reasoning collapse at high complexity while improving generalization to unseen logical compositions. |
Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-hop Question Answering (MHQA) adds layers of complexity to question answering tasks. |
| Approach: | They explore how LMs respond to multi-hop questions by permuting search results under various configurations. |
| Outcome: | The proposed model outperforms decoder-only models in MHQA tasks despite being significantly smaller in size . |
Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based learning has been an effective paradigm for large pretrained language models (LLMs), enabling few-shot or even zero-shot learning. |
| Approach: | They propose a black-box prompt search method that clusters and prunes the search space to focus exclusively on influential prompt tokens. |
| Outcome: | The proposed method achieves state-of-the-art performance across tasks and LLMs while significantly reducing search costs. |
Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to predict human emotions are inconsistent due to complexity of emotion and subjectivity of perception. |
| Approach: | They propose a Bayesian approach to estimate uncertainty in emotion attributes using a deep neural network model. |
| Outcome: | The proposed approach estimates uncertainty in emotion attributes along with aleatoric and epistemic uncertainties. |
ARTS: Assessing Readability & Text Simplicity (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text simplification evaluation measures do not include simplicity labels on text level, as they are mainly based on a relational concept. |
| Approach: | They propose a method for language-independent construction of datasets for simplicity assessment using pairwise comparisons of texts in conjunction with an Elo algorithm to produce a simplicity ranking and simplicity scores. |
| Outcome: | The proposed method produces a ranking and simplicity scores for human-labeled and three GPT-labelled datasets. |
Re-examining Sexism and Misogyny Classification with Annotator Attitudes (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets for content moderation fail to capture plurality of possible annotator perspectives or ensure representation of affected groups. |
| Approach: | They examine the relationship between annotator identities and attitudes and the responses they give to two GBV labelling tasks. |
| Outcome: | The results show that higher Right Wing Authoritarianism scores are associated with a higher propensity to label text as sexist . higher scores are also associated with negative attitudes towards sexism and neosexist attitudes . |
Data Factors for Better Compositional Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent diagnostic datasets on compositional generalization expose severe problems . state-of-the-art models trained on larger and more general datasets show better generalization ability . |
| Approach: | They conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors including dataset scale, pattern complexity, example difficulty, etc. |
| Outcome: | The proposed model training on larger datasets improves on compositional generalization tasks. |
Retrieval-augmented GUI Agents with Generative Guidelines (2025.emnlp-main)
Copied to clipboard
| Challenge: | GUI agents powered by vision-language models struggle with real-world tasks due to their complex nature and limited training data. |
| Approach: | They propose a lightweight vision-language model that leverages web tutorials at inferencetime to synthesize GUI agents. |
| Outcome: | The proposed agent outperforms baseline GUI agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes. |
Distance-Based Propagation for Efficient Knowledge Graph Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge graph completion (KGC) aims to predict unseen edges in knowledge graphs (KGs) . a few recent attempts to address this problem sacrifice the performance to gain efficiency. |
| Approach: | They propose a method that aggregates path information to solve this problem by aggregating paths in a fixed window for each source-target pair. |
| Outcome: | The proposed method can cut down on the number of propagated messages by 90% while achieving competitive performance on multiple KG datasets. |
Tag-Instruct: Controlled Instruction Complexity Enhancement through Structure-based Augmentation (2025.findings-acl)
Copied to clipboard
| Challenge: | High-quality instruction data is crucial for developing large language models (LLMs), yet existing approaches struggle to effectively control instruction complexity. |
| Approach: | They propose a framework that compresses instructions into a compact tag space and enhances complexity through RL-guided tag expansion. |
| Outcome: | The proposed framework outperforms existing methods in the evaluation of instruction complexity augmentation and semantic compression of text into a compact tag space. |
Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups (2024.emnlp-main)
Copied to clipboard
Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel
| Challenge: | Large language models (LLMs) are popular in the Natural Language Processing community because of their versatility and capability to solve unseen tasks in zero/few-shot settings. |
| Approach: | They investigate the use of large language models in CWI, LCP, and MWE settings by evaluating their use in zero-shot, few-shot and fine-tuning settings. |
| Outcome: | The proposed models struggle in certain conditions or achieve comparable results against existing methods. |
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for reasoning about events in narratives do not understand the complexity of the causal relationships of events in the narrative. |
| Approach: | They propose a Causal Reasoning Assessment Benchmark to evaluate causal understanding of events in narratives. |
| Outcome: | The proposed model performs worse when models are derived from complex causal structures than simple linear causal chains. |
Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) are a promising unsupervised approach for understanding the representations of layers of Large Language Models (LLMs). |
| Approach: | They propose a method that groups similar models and trains a single SAE per group based on representational similarity across layers. |
| Outcome: | Experiments on Pythia family models show that the proposed method significantly accelerates training with minimal impact on reconstruction quality and comparable downstream task performance and interpretability over baseline SAEs trained layer by layer. |
Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, but temporal reasoning remains challenging due to its intrinsic complexity. |
| Approach: | They propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporal narrative. |
| Outcome: | The proposed technique achieves the highest F1 on Schema-11 evaluation set, while securing an overall F1 of par with GPT-3.5/4. |
Mitigating Bias in RAG: Controlling the Embedder (2025.findings-acl)
Copied to clipboard
| Challenge: | a promising modular AI system enhances factuality and privacy in large language models . however, each component introduces its own biases into the RAG system, which could cause representational harm and unsafe user interactions. |
| Approach: | They study the conflict between biases of each component and their relationship to the overall bias of the retrieval augmented generation system. |
| Outcome: | The proposed model can be controlled by the embedder while maintaining utility . the proposed model is more fair than existing models, the authors show . |
(Almost) Free Modality Stitching of Foundation Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-modal foundation models often use modality-specific (uni-modal) models as sub-components, which are stitched together via a connector module. |
| Approach: | They propose a framework that allows for optimal uni-modal model selection and connector training by leveraging hypernetworks. |
| Outcome: | The proposed framework reduces the cost of searching for the best performing uni-modal model pair by 10 while matching the ranking and trained connector performance across diverse multi-modal benchmarks. |
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct (2025.findings-acl)
Copied to clipboard
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Yongbin Li, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Hamid Alinejad-Rokny, Xiaobo Xia, Jingkuan Song, Fei Huang
| Challenge: | a new framework for image-text instruction data evolution improves MLLM performance . lack of high-quality instruction data remains a major bottleneck in ML modeling . |
| Approach: | They propose a multimodal instruction data evolution framework that iteratively enhances data quality through fine-grained perception, cognitive reasoning, and interaction evolution. |
| Outcome: | The proposed approach improves MLLM performance in nine vision-language tasks while using significantly less data. |
MAGIC: Deep Geometric Evolution with Structural Consensus for Temporal Knowledge Graph Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing multi-geometry approaches face two key bottlenecks: Riemannian depth barrier and gate collapse. |
| Approach: | They propose a framework for Temporal Knowledge Graph reasoning that integrates a Tangent-Residual Engine into multi-geometric spaces to regulate gradient flow and prevent collapse. |
| Outcome: | The proposed framework improves state-of-the-art in TKG reasoning by up to 2.9 points. |
Data Interpreter: An LLM Agent for Data Science (2025.findings-acl)
Copied to clipboard
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Robert Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Yongxin Ni, Zhibin Gou, Zongze Xu, Yuyu Luo, Chenglin Wu
| Challenge: | Large Language Models (LLMs) excel in various domains but face challenges when applied to data science workflows due to their complex, multi-stage nature. |
| Approach: | They propose a hierarchical graph-based agent that represents complexity and a progressive strategy for step-by-step verification, refinement, and consistent context management. |
| Outcome: | The proposed agent surpasses state-of-the-art baselines on the MATH dataset and performs better on InfiAgent-DABench. |
Masked Path Modeling for Vision-and-Language Navigation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | A major challenge in vision-and-language navigation is the limited available training data, which hinders the models’ ability to generalize effectively. |
| Approach: | They propose a masked path modeling objective that pretrains an agent using self-collected data for subsequent navigation tasks. |
| Outcome: | The proposed model pretrains an agent using self-collected data for subsequent navigation tasks eliminating the need for external tools. |
PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving (2025.emnlp-main)
Copied to clipboard
Mihir Parmar, Xin Liu, Palash Goyal, Yanfei Chen, Long Le, Swaroop Mishra, Hossein Mobahi, Jindong Gu, Zifeng Wang, Hootan Nakhost, Chitta Baral, Chen-Yu Lee, Tomas Pfister, Hamid Palangi
| Challenge: | Existing methods for natural planning lack constraint-guided iterative verification and adaptive selection . a recent study found that LLMs are not good at such planning. |
| Approach: | They propose a model-agnostic and easily scalable agent framework with three key components: constraint, verification, and selection agents. |
| Outcome: | The proposed framework improves inference-time algorithms on NATURAL PLAN and OlympiadBench benchmarks. |
Don’t Erase, Inform! Detecting and Contextualizing Harmful Language in Cultural Heritage Collections (2025.acl-long)
Copied to clipboard
| Challenge: | Cultural Heritage metadata can contain outdated or offensive terms that reflect historical cultural and societal norms. |
| Approach: | They propose an AI-powered tool that detects offensive terms in CH metadata . the tool has processed over 7.9 million records and provides contextual insights . |
| Outcome: | The proposed tool has processed over 7.9 million records and provides contextual insights . it pairs biased language with contextual information and suggestions for appropriate usage . |
DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have significantly enhanced their knowledge and generative capabilities, leading to a surge of interest in leveraging LLMs for high-quality data synthesis. |
| Approach: | They propose a controllable data synthesis framework based on variational autoencoder which leverages diffusion models to reserve more information of original distribution and format structure in the learned latent distribution. |
| Outcome: | The proposed framework generates high-quality data with performance exceeding that of real data by 2%–7% on seven real-world datasets. |
IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing algorithms to improve the ability of LLMs to follow complex instructions are lacking. |
| Approach: | They propose a benchmark to improve the ability to follow complex instructions by using a IOPO alignment method to take input and output preference into consideration. |
| Outcome: | The proposed algorithm shows 8.15%, 2.18% improvements on in-domain data and 5.91%, 2.83% on out-of-domain datasets compared to SFT and DPO respectively. |
Semantic Inversion, Identical Replies: Revisiting Negation Blindness in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Negation is a common occurrence in the real world and is essential for logical reasoning as it helps understand the opposite or absence of a statement. |
| Approach: | They propose a verification framework that includes task design and measurement methods to verify this phenomenon negation blindness on the query. |
| Outcome: | The proposed framework can be used to verify the model fails to capture semantic contradictions in negated queries despite its accurate understanding of knowledge about positive queries. |
ActionStudio: A Lightweight Framework for Data and Training of Large Action Models (2025.emnlp-main)
Copied to clipboard
Jianguo Zhang, Thai Quoc Hoang, Ming Zhu, Zuxin Liu, Shiyu Wang, Tulika Manoj Awalgaonkar, Akshara Prabhakar, Haolin Chen, Weiran Yao, Zhiwei Liu, Juntao Tan, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | Existing infrastructure for efficient agentic data processing and model training remains underdeveloped. |
| Approach: | They propose a lightweight and extensible data and training framework for large action models . they propose to unify diverse agent trajectories using Unified Format 2.0 . |
| Outcome: | The proposed framework shows 9 higher throughput than existing frameworks and performs well across public and realistic agent benchmarks. |
Scope-enhanced Compositional Semantic Parsing for DRT (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional semantic parsers for DRT struggle to produce well-formed representations due to the complexity of the sentence. |
| Approach: | They propose a compositional, neurosymbolic semantic parser for DRT that uses a novel mechanism for predicting quantifier scope. |
| Outcome: | The proposed model produces well-formed outputs and performs well on complex sentences. |
Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics struggle to evaluate adversarial negative examples . existing metrics struggle in handling adversarials, resulting in low correlations with human judgments. |
| Approach: | They propose a framework that integrates AMR and domain-specific language models for automatic open-domain dialogue evaluation. |
| Outcome: | The proposed evaluation framework achieves strong correlations with human judgments across multiple datasets. |
The Empirical Variability of Narrative Perceptions of Social Media Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Identifying stories in social media texts provides a lens through which we can study how individuals and communities process and communicate experiences. |
| Approach: | They construct a taxonomy of crowd workers’ varied and nuanced perceptions of storytelling by open-coding their free-text rationales. |
| Outcome: | The proposed model shows that crowd workers disagree on categorical labels, free-text storytelling rationales, authorial intent, and more. |
Overcoming Black-box Attack Inefficiency with Hybrid and Dynamic Select Algorithms (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing black-box attack methods require large number of queries, making them inefficient and impractical for researchers. |
| Approach: | proposed new attack selection strategies combine strengths of previous selection algorithms . Hybrid Select merges generalized BinarySelect techniques with GreedySelection . Dynamic Select learns which lengths of texts each selection method should be applied to . |
| Outcome: | proposed methods combine the strengths of previous selection algorithms to improve performance. |
Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for measuring gender stereotypical bias in language models are inconsistencies . lack of explicit standards in data gathering can have detrimental effects on results . |
| Approach: | They propose that currently available benchmarks capture only partial facets of gender stereotypes . they apply a framework from social psychology to balance data across components of gender stereotypes based on stereotypical benchmarks. |
| Outcome: | The proposed framework improves correlation between different benchmarks by using simple balancing techniques. |
AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to decomposing model activations into interpretable features fail to account for input complexity. |
| Approach: | They propose a framework that dynamically adjusts sparsity levels based on the semantic complexity of each input. |
| Outcome: | The proposed framework outperforms fixed-sparsity approaches on reconstruction fidelity, explained variance, cosine similarity and interpretability metrics while eliminating the burden of extensive hyperparameter tuning. |
Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Existing red-team methods rely on modifying user prompts, which lack adaptability to new data and may impact the agent’s performance. |
| Approach: | They propose a framework that implicitly manipulates the agent’s reasoning trajectory and memory retrieval with three key stages: Trigger Extraction, Reasoning Hijacking, and Constraint Tightening. |
| Outcome: | The proposed framework shows outstanding performance in cross-model and cross-scenario environments. |
REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for reproducing social science papers focus on reproducing results using provided code and data without assessing their consistency with the paper. |
| Approach: | They propose a benchmark to evaluate agentic AI systems' ability to automate reproducibility assessment. |
| Outcome: | The proposed benchmark oversimplifies real-world scenarios and lacks diversity in data formats and programming languages. |
NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities (2025.findings-emnlp)
Copied to clipboard
Changyu Zeng, Yifan Wang, Zimu Wang, Wei Wang, Zhengni Yang, Muyi Bao, Jimin Xiao, Anh Nguyen, Yutao Yue
| Challenge: | Existing 3D benchmarks lack fine-grained numerical reasoning task annotations, limiting MLLMs’ ability to perform precise spatial measurements and complex numerical reasoning. |
| Approach: | They propose a 3D-based benchmark to enhance indoor perceptual understanding by using multi-scale annotations and question-answer pairs. |
| Outcome: | The proposed benchmark improves indoor perceptual understanding by incorporating multi-scale annotations and question-answer pairs. |
SCOPE: Compress Mathematical Reasoning Steps for Efficient Automated Process Annotation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing process annotation approaches are computationally expensive. |
| Approach: | They propose a compression-based approach that transforms reasoning steps into code and normalizes them through Abstract Syntax Tree. |
| Outcome: | The proposed method outperforms existing methods on Best-of-N strategy and ProcessBench. |
Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer’s Disease Detection (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for in-context learning (ICL) perform poorly for AD diagnosis due to inherent complexity of task. |
| Approach: | They propose a demonstration selection strategy that leverages a delta score to assess the relative gains of each training example and a KNN-based retriever that dynamically selects optimal “representatives” for a given input. |
| Outcome: | The proposed model outperforms existing methods on two AD detection datasets and surpasses even supervised classifiers. |
Beyond Single View: A Comprehensive Benchmark for Medical Multimodal Large Language Models on Multi-Image Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal large language models are limited to multiview diagnostics. |
| Approach: | They propose a benchmark specifically designed for medical multi-image understanding that evaluates MLLMs across four dimensions. |
| Outcome: | The proposed model performs better in multi-image contexts than open-source models . the model perform better when processing increased visual loads than closed-source ones . |
Syntactic Control of Language Models by Posterior Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Syntactic control of generated text is crucial for many domain-specific applications of language models, where structural constraints, such as formality, grammatical correctness, or adherence to a given template, can significantly affect usability and readability. |
| Approach: | They propose to use posterior inference to enforce target constituency structure by sampling from a proposal distribution with a syntactic tagger to ensure that each generated token aligns with the desired syntakic structure. |
| Outcome: | The proposed method increases the F1 score from 12.31 (GPT2-large) and 35.33 (Llama3-8B) to about 93 in both cases without compromising the language model’s fluency. |
LPOI: Listwise Preference Optimization for Vision Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for aligning large VLMs with human preferences often overfit to textual information or exacerbate hallucinations. |
| Approach: | They propose an object-aware listwise preference optimization for reducing hallucinations in VLMs . they mask a critical object in an image and interpolate the masked region to form more complete images . |
| Outcome: | The proposed method outperforms existing methods in reducing hallucinations and enhancing performance on MMHalBench, AMBER, and Object HalBench. |
Enhancing RAG Efficiency with Adaptive Context Compression (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods apply fixed compression rates, over-compressing simple queries or under-compressed complex ones. |
| Approach: | a new framework uses a hierarchical compressor and a context selector to optimize inference efficiency . a framework that dynamically adjusts compression rates based on input complexity optimizes inference without loss of accuracy. |
| Outcome: | Adaptive Context Compression for RAG outperforms fixed-rate methods on Wikipedia and five QA datasets . |
SPICED: News Similarity Detection Dataset with Multiple Topics and Complexity Levels (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing semantic textual similarity (STS) datasets are not suitable for news similarity detection due to their specificity to a single topic. |
| Approach: | They propose to segment news similarity datasets into topics to improve model training . they propose four different levels of complexity specifically designed for news similarities detection task . |
| Outcome: | The proposed dataset includes seven topics: Crime & Law, Culture & Entertainment, Disasters & Accidents, Economy & Business, Politics / Conflicts, Science & Technology, Sports. |
Guaranteed Guess: A Language Modeling Approach for CISC-to-RISC Transpilation with Testing Guarantees (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ISA-centric transpilation pipelines are used to translate low-level programs between ISAs . GG provides high code coverage across unit tests and better energy efficiency . |
| Approach: | They propose a ISA-centric transpilation pipeline that embeds large language models into software testing frameworks to ensure accuracy. |
| Outcome: | The proposed method achieves high code coverage across unit tests and functional/semantic correctness of 99% on HumanEval and 49% on BringupBench programs. |
Knowledge Tracing in Programming Education Integrating Students’ Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge tracing models that ignore student questions are suboptimal for programming education because of the complexity of coding tasks and the diverse methods students use to solve problems. |
| Approach: | They propose a model that leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems. |
| Outcome: | The proposed model improves student performance across Python programming courses of differing difficulty levels. |
SILO-BENCH: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems (2026.acl-long)
Copied to clipboard
Yuzhe Zhang, Feiran Liu, Yi Shan, Xinyi Huang, Xin Yang, Yueqi Zhu, Xuxin Cheng, Cao Liu, Ke Zeng, Terry Jingchen Zhang, Wenyuan Jiang
| Challenge: | Existing benchmarks conflate coordination ability with role-based priors. |
| Approach: | They propose a role-free benchmark for evaluating free-form collaboration under information silos. |
| Outcome: | The proposed benchmark systematically probes coordination capabilities under information silos using 54 configurations and 3 frontier LLMs. |
Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies (2025.acl-long)
Copied to clipboard
Massimiliano Pronesti, Joao H Bettencourt-Silva, Paul Flanagan, Alessandra Pascale, Oisín Redmond, Anya Belz, Yufang Hou
| Challenge: | Systematic reviews are widely regarded as the gold standard in evidence-based medicine, heavily influencing medical decisions made by doctors, health authorities, and patients. |
| Approach: | They propose a retrieval-augmented generation framework to tackle the unique challenges of evidence extraction by leveraging forest plots from Cochrane systematic reviews. |
| Outcome: | The proposed framework outperforms existing methods by up to 10.3% in the F1 score on this task. |
DEFT: Demystifying VLN Failures via a Unified Dual-View Explainability Framework for LLM-based Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Existing interpretability methods isolate temporal criticality from feature salience, creating an alignment gap and failing to account for the behavioral instability of black-box agents. |
| Approach: | They propose a unified dual-view framework that jointly analyzes when a decision is pivotal and what visual evidence grounds it. |
| Outcome: | Extensive experiments on MatterPort3D show that DEFT outperforms baselines in both temporal and feature fidelity. |
Uncovering Scaling Laws for Large Language Models via Inverse Problems (2025.findings-emnlp)
Copied to clipboard
Arun Verma, Zhaoxuan Wu, Zijian Zhou, Xiaoqiang Lin, Zhiliang Chen, Rachael Hwee Ling Sim, Rui Qiao, Jingtan Wang, Nhung Bui, Xinyuan Niu, Wenyang Hu, Gregory Kang Ruey Lau, Zi-Yu Khoo, Zitong Zhao, Xinyi Xu, Apivich Hemachandra, See-Kiong Ng, Bryan Kian Hsiang Low
| Challenge: | Large Language Models (LLMs) have achieved remarkable success across diverse domains. |
| Approach: | inverse problems can efficiently uncover scaling laws that guide the building of LLMs, authors argue . authors propose brute-force approaches to improve LLM training costs due to high costs . |
| Outcome: | This paper advocates that inverse problems can efficiently uncover scaling laws that guide the building of LLMs to achieve the desirable performance with significantly better cost-effectiveness. |
ParaCook: On Time-Efficient Planning for Multi-Agent Systems (2026.findings-acl)
Copied to clipboard
Shiqi Zhang, Xinbei Ma, Yunqing Xu, Zouying Cao, Pengrui Lu, Haobo Yuan, Tiancheng Shen, Zhuosheng Zhang, Hai Zhao, Ming-Hsuan Yang
| Challenge: | Existing agent benchmarks focus on task completion while neglecting time efficiency in parallel and asynchronous operations. |
| Approach: | They propose a framework for large language models that allows agents to plan long-horizon tasks in a scalable way. |
| Outcome: | The proposed framework is based on the Overcooked game and can be used to evaluate time efficiency-aware multi-agent planning. |
To Judge or Not to Judge: Can Large Language Models Leverage the Dispute Focus in Legal Judgment? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing research on large language models for legal judgment prediction fails to address the complexity of civil judicial cases. |
| Approach: | They propose a framework that leverages the dispute focus to guide LLMs through a structured, judge-like cognitive workflow. |
| Outcome: | The proposed framework can guide LLMs through a structured, judge-like cognitive workflow. |
RubricBench: Aligning Model-Generated Rubrics with Human Standards (2026.acl-long)
Copied to clipboard
Junyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, Chen Ma
| Challenge: | Existing benchmarks lack discriminative complexity and ground-truth rubric annotations required for rigorous evaluation. |
| Approach: | They propose a curated benchmark with 1,147 pairwise comparisons to assess the reliability of rubric-based evaluation. |
| Outcome: | The proposed benchmarks show that they support diverse domains, exhibit discriminative ability, provide high-quality annotations, and include human-authored rubrics. |
Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) and AI systems have led to a paradigm shift in the design and optimization of complex workflows. |
| Approach: | They propose a systematic review of recent progress in optimizing compound AI systems . they formalize the notion of compound AI system optimization and classify existing methods along several key dimensions . |
| Outcome: | The proposed methods outperform existing methods in the field of compound AI and highlight open research challenges and future directions. |
PlanningArena: A Modular Benchmark for Multidimensional Evaluation of Planning and Tool Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs can be significantly improved by integrating external tools. |
| Approach: | They propose a framework that integrates external tools into large language models to evaluate their ability to generate action plans. |
| Outcome: | The proposed framework evaluates the ability of large language models to generate action plans and generate action plan templates. |
What Are the Implications of Your Question? Non-Information Seeking Question-Type Identification in CNN Transcripts (2024.lrec-main)
Copied to clipboard
| Challenge: | Non-information seeking questions capture subtle dynamics of human discourse . authors use dataset of over 1,500 information-seeking questions and NISQs as benchmark . |
| Approach: | They use a dataset of over 1,500 information-seeking question(ISQ) and NISQ to evaluate human and machine performance on classifying fine-grained NISq types. |
| Outcome: | The proposed corpus is the first publicly available for annotation of non-information seeking questions . it evaluates human and machine performance on classifying fine-grained questions based on models . |
What Is Needed for Intra-document Disambiguation of Math Identifiers? (2024.lrec-main)
Copied to clipboard
| Challenge: | Ambiguity in math identifiers within a document poses significant challenges to understanding formulae . ambiguity in mathematical expressions can be difficult to disambiguate, requiring intra-document disambiguation . |
| Approach: | They propose to use position data and local formula structure to disambiguate math identifiers . they train a model that performs similarly to humans with an 85% accuracy . |
| Outcome: | The proposed model outperforms rule-based models in natural language processing. |
SSSD: Simply-Scalable Speculative Decoding (2026.acl-long)
Copied to clipboard
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
| Challenge: | Existing methods for accelerating inference in Large Language Models require additional training and training, resulting in a higher deployment and maintenance cost. |
| Approach: | They propose a training-free method that combines lightweight n-gram matching with hardware-aware speculation. |
| Outcome: | SSSD reduces latency by up to 2.9 and is faster than autoregressive decoding methods. |
ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus (2024.lrec-main)
Copied to clipboard
| Challenge: | a corpus of multilingual Arabic-English speech is presented in a new paper . a major bottleneck is the lack of data needed for training NLP models . |
| Approach: | They propose a multilingual multidialectal Arabic-English speech corpus with a set of guidelines for automatic speech recognition. |
| Outcome: | The proposed corpus includes two languages with Arabic and English spoken in multiple variants and Arabic and Arabic with various accents. |
Explicit Learning and the LLM in Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a growing number of researchers are examining whether large language models can learn to translate a "new" language using grammar books. |
| Approach: | They examine an LLM's ability to learn new languages using grammar books . authors suggest alternative fine-tuning strategies to improve explicit learning . |
| Outcome: | The proposed model can learn low-resource languages described in grammar books but lacking extensive corpora. |
Orthogonal Finetuning Made Scalable (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent shift in foundation models has slowed the adoption of finetuning methods . however, its high runtime and memory demands limit its scalability . |
| Approach: | They propose an input-centric reformulation that uses matrix-vector multiplications instead of cubic multiplication . they extend OFTv2 to support finetuning quantized foundation models and show it outperforms QLoRA . |
| Outcome: | The proposed model outperforms the popular QLoRA in training stability, efficiency, and memory usage. |
KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Understanding and reasoning over text within visual contexts poses a significant challenge for Vision-Language Models. |
| Approach: | They propose a benchmark for Korean Reading and rEasoning in Text-rich VQA Attuned to diverse visual contexts to address this challenge. |
| Outcome: | The proposed benchmark is tailored for Korean reading and rEasoning in text-rich VQA attuned to diverse visual contexts. |
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls (2025.emnlp-main)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, Pavan Kapanipathi
| Challenge: | Existing benchmarks and datasets for tool calling have lagged behind . nested sequencing is a common problem in LLMs, but it is not enough to evaluate them. |
| Approach: | They propose a benchmark to evaluate LLMs on nested sequences of API calls, i.e. sequences where the output of one API call is passed as input to a subsequent call. |
| Outcome: | The proposed model achieves a full sequence match accuracy of 28% and a win-rate of 60% on nested sequences of API calls. |
STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Text2SQL systems do not support SQL analysts in their primary work of performing complex analytics on specialized databases. |
| Approach: | They propose to decompose STARQA questions using SQL and Python to perform reasoning on specialized relational databases. |
| Outcome: | The proposed approach decomposes the task through a combination of SQL and Python, and achieves better performance on the more difficult questions. |
CAMEC: Complexity-Aware Multi-Expert Collaboration for Reliable Chinese Medical Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models are promising for medical question answering in china, but remain unreliable due to hallucinations, weak factual grounding and difficulty handling clinically complex cases. |
| Approach: | They propose a framework that combines hierarchical medical adaptation with complexity-aware expert routing for reliable Chinese medical QA. |
| Outcome: | The proposed framework outperforms strong general and medical LLM baselines on four Chinese medical benchmarks. |
AI Agents for the Science of Science: A Survey of Tasks, Architectures, Evaluations, and Challenges (2026.findings-acl)
Copied to clipboard
| Challenge: | The Science of Science (SciSc) examines how scientific knowledge is produced, evaluated, and transformed by utilizing large-scale scholarly and bibliometric data. |
| Approach: | They propose a task-centered taxonomy for AI agents that model citations, collaborations, and community dynamics. |
| Outcome: | The proposed taxonomy distinguishes agents as simulations from tools for empirical analysis and scientific workflows. |
A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification (2026.acl-long)
Copied to clipboard
| Challenge: | Modern LLM deployments are rarely a single model in isolation. |
| Approach: | They propose a model that reuses computation already paid for by the serving LLM . they instantiate a template with pooling, a scoring-attention gate, and a downcast multi-head self-attention probe . |
| Outcome: | The proposed model improves safety and sentiment benchmarks on dense and mixture-of-experts architectures while preserving near-serving latency. |
SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility (2026.acl-long)
Copied to clipboard
Xuyang Zhi, Peilun Zhou, Chengqiang Lu, Hang Lv, Yiwei Liang, Rongyang Zhang, Yan Gao, null Yiwu, Yao Hu, Hongchao Gu, Defu Lian, Hao Wang, Enhong Chen
| Challenge: | Large Language Models (LLMs) are shifting the focus from single verifiable tasks toward complex, open-ended real-world scenarios. |
| Approach: | They propose a framework that automatically adjusts reward weights and data importance to synchronize learning intent with data utility for optimal performance. |
| Outcome: | The proposed framework improves model capabilities across all domains and scales. |