Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Understanding Figurative Meaning through Explainable Visual Entailment (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for visual entailment and visual question-answering have limited ability to understand figurative meaning in images and captions. |
| Approach: | They propose a task framing the figurative meaning understanding problem as an explainable visual entailment task where the model has to predict whether the image entitles a caption and justify the predicted label with a textual explanation. |
| Outcome: | The proposed dataset contains 6,027 image, caption, label, explanation instances covering five diverse figurative phenomena. |
Benchmarking Distributional Alignment of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group remains uncertain. |
| Approach: | They construct a dataset expanding beyond political values and create human baselines for this task and evaluate the extent to which an LM can align with a particular group’s opinion distribution. |
| Outcome: | The proposed model can better describe opinion distributions than simulate demographic groups. |
World Models with Hints of Large Language Models for Goal Achieving (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods address this by adding intrinsic rewards, but they fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces lacking purposeful exploration. |
| Approach: | They propose a multi-modal model-based RL approach that integrates the proposed hinting subgoals into the model rollouts to encourage goal discovery and reaching in challenging tasks. |
| Outcome: | The proposed model outperforms existing methods in challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 41.8%, 21.1%, and 9.9%. |
CogLM: Tracking Cognitive Development of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have recently shown remarkable abilities across a wide variety of tasks, but few studies have explored the reasons behind the evolutionary relationship among various abilities. |
| Approach: | They construct a benchmark CogLM based on Piaget's Theory of Cognitive Development (PTC) which measures the cognitive levels of Large Language Models (LLMs) using 1,220 questions spanning 10 cognitive abilities crafted by more than 20 human experts. |
| Outcome: | The proposed framework provides a comprehensive testbed for the cognitive levels of LLMs. |
Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in representing individuals and communities, but evaluating their fidelity remains a challenge. |
| Approach: | They propose a framework for aligning large language models with online communities via instruction-tuning and comprehensively evaluating alignment across various aspects of language. |
| Outcome: | The proposed framework shows that it can be used to create high-fidelity representations of people and communities. |
Improving Retrospective Language Agents via Joint Policy Gradient Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have sparked interest in creating autonomous agents. |
| Approach: | They propose a framework that jointly optimizes both task-planning and self-reflective evolution capabilities in language agents. |
| Outcome: | The proposed framework improves task planning and self-reflective evolution capabilities in language agents. |
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases (2025.naacl-long)
Copied to clipboard
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Qizhe Shieh, Wenmeng Zhou
| Challenge: | Large Language Models excel in stand-alone code tasks but struggle with handling entire code repositories. |
| Approach: | They propose a system that integrates LLM agents with graph database interfaces extracted from code repositories. |
| Outcome: | The proposed system integrates LLM agents with graph database interfaces extracted from code repositories. |
Instantly Learning Preference Alignment via In-context DPO (2025.naacl-long)
Copied to clipboard
| Challenge: | Presently, mainstream approaches to HPA heavily depend on fine-tuning . however, the huge computational and annotation costs of fine-timing are hard to ignore . |
| Approach: | They propose a tuning-free approach to HPA using LLMs' decoding . they first rethink the derivation procedures of DPO and build an instant scorer . |
| Outcome: | The proposed approach outperforms existing methods even with tuning-free baselines and an upgraded scorer. |
ALTER: Augmentation for Large-Table-Based Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on the use of large language models (LLMs) for table-based reasoning, but most approaches struggle with scalability when applied to large tables. |
| Approach: | They propose a framework to harness latent augmentation potential in tabular data . they use only a small subset of relevant data from the table to supplement it with schema . |
| Outcome: | The proposed framework outperforms all other approaches and exhibits robustness and efficiency against perturbations in large-table scenarios. |
What the #?*!: Disentangling Hate Across Target Identities (2025.naacl-long)
Copied to clipboard
| Challenge: | Hate speech classifiers do not perform equally well in detecting hateful expressions towards different target identities. |
| Approach: | They propose to use two recently proposed functionality test datasets to analyze the impact of different factors on HS prediction. |
| Outcome: | The proposed classifiers do not perform equally well across different datasets and different target identities. |
MAD Speech: Measures of Acoustic Diversity of Speech (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in generative spoken language modeling have produced models that produce speech in a wide range of voices, prosody and recording conditions. |
| Approach: | They propose acoustic diversity metrics that measure voice, gender, emotion, accent, background noise and a priori known diversity preferences for each facet. |
| Outcome: | The proposed metrics show that they achieve stronger agreement with diversity than baselines. |
The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design (2025.naacl-long)
Copied to clipboard
| Challenge: | Embedding models are used in tasks such as information retrieval and semantic textual similarity. |
| Approach: | They propose a new Russian-focused embedding model called ru-en-RoSBERTa and a benchmark for Russian language . they propose to use the roMTEB benchmark to assess Russian and multilingual models . |
| Outcome: | The proposed model achieves results that are on par with state-of-the-art models in Russian. |
PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries (2025.naacl-long)
Copied to clipboard
Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, Patrick Ng, Zhiguo Wang
| Challenge: | Existing text-to-SQL systems focus on user questions with clear intentions that can be answered, but real user questions can be ambiguous with multiple interpretations or unanswerable due to a lack of relevant data. |
| Approach: | They construct a conversational text-to-SQL dataset called PRACTIQ, consisting of ambiguous and unanswerable questions inspired by real-world user questions. |
| Outcome: | The proposed system generates conversations with four turns, generating the user’s question, an assistant response seeking clarification, and the user's clarified SQL response with the natural language explanation of the execution results. |
MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems (2025.naacl-long)
Copied to clipboard
| Challenge: | Traditional retrieval-augmented generation benchmarks use heuristics as the ground truth for evaluation, but require an expensive large language model (LLM) as a judge for a reliable evaluation. |
| Approach: | They propose to use large language models as a judge for retrieval-augmented generation benchmarks . they use heuristic metrics as input and a large language model as heuriistic input . |
| Outcome: | The proposed method couples heuristic features with large language models as judge for evaluation. |
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs (2025.naacl-long)
Copied to clipboard
Do Xuan Long, Ngoc-Hai Nguyen, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F. Chen, Min-Yen Kan
| Challenge: | Using format-following capabilities, state-of-the-art large language models (LLMs) can be leveraged to tailor outputs to specific task formats. |
| Approach: | They propose to define a format bias evaluation metric and establish effective strategies to reduce it. |
| Outcome: | The proposed evaluation reduces the variance in ChatGPT’s performance among wrapping formats from 235.33 to 0.71 (%2) |
The Impact of Visual Information in Chinese Characters: Evaluating Large Models’ Ability to Recognize and Utilize Radicals (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have examined whether large language models and vision-language models can harness these sub-character features in Chinese through prompting. |
| Approach: | They establish a benchmark to evaluate large language models' understanding of visual elements in Chinese characters, including radicals, composition structures, strokes, and stroke counts. |
| Outcome: | The proposed model exhibits some, but still limited, knowledge of the visual elements in Chinese characters regardless of whether images of characters are provided. |
PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from related Example Banks (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive few-shot learning capabilities through in-context learning. |
| Approach: | They propose a novel Alternating Minimization approach for example selection that improves ICL performance on low-resource Indic languages. |
| Outcome: | The proposed approach outperforms existing frameworks for retrieving examples on low-resource Indic languages. |
Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for aligning large language models with human preferences are poor in extensibility and require significant retraining. |
| Approach: | They propose a multi-objective alignment approach that constructs an expert prompt and an adversarial prompt for each alignment objective to contrast at the decoding time. |
| Outcome: | The proposed approach is superior to existing methods in obtaining a well-distributed Pareto front among different alignment objectives. |
Fingerspelling within Sign Language Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work has studied fingerspelling recognition, but little attention has been paid to how well models understand it in context of entire sentences. |
| Approach: | They annotate instances of fingerspelling within FLEURS-ASL and use it to evaluate how well translation models understand it. |
| Outcome: | The proposed model family significantly improves understanding of fingerspelling, but the effect of the mixed model is mixed. |
MoDS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections (2025.naacl-long)
Copied to clipboard
Nishant Balepur, Alexa Siu, Nedim Lipka, Franck Dernoncourt, Tong Sun, Jordan Lee Boyd-Graber, Puneet Mathur
| Challenge: | Query-focused summarization (QFS) gives an overview of documents to answer a query, ignoring debatable ones. |
| Approach: | They propose a multi-LLM framework that uses a Query-focused summarization approach to create balanced summaries that answer debatable queries. |
| Outcome: | The proposed framework beats SOTA by 38-59% in topic paragraph coverage and balance, based on new citation metrics. |
Aligning Sentence Simplification with ESL Learner’s Proficiency for Language Acquisition (2025.naacl-long)
Copied to clipboard
| Challenge: | Text simplification is crucial for improving accessibility and comprehension for English as a Second Language (ESL) learners. |
| Approach: | They propose to simplify complex sentences to appropriate levels while also increasing vocabulary coverage of the target level. |
| Outcome: | The proposed method can increase frequency and diversity of vocabulary of the target level by more than 20% compared to baseline models, while maintaining high simplification quality. |
PeerQA: A Scientific Question Answering Dataset from Peer Reviews (2025.naacl-long)
Copied to clipboard
| Challenge: | a dataset of 579 QA pairs from 208 scientific articles contains answers that reviewers raised while thoroughly examining the scientific article. |
| Approach: | They propose a dataset that contains questions that reviewers raised while thoroughly examining the scientific article. |
| Outcome: | The proposed dataset contains 579 QA pairs from 208 academic articles . the results show that decontextualization approaches improve retrieval performance . |
ALiiCE: Evaluating Positional Fine-grained Citation Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research on citation generation is limited to sentence-level statements . positional fine-grained citations can appear anywhere within sentences . |
| Approach: | They propose a framework that allows LLMs to generate citations from sentences . they use dependency tree-based methods to parse sentence-level claims into atomic claims . |
| Outcome: | The proposed framework evaluates citation quality using three metrics including positional fine-grained citation recall, precision, and coefficient of variation of citation positions. |
An LLM-Based Approach for Insight Generation in Data Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to generate insightful data from databases are time-consuming and resource-intensive. |
| Approach: | They propose a method that leverages Large Language Models to automatically generate textual insights from databases. |
| Outcome: | The proposed approach generates more insightful insights than other approaches while maintaining correctness. |
WebQuality: A Large-scale Multi-modal Web Page Quality Assessment Dataset with Multiple Scoring Dimensions (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on web page quality assessment neglect the aspect of web page content. |
| Approach: | They propose a Chinese dataset for web page quality assessment . the dataset includes over 65,000 detailed an-notations spanning four sub-dimensions . |
| Outcome: | The proposed dataset includes over 65,000 detailed an-notations spanning four sub-dimensions and incorporates elements such as HTML+CSS, text, and visual screenshot. |
UFO: A UI-Focused Agent for Windows OS Interaction (2025.naacl-long)
Copied to clipboard
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
| Challenge: | UFO is a UI-Fcused agent designed to fulfill user requests tailored to Windows OS applications . it decomposes user requests using divide-and-conquer approach, enabling seamless navigation and addressing sub-tasks across multiple applications. |
| Approach: | They propose a UI-Fcused Windows OS agent that decomposes user requests using a divide-and-conquer approach and incorporates a control interaction module tailored for Windows OS. |
| Outcome: | The proposed agent decomposes user requests using divide-and-conquer approach, enabling seamless navigation and addressing sub-tasks across multiple applications. |
Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness (2025.naacl-long)
Copied to clipboard
| Challenge: | As models evolve, datasets can become outdated. |
| Approach: | They propose a human-grounded evaluation metric that assesses adversarialness by capturing models’ and humans’ varying abilities, while also identifying poor examples. |
| Outcome: | The proposed evaluation metric measures the accuracy of an adversarial question answering dataset and determines whether models are performing well on the dataset. |
Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing multimodal foundation models suffer from serious factual inaccuracy in radiology report generation. |
| Approach: | They propose a fact-aware multimodal retrieval-augmented pipeline for generating accurate radiology reports using RadGraph. |
| Outcome: | The proposed multimodal retrieval-augmented pipeline outperforms state-of-the-art retrievers on language generation and radiology-specific metrics. |
On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in NLP systems have led to widespread adoption by a broad spectrum of users across various domains, impacting decision-making, the job market, society, and scientific research. |
| Approach: | They examine existing interpretability paradigms, their properties, and their relevance to different stakeholders by analyzing trends from the past decade across multiple research fields. |
| Outcome: | The proposed models are complex and opaque and are often overlooked by technical surveys. |
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward (2025.naacl-long)
Copied to clipboard
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander G Hauptmann, Yonatan Bisk, Yiming Yang
| Challenge: | Existing studies have demonstrated that direct preference optimization (DPO) can be effective in generalizing large language models, but its effectiveness in video domain remains limited. |
| Approach: | They propose a framework that utilizes detailed video captions as a proxy of video content to enable language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. |
| Outcome: | The proposed framework shows that it can be used to align language models with video content and improves performance on open-ended video QA tasks. |
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing (2025.naacl-long)
Copied to clipboard
James Seale Smith, Chi-Heng Lin, Shikhar Tuli, Haris Jeelani, Shangqian Gao, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
| Challenge: | Empirical evaluations demonstrate substantial performance gains over existing methods . |
| Approach: | They propose a method to prune LLMs that selectively prunes model blocks based on an importance score and replaces them with a low-parameter replacement strategy. |
| Outcome: | The proposed method achieves state-of-the-art performance on 5/6 and 6/6 benchmarks with a compression rate of 30% and 40%. |
Conformalized Answer Set Prediction for Knowledge Graph Embedding (2025.naacl-long)
Copied to clipboard
| Challenge: | Knowledge graph embeddings (KGE) map entities and predicates into numerical vectors, providing non-classical reasoning capabilities based on similarities and analogies between entities and relations. |
| Approach: | They propose to use knowledge graph embeddings to provide non-classical reasoning capabilities by exploiting similarities and analogies between entities and relations. |
| Outcome: | The proposed model can generate answer sets with probabilistic guarantees on four benchmark datasets and is scaled well with respect to the difficulty of the query. |
Parameter-free and Accessible Prompt Learning to Enhance Adversarial Robustness for Pre-trained Vision-Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large pre-trained Vision-Language Models (VLMs) have revolutionized downstream vision-language tasks including classification, object detection, and segmentation. |
| Approach: | They propose to search for text prompts at the word level rather than optimizing continuous textual embeddings to boost adversarial robustness. |
| Outcome: | Experiments show that the proposed method outperforms hand-engineered prompts with average gains of +4.9% and +5.8%. |
Fine-grained Fallacy Detection with Human Label Variation (2025.naacl-long)
Copied to clipboard
| Challenge: | Fallacy detection is an open challenge in NLP and has shown to be intrinsically difficult for both humans and machines. |
| Approach: | They propose a framework that minimizes annotation errors whilst keeping signals of human label variation. |
| Outcome: | The proposed framework minimizes annotation errors while keeping signals of human label variation. |
Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. |
| Approach: | They propose an evaluation setting to detect semantic leakage by humans and automatically . they also curate a diverse test suite for diagnosing this behavior in 13 flagship models . |
| Outcome: | The proposed evaluation setting detects semantic leakage by humans and automatically, and measures it in 13 flagship models. |
SELFGOAL: Your Language Agents Already Know How to Achieve High-level Goals (2025.naacl-long)
Copied to clipboard
Ruihan Yang, Jiangjie Chen, Yikai Zhang, Siyu Yuan, Aili Chen, Kyle Richardson, Yanghua Xiao, Deqing Yang
| Challenge: | Existing approaches to improve the performance of language agents without training are not available. |
| Approach: | They propose an automatic approach to break down high-level goals into tree structure of more practical subgoals during interaction with environments while identifying the most useful subgoal. |
| Outcome: | The proposed approach significantly improves the performance of language agents across various tasks, including competitive, cooperative, and deferred feedback environments. |
Familarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Current research relies on large synthetic datasets to train zero-shot named entity recognition models. |
| Approach: | They propose a metric that captures the semantic similarity between entity types in training and evaluation to estimate label shift. |
| Outcome: | The proposed metric captures semantic similarity between entity types in training and evaluation, and their frequency in training data to provide an estimate of label shift. |
Learning to Summarize from LLM-generated Feedback (2025.naacl-long)
Copied to clipboard
| Challenge: | Developing effective text summarizers remains a challenge due to issues like unfaithful statements, key information omissions, and verbosity. |
| Approach: | They propose a large-scale dataset containing multi-dimensional feedback on LLM-generated summaries of varying quality across diverse domains to align them with human preferences for faithfulness, completeness, and conciseness. |
| Outcome: | The proposed model outperforms the 10x larger Llama3-70b-instruct in generating human-preferred summaries. |
Hybrid Graphs for Table-and-Text based Question Answering using LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for QA rely on fine-tuning and high-quality data, which is difficult to obtain. |
| Approach: | They propose a Hybrid Graph-based approach for Table-Text QA that leverages Large Language Models without fine-tuning. |
| Outcome: | The proposed approach improves Exact Match scores by 10% on Hybrid-QA and 5.4% on OTT-QA. |
CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models (2025.naacl-long)
Copied to clipboard
Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, Yunhe Wang, Dacheng Tao
| Challenge: | Large language models (LLMs) have achieved remarkable performance on various NLP tasks, yet their potential in more challenging task like finance, has not been fully explored. |
| Approach: | They propose a benchmark to assess the financial knowledge of large language models (LLMs) in China. |
| Outcome: | The proposed benchmark is the most comprehensive evaluation benchmark to date for LLMs in finance. |
LLM-Based Explicit Models of Opponents for Multi-Agent Games (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to model adversarial and cooperative interactions often focus on treating other agents as separate entities with their own intentions and strategies. |
| Approach: | They propose a model of opponents based on Large Language Models (LLMs) that constructs an individual model for each opponent and aligns these models working in synergy through a bi-level feedback-refinement framework. |
| Outcome: | The proposed model outperforms single-model approaches in multi-player deduction games, showing that it significantly enhances agents’ decision-making. |
SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on the potential misuse of large language models (LLMs) however, the ability to align LLMs with human values is still vulnerable to malicious attacks. |
| Approach: | They propose a red-teaming strategy to enhance LLM safety by using a framework to design jailbreak prompts automatically. |
| Outcome: | The proposed framework achieves attack success rates of 88% and 60% in cold-start scenarios. |
JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation (2025.naacl-long)
Copied to clipboard
Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, Kiyoharu Aizawa
| Challenge: | Using culture-agnostic subsets, performance drops in many LMMs when evaluated in Japanese. |
| Approach: | They introduce a Japanese benchmark to evaluate large multimodal models on expert-level tasks based on the Japanese cultural context. |
| Outcome: | The proposed benchmark enables comparisons with other benchmarks in other languages based on cultural contexts. |
EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction (2025.naacl-long)
Copied to clipboard
| Challenge: | EASYTOOL combines tools from diverse tool documentation into a single tool instruction. |
| Approach: | They propose a framework that transforms tool documentation into a unified tool instruction. |
| Outcome: | EASYTOOL combines extensive tool documentation into a concise tool instruction . it reduces token consumption and improves performance of LLM-based agents . |
Decoding Hate: Exploring Language Models’ Reactions to Hate Speech (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are trained on vast amounts of unmoderated internet data, enabling them to generate text autonomously. |
| Approach: | They investigate the responses of seven state-of-the-art Large Language Models (LLMs) to hate speech by qualitative analysis. |
| Outcome: | The proposed models can handle hate speech inputs and mitigate it through fine-tuning and guideline guardrailing. |
Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have adopted a non-interactive training paradigm, and refined pre-trained models through feedback afterward. |
| Approach: | They propose a trial-and-demonstration learning framework that incorporates student trials, teacher demonstrations, and a reward conditioned on language competence at various developmental stages. |
| Outcome: | The proposed framework accelerates word acquisition for student models of equal and smaller numbers of parameters and a strong correlation between the frequency of words in trials and learning curves. |
MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Byte-based machine translation systems can be used in multilingual settings. |
| Approach: | They propose a method that maps each character to specific byte(s) they propose byte-level tokenization that eliminates unknown words . |
| Outcome: | The proposed method outperforms existing methods without manual adjustment of hyper-parameters and surpasses subword-based models with fewer parameters in Ted-59 dataset. |
LLM-Human Pipeline for Cultural Grounding of Conversations (2025.naacl-long)
Copied to clipboard
| Challenge: | addressing parents by name is commonplace in the West, but it is rare in most Asian cultures. |
| Approach: | They propose a Cultural Context Schema for conversations that incorporates conversational information and cultural information such as social norms, violations, etc. |
| Outcome: | The proposed model significantly improves the empirical performance of a Chinese conversational norm and violation description using an interactive human-in-loop framework. |
ACCESS : A Benchmark for Abstract Causal Event Discovery and Reasoning (2025.naacl-long)
Copied to clipboard
Vy Vo, Lizhen Qu, Tao Feng, Yuncheng Hua, Xiaoxi Kang, Songhai Fan, Tim Dwyer, Lay-Ki Soon, Gholamreza Haffari
| Challenge: | Existing methods for identifying event causality in NLP are limited in their scale and rely on lexical cues. |
| Approach: | They propose a benchmark for identifying abstract causality from a large-scale dataset. |
| Outcome: | The proposed benchmark can be leveraged for enhancing QA reasoning performance in LLMs. |
Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities in simulating human behaviour and social intelligence, but they risk perpetuating societal biases, especially when demographic information is involved. |
| Approach: | They propose a framework that measures semantic shifts in responses and an LLM-judged Preference Win Rate to assess how demographic prompts affect response quality across power-disparate social scenarios. |
| Outcome: | The proposed framework measures semantic shifts in responses and an LLM-judged Preference Win Rate (WR) to assess how demographic prompts affect response quality across power-disparate social scenarios. |
GloCOM: A Short Text Neural Topic Model via Global Clustering Context (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing neural topic models often overlook uncovering hidden topics from short texts due to data sparsity, poor aggregation quality, and difficulty in inferring topic proportions for individual documents. |
| Approach: | They propose a model which constructs global clustering contexts for short texts using text embeddings from pre-trained language models. |
| Outcome: | The proposed model outperforms state-of-the-art models on short texts in topic quality and document representation. |
Reversed Attention: On The Gradient Descent Of Attention Layers In GPT (2025.naacl-long)
Copied to clipboard
| Challenge: | In this work, we examine the attention maps obtained from the backward pass of attention, which we call "Reversed Attention" (RA). |
| Approach: | They propose to use a method called "attention patching" to alter the forward pass of attention without modifying the model's weights. |
| Outcome: | The proposed method enables the model to alter the forward pass of attention without altering the model’s weights. |
Self-Harmonized Chain of Thought (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for chain-of-thought prompting have limitations . arithmetic, commonsense, and symbolic reasoning tasks are challenging . |
| Approach: | They propose a method that unifies diverse solution paths into a consistent reasoning pattern. |
| Outcome: | The proposed method outperforms existing methods by 2.8% on reasoning tasks. |
AnaScore: Understanding Semantic Parallelism in Proportional Analogies (2025.naacl-long)
Copied to clipboard
| Challenge: | AnaScore metric aims to evaluate the strength of semantic parallelism in sentence analogies. |
| Approach: | They propose an automatic metric to evaluate the strength of semantic parallelism in sentence analogies. |
| Outcome: | The proposed metric shows that formally explainable examples are more beneficial for analogical reasoning, whereas ambiguous analogies with no clear criterion tend to hinder inference. |
Generating Complex Question Decompositions in the Face of Distribution Shifts (2025.naacl-long)
Copied to clipboard
| Challenge: | Question decomposition has been found to improve large language models’ (LLMs) performance on complex question answering (QA) however, performance on the task remains dominated by supervised approaches, suggesting room for making LLMs better decomposers. |
| Approach: | They propose to generate synthetic decomposition data with only five annotated examples by extending recent advances in using LLM-as-judge and for reranking in novel ways. |
| Outcome: | The proposed approach generates synthetic decomposition data with only five examples over two benchmark datasets. |
Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to address ambiguous questions are limited in their efficiency and performance. |
| Approach: | They propose a retrieval augmented generation framework that diversifies and verifies the retrieved passages to encompass diverse interpretations and adapts the most suitable approach tailored to their quality. |
| Outcome: | The proposed approach improves accuracy and robustness by handling low quality retrieval issue in ambiguous questions while enhancing efficiency. |
Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of large language models have been limited to subjective protocols and benchmarks. |
| Approach: | They propose a unified evaluation taxonomy with eight pedagogical dimensions based on key learning sciences principles to assess the pedagical value of LLM-powered AI tutor responses grounded in student mistakes or confusions in the mathematical domain. |
| Outcome: | The proposed taxonomy, benchmark, and human-annotated labels will streamline the evaluation process and help track the progress in AI tutors’ development. |
Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model (2025.naacl-long)
Copied to clipboard
| Challenge: | Language model (LM) stores diverse factual knowledge in their parameters, which is learned during self-supervised training on unlabeled documents. |
| Approach: | They investigate the issue of "perplexity curse" in the continued training of language model (LM) they find that all studied LMs suffer from positional bias in the training document . |
| Outcome: | The proposed model is able to extract information from multiple questions with diverse queries. |
Evaluating Morphological Compositional Generalization in Large Language Models (2025.naacl-long)
Copied to clipboard
Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, Lonneke Van Der Plas
| Challenge: | Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. |
| Approach: | They define morphemes as compositional primitives and design a suite of generative and discriminative tasks to assess morphological productivity and systematicity. |
| Outcome: | The proposed models can identify individual morphological combinations better than chance, but their performance lacks systematicity, leading to significant accuracy gaps compared to humans. |
Balancing Forget Quality and Model Utility: A Reverse KL-Divergence Knowledge Distillation Approach for Better Unlearning in LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for unlearning large language models struggle with forget quality and model utility, leading to over-unlearning or partial unlearning. |
| Approach: | They propose a method that uses reverse KL-divergence based knowledge distillation for unlearning to achieve significant forget quality while maintaining model utility. |
| Outcome: | The proposed method outperforms existing methods in forget quality and model utility with larger unlearning datasets. |
AgentMove: A Large Language Model based Agentic Framework for Zero-shot Next Location Prediction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to apply large language models to zero-shot next location prediction tasks are limited due to their limited computational power. |
| Approach: | They propose a systematic agentic prediction framework to achieve generalized next location prediction. |
| Outcome: | The proposed framework surpasses the leading baseline by 3.33% to 8.57% across 8 out of 12 metrics. |
Embedding derived animacy rankings offer insights into the sources of grammatical animacy (2025.naacl-long)
Copied to clipboard
| Challenge: | a generative linguistics perspective posits that grammar is shaped by innate cognitive biases. |
| Approach: | They applied the semantic projection approach to animacy, a feature that has not been previously explored using this method. |
| Outcome: | The proposed method is effective in deriving proxies of human perception from word embeddings and provides insights into the sources of grammatical animacy. |
Generating Long-form Story Using Dynamic Hierarchical Outlining with Memory-Enhancement (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for long-form story generation rely on rigid outlines or lack macro-level planning, making it difficult to achieve contextual consistency and coherent plot development. |
| Approach: | They propose a Dynamic Hierarchical Outlining with Memory-Enhancement long-form story generation method to generate long-formed story with coherent content and plot. |
| Outcome: | The proposed method significantly improves the fluency, coherence, and overall quality of generated long stories compared to state-of-the-art methods. |
Little Giants: Synthesizing High-Quality Embedding Data at Scale (2025.naacl-long)
Copied to clipboard
| Challenge: | Synthetic data generation is an increasingly popular way of training models without the need for large, manually labeled datasets. |
| Approach: | They propose a framework that aligns open-source small models to efficiently generate large-scale embedding data. |
| Outcome: | The proposed framework outperforms state-of-the-art embedding models by using only 1/10 of the GPT API calls. |
Can LLMs Convert Graphs to Text-Attributed Graphs? (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to model graph-structured data are limited by the availability of text-attributed graph data. |
| Approach: | They propose a method to convert existing graphs into text-attributed graphs using large language models. |
| Outcome: | The proposed method outperforms existing approaches that manually design node features on text-free graphs. |
Forest for the Trees: Overarching Prompting Evokes High-Level Reasoning in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have greatly propelled the progress of natural language process (NLP). |
| Approach: | They propose a deductive paradigm that decomposes the reasoning process and a prompting method that elicits high-level thinking of large language models (LLMs). |
| Outcome: | The proposed method improves ChatGPT and CoT by 19.0% and 3.1% on MMLU’s College Physics, 8.8% and 2.3% on GSM8k, and 10.3% and 2.5% on StrategyQA, respectively. |
On the Role of Speech Data in Reducing Toxicity Detection Bias (2025.naacl-long)
Copied to clipboard
Samuel Bell, Mariano Coria Meglioli, Megan Richards, Eduardo Sánchez, Christophe Ropers, Skyler Wang, Adina Williams, Levent Sagun, Marta R. Costa-jussà
| Challenge: | Text toxicity detection systems produce disproportionate rates of false positives on demographic groups . toxicity classification systems often misinterpret benign group mentions as toxic . |
| Approach: | They use group annotations to compare text-based and speech-based toxicity detection systems. |
| Outcome: | The results show that access to speech data supports reduced bias against group mentions . the authors recommend improving classifiers, rather than transcription pipelines if possible . |
ITALIC: An Italian Culture-Aware Natural Language Benchmark (2025.naacl-long)
Copied to clipboard
| Challenge: | ITALIC is a large-scale benchmark dataset of 10,000 multiple-choice questions designed to evaluate the natural language understanding of the Italian language and culture. |
| Approach: | They propose to use a large-scale benchmark dataset to evaluate the natural language understanding of the Italian language and culture. |
| Outcome: | The ITALIC dataset spans 12 domains and uses 17 state-of-the-art LLMs to assess the natural language understanding of the italian language and culture. |
RAP: A Metric for Balancing Repetition and Performance in Open-Source Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models generate repetitive content, leading to incomplete or fragmented responses, which can negatively affect user experience. |
| Approach: | They propose a new evaluation metric that quantifies and integrates repetition penalty into the assessment of model performance, enabling tuning of RPP. |
| Outcome: | The proposed evaluation metric reduces repetition while minimizing performance loss. |
Improving Data Annotation for Low-Resource Relation Extraction with Logical Rule-Augmented Collaborative Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Low-resource relation extraction aims to identify semantic relationships using scarce labeled data. |
| Approach: | They propose a framework that iteratively integrates high-confidence predictions of rule-enhanced relation extractors with varying scales to obtain reliable pseudo annotations from massive unlabeled samples without human supervision. |
| Outcome: | The proposed framework achieves state-of-the-art on benchmark datasets in few-shot scenarios. |
CompAct: Compressed Activations for Memory-Efficient LLM Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on reducing peak memory utilization on GPUs, but most work only target the computation graph during training. |
| Approach: | They propose a technique that reduces peak memory utilization on GPUs by 25-30% for pretraining and 50% for fine-tuning of LLMs. |
| Outcome: | The proposed technique reduces peak memory utilization on GPUs by 25-30% for pretraining and 50% for fine-tuning of LLMs. |
Large Language Models Are Cross-Lingual Knowledge-Free Reasoners (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated impressive reasoning capabilities across multiple languages, but the relationship between capabilities in different languages is less explored. |
| Approach: | They decompose the process of reasoning tasks into two separate components: knowledge retrieval and knowledge-free reasoning. |
| Outcome: | The proposed model can be transferred across source-target languages despite secondary impact of resource in some specific target languages, while cross-lingual knowledge retrieval significantly hinders the transfer. |
What Did I Do Wrong? Quantifying LLMs’ Sensitivity and Consistency to Prompt Engineering (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have significantly improved productivity in a number of routine tasks. |
| Approach: | They propose two metrics for classification tasks, namely *sensitivity* and *consistency*, which are complementary to task performance. |
| Outcome: | The proposed metrics are complementary to task performance and can be used to guide prompt engineering and obtain LLMs that balance robustness and performance. |
Detect, Disambiguate, and Translate: On-Demand Visual Reasoning for Multimodal Machine Translation with Large Vision-Language Models (2025.naacl-long)
Copied to clipboard
Danyang Liu, Fanjie Kong, Xiaohang Sun, Dhruva Patil, Avijit Vajpayee, Zhu Liu, Vimal Bhat, Najmeh Sadoughi
| Challenge: | Multimodal machine translation (MMT) aims to leverage additional modalities beyond text . current MMT systems rely heavily on monolingual English captioning data . |
| Approach: | They propose a reasoning-based framework to leverage large-scale vision-language models for MMT . they propose Detect, Disambiguate, and Translate framework to detect ambiguity in input sentence . |
| Outcome: | The proposed framework outperforms state-of-the-art models in disambiguation accuracy and translation quality. |
Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding (2025.naacl-long)
Copied to clipboard
| Challenge: | Multi-modal large language models (MLLMs) generate plausible but incorrect content, resulting in hallucinations . recent advances in MLLM technology have demonstrated their outstanding performance in a variety of visual tasks, such as object detection. |
| Approach: | They propose a plug-and-play method which leverages MLLMs’ internal representations to mitigate hallucinations by analyzing input and output tokens. |
| Outcome: | The proposed method exploits MLLMs’ internal representations to mitigate hallucinations. |
A Multi-modal Large Language Model with Graph-of-Thought for Effective Recommendation (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated a remarkable capability in language understanding and text generation in various realworld scenarios. |
| Approach: | They propose a Graph-of-Thought prompting technique in a Multi-modal LLM to leverage the complex structure of interaction graphs. |
| Outcome: | The proposed model outperforms 12 existing state-of-the-art models on 6 benchmark datasets. |
Investigating Human Values in Online Communities (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing value frameworks struggle with sample sizes and rely on selfreported surveys to calculate values. |
| Approach: | They propose a method to computationally analyse values on Reddit using in-domain and out-of-domain human annotations to train a value relevance and a polarity classifier. |
| Outcome: | The proposed method can be used to analyse values on reddit using human annotations and human annotation. |
Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to improve language models' performance do not exploit this phenomenon . |
| Approach: | They propose to use contextual information to select and construct prompts that improve model performance. |
| Outcome: | The proposed methods show that the mutual information between a context and a question is an effective gauge for language model performance. |
MATO: A Model-Agnostic Training Optimization for Aspect Sentiment Triplet Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models with strong in-house performance may struggle to generalize to diverse expressions. |
| Approach: | They propose a model-agnostic t**raining method to improve ASTE model inference . they propose to compute the violation rate (VR) on each element of one triplet . |
| Outcome: | The proposed method can improve aspect sentiment triplet extraction models consistent with expected results facing triplet element diversity. |
Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts (2025.naacl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) models are constrained by their fixed model capacities when the number of tasks grows in instruction tuning. |
| Approach: | They propose to combine all training tasks and apply fixed sampling weights without considering the importance of different tasks as the model training state changes. |
| Outcome: | The proposed method can be used on knowledge & reasoning tasks and open-ended queries with limited training budget. |
EmoDynamiX: Emotional Support Dialogue Strategy Prediction by Modelling MiXed Emotions and Discourse Dynamics (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that implicit strategy planning lacks transparency and that LLMs’ inherent preference bias towards certain socio-emotional strategies hinders the delivery of high-quality emotional support. |
| Approach: | They propose to decouple strategy prediction from language generation and introduce a new dialogue strategy prediction framework, EmoDynamiX, which models the discourse dynamics between user fine-grained emotions and system strategies using a heterogeneous graph for better performance and transparency. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two ESC datasets with a significant margin (better proficiency and lower preference bias). |
ReasVQA: Advancing VideoQA with Imperfect Reasoning Process (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to VideoQA often fail when complex reasoning or temporal relationships are involved. |
| Approach: | They propose a method that leverages reasoning processes generated by Multimodal Large Language Models to improve VideoQA models. |
| Outcome: | The proposed method improves VideoQA models on three benchmarks. |
Divergent Thoughts toward One Goal: LLM-based Multi-Agent Collaboration System for Electronic Design Automation (2025.naacl-long)
Copied to clipboard
| Challenge: | Electronic design automation (EDA) is indispensable for the design of integrated circuits. |
| Approach: | They propose a multi-agent collaboration system where multiple agents harbor divergent thoughts converge towards a common goal. |
| Outcome: | The proposed system shows superior performance compared to single-agent systems. |
A Survey of QUD Models for Discourse Processing (2025.naacl-long)
Copied to clipboard
| Challenge: | Question Under Discussion (QUD) is a linguistic analytic framework for explaining pragmatic phenomena and information structural analysis. |
| Approach: | They propose to use Question Under Discussion (QUD) to model discourse units, such as sentences, as answers to some implicit or explicit questions. |
| Outcome: | The proposed model is compared with RST, PDTB and SDRT . questions that may require further study are suggested. |
SafetyQuizzer: Timely and Dynamic Evaluation on the Safety of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used to evaluate the safety of their users . however, evaluation questions in current benchmarks are too straightforward and difficult to update with practical relevance due to their lack of correlation with real-world events. |
| Approach: | They propose a question-generation framework to evaluate the safety of LLMs in the Chinese context. |
| Outcome: | The proposed framework reduces decline rate while maintaining similar attack success rate. |
Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing privacy studies focus on sub-fields, but they focus on a few sub-domains. |
| Approach: | They propose to use the Health Insurance Portability and Accountability Act of 1996 as an example to develop a checklist that covers social identities, private attributes, and existing privacy regulations. |
| Outcome: | The proposed checklist covers social identities, private attributes, and existing privacy regulations. |
Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing frameworks for evaluating the decomposition and composition capabilities of large language models (LLMs) in N2F are inadequate, and there are errors that can be attributed to deficiencies in natural language understanding and the learning and use of symbolic systems. |
| Approach: | They propose a framework that semi-automatically performs sample and task construction . main findings include that LLMs are deficient in both decomposition and composition . |
| Outcome: | The proposed framework evaluates the most advanced LLMs on a variety of common formal languages. |
Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring (2025.naacl-long)
Copied to clipboard
Honglin Mu, Han He, Yuxin Zhou, Yunlong Feng, Yang Xu, Libo Qin, Xiaoming Shi, Zeming Liu, Xudong Han, Qi Shi, Qingfu Zhu, Wanxiang Che
| Challenge: | Existing black-box jailbreak methods often rely on model feedback . existing methods may be intercepted by content moderators during the search process . |
| Approach: | They propose a method that guides malicious prompt construction by local training a mirror model of the target black-box model through benign data distillation. |
| Outcome: | The proposed method achieves a 92% attack success rate and 80% stealth rate on a subset of AdvBench. |
VividMed: Vision Language Model with Versatile Visual Grounding for Medicine (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) have demonstrated promise in generating visually grounded responses, but their application in the medical domain is hindered by unique challenges. |
| Approach: | They propose a vision language model with versatile visual grounding for medicine that generates semantic segmentation masks and instance-level bounding boxes. |
| Outcome: | The proposed model can generate semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. |
Mixture of Multimodal Adapters for Sentiment Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have been used for text sentiment analysis but sentiment is hidden in other modalities. |
| Approach: | They propose to fuse emotions from different data to analyze sentiments . they use compression parameter for each expert to reduce training burden . |
| Outcome: | The proposed method achieves state-of-the-art with a tiny trainable parameter count compared to current methods . emotions hidden in body movements or vocal timbres eclipse traditional methods compared with text sentiment analysis . |
The Impact of Inference Acceleration on Bias of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work suggests strategies to increase inference efficiency with LLMs . however, these strategies may inadvertently lead to some side-effects. |
| Approach: | They propose to optimize inference acceleration strategies such as quantization, pruning, and caching to reduce inference cost and latency while maintaining predictive performance. |
| Outcome: | The proposed strategies reduce cost and latency while maintaining predictive performance while preserving the model size. |
AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages (2025.naacl-long)
Copied to clipboard
Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, David Ifeoluwa Adelani, Ibrahim Said Ahmad, Saminu Mohammad Aliyu, Paul Röttger, Abigail Oppong, Andiswa Bukula, Chiamaka Ijeoma Chukwuneke, Ebrahim Chekol Jibril, Elyas Abdi Ismail, Esubalew Alemneh, Hagos Tesfahun Gebremichael, Lukman Jibril Aliyu, Meriem Beloucif, Oumaima Hourrane, Rooweither Mabuya, Salomey Osei, Samuel Rutunda, Tadesse Destaw Belay, Tadesse Kebede Guge, Tesfa Tegegne Asfaw, Lilian Diana Awuor Wanzare, Nelson Odhiambo Onyango, Seid Muhie Yimam, Nedjma Ousidhoum
| Challenge: | Hate speech and abusive language are global phenomena that need sociocultural background knowledge to be understood, identified, and moderated. |
| Approach: | They propose to use a multilingual dataset to collect hate speech and abusive language in 15 African languages to help improve model performance. |
| Outcome: | The proposed datasets are based on tweets annotated by native speakers familiar with the regional culture and show that they perform well in low-resource settings. |
Revealing the Barriers of Language Agents in Planning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show language agents lack human-level planning abilities . limitations and mechanisms to address them remain insufficiently understood . |
| Approach: | They apply a feature attribution study to identify key factors hindering agent planning . they identify the limited role of constraints and diminishing influence of questions . |
| Outcome: | The proposed model achieves 15.6% on a real-world planning benchmark. |
You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQL (2025.naacl-long)
Copied to clipboard
Hideo Kobayashi, Wuwei Lan, Peng Shi, Shuaichen Chang, Jiang Guo, Henghui Zhu, Zhiguo Wang, Patrick Ng
| Challenge: | Existing text-to-SQL systems encode the same schema for every question, resulting in unnecessary high inference cost and missing crucial database knowledge. |
| Approach: | They propose a paradigm that directly internalizes database knowledge into the parametric knowledge of a text-to-SQL model during training and eliminates the need for schema encoding during inference. |
| Outcome: | The proposed paradigm significantly reduces the input token length by 66%-98% and outperforms traditional systems on three benchmarks. |
Option Symbol Matters: Investigating and Mitigating Multiple-Choice Option Symbol Bias of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Multiple-Choice Question Answering (MCQA) is a widely used task in the evaluation of large language models (LLMs). |
| Approach: | They propose a tuning-free, causal effect driven debiasing method which intervenes the activations of identified components according to their causal effects. |
| Outcome: | The proposed method alleviates the aforementioned bias and improves the performance of LLMs. |
DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to conduct in-context learning without using human-annotated demonstrations are unreliable and lead to error accumulation. |
| Approach: | They propose a method to conduct in-context learning without using human-annotated demonstrations. |
| Outcome: | The proposed method outperforms existing methods using human-annotated demonstrations. |
LLaSA: Large Language and Structured Data Assistant (2025.naacl-long)
Copied to clipboard
Yao Xu, Shizhu He, Jiabei Chen, ZengXiangrong ZengXiangrong, Bingning Wang, Guang Liu, Jun Zhao, Kang Liu
| Challenge: | Structured knowledge grounding (SKG) tasks are a key part of many NLP applications. |
| Approach: | They propose a framework for enhancing LLMs' ability to handle structured data . they represent various types of structured data in a unified hypergraph format . |
| Outcome: | The proposed framework outperforms existing methods on SKG tasks using LoRA finetuning. |
Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss (2025.naacl-long)
Copied to clipboard
| Challenge: | APA and MDD are two of the main tasks of computer-assisted pronunciation training (CAPT) systems. |
| Approach: | They propose a computer-assisted pronunciation training approach that integrates APA and MDD tasks in parallel. |
| Outcome: | The proposed approach improves on APA and MDD tasks, and achieves an F1 score of 63.85%. |
Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models (2025.naacl-long)
Copied to clipboard
Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Maria Antoniak, Bill Yuchen Lin, Niloofar Mireshghallah, Chandra Bhagavatula, Yejin Choi
| Challenge: | Lack of transparency in training data is limiting external oversight and inspection of LLMs for issues such as copyright infringement and data contamination. |
| Approach: | They propose a method to identify training data known to proprietary LLMs without requiring access to model weights or token probabilities by using information-guided probes. |
| Outcome: | The proposed method can identify training data known to proprietary LLMs without access to model weights or token probabilities. |
An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on second language (SL) assessment of conversational fluency and interactivity have focused on written correction or pronunciation from ASR. |
| Approach: | They propose a framework that assesses the relationships between micro-level linguistic features and macro-level interactivity labels for Chinese-as-a-second-language dialogues. |
| Outcome: | The proposed framework is interpretable and can be adapted to other languages for second-language dialogue evaluation. |
From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection (2025.naacl-long)
Copied to clipboard
| Challenge: | Toolcalling has changed Large Language Model (LLM) applications by integrating external tools, but it also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. |
| Approach: | They propose a framework that exploits vulnerabilities in Large Language Models through adversarial tool injection to execute privacy theft, launch denial-of-service attacks, and manipulate business competition. |
| Outcome: | The proposed framework exploits vulnerabilities in LLM tool-calling systems through adversarial tool injection. |
COVE: COntext and VEracity prediction for out-of-context images (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing automated fact-checking methods fail to tackle both objectives explicitly. |
| Approach: | They propose a method that predicts first the true COntext of the image and then uses it to predict the VEracity of the caption. |
| Outcome: | The proposed method beats the SOTA context prediction model on all context items, often by more than five percentage points, and is reusable and interpretable to verify new out-of-context captions for the same image. |
Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing summarization systems can generate fluent summaries, but their ability to produce factually consistent summary remains questionable. |
| Approach: | They propose a framework that decomposes long texts into discourse-inspired chunks and utilizes discourse information to better aggregate sentence-level scores predicted by NLI models. |
| Outcome: | The proposed framework shows better performance over multiple benchmarks, focusing on long document summarization. |
Language Models are Crossword Solvers (2025.naacl-long)
Copied to clipboard
| Challenge: | Modern crossword models demonstrate astounding skills in reasoning, coding, wordplay, question answering, and a multitude of other tasks. |
| Approach: | They propose a search algorithm that generalizes well and can support answers with sound rationale by solving full crossword grids with out-of-the-box LLMs. |
| Outcome: | The proposed model outperforms state-of-the-art models in solving crossword grids for the first time and generalizes well. |
WHoW: A Cross-domain Approach for Analysing Conversation Moderation (2025.naacl-long)
Copied to clipboard
| Challenge: | Using this framework, we annotated 5,657 sentences with human judges and 15,494 sentences with GPT-4o from two domains: TV debates and radio panel discussions. |
| Approach: | They propose an evaluation framework for analyzing the facilitation strategies of moderators across different domains/scenarios by examining their motives (Why), dialogue acts (How) and target speaker (Who). |
| Outcome: | The framework is generalisable across domains and reveals distinct modes of moderation: debate moderators emphasise coordination and facilitate interaction through questions and instructions, panel discussion moderator prioritize information provision and actively participate in discussions. |
Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Unequal representation of cultures and socioeconomic groups in training data leads to biased Large Multi-modal (LMM) models. |
| Approach: | They propose and evaluate several prompting strategies that use non-English, geographic, and socioeconomic attributes to improve LMM model performance on underrepresented data. |
| Outcome: | The proposed prompts favor retrieving topic appearances from low-income data on lower-income datasets. |
MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translation (2025.naacl-long)
Copied to clipboard
Satya Krishna Gorti, Ilan Gofman, Zhaoyan Liu, Jiapeng Wu, Noël Vouitsis, Guangwei Yu, Jesse C. Cresswell, Rasa Hosseinzadeh
| Challenge: | Recent advances in text-to-SQL generation rely on large closed-source models that present challenges in accessibility, privacy, and latency. |
| Approach: | They propose to use open-source text-to-SQL models to critique SQL queries . their method evaluates multiple outputs simultaneously and is competitive with larger models . |
| Outcome: | The proposed method achieves state-of-the-art performance compared to open-source models while remaining competitive with larger models at a much lower cost. |
Mitigating Heterogeneity among Factor Tensors via Lie Group Manifolds for Tensor Decomposition Based Temporal Knowledge Graph Embedding (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have highlighted the effectiveness of tensor decomposition methods in the Temporal Knowledge Graphs Embedding task. |
| Approach: | They propose a method that maps factor tensors onto a unified smooth Lie group manifold to approximate homogeneous in tensian decomposition. |
| Outcome: | The proposed method can be directly integrated into existing tensor decomposition based TKGE methods without introducing extra parameters. |
What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work on LM and acceptability judgments treat these effects uniformly across models, making a strong assumption that models require the same degree of adjustment to control for length and unigram frequency effects. |
| Approach: | They propose a linking theory where the optimal level of adjustment is estimated from data via learned parameters for length and unigram frequency. |
| Outcome: | The proposed theory outperforms a commonly used linking theory for acceptability—SLOR—across two families of transformer LMs. |
WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching (2025.naacl-long)
Copied to clipboard
| Challenge: | Flow matching is a robust and stable approach to training diffusion models, but it can result in subpar audio quality. |
| Approach: | They propose a reparameterized flow matching model for mel-spectrogram conditioned speech synthesis that uses a mel prior instead of a standard Gaussian prior to minimize unnecessary transportation costs. |
| Outcome: | The proposed model improves sample quality and generation speed for speech vocoders while reducing transportation costs. |
Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have not investigated the differences between different correlation measures in meta-evaluation. |
| Approach: | They analyze 12 common correlation measures using real-world data from six widely-used NLG evaluation datasets and 32 evaluation metrics. |
| Outcome: | The proposed measures exhibit the best performance in discriminative power and ranking consistency . the measures using system-level grouping or Kendall correlation are the least sensitive to score granularity . |
Cascading Large Language Models for Salient Event Graph Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on event graph generation rely on distant supervision for event graphs . |
| Approach: | They propose a CAscading Large Language Model framework for SAlient Event graph generation which leverages the capabilities of LLMs and eliminates the need for costly human annotations. |
| Outcome: | The proposed method outperforms baseline models on a human-annotated test set. |
Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models (2025.naacl-long)
Copied to clipboard
Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
| Challenge: | Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). |
| Approach: | They propose to use a well-established method for text generation to extract token embeddings from multiple layers of LLMs and compute MD scores for each token. |
| Outcome: | The proposed method improves on existing methods and provides accurate and computationally efficient uncertainty scores for sequence-level selective generation and claim-level fact-checking tasks. |
How Can We Diagnose and Treat Bias in Large Language Models for Clinical Decision-Making? (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs exhibit social biases inherited from training data. |
| Approach: | They propose a framework for evaluation and mitigation of bias in Large Language Models applied to complex clinical cases using a dataset based on the JAMA Clinical Challenge. |
| Outcome: | The proposed framework employs multiple choice questions and explanations to evaluate gender and ethnicity biases in LLMs. |
From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks (2025.naacl-long)
Copied to clipboard
Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, Jieping Ye
| Challenge: | Large visionlanguage models (LVLMs) are a powerful visual-language reasoning tool. |
| Approach: | They propose to integrate attention analysis with LLaVA-CAM to determine interactions between visual representations. |
| Outcome: | The proposed approach can be used to determine interactions between visual representations. |
Patent-CR: A Dataset for Patent Claim Revision (2025.naacl-long)
Copied to clipboard
| Challenge: | Patent-CR is the first dataset created for the patent claim revision task in English. |
| Approach: | They propose to create a dataset for the patent claim revision task in English that includes both initial patent applications rejected by examiners and the final granted versions. |
| Outcome: | The proposed dataset includes both initial patent applications rejected by examiners and the final granted versions. |
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025.naacl-long)
Copied to clipboard
Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu
| Challenge: | State-of-the-art methods for merging expert models with different architectures do not address parameter interference and require extensive fine-tuning to restore performance. |
| Approach: | They propose a method for merging experts with different architectures into a unified Mixture-of-Experts model with a goal of enhancing performance in each domain while retaining effectiveness on general tasks. |
| Outcome: | Experiments across multiple domains show that the proposed methods reduce fine-tuning costs and improve performance over state-of-the-art methods. |
Fine-Tuned LLMs are “Time Capsules” for Tracking Societal Bias Through Books (2025.naacl-long)
Copied to clipboard
| Challenge: | We develop a corpus comprising 593 fictional books across seven decades (1950-2019) to track bias evolution. |
| Approach: | They develop a method to trace and quantify bias evolution using fine-tuned LLMs on fictional books across seven decades to track bias evolution. |
| Outcome: | The proposed method traces and quantifies bias evolution in a corpus of 593 fictional books across seven decades. |
Exploring the Cost-Effectiveness of Perspective Taking in Crowdsourcing Subjective Assessment: A Case Study of Toxicity Detection (2025.naacl-long)
Copied to clipboard
| Challenge: | toxicity evaluation tasks require annotations to accurately reflect opinions of subgroups . toxicity tasks require annotators to take the opinions of a subgroup simultaneously . |
| Approach: | They propose to use perspective taking to obtain opinions from subgroups . they propose to prompt annotators to take perspectives of contrasting subgroup simultaneously . |
| Outcome: | The proposed approach can be cost-effective and improve quality under limited budget. |
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely used and engage millions of users from diverse contexts and cultures. |
| Approach: | They propose an evaluation framework to assess LLMs’ cultural adaptability by measuring their ability to judge social acceptability across varying levels of cultural norm specificity. |
| Outcome: | The proposed model shows stronger adaptability to English-centric cultures over those from the Global South. |
LiPO: Listwise Preference Optimization through Learning-to-Rank (2025.naacl-long)
Copied to clipboard
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J Liu, Xuanhui Wang
| Challenge: | Recent work on language models with curated feedback provides promising alternatives to RLHF . multiple responses can be ranked by reward models or AI feedback, but there is no study on directly fitting upon a list of responses. |
| Approach: | They propose a method that aligns language models with curated human feedback . they propose SLiC and DPO as promising alternatives to traditional RLHF . |
| Outcome: | The proposed method outperforms DPO and SLiC on several preference alignment tasks with curated and real rankwise preference data. |
Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection (2025.naacl-long)
Copied to clipboard
Maximilian Spliethöver, Tim Knebler, Fabian Fumagalli, Maximilian Muschalik, Barbara Hammer, Eyke Hüllermeier, Henning Wachsmuth
| Challenge: | Existing prompting techniques for large language models depend on several parameters, such as the task, language model, and context provided. |
| Approach: | They propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input. |
| Outcome: | The proposed approach ensures high detection performance and is best in several settings. |
Enhancing Discriminative Representation in Similar Relation Clusters for Few-Shot Continual Relation Extraction (2025.naacl-long)
Copied to clipboard
Anh Duc Le, Nam Le Hai, Thanh Xuan Nguyen, Linh Ngo Van, Nguyen Thi Ngoc Diep, Sang Dinh, Thien Huu Nguyen
| Challenge: | Existing methods for relation extraction (RE) fail to address the problem of similar relations, which contributes to catastrophic forgetting. |
| Approach: | They propose a relation extraction method that utilizes relation descriptions and dynamic clustering to identify similar relations. |
| Outcome: | The proposed method mitigates catastrophic forgetting and outperforms state-of-the-art methods by a large margin. |
SymBa: Symbolic Backward Chaining for Structured Natural Language Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Among different methods for structured reasoning, we focus on backward chaining, where the goal is recursively decomposed into subgoals by searching and applying rules. |
| Approach: | They propose a backward chaining system that integrates a symbolic solver and an LLM to improve the performance of LLM-based reasoning. |
| Outcome: | The proposed system improves deductive, relational, and arithmetic reasoning benchmarks compared to baselines. |
MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference (2025.naacl-long)
Copied to clipboard
| Challenge: | Long-context Multimodal Large Language Models (MLLMs) require substantial computational resources as their multimodal Key-Value (KV) cache grows with increasing input lengths, challenging memory and time efficiency. |
| Approach: | They propose a dynamic multimodal KV cache allocation strategy that dynamically allocating KV size based on attention entropy to better adapt to multimodal interactions. |
| Outcome: | The proposed model achieves up to 72% KV cache memory reduction and 2.82 faster decoding speeds while maintaining or enhancing performance on various multimodal tasks in a long context. |
Language Models Largely Exhibit Human-like Constituent Ordering Preferences (2025.naacl-long)
Copied to clipboard
| Challenge: | English sentences are typically inflexible vis-à-vis word order, but constituents show far more variability in ordering. |
| Approach: | They compare LLMs with four types of constituent movement to evaluate their performance on heavy NP shift, particle movement, dative alternation, and multiple PPs. |
| Outcome: | The proposed model performs well on four types of constituent movement: heavy NP shift, particle movement, dative alternation, and multiple PPs. |
SafeQuant: LLM Safety Analysis via Quantized Gradient Inspection (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to jailbreak Large Language Models (LLMs) use computationally intensive verification or require adversarial fine-tuning, leaving models vulnerable to advanced attacks. |
| Approach: | They propose a framework that leverages quantized gradient patterns to identify harmful prompts efficiently. |
| Outcome: | The proposed framework outperforms existing defenses across multiple benchmarks while maintaining model utility. |
Exploring Large Language Models for Effective Rumor Detection on Social Media (2025.naacl-long)
Copied to clipboard
| Challenge: | Large-scale contexts hinder LLMs’ reasoning abilities while moderate contexts perform better for LLM. |
| Approach: | They propose a semantic-propagation collaboration-base framework that integrates small language models with LLMs for effective rumor detection. |
| Outcome: | The proposed framework bridges the gap between LLMs and LLM in facing long, structured data and offers a novel solution for rumor detection on social media. |
No Simple Answer to Data Complexity: An Examination of Instance-Level Complexity Metrics for Classification Tasks (2025.naacl-long)
Copied to clipboard
| Challenge: | Understanding data complexity at the instance level has become increasingly important in Natural Language Processing (NLP) and machine learning (ML). |
| Approach: | They empirically examine the relationship between instance-level complexity scores and metric selection for classification tasks. |
| Outcome: | The results show that storing training loss provides similar complexity rankings to other methods, but not demographic fairness, even in downstream predictions. |
NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals (2025.naacl-long)
Copied to clipboard
| Challenge: | Decomposing text into atomic propositions allows for finergrained inspection of text. |
| Approach: | They propose to decompose atomic propositions into atomic sub-problems that models must weigh when solving the overall problem. |
| Outcome: | The proposed method measures the inferential consistency of models and the diversity of examples in benchmark datasets. |
HISTOIRESMORALES: A French Dataset for Assessing Moral Alignment (2025.naacl-long)
Copied to clipboard
Thibaud Leteno, Irina Proskurina, Antoine Gourru, Julien Velcin, Charlotte Laclau, Guillaume Metzler, Christophe Gravier
| Challenge: | HistoiresMorales is a dataset based on moralStories in French . it is based upon annotations of moral values within the dataset . |
| Approach: | They propose a dataset in French that aims to align language models with moral values . they use annotations to ensure their alignment with French norms . |
| Outcome: | The proposed dataset guarantees grammatical accuracy and adaptation to the French cultural context. |
Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent phoneme classifiers treat allophonic variation as a single phoneme . atypical pronunciation assessment requires distinguishing between a typical and asymmetric pronunciations . |
| Approach: | They propose a new approach that leverages Gaussian mixture models to model phoneme distributions with multiple subclusters. |
| Outcome: | The proposed approach achieves state-of-the-art across dysarthric and non-native speech datasets. |
SAPIENT: Mastering Multi-turn Conversational Recommendation with Strategic Planning and Monte Carlo Tree Search (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods train RL-based agents with greedy action selection or sampling strategy and suffer from suboptimal conversational planning. |
| Approach: | They propose a Monte Carlo Tree Search-based CRS framework called SAPIENT . it consists of a conversational agent and a communication planner . |
| Outcome: | The proposed framework outperforms the state-of-the-art methods on four benchmark datasets. |
Reliability of Topic Modeling (2025.naacl-long)
Copied to clipboard
| Challenge: | Topic models allow researchers to extract latent factors from text data and use those variables in downstream statistical analyses. |
| Approach: | They propose to use McDonald's as a benchmark to evaluate topic model reliability. |
| Outcome: | The proposed model is based on McDonald's , which provides the best encapsulation of reliability on synthetic and real-world data. |
Style Transfer with Multi-iteration Preference Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Numerous recent techniques for text style transfer characterize their approaches as variants of reinforcement learning and preference optimization. |
| Approach: | They propose to use a pseudo-parallel data generation method and a dynamic weighted reward aggregation method to improve upon established preference optimization techniques. |
| Outcome: | The proposed model outperforms existing models on two commonly used text style transfer datasets and is compared with state-of-the-art models. |
DTELS: Towards Dynamic Granularity of Timeline Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing timeline summarizations lack flexibility to meet diverse granularity needs . a fine-grained timeline showing the technical details is preferred for news topics . |
| Approach: | They propose a new paradigm to construct adaptive timelines based on user instructions or requirements. |
| Outcome: | The proposed timelines are informative and granularly consistent, but they struggle to generate consistent timelines. |
ALERT: An LLM-powered Benchmark for Automatic Evaluation of Recommendation Explanations (2025.naacl-long)
Copied to clipboard
Yichuan Li, Xinyang Zhang, Chenwei Zhang, Mao Li, Tianyi Liu, Pei Chen, Yifan Gao, Kyumin Lee, Kaize Ding, Zhengyang Wang, Zhihan Zhang, Jingbo Shang, Xian Li, Trishul Chilimbi
| Challenge: | Existing benchmarks for recommendation explanation evaluation lack item diversity and user preferences data. |
| Approach: | They propose a model-agnostic recommendation explanation evaluation benchmark based on Amazon e-commerce categories with implicit preferences . they propose two novel automatic evaluators that enable scalable and human-preference aligned evaluation of explanations . |
| Outcome: | The proposed model-agnostic evaluation benchmark outperforms existing methods in a variety of domains. |
DETQUS: Decomposition-Enhanced Transformers for QUery-focused Summarization (2025.naacl-long)
Copied to clipboard
Yasir Khan, Xinlei Wu, Sangpil Youm, Justin Ho, Aryaan Mehboob Shaikh, Jairo Garciga, Rohan Sharma, Bonnie J Dorr
| Challenge: | Query-focused tabular summarization is an emerging task in table-to-text generation . traditional transformer-based approaches face challenges due to token limitations and the complexity of reasoning over large tables. |
| Approach: | They propose a system that leverages tabular decomposition alongside a fine-tuned encoder-decoder model to improve summarization accuracy. |
| Outcome: | a new system outperforms the state-of-the-art REFACTOR model in a Query-focused tabular summarization task . the proposed system achieves a ROUGE-L score of 0.4437, outperforming the previous state- of-the art model . |
IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models (2025.naacl-long)
Copied to clipboard
David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Salomey Osei, Shamsuddeen Hassan Muhammad, Sokhar Samb, Tadesse Kebede Guge, Tombekai Vangoni Sherman, Pontus Stenetorp
| Challenge: | Large language models (LLMs) are limited to a few high-resource languages . many low-resourced languages are evaluated only on basic text classification tasks . |
| Approach: | They propose to use IrokoBench to evaluate 17 low-resource African languages . they use human-translated benchmark datasets to evaluate zero-shot, few-shot and translate-test settings . |
| Outcome: | The proposed model performs well in English and French, but the highest performing model perform poorly in proprietary models. |
The Impact of Domain-Specific Terminology on Machine Translation for Finance in European Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing datasets for evaluating MT systems in this domain are limited. |
| Approach: | They propose to use a multi-parallel corpus from the European Central Bank to analyze the impact of domain-specific terminology on multilingual machine translation for finance. |
| Outcome: | The proposed method compares open-source multilingual MT systems with large language models (LLMs) that possess multilingual capabilities. |
Benchmarking Language Model Creativity: A Case Study on Code Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies on LLM creativity evaluation focus on open-ended generation tasks . however, the degree to which LLMs possess and utilize creativity for problem-solving remains unclear . |
| Approach: | They propose a framework for quantifying LLM creativity that incorporates design ingredients . they introduce DENIAL PROMPTING which pushes LLMs to develop more creative solutions . |
| Outcome: | The proposed framework quantifies creativity in LLMs on Codeforces problems . it also finds that even the most creative model fails to demonstrate human-like creativity . |
Have LLMs Reopened the Pandora’s Box of AI-Generated Fake News? (2025.naacl-long)
Copied to clipboard
Xinyu Wang, Wenbo Zhang, Sai Koneru, Hangzhi Guo, Bonam Mingole, S. Shyam Sundar, Sarah Rajtmajer, Amulya Yadav
| Challenge: | Large language models (LLMs) are increasingly being used by fake news creators to generate deceptive and persuasive content at scale. |
| Approach: | They propose to use large language models to generate fake news at scale and to assess the ability of human annotators and AI models to detect it. |
| Outcome: | The results show that LLMs are 68% more effective at detecting real news than humans, compared to humans and AI models for fake news detection. |
Probe-Free Low-Rank Activation Intervention (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing activation intervention methods involve an activation probe to detect undesirable generation, triggering the activation modification to steer subsequent generation. |
| Approach: | They propose a probe-free intervention method FLORAIN for all attention heads in a specific activation layer. |
| Outcome: | The proposed method outperforms baseline methods in enhancing model truthfulness and quality across generation and multiple-choice tasks. |
FactTrack: Time-Aware World State Tracking in Story Outlines (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing language models still struggle to reason over long context windows . et al., 2022, show that long context generation is a challenge for LLMs . |
| Approach: | They propose a method for tracking atomic facts and addressing factual contradictions . they use a four-step pipeline to update a world state data structure for each new event . |
| Outcome: | The proposed method outperforms a baseline and fair method on story outlines. |
A Bayesian Optimization Approach to Machine Translation Reranking (2025.naacl-long)
Copied to clipboard
| Challenge: | reranking is a method of improving prediction quality but can add computational cost. |
| Approach: | They propose to score a list of prediction candidates with an external scoring model and return the highest-scoring candidate. |
| Outcome: | The proposed method achieves the same CometKiwi score using 70 evaluations on average compared to scoring a subset of 180 candidates. |
Multi-Conditional Ranking with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to rank documents using large language models are limited by the complexity of the items and conditions. |
| Approach: | They propose a novel decomposed reasoning method to evaluate multi-conditional ranking across various item types and conditions to overcome this limitation. |
| Outcome: | The proposed method improves LLMs performance 14.4% over existing methods. |
ReGLA: Refining Gated Linear Attention (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) are known for their computational and storage requirements due to the quadratic computation complexity of softmax attention. |
| Approach: | They propose to reduce the quadratic computation complexity of softmax attention by using feature maps, normalization and the gating mechanism to improve performance. |
| Outcome: | The proposed model outperforms existing gated linear attention models in extensive tasks including training from scratch and post-linearization with continual pre-training. |
Intrinsic Bias is Predicted by Pretraining Data and Correlates with Downstream Performance in Vision-Language Encoders (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work has found that vision-language models trained under the Contrastive Language Image Pre-training framework contain intrinsic social biases, but how these biase relates to downstream performance has been unclear. |
| Approach: | They present the largest comprehensive analysis to-date of how upstream pre-training factors and downstream performance of CLIP models relate to their intrinsic biases. |
| Outcome: | The proposed model performance analysis shows that the choice of pre-training dataset is the most significant upstream predictor of bias, whereas architectural variations have minimal impact. |
Benchmarking Failures in Tool-Augmented Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | FAIL-TaLMs contains 1,749 examples using 906 tools across 21 categories, including single- and multi-tool usage. |
| Approach: | They introduce a benchmark to examine the shortcomings of tool-augmented language models (TaLMs) that assume 'perfect' information access and tool availability. |
| Outcome: | The proposed benchmark systematically evaluates 1,749 examples using 906 tools across 21 categories, including single- and multi-tool usage. |
Entity Decomposition with Filtering: A Zero-Shot Clinical Named Entity Recognition Framework (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that large language models (LLMs) can perform in named entity recognition tasks. |
| Approach: | They propose a framework for clinical named entity recognition that decomposes the entity recognition task into several retrievals of sub-types and then filters them. |
| Outcome: | The proposed framework improves on the clinical named entity recognition task. |
Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective (2025.naacl-long)
Copied to clipboard
Shenglai Zeng, Jiankun Zhang, Bingheng Li, Yuping Lin, Tianqi Zheng, Dante Everaert, Hanqing Lu, Hui Liu, Hui Liu, Yue Xing, Monica Xiao Cheng, Jiliang Tang
| Challenge: | Existing studies have shown that LLMs struggle to identify the boundaries of their own knowledge and tend to prioritize external information over internal knowledge learned during pre-training. |
| Approach: | They conduct a comprehensive analysis of LLM representation behaviors and demonstrate the significance of using representations in knowledge checking. |
| Outcome: | The proposed classifiers improve performance even when dealing with noisy knowledge databases. |
The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Multimodal Models exhibit impressive performance across multimodal tasks . effectiveness in cross-cultural contexts limited due to predominantly Western-centric nature of data and models . multi-agent models have shown significant capability in solving complex tasks despite limitations in crosscultural context . |
| Approach: | They propose to use a multi-agent framework to enhance cross-cultural image captioning using LMMs with distinct cultural personas to evaluate cultural information within image captions. |
| Outcome: | The proposed model outperforms single-agent models across different metrics and offers valuable insights for future research. |
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been successful in NLP tasks, but there is growing interest in extending their capabilities to speech. |
| Approach: | They propose to use dense feature prepending (DFP) to integrate speech into LLMs to enable end-to-end training with a speech encoder. |
| Outcome: | The proposed approach does not show a clear advantage over cross-attention. |
CORRECT: Context- and Reference-Augmented Reasoning and Prompting for Fact-Checking (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing fact-checking models focus on reasoning within evidence sentences, but they ignore auxiliary contexts and references. |
| Approach: | They propose a method to verify the truthfulness of claims using evidence . they construct a three-layer evidence graph with evidence, context, and reference layers . |
| Outcome: | The proposed method can verify the truthfulness of claims using evidence . it integrates evidence, context, and reference layers into a unified embedding . |
Racing Thoughts: Explaining Contextualization Errors in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated a remarkable capacity for accomplishing a wide variety of language generation and classification tasks. |
| Approach: | They propose a race conditions hypothesis to explain contextualization errors . they propose to use a variety of techniques to test the hypothesis . |
| Outcome: | The proposed model fails to properly contextualize a financial institution if it does not include a bank . the proposed model is based on the race conditions hypothesis . |
DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in video-text retrieval models have limited training data annotations. |
| Approach: | They propose a Video-Text Retrieval Paradigm with Relevance-based Augmentation which enhances video and text data using large foundation models to learn more generalized features. |
| Outcome: | The proposed method improves video-text retrieval performance over existing methods. |
ToW: Thoughts of Words Improve Reasoning in Large Language Models (2025.naacl-long)
Copied to clipboard
Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin Rrv, Chitta Baral, Ben Zhou
| Challenge: | Unlike other data augmentation methods, thoughts of words (TOW) views next-word prediction as a core reasoning task and injects fine-grained thoughts into pre-training texts. |
| Approach: | They propose a training-time data-augmentation method called thoughts of words (TOW) that injects fine-grained thoughts directly into a next-word prediction task and teaches the model to understand how the observed next word is related to previous contexts. |
| Outcome: | The proposed method reduces model hallucination by 10% and improves reasoning performance by 7% to 9% on average. |
A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagation (2025.naacl-long)
Copied to clipboard
| Challenge: | Current large language models (LLMs) produce factually incorrect statements . |
| Approach: | They propose a probabilistic framework for LLM hallucination detection that generates a belief tree by expanding a statement into logically related claims and reasoning globally about the relationships between these claims. |
| Outcome: | The proposed method improves on multiple hallucination detection benchmarks by 3%-9% over state-of-the-art models. |
ERAS: Evaluating the Robustness of Chinese NLP Models to Morphological Garden Path Errors (2025.naacl-long)
Copied to clipboard
| Challenge: | In languages without orthographic word boundaries, NLP models perform _word segmentation_ as an explicit preprocessing step or as an implicit step in an end-to-end computation. |
| Approach: | They propose a benchmark that tests a model’s vulnerability to morphological garden path errors by comparing its behavior on local segmentation ambiguities. |
| Outcome: | The proposed benchmark compares a model's behavior on sentences with and without local segmentation ambiguities and shows that models with character-level tokenization make implicit garden path errors even without an explicit word segmentation step. |
Superlatives in Context: Modeling the Implicit Semantics of Superlatives (2025.naacl-long)
Copied to clipboard
| Challenge: | a study of superlatives shows that the semantics of superlations in context can be challenging for contemporary models. |
| Approach: | They propose a unified account of superlative semantics which allows for a broad-coverage annotation schema. |
| Outcome: | The proposed schema allows for interpreting superlative expressions and their semantic interpretations. |
LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities in mathematical reasoning, but their effectiveness is limited to specific mathematical topics. |
| Approach: | They propose to use the MaTT benchmark to assess large language models' accuracy in multiple-choice scenarios. |
| Outcome: | The proposed model achieved 54% accuracy in a multiple-choice scenario, while the Chain-of-Thought prompting did not improve. |
Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work has focused on using large language models to simulate human behaviors . but, LLMs are known to generate erroneous, stereotypical, or overconfident answers . |
| Approach: | They propose to specialize large language models for simulating survey response distributions by first-token probabilities. |
| Outcome: | The proposed model outperforms other methods and zero-shot classifiers on unseen questions, countries, and a completely unseened survey. |
Representing Rule-based Chatbots with Transformers (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing work on how Transformers can solve synthetic tasks has not explored how to extend this to a conversational setting. |
| Approach: | They propose to use ELIZA as a framework for formal mechanistic analysis of Transformers . they propose to model local pattern matching and long-term dialogue state tracking . |
| Outcome: | The proposed model can be extended to model key aspects of conversation, the authors show . their model favors an induction head mechanism over a more precise copying mechanism . |
Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a recent study has found LMs focus on processing syntactic relations, but not structural information. |
| Approach: | They examine the mechanisms underlying garden path sentence processing in LMs . they use sparse autoencoders to identify interpretable features that determine which continuation . |
| Outcome: | The proposed model lacks syntactic features and shallow heuristics to perform incremental sentence processing. |
Entangled Relations: Leveraging NLI and Meta-analysis to Enhance Biomedical Relation Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research has explored the potential of leveraging natural language inference (NLI) techniques to enhance relation extraction (RE). |
| Approach: | They propose a method that verbalizes relation classes into class-indicative hypotheses to align a traditionally multi-class classification task to one of textual entailment. |
| Outcome: | The proposed method improves relation extraction performance on BioRED and ReTACRED. |
Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models (2025.naacl-long)
Copied to clipboard
Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, Hao Wang
| Challenge: | Multimodal Large Language Models have shown significant promise in various applications, but a comprehensive evaluation of their long-context capabilities remains underexplored. |
| Approach: | They propose a benchmark to assess the long-context capabilities of multimodal large language models. |
| Outcome: | The proposed benchmark compared MLLMs with API-based and open-source models in a long-context scenario. |
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines (2025.naacl-long)
Copied to clipboard
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abolade, Emmanuele Chersoni, Enrico Santus, Fariz Ikhwantri, Garry Kuwanto, Hanyang Zhao, Haryo Akbarianto Wibowo, Holy Lovenia, Jan Christian Blaise Cruz, Jan Wira Gotama Putra, Junho Myung, Lucky Susanto, Maria Angelica Riera Machin, Marina Zhukova, Michael Anugraha, Muhammad Farid Adilazuarda, Natasha Christabelle Santosa, Peerat Limkonchotiwat, Raj Dabre, Rio Alexander Audino, Samuel Cahyawijaya, Shi-Xiong Zhang, Stephanie Yulia Salim, Yi Zhou, Yinxuan Gui, David Ifeoluwa Adelani, En-Shiun Annie Lee, Shogo Okada, Ayu Purwarianti, Alham Fikri Aji, Taro Watanabe, Derry Tanti Wijaya, Alice Oh, Chong-Wah Ngo
| Challenge: | Vision Language Models struggle with cultural-specific knowledge, especially in languages other than English and in underrepresented cultural contexts. |
| Approach: | They propose a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects and a training dataset. |
| Outcome: | The proposed model performs better with correct location context, but struggles with adversarial contexts and predicting specific regional cuisines and languages. |
Extracting and Understanding the Superficial Knowledge in Alignment (2025.naacl-long)
Copied to clipboard
Runjin Chen, Gabriel Jacob Perin, Xuxi Chen, Xilun Chen, Yan Han, Nina S. T. Hirata, Junyuan Hong, Bhavya Kailkhura
| Challenge: | Recent studies have shown that alignment of large language models with human values and preferences requires substantial data and computation resources. |
| Approach: | They propose a method to extract and isolate superficial knowledge from aligned models by focusing on the shallow modifications to the final token selection process. |
| Outcome: | The proposed method extracts and isolates superficial knowledge from aligned models, focusing on the shallow modifications to the final token selection process. |
Smurfs: Multi-Agent System using Context-Efficient DFSDT for Tool Planning (2025.naacl-long)
Copied to clipboard
| Challenge: | Teaching large language models to use tools for solving complex problems can grant them human-like reasoning abilities. |
| Approach: | They propose a multi-agent system that enhances the Deep First Search Decision Tree (DFSDT) to address issues like error propagation and limited exploration in ReAct . |
| Outcome: | The proposed system reduces token usage by 60.9% compared to existing methods and performs on par with GPT-4-DFSDT. |
From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | Motivated by in-context learning capabilities of Large Language Models (LLMs), multimodal LLMs with additional visual modality are also exhibited with similar ICL abilities when multiple image-text pairs are provided as demonstrations. |
| Approach: | They conduct systematic and principled evaluation of multimodal ICL for models of different scales on a broad spectrum of new yet critical tasks. |
| Outcome: | The proposed model performance improves on a broad spectrum of new yet critical tasks. |
Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets (2025.naacl-long)
Copied to clipboard
| Challenge: | a lack of data across domains creates significant imbalances in training data sizes . a recent study shows that temperature sampling and scaling are equivalent but differ under stochastic gradient descent due to differences in gradient variance. |
| Approach: | They propose a method that upsamples low-resource languages and upweights their loss functions to address this disparity. |
| Outcome: | The proposed method competes effectively with existing data re-weighting techniques while offering computational efficiency. |
LLM The Genius Paradox: A Linguistic and Math Expert’s Struggle with Simple Word-based Counting Problems (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing conjectures about the reason for deficiency of LLMs in simple word-based counting problems are invalid. |
| Approach: | They propose to evaluate model transferability from specialized LLMs to simple counting tasks by comparing their results to popular conjectures . |
| Outcome: | The proposed model evaluations show that engaging reasoning is the most robust and efficient way to help LLMs better perceive tasks with more accurate responses. |
PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research has studied privacy in LLM training data memorization, but it does not prevent users from disclosing PII at inference time. |
| Approach: | They propose a task for chaining API-based and local LLMs that uses public data to construct a benchmark that contains personally identifiable information (PII) |
| Outcome: | The proposed model maintains high response quality for 85.5% of user queries while restricting privacy leakage to only 7.5%. |
When2Call: When (not) to Call Tools (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on the accuracy of tool calling and less on when LMs should (not) call tools. |
| Approach: | They develop a benchmark which evaluates tool-calling decision-making . they leverage multiple-choice nature of the benchmark to develop 'preference optimization' training regime . |
| Outcome: | The proposed benchmark shows that state-of-the-art LMs show room for improvement on When2Call. |
Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Machine Translation (MT) systems based on fine-tuned large language models (LLMs) are at a higher risk of generating hallucinations, which can severely undermine user’s trust and safety. |
| Approach: | They propose a method that intrinsically learns to mitigate hallucinations during the model training phase. |
| Outcome: | The proposed method reduces hallucinations by 89% on an average across three unseen target languages while preserving translation quality. |
Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Tools (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle to generate correct plans for multi-constraint planning problems . a recent study showed that large language models have significant potential in solving planning problems. |
| Approach: | They propose an LLM-based planning framework that formalizes and solves multi-constraint planning problems as constrained satisfiability problems. |
| Outcome: | The proposed framework achieves a success rate of 93.9% and is effective with diverse paraphrased prompts. |
Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs? (2025.naacl-long)
Copied to clipboard
| Challenge: | Among various types of ambiguity, this study focuses on syntactic ambiguities, specifically relative 1 Dataset available at https://github.com/PortNLP/ MultiWHO. |
| Approach: | They propose to use a dataset to fine-grained evaluate relative clause attachment preferences in ambiguous and unambiguous contexts. |
| Outcome: | The proposed dataset shows that large language models perform well in unambiguous cases, but lack flexibility in human language processing. |
Beyond Benchmarks: Building a Richer Cross-Document Event Coreference Dataset with Decontextualization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing datasets for Cross-Document Event Coreference (CDEC) are small and lacking diversity. |
| Approach: | They propose a new approach leveraging large language models to decontextualize event mentions by simplifying the document-level annotation task to sentence pairs with enriched context. |
| Outcome: | The proposed approach improves the quality of the dataset and generalizability of the model. |
Can Unconfident LLM Annotations Be Used for Confident Conclusions? (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. |
| Approach: | They propose a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations to use. |
| Outcome: | The proposed method produces accurate estimates and valid confidence intervals while reducing the number of human annotations by over 25%. |
Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding (2025.naacl-long)
Copied to clipboard
| Challenge: | Flowcharts are typically presented as images, driving the trend of using vision-language models for end-to-end flowchart understanding. |
| Approach: | They propose a vision-language model (VLM) that generates textual representations from flowchart images and a textual Reasoner that performs question-answering based on the text representations. |
| Outcome: | Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow’s state-of-the-art performance as well as its robustness. |
Ihquin tlahtouah in Tetelahtzincocah: An annotated, multi-purpose audio and text corpus of Western Sierra Puebla Nahuatl (2025.naacl-long)
Copied to clipboard
| Challenge: | a corpus of audio and annotated transcriptions of an endangered Nahuatl is presented . data made available in this corpus are useful for ASR, spelling normalization, and word-level language identification. |
| Approach: | They present a corpus of audio and annotated transcriptions of an endangered Nahuatl in Mexico . the data are useful for ASR, spelling normalization, and word-level language identification . |
| Outcome: | The corpus is made available for use in ASR, spelling normalization, and word-level language identification tasks. |
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions (2025.naacl-long)
Copied to clipboard
| Challenge: | Medical board exams or general clinical questions do not capture the complexity of real clinical cases. |
| Approach: | They construct two datasets that are structured as multiple-choice question-answering tasks accompanied by expert-written explanations. |
| Outcome: | The proposed datasets are harder than previous benchmarks. |
Unfamiliar Finetuning Examples Control How Language Models Hallucinate (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate plausible-sounding responses that are factually incorrect. |
| Approach: | They propose an approach to learn more reliable reward models by modifying how unfamiliar finetuning examples are supervised to influence model responses to unfamiliar queries. |
| Outcome: | The proposed approach improves the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks. |
Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate reasoning paths before answers, but lack a systematic approach to determine optimal number of samples or select the most faithful rationale. |
| Approach: | They propose a framework that evaluates the quality of reasoning and consistency of answers for each generated sample and uses criteria-based stopping and weighted majority voting to guide early stopping decisions and rationale selection. |
| Outcome: | The proposed framework outperforms existing methods while maintaining accuracy. |
MatViX: Multimodal Information Extraction from Visually Rich Articles (2025.naacl-long)
Copied to clipboard
Ghazal Khalighinejad, Sharon Scott, Ollie Liu, Kelly L. Anderson, Rickard Stureborg, Aman Tyagi, Bhuwan Dhingra
| Challenge: | Existing methods for multimodal information extraction are limited due to the multimodal nature of scientific articles and complex interconnections between data points. |
| Approach: | They propose a benchmark to extract structured information from scientific articles . they use curated JSON files extracted from text, tables, and figures . |
| Outcome: | The proposed benchmark is based on 324 full-length research articles and 1,688 complex structured JSON files curated by experts in polymer nanocomposites and biodegradation. |
Towards Rationality in Language and Multimodal Agents: A Survey (2025.naacl-long)
Copied to clipboard
Bowen Jiang, Yangxinyu Xie, Xiaomeng Wang, Yuan Yuan, Zhuoqun Hao, Xinyi Bai, Weijie J Su, Camillo Jose Taylor, Tanwi Mallick
| Challenge: | despite advances in language and multimodal agents, large language models lack rationality . despite their progress, large-scale models lack real-world grounding and feedback mechanisms . |
| Approach: | They propose to build more rational language and multimodal agents . they also examine what criteria define rationality in intelligent systems . |
| Outcome: | This paper assesses the state-of-the-art in language and multimodal agents . it also outlines open challenges and future research directions . |
CluSanT: Differentially Private and Semantically Coherent Text Sanitization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing implementations of Differential Privacy (DP) in NLP typically degrade semantic integrity and readability for humans, posing significant challenges for applications requiring high-quality, coherent text processing. |
| Approach: | They propose a text sanitization framework based on Metric Local Differential Privacy (MLDP) that uses large language models to create a set of potential substitute tokens and a parameterized cluster embedding to samaritize/substitute sensitive tokens. |
| Outcome: | The proposed framework can be tuned with parameters such that existing state-of-the-art token sanitization algorithms can be described and improved. |
TurkingBench: A Challenge Benchmark for Web Agents (2025.naacl-long)
Copied to clipboard
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
| Challenge: | TurkingBench is a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. |
| Approach: | They propose to use HTML pages to perform various annotation tasks on crowdsourcing platforms. |
| Outcome: | The proposed model outperforms other models on the TurkingBench benchmark. |
CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | coding tasks require generated code to be fully executable and functionally correct . current agentic approaches struggle with multi-stage planning, generating, and debugging . |
| Approach: | They propose a framework for LLM agents to efficiently explore the search space in different stages of the code generation process. |
| Outcome: | The proposed framework achieves top results on 7 code generation benchmarks and a 31.9% solving rate on the SWEBench benchmark. |
DPL: Diverse Preference Learning Without A Reference Model (2025.naacl-long)
Copied to clipboard
Abhijnan Nath, Andrey Volozin, Saumajit Saha, Albert Aristotle Nanda, Galina Grunin, Rahul Bhotika, Nikhil Krishnaswamy
| Challenge: | Existing methods to direct preference alignment do not utilize diversity in preference annotations which limits their applicability. |
| Approach: | They propose a reference-model-free method that learns a baseline desirability in LLM responses while being robust to the diversity of preference annotations. |
| Outcome: | The proposed method learns a baseline desirability in LLM responses while being robust to the diversity of preference annotations. |
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent efforts to verify text accuracy provide no guarantees on their correctness . a new method to improve LLMs' verifiability is to use quotes to ground models . |
| Approach: | They propose a method that allows models to quote verbatim statements from trusted sources . they leverage a fast membership inference function to verify text against trusted corpora . |
| Outcome: | The proposed method significantly increases verbatim quotes from high-quality documents by up to 130% relative to base models while maintaining response quality. |
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite the impressive capabilities of large multi-modal models, their effectiveness in handling complex tasks has been limited by the prevailing singlestep reasoning paradigm. |
| Approach: | They propose a visuallygrounded object-centric Chain-of-Thought reasoning framework for LMMs that is based on a multi-modal interleaved and aligned representation of object concepts. |
| Outcome: | The proposed model outperforms SOTA models in CLEVR and EmbSpatial benchmarks. |
ACCORD: Closing the Commonsense Measurability Gap (2025.naacl-long)
Copied to clipboard
| Challenge: | ACCORD is a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs). |
| Approach: | They propose a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) they introduce formal elements to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. |
| Outcome: | The proposed framework can scale with future LLM improvements. |
CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments (2025.naacl-long)
Copied to clipboard
Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, Chien-Sheng Wu
| Challenge: | Existing benchmarks for evaluating CRM agents on work-related tasks are limited due to data privacy concerns. |
| Approach: | They propose a benchmark to evaluate AI agents on real-world CRM tasks . they use 16 commonly used industrial objects with high interconnectivity to simulate real data distributions. |
| Outcome: | The new benchmark evaluates AI agents on real-world customer service tasks . it includes 16 commonly used industrial objects with high interconnectivity . the results highlight the need for enhanced agent capabilities in function-calling and rule-following . |
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large pre-trained models have achieved outstanding results in sequence modeling . alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address these inefficiencies. |
| Approach: | They propose to reduce the size and computational overhead of large pre-trained models by removing selected components at different granularities. |
| Outcome: | The proposed models achieve a speedup of up to 1.4x during inference while maintaining accuracy. |
CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy (2025.naacl-long)
Copied to clipboard
Mian Zhang, Xianjun Yang, Xinlu Zhang, Travis Labrum, Jamie C. Chiu, Shaun M. Eack, Fei Fang, William Yang Wang, Zhiyu Chen
| Challenge: | Existing research has explored mental health condition classifications, empathetic conversations, and chatbots designed for simple discourse structures. |
| Approach: | They propose a benchmark for systematic evaluation of cognitive behavioral therapy assistance using Large Language Models (LLMs). |
| Outcome: | The proposed benchmark includes three levels of tasks covering key aspects of cognitive behavioral therapy that could be enhanced through AI assistance. |
An Efficient Gloss-Free Sign Language Translation Using Spatial Configurations and Motion Dynamics with LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for sign language translation rely on glosses, which are written representations of signs. |
| Approach: | They propose a new LLM-based SLT framework that uses off-the-shelf visual encoders to extract spatial and motion features from sign videos. |
| Outcome: | The proposed framework captures spatial configurations and motion dynamics in sign language without domain-specific tuning. |
Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. |
| Approach: | They propose a benchmark that evaluates state-of-the-art Vision Language Models on converting sketches into webpage prototypes. |
| Outcome: | The benchmark evaluates state-of-the-art Vision Language Models on automating the conversion of rudimentary sketches into webpage prototypes. |
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025.naacl-long)
Copied to clipboard
| Challenge: | Generative AI has made rapid advances in multimodal understanding and code generation. |
| Approach: | They construct a first real-world benchmark for multimodal large language models that directly convert visual designs into code implementations by manually curating 484 diverse real-life webpages as test cases. |
| Outcome: | The proposed model can generate code implementations that directly render into the given reference webpages, given the screenshots as input. |
Temporal-Aware Soft Prompt Tuning for Automatic Text Dating (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for automatic text dating overlook the evolution of word meanings in texts spanning long periods. |
| Approach: | They propose a temporal-aware text representation that dynamically captures both semantic variance and invariance. |
| Outcome: | The proposed approach outperforms existing methods on two diachronic datasets. |
Sparser Mixture-of-Adapters with Cross-Layer Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for training large language models do not allow sharing adapters across layers . existing methods do not support sharing adapter pools, leading to redundancy and poor generalization . |
| Approach: | They propose a mixture-of-adapter framework that trains a pool of lightweight adapters at each layer and selects the most suitable ones for each input. |
| Outcome: | The proposed framework reduces active adapters by over 85% while boosting task accuracy. |
How to Align Multiple Signed Language Corpora for Better Sign-to-Sign Translations? (2025.naacl-long)
Copied to clipboard
| Challenge: | despite the growing need for advanced signing technologies, signed language resources remain scarce. |
| Approach: | They propose a linguistically informed alignment algorithm that matches instances between signed languages . they compare similarities and differences across three signed languages to develop a model . |
| Outcome: | The proposed algorithm performs well on automatic metrics for sign-to-sign translation and generation. |
Communication Makes Perfect: Persuasion Dataset Construction via Multi-LLM Communication (2025.naacl-long)
Copied to clipboard
Weicheng Ma, Hefan Zhang, Ivory Yang, Shiyu Ji, Joice Chen, Farnoosh Hashemi, Shubham Mohole, Ethan Gearey, Michael Macy, Saeed Hassanpour, Soroush Vosoughi
| Challenge: | Large Language Models (LLMs) have shown proficiency in generating persuasive dialogue, yet concerns about the fluency and sophistication of their outputs persist. |
| Approach: | They propose a multi-LLM communication framework that facilitates the efficient production of high-quality, diverse linguistic content with minimal human oversight. |
| Outcome: | The proposed framework excels in naturalness, linguistic diversity, and the strategic use of persuasion, even in complex scenarios involving social taboos. |
Soft Prompting for Unlearning in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing ethical and safety considerations for large language models are important for deployment . however, some ethical concerns have been raised due to the presence of private, sensitive, or harmful information in the training data. |
| Approach: | They propose a framework that learns prompt tokens that are prepended to a query to induce unlearning in LLMs. |
| Outcome: | The proposed method improves the trade-off between utility and forgetting for text classification and question-answering. |
Mutual-pairing Data Augmentation for Fewshot Continual Relation Extraction (2025.naacl-long)
Copied to clipboard
Nguyen Hoang Anh, Quyen Tran, Thanh Xuan Nguyen, Nguyen Thi Ngoc Diep, Linh Ngo Van, Thien Huu Nguyen, Trung Le
| Challenge: | Existing methods for Few-shot Continual Relation Extraction struggle with catastrophic forgetting and overfitting. |
| Approach: | They propose a method that transforms single input sentences into complex texts by integrating old and new data. |
| Outcome: | The proposed method sharpens model focus and improves model performance . it also uncovers fascinating behaviors of Sharpness-Aware Minimization (SAM) in Few-shot Continual Learning. |
KMMLU: Measuring Massive Multitask Language Understanding in Korean (2025.naacl-long)
Copied to clipboard
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, Stella Biderman
| Challenge: | Recent models struggle to show performance over 60%, significantly below the pass mark of the source exams (80%), highlighting the room for improvement. |
| Approach: | They propose to use Korean exams to collect 35,030 questions from an expert-level multiple choice model to capture linguistic and cultural aspects of the Korean language. |
| Outcome: | The proposed benchmark is based on 35,030 questions from original Korean exams. |
Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and Multimodal Large Language models (MLLMs) trained on vast web corpora can memorize and disclose individuals’ confidential and private data, raising legal and ethical concerns. |
| Approach: | They propose a benchmark to assess unlearning algorithms from multiple perspectives and provide a baseline for existing generative models. |
| Outcome: | The proposed benchmark consists of 500 fictitious profiles and 153 profiles for public celebrities, evaluated from both multimodal (image+text) and unimodal (text) perspectives. |
LLM4DistReconfig: A Fine-tuned Large Language Model for Power Distribution Network Reconfiguration (2025.naacl-long)
Copied to clipboard
| Challenge: | Power distribution network reconfiguration is crucial for maintaining operational efficiency, reliability and adaptability in modern power networks. |
| Approach: | They propose a deep learning-based approach to solve a distribution network reconfiguration problem using inputs from a LLM. |
| Outcome: | The proposed model generates optimal configurations minimizing system loss for five individual and a combined test dataset. |
WaterPool: A Language Model Watermark Mitigating Trade-Offs among Imperceptibility, Efficacy and Robustness (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to trace the usage of large language models often face trade-offs between imperceptibility and robustness. |
| Approach: | They propose a key-centered scheme to unify existing methods by decomposing a watermark into two components: a 'key module' and a "mark module". |
| Outcome: | The proposed method can be integrated with existing methods and achieve near-optimal imperceptibility and detection efficacy. |
Tricking Retrievers with Influential Tokens: An Efficient Black-Box Corpus Poisoning Attack (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for crafting adversarial passages are slow and computationally expensive, requiring either access to retriever’s gradients or large computational resources. |
| Approach: | They propose a method that leverages two key characteristics of retrievers: insensitivity to token order and bias towards influential tokens to generate effective adversarial passages. |
| Outcome: | The proposed method achieves superior efficiency and scalability compared to existing methods while maintaining comparable or better attack success rates across multiple datasets. |
The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluations of large language models (LLMs) focus on a single output per example, which limits our understanding of LLM performance variability in real-world applications. |
| Approach: | They explore the performance differences between greedy decoding and sampling and identify benchmarks’ consistency regarding non-determinism and examine unique model behaviors. |
| Outcome: | The proposed model outperforms sampling methods and greedy decoding outperformed other models. |
CVE-Bench: Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and LLM agents have demonstrated significant potential in this domain by understanding descriptions in natural language and generating corresponding formal code. |
| Approach: | They propose an evaluation framework that provides LLM agents with a test environment that simulates the real-world vulnerability repair process. |
| Outcome: | The proposed framework can repair 21% of vulnerabilities at its best, but lacks expert knowledge . the evaluation framework can only repair 29% of vulnerabilities, but it can be used in real-world scenarios . |
PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models fail to follow instructions or meet developer expectations when running in production . a dataset of 2087 LLM pipeline prompts with 12623 assertion criteria is larger than previous collections . |
| Approach: | They propose a dataset of 2087 LLM pipeline prompts with 12623 assertion criteria . they fine-tuned Mistral and Llama 3 models outperform GPT-4o by 20.93% on average . |
| Outcome: | The proposed dataset outperforms GPT-4o and mistral models in generating assertions and offers reduced latency and improved performance. |
ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesis (2025.naacl-long)
Copied to clipboard
Zezhong Wang, Xingshan Zeng, Weiwen Liu, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Large Language Models (LLMs) can be enhanced by using supervised fine-tuning . however, access to fine-timing data can be limited. |
| Approach: | They propose a Graph-based Sampling strategy and a Planned-generation strategy to enhance the coherence between dialogues by using 8,000 synthetic dialogues. |
| Outcome: | The proposed model achieves tool-calling performance comparable to or surpassing GPT-4 while maintaining strong general capabilities. |
Fighting Spurious Correlations in Text Classification via a Causal Learning Perspective (2025.naacl-long)
Copied to clipboard
| Challenge: | In text classification tasks, models often rely on spurious correlations for predictions, incorrectly associating irrelevant features with the target labels. |
| Approach: | They propose a Causally Calibrated Robust Classifier which integrates a causal feature selection method based on counterfactual reasoning and an unbiased inverse propensity weighting (IPW) loss function. |
| Outcome: | The proposed method achieves state-of-the-art performance among methods without group labels and can compete with the models that utilize group labels. |
Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to generate query expansions focus on enhancing textual similarities between search queries and document corpus, overlooking document relations. |
| Approach: | They propose a knowledge-aware query expansion framework augmenting LLMs with structured document relations from knowledge graph (KG) they leverage document texts as rich KG node representations and use document-based relation filtering for their method. |
| Outcome: | The proposed framework augments LLMs with structured document relations from knowledge graph (KG) Extensive experiments on three datasets of diverse domains show the advantages compared against state-of-the-art methods on textual and relational semi-structured retrieval. |
SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for compressing Large Language Models suffer from significant truncation losses. |
| Approach: | They propose a novel method that optimizes singular value truncation in SVD compression . they use dynamic compression ratio allocation to balance the large tuncation loss . |
| Outcome: | The proposed method outperforms current state-of-the-art methods on ten datasets and five models on various scales. |
AudioBench: A Universal Benchmark for Audio Large Language Models (2025.naacl-long)
Copied to clipboard
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, Nancy F. Chen
| Challenge: | Existing evaluation regimes for audio large language models do not cover the breadth of their possible use cases. |
| Approach: | They propose to use AudioBench to evaluate audio large language models . they found that no single model excels consistently across all tasks . |
| Outcome: | The proposed evaluation targets speech understanding, audio scene understanding, and voice understanding (paralinguistic) . no single model excels consistently across all tasks, the paper found . |
Efficient Prompting for Continual Adaptation to Missing Modalities (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods combine various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and catastrophic forgetting. |
| Approach: | They propose a continual multimodal missing modality task that uses prompts to learn modalities . existing methods often aggregate various missing cases to train recovery modules . authors conduct extensive experiments on three public datasets . |
| Outcome: | The proposed method consistently outperforms state-of-the-art methods on three public datasets. |
Benchmarking and Building Zero-Shot Hindi Retrieval Model with Hindi-BEIR and NLLB-E5 (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating retrieval models in Hindi are lacking . despite efforts to build multilingual retrieval systems, this is still a work in progress . |
| Approach: | They evaluate Hindi retrieval models on the Hindi-BEIR benchmark and introduce a multilingual model that leverages a zero-shot approach to support Hindi without the need for Hindi training data. |
| Outcome: | The proposed model leverages a zero-shot approach to support Hindi without the need for Hindi training data. |
Retrieval, Reasoning, Re-ranking: A Context-Enriched Framework for Knowledge Graph Completion (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing embedding-based methods rely on triples in the KG, which is vulnerable to specious relation patterns and long-tail entities. |
| Approach: | They propose a context-enriched framework for KGC that uses a large language model to generate potential answers for each query triple. |
| Outcome: | The proposed framework improves on FB15k237 and WN18RR datasets. |
See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models often rely on a single modality rather than treating and utilizing them equally, leading to dominance of a specific modality on the overall performance. |
| Approach: | They propose a framework to mitigate dominant modality bias by adjusting the gradient of KL divergence based on each modality's contribution and aligning task directions in a non-conflicting manner. |
| Outcome: | The proposed framework mitigates dominant modality bias on UPMC Food-101, Hateful Memes, and MM-IMDb datasets. |
Harnessing and Evaluating the Intrinsic Extrapolation Ability of Large Language Models for Vehicle Trajectory Prediction (2025.naacl-long)
Copied to clipboard
| Challenge: | Emergent abilities of large language models (LLMs) have advanced their application in autonomous vehicle research. |
| Approach: | They propose a framework that leverages LLMs’ built-in extrapolation capabilities for vehicle trajectory prediction, enabling them to understand traffic agents' behavior and interactions over time. |
| Outcome: | The proposed framework enables off-the-shelf, frozen LLMs to achieve competitive trajectory prediction performance with qualitative analyses revealing their enhanced understanding of complex, multi-agent traffic scenarios. |
Stronger Models are Not Always Stronger Teachers for Instruction Tuning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to optimize instruction-following capabilities of large language models (LLMs) assume that larger or stronger models are stronger teachers and therefore adopt smaller models as response generators. |
| Approach: | They propose to use large-scale instruction datasets to tune large language models to align with specific tasks and user intents. |
| Outcome: | The proposed metric outperforms most baselines in identifying the effectiveness of response generators. |
Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models overlook intrinsic semantic associations between soft prompt tokens, leading to high discreteness and limited interactions. |
| Approach: | They propose a low-parameters Prompt Tuning method which leverages prompt decomposition and compressed outer product to facilitate multiple interactions among prompt tokens. |
| Outcome: | Experiments on six architectures and eight datasets show that the proposed method outperforms state-of-the-art methods in performance and efficiency. |
Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs (2025.naacl-long)
Copied to clipboard
| Challenge: | Randomly concatenating data points can lead to cross-contamination due to the significant difference in their subject matter. |
| Approach: | They propose a method that randomly concatenates data of varying lengths until reaching the designed maximum length to optimize context length and reduce padding. |
| Outcome: | The proposed method significantly improves performance on GSM8K and HumanEval, and also improves fairness and accuracy by 15%. |
Transferable Post-training via Inverse Value Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing algorithms for post-training large datasets are requiring a large computational effort. |
| Approach: | They propose to model the changes at logits level during post-training using a separate neural network . they demonstrate that the value network can be seamlessly integrated with another pre-trained model . |
| Outcome: | The proposed model can be integrated with another pre-trained model during inference, enabling similar capability enhancements. |
FLEX: Expert-level False-Less EXecution Metric for Text-to-SQL Benchmark (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for text-to-SQL systems show many false positives and negatives . however, the Execution Accuracy (EX) metric is flawed and can diverge from human experts. |
| Approach: | They propose a method to evaluate text-to-SQL systems using large language models to emulate human expert-level evaluation of SQL queries. |
| Outcome: | The proposed metric improves agreement with human experts with comprehensive context and sophisticated criteria. |
AID: Adaptive Integration of Detectors for Safe AI with Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used to generate human-like text, but safety concerns have emerged with the deployment of LLMs. |
| Approach: | They propose an approach that orchestrates the strengths of multiple pretrained detectors to ensure comprehensive effectiveness in diverse scenarios. |
| Outcome: | The proposed approach can improve the area under the curve (AUC) by 0.07 to 0.21, with a median of 0.12, compared to the best individual detectors developed for specific safety aspects. |
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
Sharpness-Aware Minimization for Topic Models with High-Quality Document Representations (2025.naacl-long)
Copied to clipboard
Tung Nguyen, Tue Le, Hoang Tran Vuong, Quang Duc Nguyen, Duc Anh Nguyen, Linh Ngo Van, Sang Dinh, Thien Huu Nguyen
| Challenge: | Recent advances in topic models have significantly enhanced the performance compared to conventional probabilistic approaches. |
| Approach: | They propose to use a topic model optimization method to improve generalization and performance. |
| Outcome: | The proposed method improves topic models on popular benchmark datasets. |
C2: Scalable Auto-Feedback for LLM-based Chart Generation (2025.naacl-long)
Copied to clipboard
Woosung Koh, Janghan Yoon, MinHyung Lee, Youngjin Song, Jaegwan Cho, Jaehyun Kang, Taehyeon Kim, Se-Young Yun, Youngjae Yu, Bongshin Lee
| Challenge: | generating high-quality charts with Large Language Models presents significant challenges due to limited data and the high cost of curation. |
| Approach: | They propose a referencefree automatic feedback generator to generate high-quality charts with Large Language Models. |
| Outcome: | The proposed framework outperforms baselines and shows that it significantly improves data diversity. |
A Top-down Graph-based Tool for Modeling Classical Semantic Maps: A Case Study of Supplementary Adverbs (2025.naacl-long)
Copied to clipboard
| Challenge: | Semantic map models (SMMs) construct a network-like conceptual space from cross-linguistic instances or forms based on the connectivity hypothesis. |
| Approach: | They propose a graph-based algorithm that automatically generates conceptual spaces and SMMs in a top-down manner. |
| Outcome: | The proposed model is compared with human annotations and other automated methods on cross-linguistic supplementary adverbs. |
UniHGKR: Unified Instruction-aware Heterogeneous Knowledge Retrievers (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing information retrieval models assume a homogeneous structure for knowledge sources and user queries, limiting their applicability in real-world settings. |
| Approach: | They propose a unified instruction-aware heterogeneous knowledge retriever that builds a heterogenous retrieval space for heterogenized knowledge and follows diverse user instructions to retrieve knowledge in specified types. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on CompMix-IR . it achieves 6.36% relative improvements and 54.23% relative improvements . |
Improving Model Evaluation using SMART Filtering of Benchmark Datasets (2025.naacl-long)
Copied to clipboard
| Challenge: | Creating high quality human-annotated datasets is difficult due to dataset saturation. |
| Approach: | They propose a method to filter a subset of test examples from existing benchmarks by removing less informative and lower quality examples. |
| Outcome: | The proposed method reduces dataset size by 48% while increasing Pearson correlation with rankings from ChatBot Arena. |
Entropy-Based Decoding for Retrieval-Augmented Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and intrinsic knowledge sources. |
| Approach: | They propose a entropy-based document-parallel ensemble decoding method that prioritizes low-entropies from retrieved documents and incorporates a contrastive decoding mechanism that contrasts the obtained low- and high-entropic ensemble distributions with the high-end internal knowledge across layers. |
| Outcome: | The proposed method improves on open-domain question answering datasets and shows that it is highly efficient. |
What We Talk About When We Talk About LMs: Implicit Paradigm Shifts and the Ship of Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a new study examines the evolution of language models as a time-specific collection of models of interest. |
| Approach: | They investigate the problem of "Ship of Language Models" where scientific evolution takes form of continuous retrofits of key *existing* terms. |
| Outcome: | The proposed model is based on recent NLP publications and is quantitatively analyzed. |
Diversity Helps Jailbreak Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for jailbreaking large language models rely on laborious human engineering and whitebox access to model internals. |
| Approach: | They propose a method that instructs large language models to deviate from prior context and generate harmful outputs by instructing them to deviat from previous attacks. |
| Outcome: | The proposed method achieves a 62.83% higher success rate in compromising ten leading chatbots, while using only 12.9% of the queries. |
Constrained Decoding with Speculative Lookaheads (2025.naacl-long)
Copied to clipboard
Nishanth Sridhar Nakshatri, Shamik Roy, Rajarshi Das, Suthee Chaidaroon, Leonid Boytsov, Rashmi Gangadharaiah
| Challenge: | Constrained decoding with lookahead heuristics is effective for aligning LLM generations to human preferences, but the extensive lookaheaded roll-out operations for each generated token make it prohibitively expensive. |
| Approach: | They propose a technique that uses lookaheads to align LLMs to human preferences . they propose 2.2x to 12.15x speedup over greedy decoding . |
| Outcome: | The proposed technique achieves 2.2x to 12.15x speedup over greedy decoding without significant performance reduction. |
DyPCL: Dynamic Phoneme-level Contrastive Learning for Dysarthric Speech Recognition (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on data augmentation and feature extraction methods to improve dysarthric speech recognition. |
| Approach: | They propose a Dynamic Phoneme-level Contrastive Learning method which decomposes the speech utterance into phoneme segments for phoneme- level contrastive learning. |
| Outcome: | The proposed method outperforms baseline models and achieves an average 22.10% reduction in word error rate (WER) across the overall dysarthria group. |
Revisiting Early Detection of Sexual Predators via Turn-level Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to detect online grooming rely on chat-level risk labels and fail to identify optimal intervention points. |
| Approach: | They propose a speed control reinforcement learning strategy based on luring communication theory to capture the predator’s turn-level entrapment and a new reward function that balances the trade-off between speed and accuracy based upon the LCT. |
| Outcome: | The proposed method preempts online grooming while identifying optimal early intervention points. |
StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in text-to-speech (TTS) models have led to improvements in speaker prosody and voices modeling. |
| Approach: | They propose an efficient zero-shot TTS model that leverages distilled time-varying style diffusion to capture diverse speaker identities and prosodies. |
| Outcome: | The proposed model surpasses state-of-the-art models in both naturalness and similarity while reducing inference speed by 90%. |
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities across various cognitive tasks. |
| Approach: | They propose a high-quality evaluation dataset to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. |
| Outcome: | The proposed framework improves performance in end-to-end RAG scenarios. |
ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing mobile AI agents focus on most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. |
| Approach: | They propose a mobile AI agent that breaks tasks into page reaching and operation subtasks and a framework that focuses on improving its task-completion abilities. |
| Outcome: | The proposed framework improves IoU accuracy and text accuracy by 7.12% and 7.69% on step-level and 4.72% and 4.63% on task-level compared to the SOTA agent. |
Learning to Solve Domain-Specific Calculation Problems with Knowledge-Intensive Programs Generator (2025.naacl-long)
Copied to clipboard
| Challenge: | Domain Large Language Models (LLMs) are developed for domain-specific tasks based on general LLMs, but it still requires professional knowledge to facilitate the expertise for some domain- specific tasks. |
| Approach: | They propose a pipeline to solve domain-specific calculation problems with KIPG . they use it to extract key variables and calculate outcomes dependent on domain knowledge . |
| Outcome: | The proposed pipeline solves domain-specific calculation problems more effectively . it generates knowledge-intensive programs according to the domain- specific documents . |
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture (2025.naacl-long)
Copied to clipboard
| Challenge: | balancing the training budget, downstream performance, and general capabilities of large language models remains a challenge in many applications. |
| Approach: | They propose a mixture of expert framework based on Soft LoRA and Identity Mixture . SLIM allows dynamic routing between LoRA adapters and identity layers . |
| Outcome: | The proposed framework reduces training cost while maintaining general capabilities . it can be open-sourced upon publication. |
MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation (2025.naacl-long)
Copied to clipboard
Jinsheng Huang, Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, Jingyang Yuan, Wei Ju, Luchen Liu, Tianyu Liu, Baobao Chang, Ming Zhang
| Challenge: | Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding and reasoning abilities, but many benchmarks suffer from systematic biases. |
| Approach: | They propose a benchmark to avoid Type-I errors by creating one perception question and one knowledge anchor question through a meticulous annotation process. |
| Outcome: | The proposed benchmark avoids Type-I errors while maintaining reliability of MCQ evaluations. |
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (2025.naacl-long)
Copied to clipboard
| Challenge: | Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory costs. |
| Approach: | They propose a simple yet effective method that initializes low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrics frozen. |
| Outcome: | The proposed approach only updates the minor components of the weight matrix while keeping the principal singular components frozen. |
Analyzing (In)Abilities of SAEs via Formal Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | Autoencoders have been used for finding interpretable and disentangled features underlying neural network representations in both image and text domains, but there is a lack of corresponding results for the text domain. |
| Approach: | They propose to train sparse autoencoders (SAEs) on a synthetic testbed of formal languages to find interpretable latents in models trained on formal languages. |
| Outcome: | The proposed approach promotes learning of causally relevant features in a formal language setting. |
Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on text-based cognitive reframing, but neglected the importance of non-verbal evidence in real-life therapy. |
| Approach: | They propose a dataset that pairs each GPT-4-generated dialogue with an image that reflects the virtual client’s facial expressions to better mirror real psychotherapy, where facial expression leads to interpreting implicit emotional evidence. |
| Outcome: | The proposed approach outperforms existing methods with LLMs and vision-language models and provides more thoughtful and empathetic suggestions. |
Explanation based In-Context Demonstrations Retrieval for Multilingual Grammatical Error Correction (2025.naacl-long)
Copied to clipboard
| Challenge: | Grammatical error correction (GEC) aims to correct grammatical, spelling, and semantic errors in natural language text. |
| Approach: | They propose a retrieval method based on natural language grammatical error explanations to match inputs with pre-constructed databases where explanations for erroneous samples are generated by LLMs. |
| Outcome: | The proposed method outperforms existing semantic and BM25-based retrieval techniques without additional training or language adaptation. |
A Unified Supervised and Unsupervised Dialogue Topic Segmentation Framework Based on Utterance Pair Modeling (2025.naacl-long)
Copied to clipboard
| Challenge: | Unsupervised methods for dialogue topic segmentation are difficult to surpass due to short sentences, serious references and non-standard language. |
| Approach: | They propose a method to divide a dialogue into different topic paragraphs to better understand its structure and content. |
| Outcome: | The proposed method achieves the best results on multiple benchmark datasets across different scenarios. |
Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) provide superior summarization quality, but their high computational resource requirements limit practical use applications. |
| Approach: | They evaluate 19 small language models for news summarization across 2,000 news samples . they find that top-performing models achieve comparable results to those of 70B LLMs . |
| Outcome: | The proposed models achieve comparable results to 70B LLMs while generating more concise summaries. |
Dynamic Fisher-weighted Model Merging via Bayesian Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing merging approaches involve scaling the parameters model-wise or integrating parameter importance parameter-wise. |
| Approach: | They propose a method for merging model-based models at the parameter level without training data or joint training. |
| Outcome: | The proposed model merging framework outperforms baseline models on validation sets. |
AI-Assisted Human Evaluation of Machine Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Annotation metrics are misaligned with the ideal measure of text quality and human evaluation remains the most accurate, reliable, and ultimate standard. |
| Approach: | They propose an annotation protocol that helps annotators mark erroneous parts of the translation and assign a final score. |
| Outcome: | The proposed protocol reduces the time per span annotation by half . the method reduces annotation budget by 25% with filtering of examples that the AI deems to be likely to be correct. |
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios (2025.naacl-long)
Copied to clipboard
Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, Zhongyu Wei
| Challenge: | Large language models are increasingly employed to empower autonomous agents to simulate human behavior. |
| Approach: | They propose to evaluate LLM-driven agents through multi-turn interactions using a bottom-up approach to create diverse social scenarios constructed from extensive scripts. |
| Outcome: | The proposed model evaluates LLM-driven agents through multi-turn interactions emphasizing goal completion and implicit reasoning. |
FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior research on training grounded factuality classification models to detect hallucinations in large language models (LLMs) has relied on public natural language inference (NLI) data and synthetic data. |
| Approach: | They propose a method that leverages multi-hop reasoning on context graphs extracted from documents to generate complex multi-level claims without relying on LLMs to decide data labels. |
| Outcome: | The proposed model outperforms GPT-4-o on the LLM-Aggrefact benchmark with much smaller model size. |
Label Drop for Multi-Aspect Relation Modeling in Universal Information Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Extractive UIEs can solve model explosion problems using a relatively small model . single-target instruction UIE enables the extraction of only one type of relation at a time . |
| Approach: | They propose a model that assigns different relations to different levels for understanding and decision-making. |
| Outcome: | Experiments show that LDNet outperforms state-of-the-art systems on 9 tasks, 33 datasets . LDnet outperformed state- of-the art systems on single-modal and multi-modal tasks . |
Test-Time Code-Switching for Cross-lingual Aspect Sentiment Triplet Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Aspect Sentiment Triplet Extraction (ASTE) is a thriving research area . current code-switching methods suffer from term boundary detection issues and out-of-dictionary problems. |
| Approach: | They propose a test-time code-switching framework which bridges the gap between bilingual training and monolingual test- time prediction. |
| Outcome: | The proposed framework achieves an average improvement of 3.7% on four cross-lingual datasets. |
VisCGEC: Benchmarking the Visual Chinese Grammatical Error Correction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on Chinese grammatical error correction ignore multi-modality and faked errors, which pushes techniques far away from real-world scenarios. |
| Approach: | They propose to benchmark Chinese grammatical error correction for Chinese as a foreign language learner (CFL) using a dataset, they propose to use two CGEC frameworks to conduct experiments . |
| Outcome: | The proposed approach achieves an F 0.5 score of only 28.9%. |
Are We Done with MMLU? (2025.naacl-long)
Copied to clipboard
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, Pasquale Minervini
| Challenge: | MMLU is widely adopted but its ground truth errors obscure the true capabilities of LLMs. |
| Approach: | They propose a framework for identifying dataset errors using a novel error annotation protocol and a subset of 5,700 manually re-annotated questions. |
| Outcome: | The proposed framework is based on 5,700 re-annotated questions from the MMLU benchmark. |
MeNTi: Bridging Medical Calculator and LLM Agent with Nested Tool Calling (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been widely used in medicine but are limited in their ability to fully address the complexities of the real world. |
| Approach: | They propose a universal agent architecture for Large Language Models that integrates a specialized medical toolkit and employs meta-tool and nested calling mechanisms to enhance LLM tool utilization. |
| Outcome: | The proposed framework improves the accuracy and performance of medical calculators in complex medical scenarios. |
Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering (2025.naacl-long)
Copied to clipboard
Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, Pasquale Minervini
| Challenge: | Large language models store factual knowledge in their parameters but their parametric knowledge can conflict with the information provided in the context. |
| Approach: | They propose a training-free representation engineering method that uses pre-trained sparse auto-encoders to control the knowledge selection behaviour of large language models. |
| Outcome: | The proposed method can control the use of both knowledge sources to resolve knowledge conflict in open-domain question-answering tasks surpassing existing representation engineering methods (+10%) and contrastive decoding methods (+5%). |
MoDification: Mixture of Depths Made Easy (2025.naacl-long)
Copied to clipboard
Chen Zhang, Meizhi Zhong, Qimeng Wang, Xuantao Lu, Zheyu Ye, Chengqiang Lu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang, Dawei Song
| Challenge: | Long-context efficiency is a trending topic in large language model (LLM) serving. |
| Approach: | They propose a method to combine long-context efficiency and mixture of depths to bring down both latency and memory. |
| Outcome: | The proposed method achieves 1.2 speedup in latency and 1.8 reduction in memory compared to original LLMs especially in long-context applications. |
On the Vulnerability of Text Sanitization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing reconstruction attacks on text sanitization are not able to accurately assess their effectiveness. |
| Approach: | They propose to use ASR to measure the effectiveness of reconstruction attacks to evaluate sanitization performance. |
| Outcome: | The proposed reconstruction attacks achieve a 46.4% improvement in ASR over the state-of-the-art baseline with a privacy budget of =4.0 on the SST-2 dataset. |
Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent large language models demonstrate remarkable abilities in responding to queries in diverse languages, but their ability to handle long multilingual contexts is unexplored. |
| Approach: | They propose a multilingual Needle-in-a-Haystack (MLNeedle) test to assess a model's ability to retrieve relevant information from a collection of multilingual distractor texts. |
| Outcome: | The proposed model performance is the lowest when the needle is in a language outside the English language family and (ii) located in the middle of the input context. |
Verify-in-the-Graph: Entity Disambiguation Enhancement for Complex Claim Verification with Interactive Graph Representation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to claim verification are based on decomposing claims into sub-claims and querying a knowledge base to resolve hidden or ambiguous entities. |
| Approach: | They propose a framework that leverages the reasoning and comprehension abilities of LLM agents to solve ambiguous entities in a graph. |
| Outcome: | The proposed framework achieves competitive performance compared to baselines across benchmarks. |
Exploring the Potential of Large Language Models for Heterophilic Graphs (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches for heterophilic graphs overlook rich textual data associated with nodes, which could unlock deeper insights into their heterophilistic contexts. |
| Approach: | They propose a two-stage framework to enhance node classification on heterophilic graphs by leveraging open-world knowledge encoded by large language models. |
| Outcome: | The proposed framework can be used to better characterize heterophilic graphs, where neighboring nodes often exhibit different labels. |
Exploiting Edited Large Language Models as General Scientific Optimizers (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for solving optimization problems in scientific scenarios use observational feedback as additional textual descriptions, but these methods struggle to utilize it effectively. |
| Approach: | They propose a generalized approach to boost mathematical optimization in scientific scenarios by using observational feedback from LLMs as additional textual descriptions. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods on six different tasks using six different LLM backbones. |
DIRAS: Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | RAG systems leave out important relevant information (low recall) and excessively related but irrelevant information (high precision) authors propose a manual annotation-free schema that can be used for RAGs with limited performance. |
| Approach: | They propose a manual annotation-free schema that annotates unseen queries with calibrated relevance scores. |
| Outcome: | Evaluators show that DIRAS can achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs. |
Hello Again! LLM-powered Personalized Agent for Long-term Dialogue (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing dialogue systems focus on brief single-session interactions, neglecting real-world needs for long-term companionship and personalized interactions. |
| Approach: | They propose a model-agnostic framework for long-term dialogue agents . they use event summary and persona management to enable reasoning . |
| Outcome: | The proposed framework incorporates three independently tunable modules dedicated to event perception, persona extraction, and response generation. |
My LLM might Mimic AAE - But When Should It? (2025.naacl-long)
Copied to clipboard
| Challenge: | a study examines the representation of African American English in large language models . a survey of black americans and annotation of LLM outputs shows that Black Americans prefer to use AAE in formal settings . |
| Approach: | They examine Black Americans' perceptions of how effective AI tools are at producing authentic African American English in large language models. |
| Outcome: | The results show that Black Americans prefer to use LLMs in formal settings over informal ones . the results show they prefer to produce AAE in less formal settings . |
High-Dimension Human Value Representation in Large Language Models (2025.naacl-long)
Copied to clipboard
Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishii, Pascale Fung
| Challenge: | Existing approaches to align large language models with human values and preferences are not able to be applied to all tasks and fields. |
| Approach: | They propose a high-dimensional representation of symbolic human value distributions in LLMs that is orthogonal to model architecture and training data. |
| Outcome: | The proposed representations are evaluated on 15 open-source and commercial LLMs and are self-supervised from the value-relevant output of 8 LLM models. |
Not all Hallucinations are Good to Throw Away When it Comes to Legal Abstractive Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for summarization of legal documents rely on external knowledge to generate abstracts. |
| Approach: | They propose an entity-driven approach that learns the model to generate factual hallucinations . they evaluate legal documents in English and French to evaluate their results . |
| Outcome: | The proposed approach reduces non-factual hallucinations and maximizes summary coverage and factual hallucines at entity-level. |
Query-focused Referentiability Learning for Zero-shot Retrieval (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing dense representations have shown limitations in zero-shot scenarios . however, passage representations fail to align with their gold queries . |
| Approach: | They propose a query-focused concept of 'referentiable' which ensures passage representations are referenced by their gold queries. |
| Outcome: | The proposed model outperforms existing models on the BEIR benchmark. |
A Novel Computational Modeling Foundation for Automatic Coherence Assessment (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for text coherence assessment rely on a proxy task . however, this approach does not capture the full range of factors contributing to coherency. |
| Approach: | They propose a formal linguistic definition of what makes a discourse coherent and formalize these conditions as respective computational tasks that are jointly trained. |
| Outcome: | The proposed model improves on two human-rated coherence benchmarks. |
Token-based Decision Criteria Are Suboptimal in In-context Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) typically utilizes output probabilities of manually selected label tokens, but such calibrations lead to suboptimal decision boundaries. |
| Approach: | They propose a method which renounces token probabilities and uses the nearest centroid classifier on the Language Model’s last hidden states to predict the label of the nearest ctroid. |
| Outcome: | The proposed method outperforms current token-based baselines by about 20%50% and provides a strong state-of-the-art in ICL. |
CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluation methods do not capture complex attributes of counterspeech quality, such as contextual relevance, aggressiveness, or argumentative coherence. |
| Approach: | They propose to use a dataset and framework to evaluate counterspeech quality across four dimensions: contextual relevance, aggressiveness, argument-coherence, and suitability. |
| Outcome: | The proposed method outperforms ROUGE, METEOR, and BertScore in correlating with human judgement, indicating a significant improvement in automated counterspeech evaluation. |
Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown continuously improving multilingual capabilities. |
| Approach: | They evaluate the ability of open LLMs to handle multilingual machine translation tasks using a parallel-first monolingual-second data mixing strategy. |
| Outcome: | The proposed model outperforms state-of-the-art models and achieves competitive performance with Google Translate and GPT-4-turbo. |
RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Efforts to ensure the safety of large language models (LLMs) include safety fine-tuning, evaluation, and red teaming. |
| Approach: | They conduct a comparative analysis of RAG and non-RAG frameworks with eleven LLMs to examine how RAG can make models less safe and change their safety profile. |
| Outcome: | The proposed methods are less effective than those used for non-RAG settings. |
Evaluating Evidence Attribution in Generated Fact Checking Explanations (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing fact-checking systems struggle with attribution quality, as their generated explanations can include hallucinations. |
| Approach: | They propose a protocol to assess attribution quality in fact-checking explanations using human annotation and automatic annotation. |
| Outcome: | The proposed protocol can be automated, the authors show . best-performing LLMs still generate explanations that are not always accurate . |
ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods do not assess whether large language models fully utilize contextual information. |
| Approach: | They introduce a new metric to assess LLMs' ability to fully utilize contextual information. |
| Outcome: | The proposed benchmark comprises 1,986 test instances spanning four long-context tasks with high IC scores in the domains of books, debates, medicine, and law. |
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models’ Posteriors (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context Learning (ICL) is the primary method for performing natural language tasks with Large Language Models. |
| Approach: | They examine whether aggregation is a confounding factor in the modeling of subjective tasks . they find it is possible for minority annotators to better align with LLMs . |
| Outcome: | The proposed method is based on aggregation of annotations in a dataset with appropriate priors. |
Arabic Dataset for LLM Safeguard Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on large language models have focused on English, but the safety of LLMs in Arabic remains under-explored. |
| Approach: | They propose to use Arabic-region-specific questions to evaluate LLMs' safety . they use a dual-perspective evaluation framework to examine differences between LLM responses . |
| Outcome: | The proposed framework assesses the LLM responses from both governmental and opposition viewpoints. |
Anticipating Future with Large Language Model for Simultaneous Machine Translation (2025.naacl-long)
Copied to clipboard
Siqi Ouyang, Oleksii Hrinchuk, Zhehuai Chen, Vitaly Lavrukhin, Jagadeesh Balam, Lei Li, Boris Ginsburg
| Challenge: | Existing methods only use the partial utterance that has already arrived at the input and the generated hypothesis. |
| Approach: | They propose to use a large language model to predict future source words and opportunistically translate without introducing too much risk. |
| Outcome: | The proposed method outperforms baselines on four language directions and achieves the best translation quality-latency trade-off by up to 5 BLEU points at the same latency. |
GuideLLM: Exploring LLM-Guided Conversation with Applications in Autobiography Interviewing (2025.naacl-long)
Copied to clipboard
Jinhao Duan, Xinyu Zhao, Zhuoxuan Zhang, Eunhye Grace Ko, Lily Boddy, Chenan Wang, Tianhao Li, Alexander Rasgon, Junyuan Hong, Min Kyung Lee, Chenxi Yuan, Qi Long, Ying Ding, Tianlong Chen, Kaidi Xu
| Challenge: | Large Language Models (LLMs) have demonstrated their effectiveness in human-guided dialogues, but tasks in the real world are more complex and require greater autonomy from LLMs. |
| Approach: | They propose to characterize LLM-guided conversation into three fundamental components: Goal Navigation, Context Management, Empathetic Engagement and implement an interviewing environment for the evaluation of LLMs. |
| Outcome: | The proposed LLM outperforms baseline LLMs in interviewing quality and autobiography generation quality. |
Fine-Tuning Large Language Models with Sequential Instructions (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing instruction-tuned models struggle to adhere to a query with multiple intentions, which impairs their performance when the completion of several tasks is demanded by a single command. |
| Approach: | They develop an automatic process that turns existing data into diverse and complex task chains and a new benchmark to evaluate a model’s ability to follow all the instructions in a sequence. |
| Outcome: | The proposed model can follow instructions better and deliver higher results in coding, maths, and open-ended generation. |
Diverse In-Context Example Selection After Decomposing Programs and Aligned Utterances Improves Semantic Parsing (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are well suited for seq2seq translation . a lack of pretraining corpora can hinder the use of LLMs for structured interpretation . |
| Approach: | They propose to decompose available ICE trees into fragments and use additional invocations to map them to corresponding utterances. |
| Outcome: | The proposed method shows visible gains on diverse parsing benchmarks on popular languages. |
Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to large language models focus on semantic similarity, neglecting the intricate logical structures and reasoning essential for addressing complex legal issues. |
| Approach: | They propose a Logical-Semantic Integration Model (LSIM) that bridges semantic and logical coherence and a supervised framework that integrates semantic features with in-context learning. |
| Outcome: | The proposed framework significantly improves accuracy and reliability on a real-world legal QA dataset. |
Efficient One-shot Compression via Low-Rank Local Feature Distillation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing structured pruning approaches for large language models require calibration data and costly continued pretraining on billions of tokens to recover lost performance. |
| Approach: | They propose a method that locally distills activations with low-rank weights . they compress Mixtral-8x7B on a single GPU and Phi-2 3B by 40% . |
| Outcome: | The proposed method compresses Mixtral-8x7B on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance. |
Waste Not, Want Not; Recycled Gumbel Noise Improves Consistency in Natural Language Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Consistency in the output of language models can vary significantly in style, factual accuracy, and tone, even for similar inputs. |
| Approach: | They propose a decoding algorithm that enhances response consistency across different prompts with no degradation in response quality. |
| Outcome: | The proposed method outperforms standard sampling methods by 10% across semantic and stylistic consistency benchmarks. |
ConQRet: A New Benchmark for Fine-Grained Automatic Evaluation of Retrieval Augmented Computational Argumentation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating RAArg are costly and lack long, complex arguments and real-world evidence. |
| Approach: | They propose to use multiple fine-grained LLM judges to evaluate RAArg using a new benchmark that features long and complex human-authored arguments on debated topics. |
| Outcome: | The proposed methods provide better and more interpretable assessments than traditional single-score metrics and even previously reported human crowdsourcing. |
SynthDetoxM: Modern LLMs are Few-Shot Parallel Detoxification Data Annotators (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to multilingual text detoxification are hampered by the scarcity of parallel multilingual datasets. |
| Approach: | They propose a pipeline for the generation of multilingual parallel detoxification data and a dataset for SynthDetoxM which is manually generated and rewritten with open-source LLMs. |
| Outcome: | The proposed pipeline outperforms human-annotated datasets even in data limited setting. |
BEMEAE: Moving Beyond Exact Span Match for Event Argument Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Event Argument Extraction (EAE) is a complex task that requires deep comprehension of text to accurately identify and classify event arguments. |
| Approach: | They propose a new evaluation metric that integrates deterministic components with a semantic matching component for more accurate assessment. |
| Outcome: | The proposed evaluation metric leads to higher F1 scores and significant changes in model rankings, underscoring ESM’s inadequacy for comprehensive evaluation of EAE. |
uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work on distilling Whisper’s knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. |
| Approach: | They propose a framework that distills Whisper’s knowledge into small models using pseudo-labels and reduces the size by up to 50%. |
| Outcome: | The proposed model outperforms the teacher model by 5-7 WER points and is 25-50% more efficient when scaling the data. |
Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities (2025.naacl-long)
Copied to clipboard
Chung-En Sun, Xiaodong Liu, Weiwei Yang, Tsui-Wei Weng, Hao Cheng, Aidan San, Michel Galley, Jianfeng Gao
| Challenge: | Recent research shows that Large Language Models (LLMs) are vulnerable to automated jailbreak attacks. |
| Approach: | They propose a framework that crafts adversarial LLMs with enhanced jailbreak ability. |
| Outcome: | ADV-LLM significantly reduces the computational cost of generating adversarial suffixes while achieving nearly 100% ASR on various open-source LLMs. |
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning (2025.naacl-long)
Copied to clipboard
Yifan Peng, Krishna C Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg
| Challenge: | Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models. |
| Approach: | They propose a single-stage joint speech-text SFT approach for training SpeechLMs . their model combines text-only SFT data with three types of speech-related data . |
| Outcome: | The proposed model outperforms previous SpeechLMs on speech-based QA tasks while maintaining original speech-only capabilities. |
Rethinking Word Similarity: Semantic Similarity through Classification Confusion (2025.naacl-long)
Copied to clipboard
| Challenge: | Word similarity measures cannot capture context-dependent, asymmetrical, polysemous nature of semantic similarity. |
| Approach: | They propose a new measure of similarity that reframes semantic similarity in terms of feature-based classification confusion. |
| Outcome: | The proposed model is comparable to cosine similarity in matching human similarity judgments across several datasets and can measure similarity using predetermined features of interest. |
SUNAR: Semantic Uncertainty based Neighborhood Aware Retrieval for Complex QA (2025.naacl-long)
Copied to clipboard
| Challenge: | Open-domain complex question-answering systems face challenges in retrieving and reasoning over information that addresses multifaceted queries. |
| Approach: | They propose a method that leverages large language models to guide a Neighborhood Aware Retrieval process. |
| Outcome: | The proposed approach outperforms retrieve-and-reason baselines on two complex QA datasets. |
Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of retrieval-augmented generation systems are limited . sub-question coverage measures how well a RAG system addresses different facets of a question. |
| Approach: | They propose a framework for evaluation based on sub-question coverage . they propose to decompose questions into sub-questions and classify them into three types . |
| Outcome: | The proposed evaluation framework measures how well a RAG system addresses different facets of a question. |
Stronger Universal and Transferable Attacks by Suppressing Refusals (2025.naacl-long)
Copied to clipboard
| Challenge: | Efforts have focused on aligning models to human preferences (RLHF) . yet, it is believed that such optimization-based attacks are sample-specific. |
| Approach: | They propose an algorithm to embed a "safety feature" into models to make them safe for mass deployment. |
| Outcome: | The proposed attack achieves 25% success rate against the state-of-the-art Circuit Breaker defense, compared to 2.5% by white-box GCG. |
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback (2025.naacl-long)
Copied to clipboard
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd Van Steenkiste, Ranjay Krishna, Cyrus Rashtchian
| Challenge: | Text-to-Image models (T2I) still struggle to produce images that are both aesthetically pleasing and faithful to the user’s input text. |
| Approach: | They propose a training algorithm that trains T2I models to be faithful to the input text. |
| Outcome: | The proposed model improves both the semantic alignment and aesthetic appeal of two diffusion-based T2I models, evidenced by multiple benchmarks (+1.7% on TIFA, +2.9% on DSG1K, +3.4% on VILA aesthetic). |
Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs. |
| Approach: | They propose large vision-Language Models to augment LLMs with visual inputs. |
| Outcome: | The proposed models condition generated text on both an input image and a visual prompt, enabling a variety of use cases such as visual question answering and multimodal chat. |
AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails (2025.naacl-long)
Copied to clipboard
Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien
| Challenge: | Existing safety-related content safety models are not well-suited for commercial use. |
| Approach: | They propose a taxonomy that can be used to categorize safety risks . it combines human annotations with a multi-LLM "jury" system to assess safety . they plan to open-source Aegis2.0 data and models to aid in safety guardrailing . |
| Outcome: | The proposed taxonomy can be used to assess the safety of human-LLM interactions . it can be trained on large, non-commercial datasets and is open-source . |
UOREX: Towards Uncertainty-Aware Open Relation Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for relation extraction are limited by their inability to accurately self-assess their performance. |
| Approach: | They propose an approach that effectively models a part of the epistemic uncertainty within OpenRE by preventing overconfident errors. |
| Outcome: | The proposed approach improves OpenRE reliability by preventing overconfident errors. |
Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training (2025.naacl-long)
Copied to clipboard
Yuchen Zhuang, Jingfeng Yang, Haoming Jiang, Xin Liu, Kewei Cheng, Sanket Lokegaonkar, Yifan Gao, Qing Ping, Tianyi Liu, Binxuan Huang, Zheng Li, Zhengyang Wang, Pei Chen, Ruijie Wang, Rongzhi Zhang, Nasser Zalmout, Priyanka Nigam, Bing Yin, Chao Zhang
| Challenge: | Existing LLMs often rely on complex prompting or extensive fine-tuning to introduce new capabilities while preserving strong generalizability. |
| Approach: | They propose a large-scale pre-training corpus to enhance LLM agents' capabilities . they use 103B agent-specific data encompassing 76,537 APIs . |
| Outcome: | The proposed training corpus outperforms open-source LLMs and commercial LLM agents on three agent benchmarks. |
TinyThinker: Distilling Reasoning through Coarse-to-Fine Knowledge Internalization with Self-Reflection (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models exhibit impressive reasoning capabilities across diverse tasks . direct training on synthesized reasoning data may lead to superficial imitation of reasoning process, authors argue . |
| Approach: | They propose a framework that introduces a three-stage process that incrementally guides the student model through the reasoning process, progressively refining knowledge from coarse to fine granularity. |
| Outcome: | The proposed framework achieves superior performance on commonsense reasoning benchmarks and can be extended to other knowledge-intensive reasoning tasks. |
VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for document QA for visually rich documents outperform unimodal and long-context LLMs by 12-20%. |
| Approach: | They propose a multimodal Retrieval Augmented Generation approach that integrates visual and textual retrieval with linguistic reasoning. |
| Outcome: | The proposed approach outperforms unimodal and long-context LLM benchmarks for document QA by 12-20%. |
VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing text simplification and paraphrase datasets focus on sentence-level translation. |
| Approach: | They propose a novel academic-to-general-audience text paraphrase dataset . they also propose DSPT5 dynamic soft prompt generative language model . |
| Outcome: | The proposed dataset is the first academic-to-general-audience text paraphrase dataset . it is based on document-level these and dissertation abstract pairs from 8 colleges . |
Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | In the brains of human bilinguals, syntax processing may occur in similar regions for their first and second language, depending on factors like when the second language was learned and language proficiency. |
| Approach: | They propose to use sparse autoencoders to train Llama-3-8B and Aya-23-8B models to train multilingual models that share morphsyntactic representations of grammatical concepts. |
| Outcome: | The proposed model can predict plural verbs in different languages by activating the same plural feature. |
Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing classification models only consider the temporal variations of existing data . current models focus on English corpora, leaving time as domains unexplored . |
| Approach: | They propose a framework to generalize classifiers over time on four languages, English, Danish, French, and German. |
| Outcome: | The proposed framework can generalize classifiers over time on four languages, English, Danish, French, and German. |
FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Sign language translation has traditionally been peripheral to mainstream machine translation research. |
| Approach: | They propose a sign language benchmark extension that supports their first sign language, American Sign Language . they provide baselines for tasks from ASL to English text using a unified modeling approach . |
| Outcome: | The proposed model exceeds phrase-level benchmarks while supporting new tasks. |
EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing work on extending specialized agents to multi-agent systems is dependent on human-designed frameworks, limiting the functional scope and scalability of agent systems. |
| Approach: | They propose a generic method to automatically extend specialized agents to multi-agent systems via evolutionary algorithm . they consider existing agent frameworks as the initial individual and apply evolutionary operators to generate multiple agents with diverse settings. |
| Outcome: | The proposed method can extend specialized agents to multi-agent systems . it can generate multiple agents with diverse settings, and improves performance across tasks . |
EmoCharacter: Evaluating the Emotional Fidelity of Role-Playing Agents in Dialogues (2025.naacl-long)
Copied to clipboard
| Challenge: | EmoCharacter evaluates emotional fidelity of role-playing agents in dialogues . current evaluations focus on personality fidelity, tone imitation, and knowledge consistency . |
| Approach: | They propose a benchmark to assess emotional fidelity of role-playing agents in dialogues using large language models. |
| Outcome: | The proposed benchmark measures emotional fidelity of role-playing agents and the characters they portray. |
Language Models can Categorize System Inputs for Performance Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations only provide a single numerical score for broad categories. |
| Approach: | They propose to automatically search for finer-grained categories based on inputs where a system performs well or poorly and describe them in natural language. |
| Outcome: | The proposed model compares LLaMA 3-70B and Claude 3 Opus with similar Elo ratings on Chatbot Arena. |
FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2025.naacl-long)
Copied to clipboard
Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, Liwen Zhang
| Challenge: | Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored. |
| Approach: | They propose to use a benchmark to evaluate large language models' financial domain knowledge and practical abilities. |
| Outcome: | The proposed benchmark evaluates large language models' financial domain knowledge and practical abilities. |
Rethinking the Role of LLMs for Document-level Relation Extraction: a Refiner with Task Distribution and Probability Fusion (2025.naacl-long)
Copied to clipboard
| Challenge: | Document-level relation extraction (DocRE) provides a broad context for extracting relations for entities. |
| Approach: | They propose a method that utilizes LLMs as a refiner and task distribution and probability fusion to refine LLM-based relation extraction methods. |
| Outcome: | The proposed method outperforms existing LLM-based methods without fine-tuning by 25.2% F1. |
Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance? (2025.naacl-long)
Copied to clipboard
| Challenge: | Fact-checking pipelines adopt the Decompose-Then-Verify paradigm, where texts are broken down into smaller claims for individual verification and subsequently combined for a veracity decision. |
| Approach: | They propose to categorize decomposition errors and to reveal a trade-off between accuracy gains and noise introduced by decomposing. |
| Outcome: | The proposed analysis provides new insights into understanding current system’s instability and offers guidance for future studies toward improving claim decomposition in fact-checking pipelines. |
Model Surgery: Modulating LLM’s Behavior Via Simple Parameter Editing (2025.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for detoxification or preventing jailbreaking involve fine-tuning billions of parameters through gradient descent with substantial computational cost. |
| Approach: | They propose to use supervised fine-tuning and Reinforcement Learning from human feedback to modify LLMs' behavior by directly editing a small subset of parameters. |
| Outcome: | Experiments show that editing a small subset of parameters can modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreak, with only inference-level computational resources. |
Effective Skill Unlearning through Intervention and Abstention (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit remarkable skills across various domains without training on task-specific datasets. |
| Approach: | They propose two lightweight, training-free machine skill unlearning techniques for LLMs . they propose to unlearning a particular skill while retaining overall capabilities . |
| Outcome: | The proposed methods demonstrate strong unlearning capabilities for the designated skills across seven different languages. |
CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds (2025.naacl-long)
Copied to clipboard
| Challenge: | Evaluating role-playing capabilities in large language models is challenging due to complex dynamics involved in role-playering. |
| Approach: | They propose a simulation sandbox that generates situational fine-grained character behavior trajectories to enhance LLM performance. |
| Outcome: | The proposed model generates situational fine-grained character behavior trajectories to enhance performance. |
A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) are hardly comprehensively evaluated for their cognitive abilities. |
| Approach: | They propose to evaluate high-level cognitive abilities of Large Vision-Language Models (LVLMs) using images with rich semantics. |
| Outcome: | The proposed evaluation benchmark consists of 251 images along with comprehensive annotations. |
CoME: An Unlearning-based Approach to Conflict-free Model Editing (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) often retain outdated or incorrect information from pre-training, which undermines their reliability. |
| Approach: | They propose a conflict-free model editing framework that selectively removes outdated knowledge from LLMs to improve their accuracy and reliability. |
| Outcome: | The proposed framework improves both editing accuracy and model reliability when applied to existing editing methods. |
On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena (2025.naacl-long)
Copied to clipboard
| Challenge: | Language Models (LMs) have been shown to exhibit a strong preference towards entities associated with Western culture when operating in non-Western languages. |
| Approach: | They propose a parallel Arabic-English benchmark of 58,086 entities associated with Arab and Western cultures and 367 masked natural contexts for entities. |
| Outcome: | The proposed model shows that LMs struggle in Arabic with entities that appear at high frequencies in pre-training, where entities can hold multiple word senses. |
Adapting Sentence-level Automatic Metrics for Document-level Simplification Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on text simplification have focused on sentence simplification, but these metrics often underperform on longer texts. |
| Approach: | They propose to adapt existing sentence-level metrics for paragraph- or document-level simplification by incorporating a new approach to the evaluation of text simplification metrics. |
| Outcome: | The proposed approach outperforms existing sentence-level metrics in terms of correlation with human judgment and the sensitivity and robustness of various metrics to different types of errors produced by existing systems. |
Decoding Speculative Decoding (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference. |
| Approach: | They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens. |
| Outcome: | The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models. |
Leveraging LLM For Synchronizing Information Across Multilingual Tables (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research has sought to improve cross-language synchronization of Wikipedia tables using rule-based methods, but they struggle with complexity and generalization. |
| Approach: | They propose to use a dataset to simulate the process of updating outdated Wikipedia tables and introduce a task decomposition strategy that enhances coherence and accuracy. |
| Outcome: | The proposed model outperforms baselines in Information Updation (1.79%) and Information Addition (20.58%), highlighting its strength in dynamically updating and enriching data across architectures. |
ConMeC: A Dataset for Metonymy Resolution with Common Nouns (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work on metonymy resolution has focused on named entities, but common nouns are also a frequent problem. |
| Approach: | They propose a dataset that combines a metonymy dataset and a chain-of-thought based prompting method for detecting metonyms using large language models. |
| Outcome: | The proposed method can detect metonymy using large language models while still struggling with nuanced semantic understanding. |
Self-DC: When to Reason and When to Act? Self Divide-and-Conquer for Compositional Unknown Questions (2025.naacl-long)
Copied to clipboard
Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Huimin Wang, Guanhua Chen, Kam-Fai Wong
| Challenge: | Existing studies focus on leveraging internal knowledge of Large Language Models (LLMs) to answer known questions. |
| Approach: | They propose a framework that allows LLMs to choose between internal and external knowledge . they use a dataset to analyze compositional questions that are composed of unknown sub-questions . |
| Outcome: | The proposed framework can achieve comparable or even better performance with much fewer external calls compared with several strong baselines. |
TRANSIENTTABLES: Evaluating LLMs’ Reasoning on Temporally Evolving Semi-structured Tables (2025.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that large language models are limited in their ability to reason over time due to static datasets. |
| Approach: | They present a dataset that includes 3,971 questions derived from over 14,000 tables . they introduce a template-based question-generation pipeline that harnesses LLMs to refine questions . |
| Outcome: | The proposed model improves on the TRANSIENTTABLES dataset . it demonstrates that the model can reason over time, even when it is not static . |
AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligence (2025.naacl-long)
Copied to clipboard
| Challenge: | AdvisorQA aims to improve LLMs’ capability to offer advice for deeply subjective concerns, utilizing the LifeProTips Reddit forum. |
| Approach: | They propose a dataset to train LLMs' ability to offer advice for deeply subjective concerns, utilizing the LifeProTips Reddit forum. |
| Outcome: | The proposed model improves usefulness through automatic metric, GPT-4 and human evaluations, and expands independent evaluation axis to include harmlessness. |
tRAG: Term-level Retrieval-Augmented Generation for Domain-Adaptive Retrieval (2025.naacl-long)
Copied to clipboard
| Challenge: | Neural retrieval models suffer when there is a domain shift between training and test data distributions. |
| Approach: | They propose to generate domain-adapted pseudo-queries using large language models (LLMs) to improve term recall of unseen query terms by using term-level Retrieval-Augmented Generation (tRAG). |
| Outcome: | The proposed method significantly improves recall for unseen terms by 10.6% and outperforms LLM and retrieval-augmented generation baselines on overall retrieval performance. |
JRE-L: Journalist, Reader, and Editor LLMs in the Loop for Science Journalism for the General Audience (2025.naacl-long)
Copied to clipboard
| Challenge: | Science journalism reports current scientific discoveries to non-specialists, aiming to enable public comprehension of the state of the art. |
| Approach: | They propose a framework that integrates three LLMs mimicking the writing-reading-feedback-revision loop. |
| Outcome: | The proposed framework generates articles that are more accessible than existing methods, including prompting single advanced models such as GPT-4 and other LLM-collaboration strategies. |
Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons . |
| Approach: | They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments. |
| Outcome: | The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges. |
Graph Neural Network Enhanced Retrieval for Question Answering of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing retrieval methods divide reference documents into passages, treating them in isolation. Existing methods only use contiguous passages or keywords. |
| Approach: | They propose a retrieval method that leverages graph neural networks to exploit relatedness between passages to enhance retrieval. |
| Outcome: | The proposed method improves retrieval by exploiting the relatedness between passages. |
Pula: Training Large Language Models for Setswana (2025.naacl-long)
Copied to clipboard
| Challenge: | Setswana is a Bantu language spoken by an estimated five to ten million people worldwide. |
| Approach: | They propose to make setswana-based models available for the first time using data available from setswa and setswegian databases. |
| Outcome: | The proposed models outperform GPT-4o and Gemini 1.5 Pro on English-Setswana translation tasks and achieve state-of-the-art performance on Setswanan reasoning tasks. |
LegalViz: Legal Text Visualization by Text To Diagram Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Graphviz provides diagrams for legal documents that are easy to understand and understand . a novel dataset of 23 languages and 7,010 cases of legal document and visualization pairs is proposed . |
| Approach: | They propose a dataset of legal diagrams using DOT graph description language of Graphviz. |
| Outcome: | The proposed dataset outperforms existing models including GPTs in 23 languages and 7,010 cases of legal document and visualization pairs. |
Active Few-Shot Learning for Text Classification (2025.naacl-long)
Copied to clipboard
Saeed Ahmadnia, Arash Yousefi Jordehi, Mahsa Hosseini Khasheh Heyran, Seyed Abolghasem Mirroshandel, Owen Rambow, Cornelia Caragea
| Challenge: | Recent advances in Large Language Models (LLMs) have boosted the use of Few-Shot Learning (FSL) methods in natural language processing. |
| Approach: | They propose a method that identifies effective support instances from the unlabeled pool and can work with different LLMs. |
| Outcome: | The proposed method improves on five tasks on which it is tested on five LLMs. |
Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation (2025.naacl-long)
Copied to clipboard
Cong-Duy T Nguyen, Xiaobao Wu, Thong Thanh Nguyen, Shuai Zhao, Khoi M. Le, Nguyen Viet Anh, Feng Yichao, Anh Tuan Luu
| Challenge: | Existing approaches to multimodal entity linking use contrastive learning to align input sentences and entities, but are limited by their random negative sampling. |
| Approach: | They propose a method to match negative samples with similar attributes using JD-CCL . they also propose 'contextual visual-aid controllable patch transform' experimental results demonstrate the strong effectiveness of their method . |
| Outcome: | The proposed method is able to match negative samples with similar attributes on a multimodal knowledge graph. |
ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a new system that leverages the encyclopedic knowledge and linguistic reasoning capabilities of Large Language Models (LLMs) is proposed to enhance the productivity of researchers . a researcher's research idea generation process involves problem identification, method development, experiment design and iterative revision . |
| Approach: | They propose a system that leverages encyclopedic knowledge and linguistic reasoning capabilities of Large Language Models to assist researchers in their work. |
| Outcome: | The proposed system generates novel ideas based on human and model-based evaluations . it leverages encyclopedic knowledge and linguistic reasoning capabilities of Large Language Models based systems . |
Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language models (LLMs). |
| Approach: | They propose a logit separability-based method that integrates multiple class-related words into each sample-label pair to improve LLM understanding. |
| Outcome: | The proposed method improves ICL performance by providing clearer instructions and richer label information. |
Identifying Emerging Concepts in Large Corpora (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for text analysis are not specifically designed for identifying emergent concepts, instead applying general-purpose techniques that do not account for distinct temporal patterns associated with conceptual emergence. |
| Approach: | They propose a method to identify emerging concepts in large text corpora by analyzing changes in the heatmaps of the underlying embedding space. |
| Outcome: | The proposed method outperforms existing methods by analyzing speeches in the U.S. Senate from 1941 to 2015. |
CodeSCM: Causal Analysis for Multi-Modal Code Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work has shown that multimodal prompts can be highly sensitive, where small adjustments might result in drastically different responses from the model. |
| Approach: | They propose a Structural Causal Model (SCM) for analyzing multi-modal code generation using large language models (LLMs). |
| Outcome: | The proposed model is based on the principles of Causal Mediation Analysis and quantifies the causal effects of different prompt modalities on the model. |
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning. |
| Approach: | They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content . |
| Outcome: | The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content. |
Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design (2025.naacl-long)
Copied to clipboard
| Challenge: | Using Mixture-of-Experts, researchers have found that efficient MoE is difficult to achieve due to two key reasons: imbalanced expert activation and massive communication overhead. |
| Approach: | They propose a collaboration-constrained routing strategy that encourages more specialized expert groups and leverages expert specialization. |
| Outcome: | The proposed approach achieves an average performance improvement of 0.51% and 0.33% on LLaMA-MoE and Qwen-MaE respectively. |
LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation (2025.naacl-long)
Copied to clipboard
Sachit Kuhar, Wasi Uddin Ahmad, Zijian Wang, Nihal Jain, Haifeng Qian, Baishakhi Ray, Murali Krishna Ramanathan, Xiaofei Ma, Anoop Deoras
| Challenge: | Recent code completion models focus on local file contexts, but do not fully capture the complexities of real-world software development. |
| Approach: | They propose a version-specific code-completion task across eight libraries as they evolve over the years and an in-depth analysis of two widely used public libraries: PyTorch and Matplotlib. |
| Outcome: | The proposed model improves performance with public libraries, compared with existing models. |
Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects? (2025.naacl-long)
Copied to clipboard
| Challenge: | LVLMs often mistakenly determine objects as present in images where they do not exist . authors propose a new benchmark to evaluate object hallucinations by removing objects from images and asking the model whether it can still see the removed objects. |
| Approach: | They propose a benchmark to evaluate object hallucinations by removing objects from images . they propose oDPO, a direct preference optimization objective based on visual objects . |
| Outcome: | The proposed benchmark reduces the likelihood of object hallucinations by removing objects from images and asking the model whether it can still see the removed objects. |
Self-Pluralising Culture Alignment for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to align large language models don't take cultural diversity into account. |
| Approach: | They propose a framework that generates questions on various culture topics and outputs to LLMs under both culture-aware and culture-unaware settings. |
| Outcome: | The proposed framework improves the alignment of large language models to diverse cultures without compromising general abilities. |
K-COMP: Retrieval-Augmented Medical Domain Question Answering With Knowledge-Injected Compressor (2025.naacl-long)
Copied to clipboard
| Challenge: | Documents retrieved for closed domains require high expertise, so reader model may have difficulty comprehending the text. |
| Approach: | They propose a system which augments the prior knowledge required to answer correctly by adding thousands of tokens to the retrieved documents. |
| Outcome: | The proposed system provides the knowledge required to answer correctly and generates prior knowledge to facilitate the answer process prior to compression of the retrieved passages. |
DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students’ Hand-Drawn Math Images (2025.naacl-long)
Copied to clipboard
| Challenge: | DrawEduMath examines the ability of vision language models to handle real-world math problems, such as those encountered in classrooms and tutoring sessions. |
| Approach: | They present DrawEduMath, an English-language dataset of 2,030 images of students’ handwritten responses to math problems. |
| Outcome: | The proposed model can be used to evaluate teachers' QA pairs and 44,362 synthetic QAs derived from teachers' descriptions. |
Knowledge Graph Guided Evaluation of Abstention Techniques (2025.naacl-long)
Copied to clipboard
| Challenge: | Several prior studies test the safety promises of language models based on their effectiveness in blocking malicious requests. |
| Approach: | They create a benchmark based on benign concepts and ground them in a knowledge graph to evaluate abstention techniques. |
| Outcome: | The proposed framework causes models to abstain with over 80% abstention rates, but not as effective for descendants of the target concepts, where abstraction rates drop by 19%. |
Wav2Prompt: End-to-End Speech Prompt Learning and Task-based Fine-tuning for Text-based LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Text-based large language models (LLMs) can be applied to a wide range of tasks without being explicitly trained. |
| Approach: | They propose a method which integrates spoken input with a text-based large language model (LLM) it takes LLM token embeddings as training targets and utilises a continuous integrate-and-fire mechanism for explicit speech-text alignment. |
| Outcome: | The proposed model can be applied to speech translation, speech understanding and spoken-query-based question answering tasks. |
Legal Judgment Prediction based on Knowledge-enhanced Multi-Task and Multi-Label Text Classification (2025.naacl-long)
Copied to clipboard
| Challenge: | Legal judgment prediction (LJP) is an essential task for legal AI, aiming at predicting judgments based on the facts of a case. |
| Approach: | They propose a knowledge-enhanced approach that incorporates 'label-level knowledge' to enhance the representation of case facts for each task and 'task-level' knowledge to improve synergy. |
| Outcome: | The proposed method is effective in comparison to state-of-the-art (SOTA) baselines. |
SPeCtrum: A Grounded Framework for Multidimensional Identity Representation in LLM-Based Agent (2025.naacl-long)
Copied to clipboard
Keyeun Lee, Seo Hyeong Kim, Seolhee Lee, Jinsu Eun, Yena Ko, Hayeon Jeon, Esther Hehsun Kim, Seonghye Cho, Soeun Yang, Eun-mee Kim, Hajin Lim
| Challenge: | Existing methods for simulating individual identities oversimplify human complexity, leading to incomplete or flattened representations. |
| Approach: | They propose a framework for constructing authentic LLM agent personas by incorporating an individual’s multidimensional self-concept. |
| Outcome: | The framework integrates Social Identity (S), Personal Identity (P), and Personal Life Context (C) components, each contributing distinct yet interconnected aspects of identity. |
Beemo: Benchmark of Expert-edited Machine-generated Outputs (2025.naacl-long)
Copied to clipboard
Ekaterina Artemova, Jason S Lucas, Saranya Venkatraman, Jooyoung Lee, Sergei Tilga, Adaku Uchendu, Vladislav Mikhailov
| Challenge: | Existing benchmarks for machine-generated texts (MGTs) include single-author texts (human-written and machine-generated). |
| Approach: | They propose to benchmark machine-generated outputs (Beemo) which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases. |
| Outcome: | The proposed benchmark includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases, ranging from creative writing to summarization. |
SANDWiCH: Semantical Analysis of Neighbours for Disambiguating Words in Context ad Hoc (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that language understanding offered by chat-based Large Language Models is limited and far from human-like performance. |
| Approach: | They propose a framework for multilingual Word Sense Disambiguation using group algebra. |
| Outcome: | The proposed framework surpasses the performance of current alternatives even in low-resource languages while reducing the parameter count by 72%. |
Towards Automatic Evaluation for Image Transcreation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating image transcreation have relied on human evaluation. |
| Approach: | They propose a suite of automatic evaluation metrics inspired by machine translation metrics . they identify cultural relevance, semantic equivalence and visual similarity as critical dimensions of image transcreation . |
| Outcome: | The proposed evaluation metrics agree with human ratings across 7 countries. |
ImgTrojan: Jailbreaking Vision-Language Models with ONE Image (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on the safety of large language models (LLMs) with human values have focused on the integration of multi-modal user input into these models. |
| Approach: | They propose a method to bypass safety constraints of large language models by using poisoned images instead of original textual captions. |
| Outcome: | The proposed attack bypasses safety constraints of large language models (VLMs) by replacing the original textual captions with malicious jailbreak prompts. |
RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) show exceptional problem-solving capabilities but struggle with complex reasoning tasks. |
| Approach: | They propose a novel RAG approach that integrates retrieved information to guide tree-based reasoning process based on LLMs. |
| Outcome: | The proposed approach outperforms existing methods in large language models . iteratively plans intermediate sub-queries and answers based on the LLM itself . |
Mitigating Biases of Large Language Models in Stance Detection with Counterfactual Augmented Calibration (2025.naacl-long)
Copied to clipboard
Ang Li, Jingqian Zhao, Bin Liang, Lin Gui, Hui Wang, Xi Zeng, Xingwei Liang, Kam-Fai Wong, Ruifeng Xu
| Challenge: | Large language models generate biased stances due to spurious correlations and preference towards certain individuals and topics. |
| Approach: | They propose a counterfactual Augmented Calibration Network to calibrate potential bias in stance detection of large language models. |
| Outcome: | The proposed calibration network can mitigate biases of large language models, achieving state-of-the-art results. |
Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for zero-shot text classification lack prompt engineering due to prompt brittleness . however, these methods are not effective for zero shot text classifications . |
| Approach: | They propose a method that predicts token probabilities across multiple positions and simulates comprehensive sampling of generation paths in a single run of a language model. |
| Outcome: | The proposed approach improves accuracy and reduces standard deviation by 98% . it maintains comparable performance even without a prompt, reducing the need for prompt engineering . |
Investigating Hallucinations in Simultaneous Machine Translation: Knowledge Distillation Solution and Components Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate hallucinations in siMT generate fluency but unfaithful translation. |
| Approach: | They propose a method that utilizes the OMT model to mitigate hallucinations in SiMT. |
| Outcome: | The proposed method reduces hallucinations and improves the SiMT performance. |
Markov Chain of Thought for Efficient Mathematical Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have sought to enhance the mathematical reasoning capabilities of large language models. |
| Approach: | They propose a Markov Chain of Thought (MCoT) that compresses previous reasoning steps into a simplified question. |
| Outcome: | The proposed method improves efficiency and maintains comparable accuracy. |
Towards Inducing Long-Context Abilities in Multilingual Neural Machine Translation Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models traditionally use Sinusoidal Positional Embeddings . retraining with newer methods like ROPE or ALIBI is computationally expensive . |
| Approach: | They propose to transition NMT models from Sinusoidal to Relative PEs without compromising performance. |
| Outcome: | The proposed approach outperforms models trained with Sinusoidal PEs on document-level benchmarks . the results show that parameter-efficient fine-tuning can facilitate the transition . |
Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for backchannel prediction relied on turn-based or artificially balanced datasets. |
| Approach: | They propose a method for real-time, continuous backchannel prediction using a fine-tuned Voice Activity Projection model. |
| Outcome: | The proposed method outperforms baseline methods in timing and type prediction tasks in real-world environments. |
Prompt Compression for Large Language Models: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input. |
| Approach: | They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs. |
| Outcome: | The proposed methods are categorized into hard prompt methods and soft prompt methods. |
Goal-Conditioned DPO: Prioritizing Safety in Misaligned Instructions (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing defense methods focus on aligning the model’s output towards less harmful responses through post-processing or input perturbation. |
| Approach: | They propose a goal-conditioned direct preference optimization technique which is trained to prioritize the system prompt over the user prompt through goal-conditioning and reduces the average Attack Success Rate (ASR) on a wide variety of jailbreak attacks. |
| Outcome: | The proposed approach reduces the average Attack Success Rate (ASR) on a wide variety of jailbreak attacks while maintaining general performance. |
K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Strategic reasoning requires Large Language Model (LLM) agents to adapt their strategies dynamically in multi-agent environments. |
| Approach: | They propose a framework that enables Large Language Models to achieve varying levels of strategic depth by recursive mechanisms that allow agents to form higher order beliefs about others' beliefs. |
| Outcome: | The proposed framework enables LLMs to achieve varying levels of strategic depth, allowing agents to form higher order beliefs—beliefs about others’ beliefs. |
SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models are far from achieving the robustness and consistency required for safe biomedical NLI applications. |
| Approach: | They propose a framework that leverages external ontologies to instantiate diverse syllogistic arguments for biomedical NLI by identifying valid conclusions and extracting supporting evidence. |
| Outcome: | The proposed framework evaluates large language models on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. |
The State and Fate of Summarization Datasets: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Summarization is the task of shortening a text while preserving the most important information it contains. |
| Approach: | They propose a novel ontology covering sample properties, collection methods and distribution covering sample characteristics, collection method and distribution. |
| Outcome: | The proposed ontology covers sample properties, collection methods and distribution, and can be used to streamline future research into a more coherent body of work. |
MGM: Global Understanding of Audience Overlap Graphs for Predicting the Factuality and the Bias of News Media (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for profiling news media focus on textual features, causing them to overlook complex relationships between entities. |
| Approach: | They propose a framework for profiling news media from the lens of political bias and factuality. |
| Outcome: | The proposed framework improves existing models and improves them by integrating structural information from similar nodes. |
A Logical Fallacy-Informed Framework for Argument Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Argument generation is crucial in daily life and has numerous online and offline applications. |
| Approach: | They propose a fallacy-informed preference optimization that includes a classification loss to capture the fine-grained information on fallacy types to help LLMs generate logically sound arguments. |
| Outcome: | The proposed method reduces fallacy errors by 17.5% on argument generation tasks and outperforms fine-tuned baselines and other preference optimization methods, such as DPO. |
LLaMA-Berry: Pairwise Optimization for Olympiad-level Mathematical Reasoning via O1-like Monte Carlo Tree Search (2025.naacl-long)
Copied to clipboard
Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, Dongzhan Zhou
| Challenge: | LLaMA-Berry is an advanced mathematical reasoning framework to enhance the problem-solving ability of large language models (LLMs). |
| Approach: | They propose a Monte Carlo Tree Search and Self-Refine framework to optimize reasoning paths and a pairwise reward model to evaluate different paths globally. |
| Outcome: | The proposed framework overcomes inefficiencies and limitations of step-wise and greedy search algorithms, enabling more efficient exploration of solution spaces. |
Generative Prompt Internalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Prompts used in large language model based applications are often fixed and lengthy, leading to significant computational overhead. |
| Approach: | They propose a method that internalizes complex prompts using a joint training approach and a data synthesis technique that auto-collects conversational datasets by swapping roles of agent and environment. |
| Outcome: | The proposed method internalizes complex prompts across agent-based applications and generates the content along with reasons for why it should change accordingly. |
Script-Agnosticism and its Impact on Language Identification for Dravidian Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that modern systems are script-dependent in language identification (langID) many languages are written in multiple writing systems, and script diversity is common in low-resource languages. |
| Approach: | They propose to learn script-agnostic representations using different strategies . they use word-level script randomization and script exposure to a language written in multiple scripts . |
| Outcome: | The proposed methods exploit script randomization and exposure to a language written in multiple scripts to improve language identification while maintaining competitive performance on naturally occurring text. |
NAT: Enhancing Agent Tuning with Negative Samples (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning and reinforcement learning use only positive examples, limiting their efficiency in low-resource scenarios. |
| Approach: | They propose a method that leverages both successful and failed trajectories for fine-tuning, maximizing the utility of limited resources. |
| Outcome: | The proposed method surpasses existing methods, including SFT, DPO, and PPO, across various tasks. |
Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomalies (2025.naacl-long)
Copied to clipboard
Zirui Song, Guangxian Ouyang, Meng Fang, Hongbin Na, Zijing Shi, Zhenhao Chen, Fu Yujie, Zeyu Zhang, Shiyu Jiang, Miao Fang, Ling Chen, Xiuying Chen
| Challenge: | Existing household robots are inability to recognize potential problems or dangers in home environments. |
| Approach: | They propose a task of creating anomaly scenarios using generative models instead of manually labeled data to build simulated environments. |
| Outcome: | The proposed framework outperforms existing models in terms of task description and scene diversity. |
How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments (2025.naacl-long)
Copied to clipboard
Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Conventional approaches compare sentence probabilities directly, but large language models (LLMs) provide nuanced evaluation methods using prompts and templates. |
| Approach: | They propose to derive acceptability judgments from large language models using prompts and templates to comprehensively evaluate their grammatical knowledge. |
| Outcome: | The proposed methods excel in different linguistic phenomena, suggesting they access different aspects of the LLMs’ grammatical knowledge. |
Is Your LLM Outdated? A Deep Look at Temporal Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate large language models are limited due to their inherent dynamic nature and the inherent dynamicity of language and information. |
| Approach: | They introduce a new evaluation framework that employs fresh text and event prediction for assessing LLMs’ temporal adaptability. |
| Outcome: | The proposed framework shows significant temporal biases and a decline in performance over time. |
Towards a Perspectivist Turn in Argument Quality Assessment (2025.naacl-long)
Copied to clipboard
| Challenge: | Argument quality is a key aspect of computational argumentation (CA), but it still exhibits a high degree of subjectivity in perception. |
| Approach: | They propose to use a multi-layered classification to target two aspects of argument quality in a systematic review of NLP datasets. |
| Outcome: | The proposed model improves the quality of annotators and their ability to be used in perspectivist research. |
A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualization (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated powerful reasoning abilities across multiple domains, but have been underexplored for time-series reasoning (TsR) |
| Approach: | They propose a prompt-based solution for evaluating large language models’ TsR performance. |
| Outcome: | The proposed solution improves performance and costs by 140% and reduces costs by 99%. |
PlagBench: Exploring the Duality of Large Language Models in Plagiarism Generation and Detection (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have raised concerns about the potential threats large language models pose to academic integrity and copyright protection. |
| Approach: | They propose a dataset of 46.5K synthetic text pairs that represent three major types of plagiarism: verbatim copying, paraphrasing, and summarization. |
| Outcome: | The proposed dataset shows that GPT-3.5 Turbo can produce high-quality paraphrases and summaries without significantly increasing text complexity compared to GPT-4 Turbo. |
Commonality and Individuality! Integrating Humor Commonality with Speaker Individuality for Humor Recognition (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for humor recognition focus on one aspect of humor commonalities, ignoring the multifaceted nature of humor. |
| Approach: | They propose a commonality and individuality incorporated network for humor recognition that integrates multifaceted humor commonalities with speaker individuality. |
| Outcome: | The proposed model integrates multifaceted humor commonalities with speaker individuality to deepen the understanding of humor expressions. |
CAST: Corpus-Aware Self-similarity Enhanced Topic modelling (2025.naacl-long)
Copied to clipboard
Yanan Ma, Chenghao Xiao, Chenhan Yuan, Sabine N Van Der Veer, Lamiece Hassan, Chenghua Lin, Goran Nenadic
| Challenge: | Existing topic modelling methods encode contextual information of documents while ignoring contextual details of candidate centroid words. Existing methods are limited by the contextualization gap. |
| Approach: | They propose a topic modelling method that builds upon candidate centroid word embeddings contextualized on the dataset and a self-similarity-based method to filter out less meaningful tokens. |
| Outcome: | The proposed method significantly enhances the coherence and diversity of generated topics, and handles noisy data, outperforming strong baselines. |
A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to DST are limited by their computational resources or lack flexibility to adapt to new slots. |
| Approach: | They propose a system that integrates domain classification and DST in a single pipeline and uses self-refining prompts to adapt dynamically. |
| Outcome: | The proposed system improves on existing methods on multiWOZ datasets and provides 20% better Joint Goal Accuracy (JGA) over existing methods with 90% fewer requests to the LLM API. |
Navigating the Cultural Kaleidoscope: A Hitchhiker’s Guide to Sensitivity in Large Language Models (2025.naacl-long)
Copied to clipboard
Somnath Banerjee, Sayan Layek, Hari Shrawgi, Rajarshi Mandal, Avik Halder, Shanu Kumar, Sagnik Basu, Parag Agrawal, Rima Hazra, Animesh Mukherjee
| Challenge: | Cultural harm arises when LLMs misrepresent or normalize values, identities, and practices in ways that conflict with the norms of diverse cultural groups. |
| Approach: | They propose a cultural harm test dataset and a preference dataset to assess model outputs across different cultural contexts. |
| Outcome: | The proposed model improves model behavior significantly reducing the likelihood of generating culturally insensitive or harmful content. |
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. |
| Approach: | They conduct the first in-depth analysis of the role padding tokens play in T2I diffusion models by using two causal techniques to analyze how information is encoded in the representation of tokens across different components of the pipeline. |
| Outcome: | The proposed techniques reveal that padding tokens may affect the model’s output during text encoding, during the diffusion process, or be effectively ignored. |
In-Context Learning (and Unlearning) of Length Biases (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing work has demonstrated the ability of large language models to learn lexical and label biases in-context negatively impacts performance and robustness of models. |
| Approach: | They investigate the impact of length biases on in-context learning by analyzing model length information in-constext. |
| Outcome: | The proposed model learns length biases in the context window without parameter updates. |
AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing pre-trained language models outperform them in certain domains, indicating that there is significant potential for further improvement in this area. |
| Approach: | They propose to use pre-trained language models to evaluate ad texts from multiple perspectives within real-world advertising operations to define five tasks and construct a Japanese dataset. |
| Outcome: | The proposed benchmark outperforms existing pre-trained language models in several tasks, but humans outperformed them in certain domains. |
Empowering Retrieval-based Conversational Recommendation with Contrasting User Preferences (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing CRSs assume positive and negative user preferences, but assume that the entities in the dialogue history are positive. |
| Approach: | They propose a conversational recommender model that captures user sentiments and uses the reasoning capacity of the LLMs to extract user's hidden preferences. |
| Outcome: | The proposed model outperforms existing methods in three benchmark datasets, improving up to 99.72% in Recall@10. |
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding. |
| Approach: | They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices. |
| Outcome: | The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs. |
Towards Robust Knowledge Representations in Multilingual LLMs for Equivalence and Inheritance based Consistent Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have led to impressive linguistic capabilities and emergent reasoning behaviors. |
| Approach: | They propose to use "equivalence" and "inheritance" to evaluate LLMs' representations . they propose to combine "equal" and 'inheritory' to improve consistency across languages . |
| Outcome: | The proposed representations show that they produce conflicting answers across languages . the proposed representation improves performance across languages and improves learning and knowledge sharing. |
LLMs as Meta-Reviewers’ Assistants: A Case Study (2025.naacl-long)
Copied to clipboard
Eftekhar Hossain, Sanjeev Kumar Sinha, Naman Bansal, R. Alexander Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Ram Pavan Kumar Guttikonda, Mousumi Akter, Md. Mahadi Hassan, Matthew Freestone, Matthew C. Williams Jr., Dongji Feng, Santu Karmaker
| Challenge: | Meta-reviews are a critical step in the overall scientific peer-reviewed process, which focuses on understanding the consensus of expert opinions on a scholarly work and making informed judgments on its scientific merit. |
| Approach: | They propose to use large language models to generate a controlled multi-perspective-summary (MPS) of their opinions to help meta-reviewers better comprehend multiple experts' perspectives. |
| Outcome: | The proposed model can help meta-reviewers better comprehend multiple experts’ perspectives by generating a controlled multi-perspective-summary (MPS) of their opinions. |
A Survey of NLP Progress in Sino-Tibetan Low-Resource Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite the increasing effort in including more low-resource languages in NLP/CL development, most of the world’s languages are still absent. |
| Approach: | They propose to include low-resource languages in NLP/CL research as more resources are poured into the development of data-driven models. |
| Outcome: | The proposed language family is a low-resource language family with a small number of native speakers and government support. |
Enhancing Language Model Hypernetworks with Restart: A Study on Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | a comprehensive investigation into optimization strategies for hypernetworks remains lacking. |
| Approach: | They propose restart optimization strategies to improve hypernetworks' performance for language models. |
| Outcome: | The proposed restart strategy improves hypernetworks' performance for language models, compared to conventional deep neural networks. |
Functional Lexicon in Subword Tokenization (2025.naacl-long)
Copied to clipboard
| Challenge: | Function units are hard to map across languages, while being the most frequent tokens. |
| Approach: | They analyze subword tokens in terms of their productivity and try to find thresholds that best distinguish function from content tokens. |
| Outcome: | The proposed method can be used to identify functional lexical units in low-resource languages with minimal annotated data. |
Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data (2025.naacl-long)
Copied to clipboard
Haonan Wang, Minbin Huang, Runhui Huang, Lanqing Hong, Hang Xu, Tianyang Hu, Xiaodan Liang, Zhenguo Li, Hong Cheng, Kenji Kawaguchi
| Challenge: | Contrastive Language-Image Pre-training (CLIP) is a standard for cross-modal image-text representation learning. |
| Approach: | They propose a framework that enhances pre-trained CLIP models by exploiting challenging text-image pairs within existing datasets. |
| Outcome: | The proposed framework improves CLIP models by exploiting text-image pairs in training. |
Evaluating the Prompt Steerability of Large Language Models (2025.naacl-long)
Copied to clipboard
Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Kush R. Varshney, Eitan Farchi, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, Miao Liu, Prasanna Sattigeri
| Challenge: | a primary question underlying alignment research is: whose views are we aligning to? |
| Approach: | They propose to evaluate the steerability of model personas as a function of prompting by defining a benchmark and inspecting how these indices change as if steering effort is a factor. |
| Outcome: | The proposed benchmark reveals that the steerability of many current models is limited due to skew in baseline behavior and an asymmetry in their steerability across many persona dimensions. |
A Data-Driven Method for Analyzing and Quantifying Lyrics-Dance Motion Relationships (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have not explored the relationships between lyrics and dance motions . previous studies focused on synthesizing or retrieving dance motion from lyrics . |
| Approach: | They propose a method to detect parts of songs where meaningful relationships exist . they use clustering to transform lyrics and dance motions into symbols . |
| Outcome: | The proposed method outperforms existing methods on prose and non-dance dance motions. |
CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent Vision and Language models have shown impressive performance across benchmarks . however, frontier models lack cultural awareness and can affect global cultural diversity . |
| Approach: | They propose a visual question answering benchmark to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. |
| Outcome: | The proposed model shows large performance disparities between culture-specific and common concepts in the parametric setting. |
PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing systems produce generic, monotonic responses that lack individuality and fail to adapt to users’ personal attributes. |
| Approach: | They propose a dataset that incorporates user images as part of the persona, enabling personalized responses tailored to user-specific factors such as age or emotional context. |
| Outcome: | The proposed dataset enhances user experience, with personalized responses contributing to a more engaging interaction. |
Scaling LLM Inference Efficiently with Optimized Sample Compute Allocation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to optimize sample allocations for large language models fail to account for the optimal sampling configuration. |
| Approach: | They propose an algorithm that optimizes sample allocation by finding an optimal mix of different inference configurations. |
| Outcome: | The proposed algorithm achieves better accuracy on SWE-Bench with 3x less compute than the default configuration. |
Large Language Models for Persian-English Idiom Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have shown superior capabilities in translating figurative language compared to neural machine translation systems. |
| Approach: | They evaluate LLMs, NMTs and their combinations using PersianIdioms datasets . they find that automatic evaluation methods like BLEU and BERTScore are effective . |
| Outcome: | The proposed model performs better in both directions than other models. |
Follow the Beaten Path: The Role of Route Patterns on Vision-Language Navigation Agents Generalization Abilities (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision and language navigation (VLN) is a challenging task towards the creation of embodied agents. |
| Approach: | They propose a solution that combines visual and linguistic features to enable VLN . they propose augmentation of the training data to fill the gap in missing patterns . |
| Outcome: | The proposed solution fills the gap in missing patterns of training data. |
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models are biased towards a Western and Anglo-centric worldview. |
| Approach: | They propose to extend the Octopus test to measure "cultural awareness" they argue that cultural awareness is needed for AI systems to be useful across cultures . |
| Outcome: | The proposed method argues that cultural awareness is not cultural knowledge, but meta-cultural competence . the proposed method is based on the octopus test, which shows it is impossible to learn meaning from real-world concepts without knowing intent and meaning . |
Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps? (2025.naacl-long)
Copied to clipboard
| Challenge: | a new study examines the extent and patterns of gaps in understandability of book reviews . 83% of the reviews had at least one culturally-specific difficult-to-understand element . |
| Approach: | They examine extent and patterns of gaps in understandability of book reviews . 83% of reviews had at least one culturally-specific difficult-to-understand element . authors say they have a significant scope for improvement . |
| Outcome: | The proposed approach improves the understanding of book reviews from different cultures . 83% of the reviews had at least one culturally-specific difficult-to-understand element . |
HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models that memorize past tokens have “flat” memory architectures that restrict the context window. |
| Approach: | They propose a framework that imitates human memorization behavior by preserving tokens from early input segments, passing memory embeddings along the sequence, and recalling relevant information from history. |
| Outcome: | The proposed framework outperforms existing models in language modeling and question-answering tasks and achieves comparable or superior generation quality to long-context models with 2 57 fewer parameters and 2.5 116 less inference memory. |
Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent surge in multilingual large language models (LLMs) and Retrieval Augmented Generation (RAG) has significantly expanded conversational search across varied linguistic and cultural demographics. |
| Approach: | They found that LLMs displayed systemic bias towards information in the same language as query language in document retrieval and answer generation. |
| Outcome: | The results highlight the linguistic divide within multilingual LLMs in information search systems. |
Teaching Models to Balance Resisting and Accepting Persuasion (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are susceptible to persuasion, which can pose risks when faced with an adversarial interlocutor. |
| Approach: | They propose a method to balance positive and negative persuasion by using recursive dialogue trees to train models to accept persulasion. |
| Outcome: | The proposed model-based training improves resistance to misinformation and resilience to being challenged while also resulting in the best overall performance on multi-agent debates across two domains. |
Making Language Models Robust Against Negation (2025.naacl-long)
Copied to clipboard
| Challenge: | Negation is a semantic phenomenon that alters an expression to convey the opposite meaning. |
| Approach: | They propose a self-supervised method to make language models more robust against negation by pre-training models. |
| Outcome: | The proposed task outperforms the off-the-shelf versions on nine negation-related benchmarks. |
Through the Lens of History: Methods for Analyzing Temporal Variation in Content and Framing of State-run Chinese Newspapers (2025.naacl-long)
Copied to clipboard
| Challenge: | State-run newspapers are believed to strategically select and frame news articles to align with the shifting political tides of the country. |
| Approach: | They analyze more than 50 years of articles from the People's Daily and Reference News to quantify differences in content and framing over time. |
| Outcome: | The proposed methods show that the changes in name mentions and sentiment in news articles are more significant in People’s Daily than in Reference News . |
PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models (2025.naacl-long)
Copied to clipboard
Michael-Andrei Panaitescu-Liess, Pankayaraj Pathmanathan, Yigitcan Kaya, Zora Che, Bang An, Sicheng Zhu, Aakriti Agrawal, Furong Huang
| Challenge: | PoisonedParrot is the first stealthy data poisoning attack that induces an LLM to generate copyrighted content even when the model has not been directly trained on the copyright material. |
| Approach: | They propose a stealthy data poisoning attack that induces an LLM to generate copyrighted content even when it has not been directly trained on the copyright material. |
| Outcome: | The proposed model induces an LLM to generate copyrighted content with no discernible side effects and is surprisingly effective at priming the model to generate content with little side effects. |
Towards Operationalizing Right to Data Protection (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work introduces the concept of generating unlearnable datasets (by adding imperceptible spurious correlations to the clean data) this approach is limited by several practical constraints like requiring knowledge of the target model. |
| Approach: | They propose a framework that injects imperceptible spurious correlations into natural language datasets, rendering them unlearnable without affecting semantic content. |
| Outcome: | The proposed framework can restrict newer models like GPT-4o and Llama from learning on generated data, resulting in a drop in test accuracy compared to their zero-shot performance. |
Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on in-context learning mechanisms are not consistent . current research identifies two main approaches to explain the ICL mechanism . |
| Approach: | They propose a framework for evaluating in-context learning mechanisms by focusing on regression tasks. |
| Outcome: | The proposed framework can solve regression problems and then measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. |
GLiREL - Generalist Model for Zero-Shot Relation Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot named entity recognition rely on distant supervision and training data for unseen labels. |
| Approach: | They propose an efficient architecture and training paradigm for zero-shot relation classification . they use a protocol to generate multiple relation labels in a single forward pass . |
| Outcome: | The proposed architecture and training paradigm achieve state-of-the-art results on the zero-shot relation classification task. |
ComPO: Community Preferences for Language Model Personalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for training language models with human feedback rely on subjective preferences that are assumed to account for an "average" user . however, annotating preferences is inherently subjective and results in generic models that generate outputs not preferred by many user groups. |
| Approach: | They propose a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. |
| Outcome: | The proposed method improves performance by focusing on group-level preferences rather than individual feedback. |
GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing LLMs excel and often surpass human performance on benchmarks, but they are known to falter in simple tasks and under seemingly straightforward circumstances. |
| Approach: | They propose a benchmark to assess compositional and conditional reasoning within a flight booking task. |
| Outcome: | The proposed model outperforms existing models on the flight booking task with a 67% accuracy rate. |
ALPACA AGAINST VICUNA: Using LLMs to Uncover Memorization of LLMs (2025.naacl-long)
Copied to clipboard
Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, Santu Rana
| Challenge: | Existing studies have shown that pre-trained LLMs emit training data up to 150 more often than in regular operation. |
| Approach: | They propose a black-box prompt optimization method where an attacker LLM agent uncovers higher levels of memorization in a victim agent . |
| Outcome: | The proposed method shows 23.7% more overlap with training data compared to state-of-the-art baselines. |
Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Few studies have systematically compared LMs’ contextualized word embeddings for languages beyond English. |
| Approach: | They evaluate Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models. |
| Outcome: | The proposed model captures some variance in human relatedness judgments but falls short of the human benchmark. |
Understanding LLMs’ Fluid Intelligence Deficiency: An Analysis of the ARC Task (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research on fluid intelligence assessments has highlighted significant deficiencies in LLMs’ abilities. |
| Approach: | They analyze the challenges LLMs face in demonstrating fluid intelligence through controlled experiments using the most representative ARC task as an example. |
| Outcome: | The proposed model shows that it lacks the ability to combine skill composition and abstract input formats and lacks left-to-right decoding. |
FedSpaLLM: Federated Pruning of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing pruning methods assume public access to calibration data, which is impractical for privacy-sensitive applications. |
| Approach: | They propose a federated learning framework for pruning LLMs that prunes models locally based on private data while accounting for system heterogeneity and communication efficiency. |
| Outcome: | The proposed framework reduces communication overhead and personalizes pruning process based on client resources in federated settings. |
IHEval: Evaluating Language Models on Following the Instruction Hierarchy (2025.naacl-long)
Copied to clipboard
Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, Meng Jiang
| Challenge: | Instruction-tuned language models (LMs) are increasingly deployed as interactive services across various applications. |
| Approach: | They propose a benchmark to evaluate models' ability to follow the instruction hierarchy by comparing their models to a set of benchmarks. |
| Outcome: | The proposed benchmark covers 3,538 examples across nine tasks covering cases where instructions in different priorities either align or conflict. |
Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond (2025.naacl-long)
Copied to clipboard
Mardhiyah Sanni, Tassallah Abdullahi, Devendra Deepak Kayande, Emmanuel Ayodele, Naome A Etori, Michael Samwel Mollel, Moshood O. Yekini, Chibuzor Okocha, Lukman Enegi Ismaila, Folafunmi Omofoye, Boluwatife A. Adewale, Tobi Olatunji
| Challenge: | Afrispeech-Dialog is a benchmark dataset of 50 simulated medical and non-medical African-accented English conversations . a 10%+ performance degradation is found in ASR systems on long-form, accented speech . |
| Approach: | They propose to use a dataset to evaluate automatic speech recognition systems on African-accented conversations. |
| Outcome: | The proposed dataset compares state-of-the-art speech recognition systems on accented conversations with native accents and shows a 10%+ performance degradation. |
THREAD: Thinking Deeper with Recursive Spawning (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive capabilities across diverse settings, but their performance degrades as context length and complexity increases. |
| Approach: | They propose to frame model generation as a thread of execution that, based on the context, can run to completion or dynamically spawn new threads. |
| Outcome: | The proposed model outperforms existing frameworks by 10% to 50% on diverse benchmarks. |
CORG: Generating Answers from Complex, Interrelated Contexts (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to analyzing knowledge in a corpus often focus on single factors in isolation. |
| Approach: | They propose a framework that organizes multiple contexts into independently processed groups . they classify these relationships into distracting, ambiguous, counterfactual, and duplicated . |
| Outcome: | The proposed framework outperforms existing grouping methods and single-context approaches. |
Generating Diverse Hypotheses for Inductive Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that large language models (LLMs) can engage in inductive reasoning by sampling multiple hypotheses about the rules and selecting the one that best explains the observations. |
| Approach: | They propose to increase the temperature parameter to enhance diversity by sampling multiple hypotheses and selecting the one that best explains the observations. |
| Outcome: | The proposed method improves diversity while maintaining text quality while increasing temperature. |
On the Analysis and Distillation of Emergent Outlier Properties in Pre-trained Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that a small subset of dimensions within language Transformers’ representation spaces emerge as "outliers" during pretraining. |
| Approach: | They propose a method that prioritizes critical outlier dimensions in distillation using a weighted MSE loss. |
| Outcome: | The proposed method outperforms state-of-the-art distillation methods and generalizes well across Encoder-only BERT, Decoder-only GPT-2, and Encodeer-Decoder T5 architectures. |
Open-World Evaluation for Retrieving Diverse Perspectives (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing retrieval systems only cover diverse perspectives on 33.74% of the examples . existing systems only focus on relevance to the question, ignoring diversity. |
| Approach: | They build a Benchmark for Retrieval Diversity for Subjective questions (BERDS) based on a question and diverse perspectives associated with the question . they evaluate retrievers paired with a corpus to determine whether each document contains a perspective . |
| Outcome: | The proposed approach improves retrieval diversity on complex questions . existing retrieval systems only cover diverse perspectives on 33.74% of the examples . |
Analyzing the Inner Workings of Transformers in Compositional Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on compositional generalization abilities of neural models have focused on benchmarks, but the results do not reflect the underlying competence of the model. |
| Approach: | They propose to find an existing subnetwork that contributes to the generalization performance and perform causal analyses on how the model utilizes syntactic features. |
| Outcome: | The proposed model relies on syntactic features but the subnetwork with better generalization performance relies mainly on a non-compositional algorithm . |
Substance Beats Style: Why Beginning Students Fail to Code with LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing work shows that beginners struggle to prompt LLMs to solve text-to-code tasks. |
| Approach: | They propose to use a causal intervention experiment on technical vocabulary to test whether students lack the technical vocabulary needed to write good prompts and to analyze graphs that abstract how students edit prompts. |
| Outcome: | The proposed model improves student-LLM communication by predicting student failures and predicting the information content of prompts. |
Reverse Thinking Makes LLMs Stronger Reasoners (2025.naacl-long)
Copied to clipboard
Justin Chen, Zifeng Wang, Hamid Palangi, Rujun Han, Sayna Ebrahimi, Long Le, Vincent Perot, Swaroop Mishra, Mohit Bansal, Chen-Yu Lee, Tomas Pfister
| Challenge: | Reverse-Enhanced Thinking (RevThink) is a framework for large language models to perform reverse thinking. |
| Approach: | They propose a framework for enhancing forward-backward reasoning by collecting data from a teacher model and employing three objectives to train a student model in a multi-task learning fashion. |
| Outcome: | The proposed framework outperforms a fine-tuning method trained on 10x more forward reasoning on 12 datasets covering commonsense, math, and logical reasoning. |
Towards Lifelong Dialogue Agents via Timeline-based Memory Management (2025.naacl-long)
Copied to clipboard
Kai Tzu-iunn Ong, Namyoung Kim, Minju Gwak, Hyungjoo Chae, Taeyoon Kwon, Yohan Jo, Seung-won Hwang, Dongha Lee, Jinyoung Yeo
| Challenge: | Existing studies focus on getting rid of outdated memories to improve retrieval quality, but we argue that such memories provide rich, important contextual cues for response generation (RG). |
| Approach: | They propose a framework for LLM-based lifelong dialogue agents that discards memory removal and manages large-scale memories by linking them based on their temporal and cause-effect relation. |
| Outcome: | The proposed framework augments RG with memory timelines based on evolution or causality of relevant past events. |
StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples (2025.naacl-long)
Copied to clipboard
Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, Chris Callison-Burch
| Challenge: | Existing methods for embedding text are limited by the imperfect nature of data acquired under such assumptions. |
| Approach: | They propose a new approach to training stronger content-independent style embeddings using a synthetic dataset of near-exact paraphrases with controlled style variations. |
| Outcome: | The proposed model outperforms existing methods in real-world benchmarks and outperformed leading style representations in downstream applications. |
FiNE: Filtering and Improving Noisy Data Elaborately with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Currently, there are two mainstream methods for improving data integrity: data filtering and data augmentation. |
| Approach: | They propose a method to improve data integrity by combining data filtering and data augmentation with LLMs. |
| Outcome: | The proposed method surpasses the open-source chat version on HalluQA by 8.45 on the open source version. |
CAMIEval: Enhancing NLG Evaluation through Multidimensional Comparative Instruction-Following Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Evaluating the quality of texts generated by language models has always been a challenging task in natural language processing (NLP). |
| Approach: | They propose a multidimensional comparative evaluation method based on instruction-following that combines relevance, factuality, and adherence with a concrete Chain-of-Thoughts process to enhance the accuracy of evaluations. |
| Outcome: | The proposed method outperforms existing methods in correlation with human evaluations on two NLG evaluation benchmarks. |
LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025.naacl-long)
Copied to clipboard
Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, Lingyun Wang
| Challenge: | LongLeader aims to assess different LLMs' long-context comprehension abilities . long-constext comprehension is a key bottleneck for many use cases . |
| Approach: | They propose a leaderboard to assess different LLMs' long-context comprehension abilities . they offer open-source access to the benchmarks and maintain a dedicated website . |
| Outcome: | The proposed model assesses different LLMs on selected benchmarks and provides open-source access to the benchmarks. |
Language Models Can Infer Action Semantics for Symbolic Planners from Environment Feedback (2025.naacl-long)
Copied to clipboard
| Challenge: | Symbolic planners can discover sequences of actions from initial to goal states given domain-specific action semantics. |
| Approach: | They propose a domain induction task that requires an expert to define action semantics of domain to enable symbolic planning. |
| Outcome: | The proposed model boosts plan success rate from 36.4% to 100% on 7 environments. |
SLM-Mod: Small Language Models Surpass LLMs at Content Moderation (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are expensive to query in real-time and do not allow for a community-specific approach to content moderation. |
| Approach: | They propose to use small language models for community-specific content moderation tasks by fine-tuning and evaluating their performance against larger open- and closed-sourced models. |
| Outcome: | The proposed models outperform zero-shot LLMs in content moderation tasks with 11.5% higher accuracy and 25.7% higher recall across all communities. |
On Positional Bias of Faithfulness for Long-form Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit positional bias in long-context settings, under-attending to information in the middle. |
| Approach: | They compile eight human-annotated long-form summarization datasets to evaluate faithfulness . they find that LLMs faithfully summarize beginning and end of documents but neglect middle content . |
| Outcome: | The proposed methods show that LLMs under-attend to information in the middle of inputs. |
BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | Reinforcement Learning with Human Feedback (RLHF) is the key to the success of large language models (LLMs) in recent years. |
| Approach: | They propose a method to balance the number of prompts and responses to improve knowledge breadth and knowledge depth by introducing gradient-based clustering to estimate the knowledge informativeness and usefulness of each augmented sample. |
| Outcome: | The proposed method outperforms baseline methods while maintaining training efficiency. |
UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for unlearning large language models fine-tune by maximizing loss, but they are unstable . this creates instability, especially on larger datasets, leading to over-unlearning . |
| Approach: | They propose a novel unlearning method that leverages self-distillation to adjust logits . this method ensures smooth convergence and avoids catastrophic forgetting . |
| Outcome: | The proposed method achieves smooth convergence and avoids catastrophic forgetting even on large datasets and sequential unlearning requests. |
H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for tabular reasoning combine textual and symbolic reasoning in a two-stage process to address these limitations. |
| Approach: | They propose an algorithm that integrates symbolic and semantic (textual) approaches in a two-stage process to address these limitations. |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods across three tabular question-answering and fact-verification datasets, underscoring its effectiveness and efficiency. |
Kill two birds with one stone: generalized and robust AI-generated text detection via dynamic perturbations (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods focus on model generalization or focus on robustness. |
| Approach: | They propose a model-based AIGT detection method that can be generalized and robust under two adversarial attacks. |
| Outcome: | The proposed method outperforms state-of-the-art methods for generalization and robustness under two text adversarial attacks. |
Vision-Language Models Can Self-Improve Reasoning via Reflection (2025.naacl-long)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) has been shown to improve the reasoning capability of large language models (LLMs). |
| Approach: | They propose a framework which iteratively enhances the model’s Vision-language Reasoning by Reflecting on CoT Rationales. |
| Outcome: | The proposed framework improves multimodal reasoning on vision-language tasks by 23% to 60% over baselines. |
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that attention heads have a temporal induction property that allows them to learn and reproduce sequences of tokens. |
| Approach: | They analyze attention heads and transformer outputs to examine in-context temporal biases . they find that transformer output has a tendency toward in-constext serial recall . |
| Outcome: | The findings shed light on similarities and differences between LLMs and human memory and learning. |
Knowledge Graph-Guided Retrieval Augmented Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on RAG focus on semantic retrieval of isolated relevant chunks, which ignore their intrinsic relationships. |
| Approach: | They propose a framework that utilizes knowledge graphs to provide fact-level relationships between chunks, improving the diversity and coherence of the retrieved results. |
| Outcome: | Extensive experiments on the HotpotQA dataset and its variants demonstrate the advantages of KG2RAG compared to existing RAG-based approaches in terms of response quality and retrieval quality. |
Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference (2025.naacl-long)
Copied to clipboard
Zeping Li, Xinlong Yang, Ziheng Gao, Ji Liu, Guanchen Li, Zhuang Liu, Dong Li, Jinzhang Peng, Lu Tian, Emad Barsoum
| Challenge: | Existing methods such as Medusa lack adequate information interaction between different drafting heads. |
| Approach: | They propose an enhanced speculative decoding framework that builds upon Medusa and integrates a drafting block capable of parallel inference. |
| Outcome: | The proposed framework outperforms Medusa in terms of head accuracy and latency. |
CAVE: Controllable Authorship Verification Explanations (2025.naacl-long)
Copied to clipboard
| Challenge: | Authorship Verification (AV) is used for tasks such as plagiarism detection, forensic analysis, analysis of the spread of misinformation. |
| Approach: | They propose to train an offline authorship verification model that is accessible and easy to use. |
| Outcome: | The proposed model generates high quality explanations and competitive task accuracy on three difficult AV datasets. |
Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing efforts to train large language models to generate outputs containing epistemic markers have been largely overlooked. |
| Approach: | They propose a benchmark to assess the robustness of LLM-judges to epistemic markers. |
| Outcome: | EMBER benchmarks show that LLM-judges lack robustness in presence of epistemic markers . EMber QA (2,000 instances) and IF (2823 instances) are used to evaluate outputs containing epistemological markers. |
Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work has shown that in-context learning (ICL) with retriever augmentation can help LLMs better capture long-tail knowledge, reducing their reliance on pre-trained data. |
| Approach: | They propose a reinforcement learning-based dynamic uncertainty ranking method that accounts for the varying impact of each retrieved sample on LLM predictions. |
| Outcome: | The proposed method outperforms baseline models on question-answering datasets by 2.76% and 5.96% on long-tail questions that elude zero-shot inference. |
Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Current PP methods face severe bottlenecks, including pipeline bubbles and memory footprint. |
| Approach: | They propose a sequence-level one-forward-one-backward (1F1B) PP method for training LLMs on long sequences with high throughput and memory efficiency. |
| Outcome: | The proposed method achieves 1.14X training throughput with half memory footprint compared to baseline methods . it trains an LLM with 30B parameters on sequences up to 64k tokens using 64X NVIDIA A100 GPUs . |
Differentially Private Learning Needs Better Model Initialization and Self-Distillation (2025.naacl-long)
Copied to clipboard
| Challenge: | Differentially private SGD (DPSGD) enables privacy-preserving training of language models, but often reduces utility, diversity, and linguistic quality. |
| Approach: | They propose a method that initializes a model using data synthesis from a small pre-trained LM with rigorous filtering, applies DP finetuning on private data, and performs self-distillation to refine outputs. |
| Outcome: | The proposed method outperforms vanilla DPSGD with significant improvements in lexical diversity and grammar errors. |
Is a Peeled Apple Still Red? Evaluating LLMs’ Ability for Conceptual Combination with Property Type (2025.naacl-long)
Copied to clipboard
| Challenge: | Conceptual combination is a cognitive process that merges basic concepts, enabling the creation of complex expressions. |
| Approach: | They propose to use a Conceptual Combination with Property Type dataset to evaluate LLMs for conceptual combination thoroughly. |
| Outcome: | The proposed method improves performance in all generative tasks. |
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |
KS-Lottery: Finding Certified Lottery Tickets for Multilingual Transfer in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that a small subset of parameters is highly effective in fine-tuning . prior work shows that there are a few additional parameters corresponding to an intrinsic dimension in a well-trained Large Language Model. |
| Approach: | They propose a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning. |
| Outcome: | The proposed method can find the certified winning tickets in the embedding layer, and fine-tuning on the found parameters is guaranteed to perform as well as full fine- tuning. |
PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to optimize RAG generators fail to align with RAG requirements thoroughly. |
| Approach: | They propose a method for optimizing the RAG generator from multiple preference perspectives to align with RAG requirements comprehensively. |
| Outcome: | The proposed method improves the performance of RAG generators by incorporating retrieved documents into the prompt. |
B4: A Black-Box Scrubbing Attack on LLM Watermarks (2025.naacl-long)
Copied to clipboard
| Challenge: | Experimental results demonstrate superior performance of black-box scrubbing attack on watermarks compared with other baselines. |
| Approach: | They propose a black-box scrubbing attack on watermarks that embeds a hidden pattern invisible to human into generated content of a specific LLM. |
| Outcome: | The proposed method outperforms baselines in 12 different environments. |
IMRRF: Integrating Multi-Source Retrieval and Redundancy Filtering for LLM-based Fake News Detection (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to detect fake news rely on manual checking, which is time-consuming. |
| Approach: | They propose a model which integrates textual corpus retrieval with knowledge graph retrieval to retrieve more comprehensive evidence and a redundant information filtering strategy which minimizes the influence of irrelevant information on the LLM reasoning process. |
| Outcome: | The proposed method outperforms state-of-the-art fact-checking baselines on two challenging fact- checking datasets. |
Matina: A Large-Scale 73B Token Persian Text Corpus (2025.naacl-long)
Copied to clipboard
Sara Bourbour Hosseinbeigi, Fatemeh Taherinezhad, Heshaam Faili, Hamed Baghbani, Fatemeh Nadi, Mostafa Amiri
| Challenge: | Existing Persian datasets are small and lack content diversity . lack of high-quality data has slowed development of NLP models and open-source LLMs for Persian. |
| Approach: | They propose a Persian dataset of 72.9B tokens that is preprocessed and deduplicated to ensure high data quality. |
| Outcome: | The proposed model performs well on key Persian NLP tasks. |
SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | a scalable approach to classify text with sensitivity is costly because of exponential time complexity. |
| Approach: | They propose a framework for calculating word-level local and global sensitivities . they use a CHECKLIST-generated sentiment analysis dataset to test their approach . |
| Outcome: | The proposed framework can be used to calculate word-level local and global sensitivities . it improves attacks by 15.58%, while using sensitivity as an additional reward improves . |
ManaTTS Persian: a recipe for creating TTS datasets for lower resource languages (2025.naacl-long)
Copied to clipboard
| Challenge: | a new text-to-speech system is needed for visual impairments and the visually impaired . a text-based system is not available for all users, and is therefore limited to a limited audience. |
| Approach: | They propose to use ManaTTS, the most extensive publicly accessible Persian corpus . they use a fully transparent, MIT-licensed pipeline to collect transcribed speech datasets . |
| Outcome: | The proposed framework is the most extensive publicly accessible single-speaker Persian corpus . it includes tools for sentence tokenization, bounded audio segmentation, and forced alignment method . |
CultureInstruct: Curating Multi-Cultural Instructions at Scale (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit severe cultural bias, despite their success in recent years . a critical challenge of LLMs is integration of cultural knowledge into these models . |
| Approach: | They propose a large-scale instruction-tuning dataset to reduce cultural bias in large language models. |
| Outcome: | The proposed model outperforms GPT-4o Mini and GPT-42 with 18.47% and 13.07% relative improvements on cultural benchmarks. |
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | In the recent past, a popular way of evaluating natural language understanding was to consider a model’s ability to perform natural language inference (NLI) tasks. |
| Approach: | They focus on five different NLI benchmarks across six models of different scales and examine how their accuracies develop during training. |
| Outcome: | The softmax distributions of models align with human label distributions in cases where statements are ambiguous or vague. |
DenseSSM: State Space Models with Dense Hidden Connection for Efficient Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) face excessive computational and memory requirements due to the commonly used Transformer architecture. |
| Approach: | They propose a method to enhance the flow of hidden information between layers in large language models by selectively integrating shallow-layer hidden states into deeper layers. |
| Outcome: | The proposed method maintains parallelizability and inference efficiency of SSMs while significantly boosting performance on public benchmarks. |
A Mixed-Language Multi-Document News Summarization Dataset and a Graphs-Based Extract-Generate Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research on news summarization focuses on single-language single-document (SLSD), single-linguistic multi-document or cross-language multi-doc (CLSD) however, in real-world scenarios, news articles often involve multiple documents in different languages, i.e., mixed-language MLMD. |
| Approach: | They propose a mixed-language multi-document news summarization dataset with four different languages and 10,992 source document cluster and target summary pairs. |
| Outcome: | The proposed dataset contains four different languages and 10,992 source document cluster and target summary pairs. |
Measuring memorization in language models via probabilistic extraction (2025.naacl-long)
Copied to clipboard
Jamie Hayes, Marika Swanberg, Harsh Chaudhari, Itay Yona, Ilia Shumailov, Milad Nasr, Christopher A. Choquette-Choo, Katherine Lee, A. Feder Cooper
| Challenge: | Large language models (LLMs) are susceptible to memorizing training data, raising concerns about the potential extraction of sensitive information at generation time. |
| Approach: | They propose a method that splits training example into prefix and suffix, prompts LLM with suffix and deems it extractable if it generates the suffix using greedy sampling. |
| Outcome: | The proposed method is unreliable because it does not account for non-determinism in more realistic sampling schemes. |
Audio Is the Achilles’ Heel: Red Teaming Audio Large Multimodal Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LMMs) have demonstrated ability to interact with humans through text . however, safety of audio LMMs remains under-explored . |
| Approach: | They red team the safety of five audio LMMs under three settings . they find that audio Lmms suffer an average attack success rate of 69.14% on harmful questions . |
| Outcome: | a new study shows that audio LMMs suffer an average success rate on harmful questions . the authors also show that the models exhibit safety vulnerabilities when distracted . |
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a key technique for enhancing the inference speed of Large Language Models. |
| Approach: | They propose a method that adds padding tokens to ensure that the number of new tokens remains consistent across samples. |
| Outcome: | The proposed method can handle the issue of inconsistent prediction tokens without adding padding tokens. |
Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | Best-of-N (BoN) sampling is an effective strategy for aligning Large Language Models (LLMs) to human preferences at the time of decoding. |
| Approach: | They propose a variant of BoN that incorporates the Minimum Bayes Risk objective as a proximity regularizer for BoN sampling. |
| Outcome: | The proposed method outperforms both BoN sampling and MBR decoding on the AlpacaFarm and Anthropic datasets. |
MAPWise: Evaluating Vision-Language Models for Advanced Map Queries (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models excel at tasks requiring joint understanding of visual information and natural language. |
| Approach: | They propose to use choropleth maps to answer questions from three geographical regions in the United States, India, China as question templates. |
| Outcome: | The proposed model outperforms other models in the area of visual language and visual question answering. |
Pay More Attention to Images: Numerous Images-Oriented Multimodal Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing multimodal summarization approaches struggle with scenarios involving multiple images as input. |
| Approach: | They propose a task to generate multimodal summaries by integrating multiple images as input . they propose 'multimodal information evaluation' method that measures differences between generated summary and input based on multimodal input - and compares various methods . |
| Outcome: | The proposed method correlates more closely with human judgments than five widely used metrics . |
S2-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiency (2025.naacl-long)
Copied to clipboard
Yuting Zeng, Weizhe Huang, Lei Jiang, Tongxuan Liu, XiTai Jin, Chen Tianying Tiana, Jing Li, Xiaohua Xu
| Challenge: | Large language models exhibit limitations when handling complex mathematical reasoning and logical inference tasks. |
| Approach: | They propose a sparsification strategy to reduce token costs within Multi-agent Debate (MAD) this strategy minimizes ineffective exchanges of information and unproductive discussions among agents . |
| Outcome: | The proposed approach reduces token costs by up to 94.5% while maintaining performance degradation below 2.0%. |
MASTER: A Multi-Agent System with LLM Specialized MCTS (2025.naacl-long)
Copied to clipboard
Bingzheng Gan, Yufan Zhao, Tianyi Zhang, Jing Huang, Li Yusu, Shu Xian Teo, Changwang Zhang, Wei Shi
| Challenge: | Large Language Models (LLMs) are increasingly being explored for problem-solving tasks . their strategic planning capability is often viewed with skepticism due to their limited planning capabilities. |
| Approach: | They propose a framework that coordinates agent recruitment and communication through LLM specialized MCTS. |
| Outcome: | The proposed framework achieves 76% accuracy on HotpotQA and 80% on WebShop . it relies on extensive sampling simulations to approximate the true reward distribution . |
ScreenQA: Large-Scale Question-Answer Pairs Over Mobile App Screenshots (2025.naacl-long)
Copied to clipboard
Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Srinivas Sunkara, Victor Carbune, Jason Lin, Maria Wang, Yun Zhu, Jindong Chen
| Challenge: | Existing screen datasets focus on low-level structural and component understanding or on a much higher-level composite task such as navigation and task completion for autonomous agents. |
| Approach: | They propose to annotate 86k question-answer pairs over the RICO dataset to benchmark screen content understanding. |
| Outcome: | The proposed dataset covers full answers, short answer phrases, and corresponding UI contents with bounding boxes, enabling four subtasks to address various application scenarios. |
Cross-Lingual and Cross-Cultural Variation in Image Descriptions (2025.naacl-long)
Copied to clipboard
| Challenge: | Behavioural and cognitive studies report cultural effects on perception, but these are limited in scope and hard to replicate. |
| Approach: | They develop a method to accurately identify entities mentioned in captions and present in images, then measure how they vary across languages. |
| Outcome: | The proposed method corroborates previous studies showing that languages that are geographically or genetically closer mention entities more frequently than others. |
Soft Syntactic Reinforcement for Neural Event Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent event extraction methods rely on pre-trained language models but still suffer from errors due to a lack of syntactic knowledge. |
| Approach: | They propose a method to incorporate syntactic information into PLM-based models for event extraction (EE) this method uses a standard dependency corpus to select syntax-related dimensions of the model's representation. |
| Outcome: | The proposed method outperforms baseline models and existing syntactic reinforcement methods on sentence-level and document-level EE benchmark datasets. |
Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models impose substantial resource usage. |
| Approach: | They propose a parameter-efficient fine-tuning method that freezes adapters early to reduce resource usage while maintaining performance. |
| Outcome: | The proposed method reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82% while maintaining performance. |
Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Experimental results show that expert models generate accurate, informative and fluent commentary, but are prone to hallucinations due to their limited decision-making capabilities. |
| Approach: | They propose a concept-guided chess commentary generation and a GPT-based Chess Commentary Evaluation to bridge this gap between expert models and large language models. |
| Outcome: | The proposed model is accurate, informative, and fluent. |
TCProF:Time-Complexity Prediction SSL Framework (2025.naacl-long)
Copied to clipboard
| Challenge: | determining the precise time complexity of a code is theoretically undecidable . determining time complexity is a challenging task in programming efficiency analysis . |
| Approach: | They propose a time-complexity prediction SSL framework that uses code snippets labeled with their time complexity classes to predict code time. |
| Outcome: | The proposed approach improves performance by 60% over self-training methods. |
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing text-to-image models fail to produce appropriate images for cultural concepts or objects not well known or underrepresented in western cultures, such as 'hangari' (a Korean utensil). |
| Approach: | They propose a method which iteratively refines the prompt to improve the alignment between the generated images and underrepresented cultural nouns in text-to-image models. |
| Outcome: | The proposed approach improves the alignment between the generated images and cultural nouns in text-to-image models. |
Behavior-SD: Behaviorally Aware Spoken Dialogue Generation with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Spoken dialogues lack explicit modeling of behavior traits that are often overlooked in language models . et al.: our work opens new possibilities for developing behaviorally-aware dialogue systems . |
| Approach: | They propose a large-scale dataset with over 100K spoken dialogues (2,164 hours) they propose BeDLM, the first dialogue model capable of generating natural conversations . |
| Outcome: | The proposed model outperforms baseline models in generating natural dialogues . the proposed model can generate natural conversations conditioned on behavioral and narrative contexts - a key feature of spoken language models . |
Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated multilingual capabilities, yet they are mostly English-centric due to the imbalanced training corpora. |
| Approach: | They extend the evaluation to real-world user queries and non-English-centric LLMs . they show that translation into English can boost LLM performance on NLP tasks . |
| Outcome: | The proposed evaluation extends to user queries and non-English-centric LLMs . it shows that translation into English can boost performance on NLP tasks, but not universally optimal . |
AlgoPuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Algorithmic Multimodal Puzzles (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing datasets focused on visual question-answering focus on visual, language, and algorithmic knowledge . a new study examines the performance of multimodal language models in solving algorithmic puzzles . |
| Approach: | They propose a dataset to test the capabilities of multimodal language models in solving algorithmic puzzles. |
| Outcome: | The proposed dataset is generated automatically from human code. |
Towards Quantifying Commonsense Reasoning with Mechanistic Insights (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have evaluated commonsense reasoning abilities using text-based tasks. |
| Approach: | They propose to capture commonsense knowledge in a graphical representation of 37 daily human activities in graphical form and frame them to frame commonsensical queries. |
| Outcome: | The proposed model can frame an enormous number of commonsense queries ( 10 17) and perform rigorous evaluations of common sense reasoning in LLMs. |
Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs (2025.naacl-long)
Copied to clipboard
Anirudh Phukan, Divyansh Divyansh, Harshit Kumar Morj, Vaishnavi Vaishnavi, Apoorv Saxena, Koustava Goswami
| Challenge: | Large Multimodal Models are plagued by hallucinations that limit their reliability and adoption. |
| Approach: | They propose a method that leverages contextual token embeddings from LMMs to detect hallucinations. |
| Outcome: | The proposed method improves hallucination detection and grounding across diverse categories while excelling in tasks requiring contextual understanding. |
M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to collect instruction fine-tuning data are limited due to their toxicity, privacy and toxicity concerns. |
| Approach: | They propose to use a two-step taxonomy to transform a small set of human written instructions into complex and challenging conversations. |
| Outcome: | M2Lingual has 175K conversations across 70 languages with a balanced mix of high, low and mid-resourced languages. |
Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | a new study shows that cultural background significantly affects multimodal hate speech moderation models . a limited dataset excludes multi-modal forms of hate and excludes non-English-speaking cultures . the lowest pairwise label agreement between the USA and India is due to cultural factors . |
| Approach: | They use a multimodal and multilingual parallel hate speech dataset to examine cultural differences . they find that cultural background significantly affects multimodal hate speech annotation . |
| Outcome: | The proposed dataset shows that cultural background significantly affects multimodal hate speech annotation. |
Grounding Fallacies Misrepresenting Scientific Publications in Evidence (2025.naacl-long)
Copied to clipboard
| Challenge: | Current fact-checking models struggle to use misrepresented scientific passages to refute misinformation. |
| Approach: | They propose an extension of the fallacy detection dataset Missci to detect and highlight such fallacies by grounding applied fallacias in real-world passages from misrepresented studies. |
| Outcome: | The proposed model can detect and highlight misrepresented publications and verbalize them to refute them. |
Has this Fact been Edited? Detecting Knowledge Edits in Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Knowledge editing methods (KEs) can update language models’ obsolete or inaccurate knowledge learned from pre-training. |
| Approach: | They propose to detect knowledge edits in language models by using four KEs, two large language models and two datasets to classify the knowledge as unedited (based on pre-training) and edited (based upon subsequent editing). |
| Outcome: | The proposed method detects whether an output is based on edited knowledge or first-hand knowledge from pre-training. |
AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models excel in highresource languages but underperform in lowresource ones. |
| Approach: | They propose a cross-lingual transfer method that decouples "task ability" from "language ability" they propose to use adaptive adapter merging to obtain target adapters by combining other adapters. |
| Outcome: | The proposed method outperforms existing methods in highresource languages . it decouples "task ability" from "language ability" but fails to fully separate "task capability" from the "source language" |
Coverage-based Fairness in Multi-document Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies quantify summary-level fairness using Proportional Representation, but they ignore corpus-level unfairness. |
| Approach: | They propose a new summary-level fairness measure that considers redundancy in documents . they evaluate the fairness of thirteen different multi-document summarization systems . |
| Outcome: | The proposed measure is based on coverage of documents with different social attribute values and considers redundancy within documents. |
Grammar Control in Dialogue Response Generation for Language Learning Chatbots (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing language learning chatbots and research on second language acquisition benefit from these affordances. |
| Approach: | They ground a dialogue response generation model in a pedagogical repository of grammar skills and evaluate prompting, fine-tuning, and decoding strategies for grammar-controlled dialogue response generators. |
| Outcome: | The proposed model outperforms GPT-3.5 when tolerating minor response quality losses and predicts grammar-controlled responses to support grammar acquisition adapted to learner proficiency. |
Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge (2025.naacl-long)
Copied to clipboard
Li Zhou, Taelin Karidi, Wanlong Liu, Nicolas Garneau, Yong Cao, Wenyu Chen, Haizhou Li, Daniel Hershcovich
| Challenge: | Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet lack a robust methodology to dissect these phenomena comprehensively. |
| Approach: | They propose a multilingual dataset centered on food-related cultural facts and variations in food practices. |
| Outcome: | The proposed model incorporates cultural context significantly and improves its ability to access cultural knowledge. |
Palette of Language Models: A Solver for Controlled Text Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized text generation with their remarkable capabilities. |
| Approach: | They propose to combine a single-attribute model with a discriminative model to achieve a combination strategy that incorporates positive correlation and attribute enhancement. |
| Outcome: | The proposed method is adapted for single-attribute control scenario and achieves surpassing results. |
MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration (2025.naacl-long)
Copied to clipboard
| Challenge: | Multi-agent collaboration among models has shown promise in reasoning tasks but is underexplored in long-form generation tasks like summarization and question-answering. |
| Approach: | They propose a multi-agent multi-model reasoning recipe to improve faithfulness through refinement. |
| Outcome: | The proposed method improves faithfulness and error detection on three summarization datasets and on long-form question-answering tasks. |
MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for memory-augmented dialogue systems lack practical value . current evaluation methods only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors. |
| Approach: | They propose to use long-term memory to create human-like dialogues using chatbots. |
| Outcome: | The proposed benchmark covers memory retrieval and memory recognition tasks with both passive and proactive memory recall data. |
Assessing the State of the Art in Scene Segmentation (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in scene segmentation have made it difficult to detect scenes in literary texts. |
| Approach: | They propose to modify existing models to improve detection of scenes in literary texts . they propose to use a training sample generation scheme to alleviate this problem . |
| Outcome: | The proposed model is more robust to different types of texts, while its overall performance is slightly worse than that of BERT-based models. |
DCE-LLM: Dead Code Elimination with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Dead code can obscure logical errors and be exploited for obfuscation in malware. |
| Approach: | They propose a framework for automated dead code elimination using a codeBERT model with an attribution-based line selector. |
| Outcome: | Experimental results show that DCE-LLM outperforms existing tools for dead code elimination . dead code can obscure logical errors and be exploited for obfuscation in malware . |
Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instruction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches involve models iterating and improving their previous responses based on internal reflection ability or external feedback. |
| Approach: | They propose a reflection framework that leverages meta-thoughts and self-consistency to enhance the iterative reflection capability of Large LanguageModels. |
| Outcome: | The proposed framework achieves an average improvement of 10.1% over established baselines in mathematical and commonsense reasoning tasks, highlighting its efficacy and applicability. |
Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment (2025.naacl-long)
Copied to clipboard
Sangwon Yu, Jongyoon Song, Bongkyu Hwang, Hoyoung Kang, Sooah Cho, Junhwa Choi, Seongho Joe, Taehee Lee, Youngjune Gwon, Sungroh Yoon
| Challenge: | Experimental results show that large language models exhibit a negative bias in binary decision tasks . hallucination is a factor that degrades reliability of LLMs . |
| Approach: | They propose a negative attention score to systematically and quantitatively formulate negative bias by using a parameter-efficient fine-tuning technique. |
| Outcome: | The proposed method reduces the gap between precision and recall caused by negative bias while preserving generalization abilities. |
MiCEval: Unveiling Multimodal Chain of Thought’s Quality via Image Description and Reasoning Steps (2025.naacl-long)
Copied to clipboard
Xiongtao Zhou, Jie He, Lanyu Chen, Jingyu Li, Haojing Chen, Victor Gutierrez Basulto, Jeff Z. Pan, Hanjie Chen
| Challenge: | Existing methods for evaluating the quality of reasoning steps in multimodal chain-of-thought are lacking. |
| Approach: | They propose a framework to evaluate the correctness of reasoning chains by evaluating the quality of both the description and each reasoning step. |
| Outcome: | The proposed framework improves interpretability and human judgments on four state-of-the-art MLLMs. |
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)
Copied to clipboard
Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding
| Challenge: | Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks. |
| Approach: | They propose a mix-of-experts model that allows the model size to grow without raising training costs. |
| Outcome: | The proposed model outperforms existing models in perplexity and robustness tests. |
Measuring and Benchmarking Large Language Models’ Capabilities to Generate Persuasive Language (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on specific domains or types of persuasion, but a general study has focused on how LLMs produce persuasive text. |
| Approach: | They construct a dataset to measure and benchmark the ability of Large Language Models (LLMs) to produce persuasive text. |
| Outcome: | The proposed model can be used to generate persuasive text across domains and domains. |
MILU: A Multi-task Indic Language Understanding Benchmark (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on English, leaving substantial gaps in assessing LLM capabilities in low-resource and linguistically diverse languages. |
| Approach: | They propose a multi-task indic language understanding benchmark to assess LLMs in low-resource languages. |
| Outcome: | The new benchmark spans 8 domains and 41 subjects across 11 Indic languages, reflecting general and culturally specific knowledge. |
AutoEval-ToD: Automated Evaluation of Task-oriented Dialog Systems (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluation methodologies heavily depend on human annotators, which can be inefficient, subjective, and expensive to scale. |
| Approach: | They propose an automated end-to-end evaluation framework that interacts with the ToD system and then assesses its performance across key dimensions. |
| Outcome: | The proposed framework first interacts with the ToD system and assesses its performance across key dimensions by analyzing both its responses and internal states. |
Self-calibration for Language Model Quantization and Pruning (2025.naacl-long)
Copied to clipboard
| Challenge: | Quantization and pruning are fundamental approaches for model compression, but they require large computational resources. |
| Approach: | They propose to use model calibration data to generate synthetic calibrations to improve model performance. |
| Outcome: | The proposed method outperforms other methods using real data in a post-training setting. |
Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models (2025.naacl-long)
Copied to clipboard
Tongxuan Liu, Wenjiang Xu, Weizhe Huang, Yuting Zeng, Jiaxing Wang, Xingyu Wang, Hailong Yang, Jing Li
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. |
| Approach: | They propose a propositional logic prompting method which generates expanded logical information descriptions and utilizes them as an additional augmentation to original contexts. |
| Outcome: | Extensive experiments show that Logic-of-Thought boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. |
IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval (2025.naacl-long)
Copied to clipboard
| Challenge: | Current information retrieval systems struggle to handle complex instructions, despite its critical importance . current models struggle to follow complex instructions in real-world applications, resulting in user-specific tasks. |
| Approach: | They propose a benchmark to evaluate instruction-following information retrieval in expert domains. |
| Outcome: | The proposed method improves on existing models and provides valuable insights to guide future advancements in retrieval. |
QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Currently, large vision-language models are limited in their ability to provide correct answers for multimodal tasks . however, they can still provide correct responses for multiple images associated with a single image . a query-agnostic visual attack (QAVA) provides robust adversarial examples that generate incorrect responses to unspecified and unknown questions. |
| Approach: | They propose a query-agnostic visual attack to create adversarial examples that generate incorrect answers to unspecified and unknown questions. |
| Outcome: | The proposed model improves performance on images when the question is unknown compared to known target questions . |
Evaluating and Improving Graph to Text Generation with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized natural language processing due to their zero-and-short-shot capabilities. |
| Approach: | They propose a tuning-free prompting approach for graph-to-text generation tasks. |
| Outcome: | The proposed approach improves LLMs on graph-to-text generation tasks incrementally. |
The Plagiarism Singularity Conjecture (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are being used to generate content at an unprecedented scale, raising concerns over their misuse and saturation of the content space with artificially generated material. |
| Approach: | They propose to use large language models to generate text that looks indistinguishable from that written by humans. |
| Outcome: | The proposed model can generate 10-30 sentences to breach the plagiarism limit, the authors estimate . |
Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for ensembling language models fail to address complex reasoning tasks. |
| Approach: | They propose a framework for process-level ensembling of large language models using Monte Carlo tree search. |
| Outcome: | The proposed framework outperforms both language model decoding and language model ensemble methods on five reasoning benchmarks. |
One Unified Model for Diverse Tasks: Emotion Cause Analysis via Self-Promote Cognitive Structure Modeling (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for emotion cause analysis overlook common ground rooted in cognitive emotion theories, in particular, the cognitive structure of emotions. |
| Approach: | They propose a unified model capable of tackling diverse emotion cause analysis tasks . they propose 'self-promote mechanism' that constructs the emotion cognitive structure through LLM . |
| Outcome: | The proposed model outperforms existing models and baselines on multiple emotion cause analysis tasks. |
Soft Language Prompts for Language Transfer (2025.naacl-long)
Copied to clipboard
| Challenge: | Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing. |
| Approach: | They propose to combine language-specific adapters and soft prompts to enhance cross-lingual transfer by parameter-efficient fine-tuning methods. |
| Outcome: | The proposed methods outperform language adapters and soft prompts in 16 languages and 10 low-resource languages. |
PICLe: Pseudo-annotations for In-Context Learning in Low-Resource Named Entity Detection (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is sensitive to the choice of demonstrations and can be used for tasks with few examples. |
| Approach: | They propose a framework for in-context learning with noisy, pseudo-annotated demonstrations . they annotate large quantities of demonstrations in a zero-shot first pass . |
| Outcome: | The proposed framework outperforms ICL on biomedical NED datasets with zero human-annotation. |
Can Large Language Models Invent Algorithms to Improve Themselves? (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable performance improvements, but the methods for improving LLMs are still designed by humans. |
| Approach: | They propose a framework which enables LLMs to generate and learn model-improvement algorithms by the seed model. |
| Outcome: | The proposed framework outperforms human-designed methods in model-improving tasks and improves the seed model by 6% and outperformed human-design methods by 4.3% on GSM8k. |
Simulating Classroom Education with LLM-Empowered Agents (2025.naacl-long)
Copied to clipboard
Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianxiao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, Lei Hou, Juanzi Li
| Challenge: | Initial studies have focused on task-specific, independent LLM-empowered agents, but the potential of LLMs within a multi-agent collaborative framework for classroom simulation with real user participation remains unexplored. |
| Approach: | They propose a multi-agent classroom simulation teaching framework that recognizes representative class roles and introduces a novel class control mechanism for automatic classroom teaching. |
| Outcome: | The proposed framework can simulate dynamic learning environment for users with active teacher-student and student-studente interactions. |
A Grounded Typology of Word Classes (2025.naacl-long)
Copied to clipboard
| Challenge: | Using captioned images, we can quantify language function and semantics using a grounded typology approach . linguistic typology is the study of patterns and variation across the world's languages . |
| Approach: | They propose a grounded typology approach that uses images captioned across languages to quantify meaning and semantics. |
| Outcome: | The proposed approach can quantify language function and semantics using images captioned across languages. |
SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation (2025.naacl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) has been demonstrated effective in reducing the trainable parameter number when fine-tuning a large foundation model (LLM). |
| Approach: | They propose a low-rank adaptation approach that reduces the number of trainable parameters while enhancing model performance. |
| Outcome: | The proposed approach outperforms existing parameter-efficient fine-tuning methods while achieving substantial reductions in computational cost and memory requirements. |
LLM-guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating user satisfaction with dialogue systems face challenges due to limited understanding of underlying reasons for user dissatisfaction and high costs of annotating user intentions. |
| Approach: | They propose an interpretable framework for effective user satisfaction prediction . they propose to align utterances with strategies and large language models to retrieve relevant features from utterations. |
| Outcome: | The proposed framework achieves state-of-the-art performance on three benchmarks for the USE task. |
LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models excel in generating coherent and contextually rich outputs, but their capacity to handle long-form contexts is limited by fixed-length position embeddings. |
| Approach: | They propose a method that enables the efficient processing long-form sequences beyond the model’s length limit through recurrent compression without retraining the entire model. |
| Outcome: | The proposed method significantly improves LLM’s ability to manage extended contexts, making it well-suited for tasks that require both comprehensive context understanding and query relevance. |
A Template Is All You Meme (2025.naacl-long)
Copied to clipboard
| Challenge: | Templatic memes are a form of communication capable of succinctly conveying complicated messages. |
| Approach: | They propose a method to match memes to a knowledge base of 5,200 meme templates and 54,000 examples of template instances using a distance-based lookup. |
| Outcome: | The proposed method improves general meme knowledge and sample efficiency, leading to more robust models. |
LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs? (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have compared LLM-based augmentations with established methods, but the results are contradictory. |
| Approach: | They compare the performance of LLM-based augmentation methods with established ones . they found that LLMs are worthy of deployment only when very small number of seeds is used . |
| Outcome: | The proposed methods are worthy of deployment only when very small number of seeds is used. |
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
Self-Training Meets Consistency: Improving LLMs’ Reasoning with Consistency-Driven Rationale Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches labeled rationales that produce correct answers as appropriate for training but one measure risks misjudging rationale quality, leading models to learn flawed reasoning patterns. |
| Approach: | They propose a framework that evaluates rationales through follow-up questions and leverages this evaluation to guide its training. |
| Outcome: | The proposed framework improves robustness and correctness of rationales and reasoning abilities compared to previous self-training approaches. |
Evaluating Defeasible Reasoning in LLMs with DEFREASING (2025.naacl-long)
Copied to clipboard
| Challenge: | Defeasible inferences are highly plausible but can be impacted by new information. |
| Approach: | They construct a dataset to evaluate defeasible reasoning about property inheritance . they use generics to represent the inheritance rules because their semantics include exceptions . |
| Outcome: | The proposed model performs poorly across all pattern types and achieves 0.64 F 1 . the best performing model only achieves F 1 and the model is not well tuned . |
Evaluating Input Feature Explanations through a Unified Diagnostic Evaluation Framework (2025.naacl-long)
Copied to clipboard
| Challenge: | Input feature explanations reveal how a model makes decisions based on a specific input. |
| Approach: | They propose a framework that facilitates an automated comparison between highlight and interactive explanations comprised of four diagnostic properties. |
| Outcome: | The proposed framework compares highlight and interactive explanations across two datasets and two models and shows that interactive span explanations outperform other explanation types across most diagnostic properties. |
From Evidence to Belief: A Bayesian Epistemology Approach to Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on language models have focused on factual correctness and justification, but prior research has focused on the factual truth condition and justifier. |
| Approach: | They analyze language models’ responses and confidence using verbalized confidence, token probability, and sampling to examine their knowledge of Bayesian epistemology. |
| Outcome: | The language models that follow the Bayesian confirmation assumption with true evidence show varying performance depending on the degree of irrelevance, indicating they deviate from Bayes' assumptions. |
Private Synthetic Text Generation with Diffusion Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research shows diffusion models are capable of generating synthetics texts . but are they also good in generating private data if the training was under differential privacy? |
| Approach: | They examine whether diffusion models are capable of generating synthetics texts . they critically assess previous work on private text generation with diffusion models . |
| Outcome: | The proposed model outperforms auto-regressive models in generating private images despite unmet privacy assumptions . the proposed model is open-source and can be used for other purposes . |
Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling (2025.naacl-long)
Copied to clipboard
Yiwen Ding, Zhiheng Xi, Wei He, Lizhuoyuan Lizhuoyuan, Yitao Zhai, Shi Xiaowei, Xunliang Cai, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Large language models (LLMs) generate solutions themselves and iteratively train on filtered, high-quality rationales, but performance reaches a ceiling after a few iterations. |
| Approach: | They propose a strategy to improve the efficiency of sampling heavy-tailed data by using Socratic-style guidance signals to help LLMs reasoning with complex queries. |
| Outcome: | The proposed approach is effective on difficult queries and on held-out tasks, while requiring human supervision. |
FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant advances in every natural language processing task, but they are vulnerable to small perturbations in the inputs, raising concerns about their robustness in the real world. |
| Approach: | They propose a large-scale benchmark for extensive evaluation of LLMs in the fact verification domain covering 17 realistic word-level and character-level perturbations and 4 types of subpopulations. |
| Outcome: | The proposed model is brittle to small input changes and exhibits performance variations across different subpopulations. |
Analyzing Memorization in Large Language Models through the Lens of Model Attribution (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research has focused on extracting memorized content from LLMs or developing memorization metrics without exploring the underlying architectural factors that contribute to memorizing. |
| Approach: | They analyze how attention modules at different layers impact its memorization and generalization performance by using attribution techniques. |
| Outcome: | The proposed model can be used to mitigate memorization while keeping other components like layer normalization and MLP transformations intact. |
Track-SQL: Enhancing Generative Language Models with Dual-Extractive Modules for Schema and Context Tracking in Multi-turn Text-to-SQL (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to generative language models struggle to handle the increasing complexity of multi-turn Text-to-SQL tasks. |
| Approach: | They propose a framework which enhances generative language models with dual-extractive modules designed to track schema and contextual changes in multi-turn Text-to-SQL. |
| Outcome: | The proposed framework achieves state-of-the-art performance on SparC and CoSQL datasets and significantly improves execution accuracy in multi-turn interactions by 7.1% and 9.55%. |
Prototypical Extreme Multi-label Classification with a Dynamic Margin Loss (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work in XMC addresses this problem using deep encoders that project text descriptions to an embedding space suitable for recovering the closest labels. |
| Approach: | They propose a method that uses a shallow transformer encoder to combine text-based embeddings, label centroids and learnable free vectors to improve XMC efficiency. |
| Outcome: | The proposed method achieves state-of-the-art in several public benchmarks of different sizes and domains while keeping the model efficient. |
MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025.naacl-long)
Copied to clipboard
| Challenge: | Generating multiple-choice questions (MCQG) for professional exams is challenging due to outdated knowledge, hallucination issues, and prompt sensitivity. |
| Approach: | They propose a framework for converting medical cases into high-quality USMLE-style questions using a self-refine-based framework. |
| Outcome: | The proposed framework improves human expert satisfaction regarding quality and difficulty of medical questions. |
Main Predicate and Their Arguments as Explanation Signals For Intent Classification (2025.naacl-long)
Copied to clipboard
| Challenge: | Intent classification is crucial for conversational agents, and deep learning models perform well in this area due to the lack of suitable benchmark data. |
| Approach: | They propose a technique to augment text samples from intent classification datasets with word-level explanations by marking main predicates and their arguments as explanation signals. |
| Outcome: | The proposed method augments text samples from intent classification datasets with word-level explanations. |
Handling Missing Entities in Zero-Shot Named Entity Recognition: Integrated Recall and Retrieval Augmentation (2025.naacl-long)
Copied to clipboard
| Challenge: | Zero-shot Named Entity Recognition (ZS-NER) aims to recognize entities in unseen domains without specific annotated data. |
| Approach: | They propose a novel two-stage framework leveraging large language model techniques to improve the ZS-NER’s recall rate. |
| Outcome: | The proposed framework improves the ZS-NER’s recall rate and accuracy by incorporating a large language model. |
KMI: A Dataset of Korean Motivational Interviewing Dialogues for Psychotherapy (2025.naacl-long)
Copied to clipboard
| Challenge: | Motivational Interviewing (MI) is gaining attention as a theoretical basis for mental health chatbots. |
| Approach: | They propose a framework that simulates MI sessions enriched with the expertise of professional therapists by using large language models to generate utterances through prompt engineering. |
| Outcome: | The proposed framework simulates MI sessions enriched with the expertise of professional therapists and employs large language models to generate utterances through prompt engineering. |
Automatic Input Rewriting Improves Translation with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | LLMs can rewrite inputs but in machine translation, they are primarily used to re-write outputs via post-editing. |
| Approach: | They propose to use LLMs to rewrite inputs automatically to improve machine translation (MT) they propose to simplify inputs and use quality estimation to assess translatability. |
| Outcome: | The proposed methods can be improved by using quality estimation to assess translatability. |
HIGGS: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing large language models focus on breaking down the problem into layer-wise sub-problems and minimizing per-layer error, but this approach lacks theoretical justification and the metrics employed may be sub-optimal. |
| Approach: | They propose a "linearity theorem" establishing a direct relationship between the layer-wise reconstruction error and the model perplexity increase due to quantization. |
| Outcome: | The proposed method outperforms previous data-free methods and improves accuracy-compression trade-offs on Llama-family models. |
The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Units (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have revealed their potential to perform far more than language processing tasks, showcasing abilities in reasoning and problem-solving. |
| Approach: | They identify language-selective units within 18 popular LLMs using the same localization approach that is used in neuroscience. |
| Outcome: | The proposed method shows that language-selective units are more aligned to brain recordings from the human language system than random units. |
MixLLM: Dynamic Routing in Mixed Large Language Models (2025.naacl-long)
Copied to clipboard
Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, Haifeng Chen
| Challenge: | Large Language Models (LLMs) exhibit potential artificial generic intelligence, however, their usage is costly with high response latency. |
| Approach: | They develop a dynamic contextual-bandit-based routing system for query-LLM assignment that leverages query tags to enhance query embeddings. |
| Outcome: | The proposed model maximizes response quality and minimizes cost and latency. |
Continual Learning in Multilingual Sign Language Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite the low translation quality of sign language, many machine learning approaches are still in its infancy. |
| Approach: | They propose to use continual learning for mul- tilingual SLT to improve translation quality. |
| Outcome: | The proposed methods outperform baseline and fine-tuning approaches in sign language translation. |
Few-Shot Natural Language to First-Order Logic Translation via Code Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on translation of natural language to first-order logical formula (NL-FOL) but these methods face challenges such as inconsistency between training and inference phases and data-intensive finetuning process. |
| Approach: | They propose a method for translating natural language into first-order logical formulas using code snippets. |
| Outcome: | The proposed method surpasses training-free baselines and is comparable to supervised models trained on the full training data. |
How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent research has focused on literary machine translation (MT) but evaluation of literary MT remains an open problem. |
| Approach: | They propose a paragraph-level parallel corpus containing verified human translations and 13k evaluated sentences across four language pairs. |
| Outcome: | The proposed corpus compares human evaluations with students and professionals . it shows that the adequacy of human evaluation is controlled by two factors . |
PORT: Preference Optimization on Reasoning Traces (2025.naacl-long)
Copied to clipboard
| Challenge: | Preference optimization methods have been successfully applied to improve the alignment of large language models with human values. |
| Approach: | They propose to use preference optimization methods to generate rejected answers using weak LLM prompting and digit corruption to improve the mathematical reasoning abilities of language models. |
| Outcome: | The proposed method leads to increased accuracy on the GSM8K and AQuA-RAT benchmarks without annotations. |
Guiding Through Complexity: What Makes Good Supervision for Hard Reasoning Tasks? (2025.naacl-long)
Copied to clipboard
| Challenge: | Using weak teacher models to effectively supervise LLMs can improve performance on hard reasoning tasks. |
| Approach: | They propose two data-driven supervision strategies that offer supervision data at different quality levels upon tasks of varying complexity. |
| Outcome: | The proposed methods outperform "perfectly correct" supervision on harder subtasks even when the outcome error rate is high. |
Fine-Grained Transfer Learning for Harmful Content Detection through Label-Specific Soft Prompt Tuning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing detection models are less effective and generalizable due to static data. |
| Approach: | They propose a method that leverages class-specific knowledge to enhance harmful content detection. |
| Outcome: | The proposed method improves harmful content detection across English and German datasets. |
A Systematic Examination of Preference Learning through the Lens of Instruction-Following (2025.naacl-long)
Copied to clipboard
Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, Liang Tan
| Challenge: | a recent study has found that preference learning is a key tool for enhancing LLM training and alignment. |
| Approach: | They use a synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with 23 verifiable constraints to obtain preference pairs. |
| Outcome: | The proposed pipeline generates 48,000 unique instruction-following prompts with 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. |
Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Use (2025.naacl-long)
Copied to clipboard
Mohit Chandra, Siddharth Sriraman, Gaurav Verma, Harneet Singh Khanuja, Jose Suarez Campayo, Zihang Li, Michael L. Birnbaum, Munmun De Choudhury
| Challenge: | Adverse Drug Reactions (ADRs) from psychiatric medications are the leading cause of hospitalizations among mental health patients. |
| Approach: | They propose a benchmark and a framework to evaluate LLMs' ability to detect ADRs . they find that LLM responses are more complex and harder to read than experts . |
| Outcome: | The proposed framework evaluates LLMs' ability to detect and deliver expert-aligned mitigation strategies. |
Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision (2025.naacl-long)
Copied to clipboard
Zhouhang Xie, Tushar Khot, Bhavana Dalvi Mishra, Harshit Surana, Julian McAuley, Peter Clark, Bodhisattwa Prasad Majumder
| Challenge: | Instruction-following LLMs have recently allowed systems to discover hidden concepts from a collection of unstructured documents based on a natural language description of the purpose of the discovery (i.e., goal). |
| Approach: | They propose a goal-oriented latent factor discovery system that integrates LLM’s instruction-following ability with statistical models to handle large, noisy datasets where LLM reasoning alone falls short. |
| Outcome: | The proposed system improves task performance by 5-52% over baselines and 1.8 times as often as the best alternative, on average, in human evaluation. |
LLM-Supported Natural Language to Bash Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Using the natural language to Bash command (NL2SH) for command composition is difficult due to inaccurate test data and unreliable heuristics for determining the functional equivalence of Bash commands. |
| Approach: | They propose to use a heuristic to determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heurs. |
| Outcome: | The proposed heuristic can determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heurs. |
REL-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of large language models' ability to communicate uncertainty and knowledge limitations focus on the behaviors of their human interlocutors. |
| Approach: | They propose an interaction-centered evaluation approach that quantifies whether and how humans rely on LLMs' responses. |
| Outcome: | The proposed approach quantifies whether and how humans rely on LLMs' responses. |
Eliciting Critical Reasoning in Retrieval-Augmented Generation via Contrastive Explanations (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that LLMs struggle to critically analyse RAG-based in-context information. |
| Approach: | They propose a framework that elicits critical arguments in RAG via contrastive explanations . they propose CRAG to retrieve relevant documents given a query and generate explanations that explicitly contrast relevance of passages to support the final answer. |
| Outcome: | The proposed framework improves state-of-the-art RAG models while requiring significantly fewer prompts and demonstrations and robust to perturbations in the retrieved documents. |
A Distributional Perspective on Word Learning in Neural Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Language models are increasingly being studied as models of human language learners. |
| Approach: | They propose a distributional approach to word learning that captures distributional knowledge and gradient preferences for the word’s appropriateness. |
| Outcome: | The proposed signatures capture knowledge of where the target word can and cannot occur as well as gradient preferences about the word’s appropriateness. |
Disentangling language change: sparse autoencoders quantify the semantic evolution of indigeneity in French (2025.naacl-long)
Copied to clipboard
Jacob A. Matthews, Laurent Dubreuil, Imane Terhmina, Yunci Sun, Matthew Wilkens, Marten Van Schijndel
| Challenge: | Existing methods to measure semantic change with contextual word embeddings (CWEs) are not suitable for highly imbalanced datasets and pose challenges for interpretation. |
| Approach: | They propose an interpretable, feature-level approach to analyzing language change using k-sparse autoencoders to trace the semantic evolution of the term "indigène(s)" between 1825 and 1950. |
| Outcome: | The proposed approach can learn interpretable features from over 210,000 CWEs generated using sentences from the French National Library. |
Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods struggle to ensure semantic correctness and rely on simple or unrealistic datasets. |
| Approach: | They propose a benchmark to evaluate language models’ ability to generate PDDL code from natural language descriptions of planning tasks. |
| Outcome: | The proposed benchmark evaluates the ability of language models to generate PDDL code from natural language descriptions of planning tasks against ground truth and a dataset of 145,918 text-to-PDDL pairs with varying levels of difficulty. |
One fish, two fish, but not the whole sea: Alignment reduces language models’ conceptual diversity (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies suggest large language models can capture certain behavioral patterns, but there are ongoing debates as to whether they are valid replacements for human subjects. |
| Approach: | They propose to use large language models as replacements for humans in behavioral research by relating the internal variability of simulated individuals to the population-level variability. |
| Outcome: | The proposed model can capture human-like conceptual diversity, but it is unclear whether post-training alignment affects models’ internal diversity. |
Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to analyze online reviews for aspects of quality are limited . authors propose a method to disentangle the impact of each aspect on overall perception . |
| Approach: | They propose a method to disentangle the effect of each aspect on overall perception . they use textual mentions in reviews as proxies for real-world attributes . |
| Outcome: | The proposed method improves on real-world reviews of U.S. K-12 schools. |
Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate (2025.naacl-long)
Copied to clipboard
Xiaomeng Jin, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Mingyi Hong
| Challenge: | Existing methods to remove unwanted knowledge from large language models are formulated as minimizing memorization through the loss of the model. |
| Approach: | They propose a normalized gradient difference algorithm that optimizes a forgetting objective and an automatic learning rate scheduler that allows for better control over the trade-off between the objectives. |
| Outcome: | The proposed method improves on TOFU and MUSE datasets while exhibiting stable training. |
REFFLY: Melody-Constrained Lyrics Editing Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Automatic melody-to-lyric (M2L) generation aims to create lyrics that align with a given melody. |
| Approach: | They propose a framework for automatic melody-to-lyric generation that allows for a more flexible approach to creating lyrics from plain text. |
| Outcome: | The proposed framework outperforms baselines Lyra and GPT-4 in musicality and text quality. |
Exploring Safety-Utility Trade-Offs in Personalized Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior studies have shown that large language models can exhibit bias against specific demographic groups and engage in the generation of stereotypical responses. |
| Approach: | They propose a framework to evaluate LLM performance along two axes: safety and utility. |
| Outcome: | The proposed framework evaluates the performance of LLMs along two axes: safety and utility. |
MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing chart understanding benchmarks focus on single-chart tasks, neglecting multi-hop reasoning required to extract and integrate information from multiple charts. |
| Approach: | They propose a benchmark that evaluates MLLMs’ capabilities in four key areas: direct question answering, parallel question answering and comparative reasoning. |
| Outcome: | The proposed benchmark evaluates MLLMs’ capabilities in four key areas: direct question answering, parallel question answering and comparative reasoning. |
It Is Not Only the Negative that Deserves Attention! Understanding, Generation & Evaluation of (Positive) Moderation (2025.naacl-long)
Copied to clipboard
| Challenge: | Moderation is essential for maintaining and improving the quality of online discussions. |
| Approach: | They annotate a dataset on 13 modes of discussion and use it to generate positive moderation. |
| Outcome: | The proposed model shows that professional moderation generates higher ratings than professional moderated moderation, but prefers professional moderate in pairwise comparison. |
Social Norms in Cinema: A Cross-Cultural Analysis of Shame, Pride and Prejudice (2025.naacl-long)
Copied to clipboard
| Challenge: | We examine *how* and *why* shame and pride are expressed across cultures using a blend of psychology-informed language analysis combined with large language models. |
| Approach: | They introduce a cross-cultural dataset of over 10k shame/pride-related expressions with underlying social expectations from 5.4K Bollywood and Hollywood movies. |
| Outcome: | The results show that women are more sanctioned across cultures and for violating similar social expectations. |
The Stochastic Parrot on LLM’s Shoulder: A Summative Assessment of Physical Concept Understanding (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent years have witnessed remarkable advancements in large language models (LLMs) many researchers argue that LLMs may not * Equal contribution. |
| Approach: | They propose a task that summarises the memorization issue by using grid inputs that abstractly describe physical phenomena. |
| Outcome: | The proposed task alleviates the memorization issue by using grid-format inputs that abstractly describe physical phenomena. |
mHumanEval - A Multilingual Benchmark to Evaluate Large Language Models for Code Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Current evaluations focus on English-to-Python conversion tasks with limited test cases . code generation from low-resource language prompts remains largely unexplored . |
| Approach: | They propose a benchmark that supports prompts in over 200 natural languages . they provide expert human translations for 15 diverse natural languages (NLs) |
| Outcome: | The HumanEval Benchmark is the most widely used code generation benchmark . it provides expert human translations for 15 diverse natural languages . |
What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks. |
| Approach: | They propose a Gaussian-Noise-free pipeline for mechanistic interpretability in VLMs that introduces Semantic Image Pairs corruption, the first visual counterpart to Symmetric Token Replacement for text. |
| Outcome: | The proposed pipeline identifies a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities. |
Are explicit belief representations necessary? A comparison between Large Language Models and Bayesian probabilistic models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have indirect pragmatic capabilities, but their performance on Theory of Mind tasks is mixed. |
| Approach: | They propose to use Bayesian probabilistic model to make inferences about others' beliefs to predict human belief inference. |
| Outcome: | The proposed model outperforms the Rational Speech Act (RSA) framework in predicting human belief inferences, even though it does not explicitly encode belief representations. |
Self-Generated Critiques Boost Reward Modeling for Language Models (2025.naacl-long)
Copied to clipboard
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Rui Hou
| Challenge: | Existing reward models produce scalar scores and struggle to incorporate critiques in a natural language format. |
| Approach: | They propose a framework that predicts critiques and rewards using self-generated critiques without extra supervision. |
| Outcome: | The proposed framework improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges. |
Characterizing the Role of Similarity in the Property Inferences of Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Property inheritance is a phenomenon where novel properties are projected from higher level categories to lower level ones. |
| Approach: | They investigate how LMs perform property inheritance with behavioral and causal analysis experiments. |
| Outcome: | The results provide insight into the conceptual structure of language models and may suggest new psycholinguistic experiments for human subjects. |
SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains (2025.naacl-long)
Copied to clipboard
Ran Xu, Hui Liu, Sreyashi Nag, Zhenwei Dai, Yaochen Xie, Xianfeng Tang, Chen Luo, Yang Li, Joyce C. Ho, Carl Yang, Qi He
| Challenge: | Retrieval-augmented generation (RAG) enhances the question answering abilities of large language models (LLMs) however, adapting general-purpose RAG systems to specialized fields poses unique challenges due to distribution shifts and limited access to domain-specific data. |
| Approach: | They propose a method that equips large language models with joint capabilities of question answering and question generation for domain adaptation. |
| Outcome: | Experiments on 11 datasets across three different domains verify the efficacy of SimRAG over baselines by 1.2%–8.6%. |
Learning to Substitute Words with Model-based Score Ranking (2025.naacl-long)
Copied to clipboard
| Challenge: | Experimental results show that the proposed approach outperforms both masked language models and large language models. |
| Approach: | They propose a model-based scoring approach to quantify sentence quality . they propose 'loss function' that optimizes alignment between model predictions and sentence scores . |
| Outcome: | The proposed approach outperforms masked language models and large language models in the quantitative analysis of word substitutions. |
Multilingual Reasoning via Self-training (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have introduced eclectic strategies to improve reasoning beyond English, but these methods are related to specific language that is not always optimal for reasoning. |
| Approach: | They propose a modular approach that instructs models to structure reasoning passages in a different problem space and then self-refines their capabilities to deliver step-wise reasoning passage. |
| Outcome: | The proposed approach achieves significant improvements in multilingual reasoning of various models and task, with improved reasoning consistency across languages. |
xLAM: A Family of Large Action Models to Empower AI Agent Systems (2025.naacl-long)
Copied to clipboard
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Quoc Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Manoj Awalgaonkar, Rithesh R N, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | Autonomous agents powered by large language models (LLMs) have attracted significant research interest, but there are few standards for developing specialized models for agent tasks. |
| Approach: | They propose a series of large action models with dense and mixture-of-expert architectures that unifies, augments, and synthesizes diverse datasets to enhance agent generalizability and performance. |
| Outcome: | The proposed models outperform GPT-4, Claude-3, and many other models in terms of tool use and outperformed GPT-based models on multiple agent ability benchmarks. |
ProMQA: Question Answering Dataset for Multimodal Procedural Activity Understanding (2025.naacl-long)
Copied to clipboard
Kimihiro Hasegawa, Wiradee Imrattanatrai, Zhi-Qi Cheng, Masaki Asada, Susan Holm, Yuran Wang, Ken Fukuda, Teruko Mitamura
| Challenge: | Existing studies typically provide traditional, but less practical evaluation testbeds for multimodal systems. |
| Approach: | They propose a novel evaluation dataset, ProMQA, to measure the advancement of systems in application-oriented scenarios. |
| Outcome: | The proposed evaluation dataset reveals a significant gap between human and competitive multimodal models. |
Ethical Concern Identification in NLP: A Corpus of ACL Anthology Ethics Statements (2025.naacl-long)
Copied to clipboard
Antonia Karamolegkou, Sandrine Schiller Hansen, Ariadni Christopoulou, Filippos Stamatiou, Anne Lauscher, Anders Søgaard
| Challenge: | a survey shows that laypeople express different ethical concerns than professionals . acl-code-ethics provides a taxonomy for ethical concerns . |
| Approach: | They propose to annotate a corpus of ethical concern statements from scientific papers . they extract ethical concern keywords from the statements and automate the process . |
| Outcome: | The proposed corpus of ethical concern statements compares with existing taxonomies and guidelines pointing to gaps and actionable insights. |
AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing contrastive methods that ignore the context of a large language model (LLM) fail to handle instances that vary in their amount of conflict, with static methods over-adjusting when conflict is absent. |
| Approach: | They propose a fine-grained, instance-level approach called AdaCAD which dynamically adjusts the degree of conflict based on the degree. |
| Outcome: | The proposed approach outperforms baselines and improves factuality of summaries by 6.19. |
Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent-released MLLMs have shown remarkable performance on various multimodal math reasoning benchmarks. |
| Approach: | They introduce RoMMath, the first benchmark designed to evaluate the capabilities and robustness of multimodal large language models in handling multimodal math reasoning. |
| Outcome: | The proposed model performs well on a broad spectrum of 17 MLLMs and demonstrates that they are robust to adversarial perturbations. |
LBC: Language-Based-Classifier for Out-Of-Variable Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have great success in natural language processing tasks such as response generation, but their performance on tabular data tasks has been limited due to their inferior performance compared to traditional machine learning models (TMLs). |
| Approach: | They propose a Language-Based-Classifier (LBC) that maximizes the benefits of LLMs to outperform TMLs on OOV tasks. |
| Outcome: | The proposed model outperforms TMLs on OOV tasks by using three key methods. |
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities. |
| Approach: | They propose to use supervised fine-tuning and Quantized Low-Rank Adapters to improve LLMs' task-specific performance to address privacy and safety risks. |
| Outcome: | The proposed model improves the accuracy of the chain-of-thought reasonings across four datasets and demonstrates that the faithfulness of CoT reasoning decreases. |
InfoPO: On Mutual Information Maximization for Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
Teng Xiao, Zhen Ge, Sujay Sanghavi, Tian Wang, Julian Katz-Samuels, Marc Versage, Qingjun Cui, Trishul Chilimbi
| Challenge: | Recent studies have shown that direct preference optimization and its variants can be useful for fine-tuning large language models with human preferences data. |
| Approach: | They propose a preference fine-tuning algorithm that effectively and efficiently aligns large language models using preference data. |
| Outcome: | Extensive experiments show that the proposed algorithm outperforms established baselines on reasoning tasks. |
Is In-Context Learning a Type of Error-Driven Learning? Evidence from the Inverse Frequency Effect in Structural Priming (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent pre-trained large language models have shown the capacity to perform in-context learning (ICL) this capability could provide a way to bridge the divide between language models and humans. |
| Approach: | They propose a new way of diagnosing whether ICL is error-driven learning . they simulated structural priming with ICL and found the effect was stronger . |
| Outcome: | The proposed method is based on the inverse frequency effect (IFE) phenomenon is similar to error-driven learning in large language models . |
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
Analyzing and Improving Coherence of Large Language Models in Question Answering (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models suffer from instability or lack of coherence when receiving diverse input variations. |
| Approach: | They analyze the behavior of large language models when dealing with multiple lexical variations of the same info-seeking questions. |
| Outcome: | The proposed model generates equivalent outputs when receiving diverse input variations. |
ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Parity LLM Data Valuation (2025.naacl-long)
Copied to clipboard
Yanzhou Pan, Huawei Lin, Yide Ran, Jiamin Chen, Xiaodong Yu, Weijie Zhao, Denghui Zhang, Zhaozhuo Xu
| Challenge: | Large Language Models (LLMs) heavily rely on high-quality training data, making data valuation crucial for optimizing model performance. |
| Approach: | They propose a third-party data valuation approach that assesses the value of individual data samples and proposes a learning strategy to approximate LinFiK. |
| Outcome: | The proposed approach surpasses baselines in effectiveness and efficiency, showing significant scalability advantages as LLM parameters increase. |
E-Gen: Leveraging E-Graphs to Improve Continuous Representations of Symbolic Expressions (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for embedding mathematical expressions are limited by the size and diversity of training data. |
| Approach: | They propose an e-graph-based dataset generation scheme that synthesizes large and diverse datasets. |
| Outcome: | The proposed method outperforms state-of-the-art large language models on several tasks. |
Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech (2025.naacl-long)
Copied to clipboard
Eric Battenberg, RJ Skerry-Ryan, Daisy Stanton, Soroosh Mariooryad, Matt Shannon, Julian Salazar, David Teh-Hwa Kao
| Challenge: | Autoregressive (AR) Transformer-based sequence models have difficulty generalizing to sequences longer than those seen during training. |
| Approach: | They propose a system that provides cross-attention operations with relative location information. |
| Outcome: | The proposed system matches the naturalness and expressiveness of a baseline T5-based system while eliminating problems with repeated or dropped words. |
PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent efforts to improve the quality of machine-generated natural language content have been limited due to the large token usage required by complex evaluation prompts. |
| Approach: | They propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. |
| Outcome: | The proposed approach reduces token usage and costs by 2.37 compared with larger LLMs for downstream evaluation. |
AutoParLLM: GNN-guided Context Generation for Zero-Shot Code Parallelization using LLMs (2025.naacl-long)
Copied to clipboard
Quazi Ishtiaque Mahmud, Ali TehraniJamsaz, Hung D Phan, Le Chen, Mihai Capotă, Theodore L. Willke, Nesreen K. Ahmed, Ali Jannesari
| Challenge: | In-Context Learning (ICL) is a powerful technique to augment the capabilities of LLMs for a diverse range of tasks. |
| Approach: | They propose a way to generate context using guidance from graph neural networks to generate efficient parallel codes. |
| Outcome: | The proposed method improves state-of-the-art LLMs by 19.9% and 6.48% on NAS and rodinia benchmarks. |
Causally Modeling the Linguistic and Social Factors that Predict Email Response (2025.naacl-long)
Copied to clipboard
Yinuo Xu, Hong Chen, Sushrita Rakshit, Aparna Ananthasubramaniam, Omkar Yadav, Mingqian Zheng, Michael Jiang, Lechen Zhang, Bowen Yi, Kenan Alkiek, Abraham Israeli, Bangzhao Shu, Hua Shen, Jiaxin Pei, Haotian Zhang, Miriam Schirmer, David Jurgens
| Challenge: | a key intent behind many emails is to get a reply from the recipient. |
| Approach: | They propose to model the intents, expectations, and responsiveness in email exchanges by using a dataset containing 1800 emails annotated with nuanced types of intents and expectations. |
| Outcome: | The proposed model is based on 1800 emails annotated with nuanced types of intents and expectations . it shows that social status, argumentation, and strength of social connection influence email response rates . |
AI-LieDar : Examine the Trade-off Between Utility and Truthfulness in LLM Agents (2025.naacl-long)
Copied to clipboard
| Challenge: | LieDar is a framework to study how LLM-based agents navigate these scenarios in a multi-turn interactive setting. |
| Approach: | They propose a framework to study how LLM-based agents navigate these scenarios in an interactive multi-turn setting. |
| Outcome: | The proposed framework shows that all models are truthful less than 50% of the time, although truthfulness and goal achievement rates vary across models. |
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset (2025.naacl-long)
Copied to clipboard
| Challenge: | Learning from human feedback (LHF) has been used to mitigate the harms of large language models (LLMs) but the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. |
| Approach: | They audit the Helpful and Harmless (HH) dataset by Anthropic and examine how conceptualization failures and quality issues identified in the dataset can create additional harms . |
| Outcome: | The findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in large language models. |
FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions (2025.naacl-long)
Copied to clipboard
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, Luca Soldaini
| Challenge: | Modern language models (LMs) are capable of following long and complex instructions that enable a large and diverse set of user requests. |
| Approach: | They propose a dataset that contains an instruction evaluation benchmark and a training set to help IR models learn to follow instructions. |
| Outcome: | The proposed model improves after fine-tuning on a training set and rigorous instruction evaluation benchmark. |
Few-shot Personalization of LLMs with Mis-aligned Responses (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches have limited successes in personalizing large language models due to the lack of personalized learning or the reliance on shared personal data. |
| Approach: | They propose a few-shot personalization of large language models with mis-aligned responses using LLMs by learning a set of personalized prompts for each user based on user profile and examples of previous opinions. |
| Outcome: | The proposed method significantly improves performance across benchmarks compared to best-performing baselines. |
Prompting with Phonemes: Enhancing LLMs’ Multilinguality for Non-Latin Script Languages (2025.naacl-long)
Copied to clipboard
Hoang H Nguyen, Khyati Mahajan, Vikas Yadav, Julian Salazar, Philip S. Yu, Masoud Hashemi, Rishabh Maheshwary
| Challenge: | Multilingual LLMs have achieved remarkable benchmark performance, but continue to underperform on non-Latin script languages. |
| Approach: | They propose to integrate phonemic transcriptions as complementary signals to induce script-invariant representations by integrating phonemic and orthographic transcriptions. |
| Outcome: | The proposed approach improves performance for Latin and non-Latin script languages, with 12.6% performance improvement and 15.1% performance improvement compared to randomized ICL retrieval. |
SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models (2025.naacl-long)
Copied to clipboard
Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Doughman, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aimée Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Masoud, Nikita Nangia, Anaelia Ovalle, Giada Pistilli, Dragomir Radev, Beatrice Savoldi, Vipul Raheja, Jeremy Qin, Esther Ploeger, Arjun Subramonian, Kaustubh Dhole, Kaiser Sun, Amirbek Djanibekov, Jonibek Mansurov, Kayo Yin, Emilio Villa Cueva, Sagnik Mukherjee, Jerry Huang, Xudong Shen, Jay Gala, Hamdan Al-Ali, null Tair Djanibekov, Nurdaulet Mukhituly, Shangrui Nie, Shanya Sharma, Karolina Stanczak, Eliza Szczechla, Tiago Timponi Torrent, Deepak Tunuguntla, Marcelo Viridiano, Oskar Van Der Wal, Adina Yakefu, Aurélie Névéol, Mike Zhang, Sydney Zink, Zeerak Talat
| Challenge: | Large Language Models reproduce and exacerbate social biases present in training data, and resources to quantify this issue are limited. |
| Approach: | They propose a multilingual parallel dataset to examine culturally-specific stereotypes that may be learned by LLMs. |
| Outcome: | The proposed dataset includes stereotypes from 20 regions around the world and 16 languages, spanning multiple identity categories subject to discrimination worldwide. |
Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (2025.naacl-long)
Copied to clipboard
Jacob K Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto
| Challenge: | Existing methods to accelerate large language model inference are limited by the reliance on incremental token generation in existing draft models. |
| Approach: | They propose an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences and allows parallelization of both the drafting and verification steps. |
| Outcome: | The proposed approach provides 7.2x speedups over standard generation processes and 1.75x speed ups over existing speculative decoding approaches. |
Bayelemabaga: Creating Resources for Bambara NLP (2025.naacl-long)
Copied to clipboard
| Challenge: | a lack of well-structured multilingual datasets remains a challenge for machine translation in under-resource languages. |
| Approach: | They propose to create a multilingual dataset for machine translation in the Bambara language, the vehicular language of Mali. |
| Outcome: | The proposed dataset is the most extensive curated multilingual dataset for machine translation in the Bambara language, the vehicular language of Mali. |
Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) is a challenging task of extracting sentiments along with their corresponding aspects and opinion terms from text. |
| Approach: | They propose a pipeline that expands existing evaluation sets by adding alternative valid terms for aspect and opinion. |
| Outcome: | The proposed evaluation set uncovers the capabilities of large language models (LLMs) in ABSA tasks, which is concealed by the single-answer GT sets. |
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models (2025.naacl-long)
Copied to clipboard
Jianyu Liu, Hangyu Guo, Ranjie Duan, Xingyuan Bu, Yancheng He, Shilong Li, Hui Huang, Jiaheng Liu, Yucheng Wang, Chenchen Jing, Xingwei Qu, Xiao Zhang, Pei Wang, Yanan Wu, Jihao Gu, Yangguang Li, Jianke Zhu
| Challenge: | Multimodal Large Language Models (MLLMs) pose unique safety challenges due to their integration of visual and textual data. |
| Approach: | They propose a method to disentangle risks through step-by-step reasoning within multimodal inputs. |
| Outcome: | The proposed approach improves safety alignment in MLLMs by fine-tuning and iterative Reinforcement Learning from AI feedback. |
In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is limited by context length, but it can be used for many tasks. |
| Approach: | They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains . |
| Outcome: | The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context. |
Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to address inconsistencies in preference learning datasets rely on heuristics to achieve alignment. |
| Approach: | They propose a method that preprocesses annotated datasets by leveraging proxy models trained directly on them to detect and select consistent annotations. |
| Outcome: | The proposed method shows performance improvements of up to 33% across learning algorithms and proxy capabilities. |
TurtleBench: A Visual Programming Benchmark in Turtle Geometry (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) are capable of reasoning about geometric patterns, but they are still a challenge to evaluate. |
| Approach: | They propose a benchmark to evaluate LMMs’ ability to interpret geometric patterns and generate precise code outputs. |
| Outcome: | The benchmark compared LMMs with turtle geometry and GPT-4V, and found that few-shot prompting improved their performance (2%). |
Automatically Discovering How Misogyny is Framed on Social Media (2025.naacl-long)
Copied to clipboard
| Challenge: | a study on misogyny on social media reveals that there are many forms of misogamy, each addressing another MisogYny Problem (MP) the detection of the way misogony is framed is important for identifying misos . |
| Approach: | This paper considers the automatic discovery of misogyny problems and their frames through the Dis-MP&F method . it proposes a data-driven, rich Taxonomy of MisogYny (ToM) method that can be used to generate a misomy benchmark dataset. |
| Outcome: | The proposed method can generate a data-driven, rich Taxonomy of misogyny (ToM) and produces promising results on a misomyne benchmark dataset. |
Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation (2025.naacl-long)
Copied to clipboard
Mahnaz Koupaee, Jake W. Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, Kyu J. Han, Hang Su
| Challenge: | Existing approaches to evaluate faithfulness of summaries are often fooled by the fluency of the text and struggle with identifying errors. |
| Approach: | They propose an approach to summary faithfulness evaluation where multiple LLM-based agents are assigned initial stances and forced to come up with a reason to justify belief. |
| Outcome: | The proposed approach can identify ambiguities and have even stronger performance on non-ambiguous summaries. |
ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan
| Challenge: | Existing evaluations of large language models (LLMs) for instruction following are incomplete. |
| Approach: | They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets. |
| Outcome: | The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. |
Language Models Predict Empathy Gaps Between Social In-groups and Out-groups (2025.naacl-long)
Copied to clipboard
| Challenge: | Studies of human psychology have shown that people are more motivated to extend empathy to in-group members than out-group member. |
| Approach: | They propose to use language models to study intergroup empathy gap . they use a short description of an experience to predict emotion intensity . |
| Outcome: | The proposed model exhibited strongest intergroup bias among those tested. |
HARP: Hesitation-Aware Reframing in Transformer Inference Pass (2025.naacl-long)
Copied to clipboard
| Challenge: | a recent study has shown that inference steps are not equally challenging, with some being "harder" and others "easier." |
| Approach: | They propose a modified Transformer forward pass that selectively applies additional computation when the model encounters uncertainty during token generation. |
| Outcome: | The proposed method achieves performance gains while maintaining inference times twice faster than beam search. |
JAWAHER: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking (2025.naacl-long)
Copied to clipboard
Samar Mohamed Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Taher Abdelfadil, Shady Shehata, Muhammad Abdul-Mageed
| Challenge: | Recent advances in instruction fine-tuning and alignment methods have enhanced the adaptability of large language models to user preferences. |
| Approach: | They propose a benchmark to assess LLMs’ capacity to comprehend and interpret Arabic proverbs. |
| Outcome: | The proposed model can generate accurate translations, but struggle to produce culturally nuanced and contextually relevant explanations. |
EmojiPrompt: Generative Prompt Obfuscation for Privacy-Preserving Communication with Cloud-based LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have substantially expanded their applicability across diverse fields, such as personalized recommendations, health report analysis, and financial decision-making. |
| Approach: | They propose a generative transformation paradigm that obfuscates user data with linguistic and non-linguistic elements before submitting it to cloud-based LLMs. |
| Outcome: | The proposed paradigm obfuscates user private data while maintaining performance compared to the unobflated version. |
MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models that measure confidence in tool-using agents are poorly calibrated. |
| Approach: | They propose a class of model-internal confidence estimators to assess confidence when calling tools. |
| Outcome: | The proposed class beats baselines on a simulated trial and error (STE) tool-calling dataset using Llama3 models. |
PAT: Parameter-Free Audio-Text Aligner to Boost Zero-Shot Audio Classification (2025.naacl-long)
Copied to clipboard
| Challenge: | Audio-Language Models (ALMs) have demonstrated remarkable performance in zero-shot audio classification. |
| Approach: | They propose a training-free method that enhances audio and language representations using mutual feedback. |
| Outcome: | The proposed method outperforms vanilla zero-shot evaluation with significant margins of 0.42%-27.0%. |
Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of Large Language Models (LLMs) rely on a single large model to score outputs from other LLMs, but this is prone to intra-model bias and many tasks may be too subjective for a one model to judge fairly. |
| Approach: | They propose a language model council where a group of LLMs collaborate to create tests, respond to them, and evaluate each other’s responses to produce a ranking in a democratic fashion. |
| Outcome: | The proposed model produces rankings that are more separable and robust than any individual LLM judge. |
SCIURus: Shared Circuits for Interpretable Uncertainty Representations in Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for uncertainty quantification in large language models provide little insight into factors responsible for an uncertainty estimate, limiting their usefulness as practical tools for improving trustworthiness and understanding uncertainty reasoning. |
| Approach: | They adapt causal tracing and zero-ablation techniques to study the effect of different circuits on LLM generation to identify whether factuality of generated responses and uncertainty originate in separate or shared circuits. |
| Outcome: | The proposed methods use the well-established methods of causal tracing and zero-ablation to study the effect of different circuits on LLM generation. |
ProSE: Diffusion Priors for Speech Enhancement (2025.naacl-long)
Copied to clipboard
Sonal Kumar, Sreyan Ghosh, Utkarsh Tyagi, Anton Jeran Ratnarajah, Chandra Kiran Reddy Evuru, Ramani Duraiswami, Dinesh Manocha
| Challenge: | deterministic deep learning models have been used for speech enhancement, but generative models have shown promise. |
| Approach: | They propose a method to apply diffusion probabilistic models to speech enhancement using priors in a latent space. |
| Outcome: | The proposed method achieves state-of-the-art performance on synthetic and real-world datasets while consuming less computational costs. |
Mastering the Craft of Data Synthesis for CodeLLMs (2025.naacl-long)
Copied to clipboard
Meng Chen, Philip Arthur, Qianyu Feng, Cong Duy Vu Hoang, Yu-Heng Hong, Mahdi Kazemi Moghaddam, Omid Nezami, Duc Thien Nguyen, Gioacchino Tangari, Duy Vu, Thanh Vu, Mark Johnson, Krishnaram Kenthapadi, Don Dharmasiri, Long Duong, Yuan-Fang Li
| Challenge: | Large language models (LLMs) have shown impressive performance in code understanding and generation. |
| Approach: | They propose a systematic review of large language models and their taxonomy and propose specialized LLMs for code-related tasks. |
| Outcome: | The proposed models have shown to be highly effective in coding tasks. |
ParaICL: Towards Parallel In-Context Learning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to improve ICL performance are limited by the length of the input context. |
| Approach: | They propose a method that utilizes all demonstration examples without exceeding the manageable context length. |
| Outcome: | The proposed method can be scaled up to integrate with existing methods. |
CausalEval: Towards Better Causal Reasoning in Language Models (2025.naacl-long)
Copied to clipboard
Longxuan Yu, Delin Chen, Siheng Xiong, Qingyang Wu, Dawei Li, Zhikai Chen, Xiaoze Liu, Liangming Pan
| Challenge: | Large language models (LLMs) have been used for a variety of tasks, including problem-solving, decision-making, and understanding of the world. |
| Approach: | They propose a review of existing methods aimed at enhancing LMs for causal reasoning . they categorize existing methods as reasoning engines or as helpers providing knowledge or data to traditional methods . |
| Outcome: | The proposed methods perform better than existing methods on a range of tasks. |
Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defense (2025.naacl-long)
Copied to clipboard
Yang Ouyang, Hengrui Gu, Shuhang Lin, Wenyue Hua, Jie Peng, Bhavya Kailkhura, Meijun Gao, Tianlong Chen, Kaixiong Zhou
| Challenge: | Existing methods to defend against jailbreak attacks exploit vulnerabilities to elicit unintended or harmful outputs. |
| Approach: | They propose a method to defend against jailbreak attacks by patching specific layers within large language models through self-augmented datasets. |
| Outcome: | The proposed approach reduces harmfulness and attack success rate of jailbreak attacks without compromising utility for benign queries compared to previous methods. |
DeCAP: Context-Adaptive Prompt Generation for Debiasing Zero-shot Question Answering in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing zero-shot methods for Question Answering (QA) are efficient but fail to consider context and prevent bias propagation in the answers. |
| Approach: | They propose a method for debiasing Large Language Models using context-adaptive prompt generation that takes appropriate debiased actions based on the context and aNeutral Answer Guidance Generation to suppress the LLMs make objective judgments about the context. |
| Outcome: | The proposed method achieves state-of-the-art zero-shot debiased QA performance across eight LLMs. |
Reward-Guided Tree Search for Inference Time Alignment of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Inference-time computation methods enhance performance of Large Language Models by leveraging additional computational resources. |
| Approach: | They propose an inference-time alignment method that leverages a reward model to achieve alignment through reward-guided tree search. |
| Outcome: | The proposed method outperforms other inference-time alignment methods on two benchmarks . it achieves comparable performance to preference-tuned models on both benchmarks, authors show . |
Typographic Attacks in a Multi-Image Setting (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models are susceptible to typographic attacks, which are misclassifications caused by an attack text that is added to an image. |
| Approach: | They propose a multi-image setting for studying typographic attacks by leveraging the difficulty of the target image, the strength of the attack text, and text-image similarity. |
| Outcome: | The proposed approach improves success rates by 21% over random, non-specific methods on the CLIP model while maintaining stealth in a multi-image scenario. |
Tonguescape: Exploring Language Models Understanding of Vowel Articulation (2025.naacl-long)
Copied to clipboard
| Challenge: | a study shows that language models can explain vowel pronunciation based on tongue positions . a visual LM can explain the relationship between vowels and tongue positions, but it is unclear whether they align textual information with visual information. |
| Approach: | They created video and image datasets from MRI data to examine if LMs associate real tongue positions with vowel articulation. |
| Outcome: | The proposed model can explain vowel pronunciation and the correlation between vowels and tongue positions as textual knowledge. |
CoRAC: Integrating Selective API Document Retrieval with Question Semantic Intent for Code Question Answering (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing automated code question answering methods provide accurate and relevant answers to questions about code. |
| Approach: | They propose a knowledge-based framework that generates precise code question answers by analyzing code snippets. |
| Outcome: | The proposed framework generates high-quality answers compared to large language models, such as ChatGPT. |
Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models are typically optimized for resource-rich languages like English . however, the proprietary nature of these models makes them impractical for many researchers and developers. |
| Approach: | They propose to develop large language models that can follow instructions in Basque . they focus on three key stages: pre-training, instruction tuning, and alignment with human preferences . |
| Outcome: | The proposed models improve natural language understanding (NLU) of the foundational model by 12 points . the results show that the models can follow instructions in Basque with human preferences . |
How to Make LLMs Forget: On Reversing In-Context Knowledge Edits (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context knowledge editing (IKE) is an efficient and efficient knowledge editing method (Zheng et al., 2022b; Gangadhar and Stratos, 2024) it can be misused to manipulate responses opaquely, e.g., insert misinformation or offensive content. |
| Approach: | They propose to detect and reverse IKE-edits using only the top-10 output probabilities of the next token, even in a black-box setting. |
| Outcome: | The proposed method can be detected with high accuracy even in a black-box setting, achieving over 80% accuracy in recovering original, unedited outputs across multiple LLMs. |
PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian (2025.naacl-long)
Copied to clipboard
Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh, Azadeh Shakery, Mohammad Taher Pilehvar
| Challenge: | Large language models predominantly reflect Western cultures due to the dominance of English-centric training data. |
| Approach: | They propose a dataset to assess the sensitivity of LLMs to Persian culture. |
| Outcome: | The proposed model shows a 11.3% gap between best closed-source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. |
Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are recognized for their exceptional generative capabilities and versatility across various tasks. |
| Approach: | They conduct a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks to determine the impact of different models, tasks, prompts, and system-related factors on inference. |
| Outcome: | The proposed model energy benchmarks show that quantization and optimal batch sizes can significantly reduce energy usage. |
CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated significant advances in computer science research . current agent-related applications include code writing, code base generation, code correction and more. |
| Approach: | They propose a benchmark to assess the effectiveness of Large Language Models (LLMs) they propose GitHub agent framework that automates deployment of GitHub repositories . |
| Outcome: | The proposed framework improves the deployment of GitHub code repositories and thereby boosts developer productivity. |
SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data (2025.naacl-long)
Copied to clipboard
| Challenge: | In many natural language processing tasks, model training often leads to spurious correlations . shortcuts allow models to rely on irrelevant patterns in the data, leading to biased predictions. |
| Approach: | They propose a method to generate structure-aware positive and negative sentences using tagging. |
| Outcome: | The proposed method improves model robustness and generalization across different environments while minimizing spurious correlations. |
Rationale-Guided Retrieval Augmented Generation for Medical Question Answering (2025.naacl-long)
Copied to clipboard
Jiwoong Sohn, Yein Park, Chanwoong Yoon, Sihyeon Park, Hyeon Hwang, Mujeen Sung, Hyunjae Kim, Jaewoo Kang
| Challenge: | Large language models (LLMs) struggle with hallucinations and outdated knowledge. |
| Approach: | They propose a retrieval-augmented generation framework for enhancing the reliability of RAG in biomedical contexts. |
| Outcome: | The proposed framework outperforms the previous best medical RAG model by up to 5.6% across three medical question-answering benchmarks. |
Prototype Conditioned Generative Replay for Continual Learning in NLP (2025.naacl-long)
Copied to clipboard
| Challenge: | Generative replay methods that rely on a single task-specific token or prompt often fail to generate pseudo-samples that accurately reflect the true data distribution. |
| Approach: | They propose a Prototype Conditioned Generative Replay method which incorporates task-level statistics into a prototyping process. |
| Outcome: | The proposed method outperforms state-of-the-art (SOTA) methods on two different scenarios. |
KODIS: A Multicultural Dispute Resolution Dialogue Corpus (2025.naacl-long)
Copied to clipboard
| Challenge: | KODIS is a dyadic dispute resolution corpus containing thousands of dialogues from over 75 countries. |
| Approach: | They propose to use a dyadic dispute resolution corpus to examine how conflicts escalate through conversation rather than deal-making. |
| Outcome: | The proposed corpus contains thousands of dialogues from over 75 countries. |