Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Towards Automated Error Discovery: A Study in Conversational AI (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that LLMs require information about the nature of an error or hints about its occurrence for accurate detection. |
| Approach: | They propose an encoder-based approach to detect and define errors in conversational AI. |
| Outcome: | The proposed framework outperforms baselines across multiple error-annotated dialogue datasets and shows strong generalization to unknown intent detection. |
Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models . however, these methods are constrained and lack nuanced and accurate evaluations based on specific cultural proxies. |
| Approach: | They propose to use the World Values Survey and Hofstede Cultural Dimensions as case studies to examine cultural alignment in Large Language Models. |
| Outcome: | The findings advocate for more robust evaluation frameworks that focus on cultural proxies. |
Biased Tales: Cultural and Topic Bias in Generating Children’s Stories (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalized stories are often preferred because they reflect a child's interests, experiences, and developmental needs. |
| Approach: | They analyze a dataset to examine how biases influence protagonists’ attributes and story elements in LLM-generated stories. |
| Outcome: | The proposed dataset shows that gender stereotypes influence protagonist attributes and story elements in LLM-generated stories. |
Large Language Models as Realistic Microservice Trace Generators (2025.emnlp-main)
Copied to clipboard
| Challenge: | Obtaining real-world traces is difficult due to limited public data availability and the difficulty of collecting them at large scale from diverse environments. |
| Approach: | They propose to train a large language model to generate microservice call graphs using a recursive approach to capture hierarchical structures and implicit constraints in such traces. |
| Outcome: | The proposed method outperforms existing methods in accuracy and validity. |
JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for text simplification focus on only one dimension: fluency, simplicity and meaning preservation. |
| Approach: | They introduce a dataset to assess legal meaning preservation between two legal texts . they also introduce sanity checks for two identical sentences . |
| Outcome: | The proposed metric shows superior correlation with human judgment compared to existing metrics. |
QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large and Transformer-based language models perform outstandingly in various downstream tasks, but there is limited understanding regarding how these models internalize linguistic knowledge. |
| Approach: | They propose to use a binary acceptability judgments dataset to benchmark seven language models using a standard binary acceptibility judgments framework. |
| Outcome: | The proposed dataset shows that on average, a fine-tuned Transformer-based LM outperforms other methods and that pre-trained cross-lingual LLMs do not acquire linguistic judgment capabilities during their pre-training for Quebec French. |
Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing value probing methods that capture in-context information and predict models’ real-world actions are limited and lack systematic comparisons. |
| Approach: | They compare three widely used value probing methods: token likelihood, sequence perplexity, and text generation. |
| Outcome: | The proposed methods exhibit large variances under non-semantic perturbations in prompts and option formats, with sequence perplexity being the most robust overall. |
A Systematic Analysis of Base Model Choice for Reward Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) and reward modeling are key to training powerful large language models (LLMs). |
| Approach: | They propose to combine RLHF and reward modeling to boost model selection . they also demonstrate that a small set of benchmarks could be combined to boost the model selection. |
| Outcome: | The results show that the model selection can be improved by up to 14% compared to the most common (default) choice. |
Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | specialised small models outperform general large models with few labelled samples (on average 100) performance variance is taken into account when comparing the number of labelled sample required to tune a specialised model with a larger number of samples. |
| Approach: | They find that specialised small models need only few labelled samples to outperform general large models with limited labelled data. |
| Outcome: | The proposed models outperform general large models with few labelled samples and take performance variance into account. |
Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | In many domains, determining faithfulness of a claim to a source document is a binary judgment . but, whether a document is factual or whether it is entailed given some input is highly subjective. |
| Approach: | They propose a task to manage the subjectivity involved with factuality judgments of ambiguous claims. |
| Outcome: | The proposed method improves the annotator agreement on faithfulness of a claim by 21%. |
MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating the pedagogical capabilities of AI-based tutoring models is critical for guided progress in the field. |
| Approach: | They propose an open-source benchmark for holistic tutoring model evaluation. |
| Outcome: | The proposed model can discriminate between expert and novice teachers with high accuracy. |
Preemptive Detection and Correction of Misaligned Actions in LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized human-AI collaboration by enabling autonomous agents to execute complex, multi-step tasks. |
| Approach: | They propose a method that leverages the belief reasoning ability of LLMs to detect misaligned actions. |
| Outcome: | Experiments on three widely used tasks show that InferAct outperforms other methods on Marco-F1 and emnlp2025. |
Fingerprinting LLMs through Survey Item Factor Correlation: A Case Study on Humor Style Questionnaire (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating LLMs focus on output accuracy, faithfulness, or alignment with human preferences, but these metrics do not capture fundamental differences in how models internally represent and relate psychological constructs. |
| Approach: | They propose to “fingerprint” LLMs through factor correlation patterns on standardized psychological assessments to deepen understanding of LLM's constructs representation. |
| Outcome: | The proposed method shows that LLMs represent constructs differently than humans . it also shows that no LLM recovers the constructs of the Humor Style Questionnaire . |
Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large-scale visionlanguage pre-training framework is limited by the scarcity of large-sized annotated vision-language data . noise-resistant data construction pipeline is needed to filter and caption web-sourced images . noisy text tokens can be a problem for fine-grained representation learning . |
| Approach: | They develop a noise-resistant data construction pipeline that leverages in-context learning capabilities of MLLMs to automatically filter and caption web-sourced images. |
| Outcome: | The proposed framework improves cross-modal alignment by masking noisy textual tokens based on the gradient-attention similarity score. |
From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are often optimized for direct question-answering, but their effectiveness is often undermined by strategically withholding answers. |
| Approach: | They propose an online reinforcement learning-based alignment framework that can quickly adapt LLMs into effective tutors using simulated student-tutor interactions. |
| Outcome: | The proposed model outperforms proprietary models like LearnLM and can be used to enhance interpretability and pedagogical quality. |
CompKBQA: Component-wise Task Decomposition for Knowledge Base Question Answering (2025.emnlp-main)
Copied to clipboard
Yuhang Tian, Dandan Song, Zhijing Wu, Pan Yang, Changzhi Zhou, Jun Yang, Hao Wang, Huipeng Ma, Chenhao Li, Luan Zhang
| Challenge: | Existing knowledge base question answering methods struggle with complex queries. |
| Approach: | They propose a framework that optimizes the process of fine-tuning a LLM for generating logical forms by enabling it to learn relevant sub-tasks like skeleton generation, topic entity generation, and relevant relations generation. |
| Outcome: | The proposed framework achieves state-of-the-art on two benchmark KBQA datasets, WebQSP and CWQ. |
Permutative Preference Alignment from Listwise Ranking of Human Judgments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to align Large Language Models with human preferences are based on the Bradley-Terry model, but when multiple responses are available, the B-T model fails to guarantee an accurate list ranking of the responses. |
| Approach: | They propose an offline listwise approach that incorporates the Normalized Discounted Cumulative Gain (NDCG) as an alternative training objective for LLM alignment. |
| Outcome: | The proposed approach outperforms existing pairwise and listwise methods on evaluation sets and general benchmarks such as AlpacaEval. |
ToneCraft: Cantonese Lyrics Generation with Harmony of Tones and Pitches (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current research has not addressed the challenge of generating harmonious Cantonese lyrics. |
| Approach: | They propose a framework for generating Cantonese lyrics that ensures tonal and melodic harmony. |
| Outcome: | The proposed framework ensures tonal and melodic harmony while preserving character count and quality. |
SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | SensorLLM is a timeseries classification framework that can perform human activity recognition tasks. |
| Approach: | They propose a framework that enables Large Language Models to perform human activity recognition from sensor time-series data. |
| Outcome: | The proposed framework can perform human activity recognition (HAR) tasks with human inputs. |
MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to update model-based indexes with new documents are expensive and require expensive retraining. |
| Approach: | They propose a framework that combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution-driven expansion strategy. |
| Outcome: | Experiments on NQ320k and MS MARCO Passage show that the proposed framework outperforms full-model update baselines with minimal parameter overhead and substantially lower training costs. |
ViClaim: A Multilingual Multilabel Dataset for Automatic Claim Detection in Videos (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing efforts in misinformation detection focus on written text, leaving a significant gap in addressing the complexity of spoken text in video transcripts. |
| Approach: | They propose to annotate video transcripts in three languages and six topics using a custom annotation tool. |
| Outcome: | The proposed tool shows strong cross-validation performance but challenges for generalization to unseen topics. |
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) with web search capabilities show significant potential for deep research. |
| Approach: | They introduce a framework for end-to-end training of LLM-based deep research agents . they implement a specialized multi-agent architecture where browsing agents extract relevant information from various webpage structures. |
| Outcome: | The proposed framework improves on open-domain research tasks by 28.9 points over prompt engineering and 7.2 points over RAG-based RL agents. |
Mixture of Length and Pruning Experts for Knowledge Graphs Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing graph neural networks (GNNs) adopt rigid, query-agnostic path-exploration strategies limiting their ability to adapt to diverse linguistic contexts and semantic nuances. |
| Approach: | They propose a mixture-of-experts framework that personalizes path exploration . framework uses length experts that adaptively selects and weights candidate paths . it also uses pruning experts that evaluates candidate path from a complementary perspective . |
| Outcome: | The proposed framework shows superior performance on a diverse benchmark . it uses a mixture of experts that weights and selects path lengths according to query complexity . |
MPRF: Interpretable Stance Detection through Multi-Path Reasoning Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing stance detection methods treat the task as a classification problem, where models output a stance label without providing interpretable reasoning paths. |
| Approach: | They propose a framework that generates, evaluates, and integrates multiple reasoning paths to improve accuracy, robustness, and transparency in stance detection. |
| Outcome: | The proposed framework outperforms existing models on the SEM16, VAST, and PStance datasets and is highly interpretable and reliable. |
Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels (2025.emnlp-main)
Copied to clipboard
Junjie Ye, Yuming Yang, Yang Nan, Shuo Li, Qi Zhang, Tao Gui, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan
| Challenge: | Large language models (LLMs) acquire substantial world knowledge during pretraining, which is further shaped by post-training techniques such as supervised fine-tuning (SFT). |
| Approach: | They evaluate closed-book question answering (CBQA) performance across five LLMs from the LLaMA-2 and LLama-3 families and examine the impact of supervised fine-tuning on model knowledge. |
| Outcome: | The proposed model performance is 14% worse than models fine-tuned on 1,920 samples and 12% worse on 240 samples. |
JI2S: Joint Influence‐Aware Instruction Data Selection for Efficient Fine‐Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior selection strategies score samples using generalpurpose LLMs, leveraging their strong language understanding but introducing inherent biases that misalign with the target model’s behavior and yield unstable downstream performance. |
| Approach: | They propose a framework that jointly models marginal and combinatorial influences within sample groups and evaluate them on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons. |
| Outcome: | The proposed framework outperforms fulldataset training and strong baselines on Open LLM Benchmarks, MTBench, and GPT4–judged pairwise comparisons. |
SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models (2025.emnlp-main)
Copied to clipboard
Xingjian Diao, Chunhui Zhang, Keyi Kong, Weiyi Wu, Chiyu Ma, Zhongyu Ouyang, Peijun Qing, Soroush Vosoughi, Jiang Gui
| Challenge: | Recent large language models have demonstrated impressive reasoning abilities, but their extension to the audio modality remains underexplored. |
| Approach: | They propose a rule-based reinforcement learning algorithm to equip LALMs with robust reasoning capabilities. |
| Outcome: | The proposed algorithm improves on the SoundMind benchmark. |
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for converting visual tokens into tokens are limited by their high volume . et al., 2023; Zheng e.t., 2023): a revolution in video understanding. |
| Approach: | They propose a language-aware dynamic token compression system that converts video clips into soft caption tokens as visual representations. |
| Outcome: | The proposed method reduces FLOPs by 49% while maintaining competitive performance. |
RoT: Enhancing Table Reasoning with Iterative Row-Wise Traversals (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in reasoning large language models (RLLMs) have significantly enhanced reasoning capabilities, leading to brilliant performance on table reasoning. |
| Approach: | They propose a method which performs iterative row-wise table traversal, allowing for reasoning extension and reflection-based refinement at each traversal. |
| Outcome: | Experiments show that the proposed method outperforms RLLMs on WikiTableQuestions and TableBench by 4.3% and achieves state-of-the-art results with comparable models. |
T-MAD: Target-driven Multimodal Alignment for Stance Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Multimodal Stance Detection struggle with generalizing to unseen targets and handling modality inconsistencies. |
| Approach: | They propose a multimodal stability detection model which captures target-specific relationships and balances modality contributions by iterative reasoning. |
| Outcome: | Experiments on the MMSD and MultiClimate datasets show that the proposed model outperforms state-of-the-art models with optimal results achieved using RoBERTa, ViT, and an iterative depth of 5. |
Emotion Transfer with Enhanced Prototype for Unseen Emotion Recognition in Conversation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on emotion recognition in conversation does not reach a consensus on classification theories . despite this, there is no clear consensus on how to recognize previously unseen emotions in real-world applications. |
| Approach: | They propose a prototype-based emotion transfer framework that can be used in real-world applications. |
| Outcome: | The proposed framework shows promise but still faces key challenges in the field of emotion recognition in conversation. |
PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization (2025.emnlp-main)
Copied to clipboard
Ruoxi Cheng, Yizhong Ding, Shuirong Cao, Ranjie Duan, Xiaoshuang Jia, Shaowei Yuan, Simeng Qin, Zhiqiang Wang, Xiaojun Jia
| Challenge: | Existing methods to jailbreak Large Vision Language Models do not consider interaction between images and text. |
| Approach: | They propose a prior-guided bimodal interactive black-box jailbreak attack for toxicity maximization that exploits the interaction of images and text. |
| Outcome: | The proposed method outperforms state-of-the-art jailbreak methods in black box scenarios and in closed-source LVLMs. |
Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Utility-based retrieval has emerged as a promising topic for downstream tasks . however, capturing passage utility accurately remains unexplored due to insufficient understanding . |
| Approach: | They propose a framework for training utility-based retrievers in Retrieval-Augmented Language Models . it incorporates multi-task generalization and inter-passage interaction to improve performance . |
| Outcome: | The proposed framework improves performance on ten datasets across different tasks. |
SportReason: Evaluating Retrieval-Augmented Reasoning across Tables and Text for Sports Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for retrieval-augmented reasoning on numerical sports questions focus on one or two evidence units. |
| Approach: | They propose a benchmark for retrieval-augmented reasoning on numerical sports questions . they evaluate existing retrievers and rerankers, along with agentic Retrieval-Augmented Generation systems. |
| Outcome: | The proposed benchmarks focus on the sports domain because it offers rich multi-modal resources. |
MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models produce non-existing facts when faced with questions outside their parametric knowledge, which undermines their reliability. |
| Approach: | They propose a method that separates the learning of answer prediction and confidence estimation during fine-tuning on instruction data. |
| Outcome: | Experiments on multiple models and different model sizes show that the proposed method outperforms baselines by up to 25% in average precision. |
CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior implicit CoT methods have underperformed in terms of efficiency and robustness by relying on natural language tokens for reasoning. |
| Approach: | They propose a training framework that compresses natural language CoT into continuous space by aligning hidden states of a designated token. |
| Outcome: | The proposed framework outperforms the existing state-of-the-art in 3.1x compression rate and 28.2% accuracy on GSM8k scale. |
PAFT: Prompt-Agnostic Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prompt-agnostic fine-tuning (PAFT) improves performance by reducing overfitting to specific prompts. |
| Approach: | They propose a method that enhances robustness through dynamic prompt variation during training. |
| Outcome: | The proposed method achieves higher generalization accuracy on unseen prompts than standard methods with similar training efficiency. |
Theorem-Validated Reverse Chain-of-Thought Problem Generation for Geometric Reasoning (2025.emnlp-main)
Copied to clipboard
Deng Linger, Linghao Zhu, Yuliang Liu, Yu Wang, Qunyi Xie, Jingjing Wu, Gang Zhang, Yingying Zhu, Xiang Bai
| Challenge: | Existing methods for generating geometric reasoning data through Chain-of-Thought (CoT) frameworks face three fundamental limitations: 1) lack of high-quality annotations and domain-specific expertise to ensure theorem-grounded diagrams. 2) lack of a coherent model; 3) lack of coherent model. |
| Approach: | They propose a two-stage Theorem-Validated Reverse Chain-of-Thought Reasoning Synthesis framework that synthesizes theorematic diagrams with structured descriptions and properties. |
| Outcome: | The proposed framework expands theorem-type coverage, corrects misunderstandings, and enhances geometric reasoning. |
TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal in-context learning (ICL) is a key mechanism for harnessing the capabilities of large vision–language models. |
| Approach: | They propose a transformer-based model with task-aware attention that dynamically configures ICL sequences. |
| Outcome: | Experiments on five LVLMs and nine datasets show that TACO surpasses baselines across diverse ICL tasks. |
Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) has advanced from generating natural-sounding speech to enabling fine-grained control over speech attributes. |
| Approach: | They provide a review of controllable TTS methods from traditional control techniques to emerging approaches using natural language prompts. |
| Outcome: | The proposed methods are based on models, strategies, and features, and summarize challenges, datasets, and evaluations. |
Automating Steering for Safe Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, especially when faced with adversarial multimodal inputs. |
| Approach: | They propose a modular and adaptive inference-time intervention technology, AutoSteer, that integrates a safety awareness score, an adaptive safety prober, and a lightweight Refusal Head to modulate generation when safety risks are detected. |
| Outcome: | Experiments on LLaVA-OV and Chameleon show that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats while maintaining general abilities. |
EMNLP: Educator-role Moral and Normative Large Language Models Profiling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks for evaluating the ethical and moral alignment of large language models (LLMs) in educational AI are lacking. |
| Approach: | They propose a framework for a teacher-role moral and normative LLMs profiling . they extend existing scales and construct 88 teacher-specific moral dilemmas . |
| Outcome: | The proposed framework evaluates compliance and vulnerability of teacher-role LLMs under soft prompt injection. |
TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evidence-based summarization tasks require tracing source evidence to assess their accuracy. |
| Approach: | They propose a benchmark for traceable, aspect-based summarization that pairs summaries with sentence-level citations to enable users to trace back to the original context. |
| Outcome: | The proposed benchmark can be used to evaluate document summarization with LLMs and human evaluations. |
Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
Wenbin Hu, Haoran Li, Huihao Jing, Qi Hu, Ziqian Zeng, Sirui Han, Xu Heli, Tianshu Chu, Peizhao Hu, Yangqiu Song
| Challenge: | Current mitigation strategies fail to preserve contextual reasoning capabilities in risky scenarios, leading to systemic risks for legal compliance. |
| Approach: | They propose to use reinforcement learning with a rule-based reward to incentivize contextual reasoning capabilities while enhancing compliance with safety and privacy norms. |
| Outcome: | The proposed model outperforms Qwen2.5-7B-Instruct model in safety and privacy benchmarks and achieves +8.58% accuracy improvement. |
Towards General-Domain Word Sense Disambiguation: Distilling Large Language Model into Compact Disambiguator (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Word Sense Disambiguation rely heavily on manually annotated data, which limits coverage and generalization. |
| Approach: | They propose a framework that leverages large language models as knowledge distillers to build silver-standard WSD corpora by combining generation-based distillation and annotation-based disambiguation. |
| Outcome: | The proposed framework outperforms existing methods on general-domain benchmarks by 50% on the most challenging test set and by 1000 times fewer parameters. |
SLoW: Select Low-frequency Words! Automatic Dictionary Selection for Translation on Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models only support hundreds of languages, and they are usually limited in English. |
| Approach: | They propose a task to automatically select which dictionary to use to enhance translation . they call it Select Low-frequency Words!, which inherits advantage of dictionary-based methods . |
| Outcome: | The proposed method can save tokens and improve translation performance on 100 languages. |
Parallel Continuous Chain-of-Thought with Jacobi Iteration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to continuous CoT rely on sequential decoding of latent thought tokens, which leads to long training time and low inference speed. |
| Approach: | They propose a parallel continuous chain-of-thought which updates latent thought tokens iteratively in parallel instead of sequentially and improves both training and inference efficiency. |
| Outcome: | The proposed method saves 50% of training and inference time while maintaining stability and robustness in training. |
EQA-RM: A Generative Embodied Reward Model with Test-time Scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing generic Reward Models are ill-equipped for dynamic and interactive domains. |
| Approach: | They propose a novel generative multimodal reward model specifically architected for EQA that provides interpretable, structured reward feedback. |
| Outcome: | The proposed model outperforms proprietary benchmarks, including Gemini-2.5-Flash, GPT-4o, Claude-3.5-Haiku, and open-sourced state-of-the-art models such as RoVRM and VisualPRM. |
Refusal-Aware Red Teaming: Exposing Inconsistency in Safety Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require rigorous safety evaluations to be effective. |
| Approach: | They propose a red teaming framework that detects internal model refusals and contrasts them with judgments from an external safety evaluator to generate test cases that expose such discrepancies. |
| Outcome: | The proposed framework outperforms existing reinforcement learning-based approaches in generating diverse test cases and achieves a substantially higher discovery rate of refusal gaps. |
OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinking (2025.emnlp-main)
Copied to clipboard
Zekun Xi, Wenbiao Yin, Jizhan Fang, Jialong Wu, Runnan Fang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated remarkable progress in machine writing such as open domain long-form generation. |
| Approach: | They propose a slow-thinking machine writing framework that emulates the human-like process of iterative expansion and reflection to improve the knowledge density of generated articles. |
| Outcome: | The proposed framework improves the knowledge density of generated articles without compromising metrics such as coherence and depth. |
LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Text-to-SQL models fail to address schema linking problems in large-scale multi-database environments. |
| Approach: | They propose a framework that aims to enable non-expert users to retrieve data effortlessly . they highlight four core errors leading to schema linking failures . |
| Outcome: | The proposed framework outperforms baselines on all schema linking metrics. |
On Relation-Specific Neurons in Large Language Models (2025.emnlp-main)
Copied to clipboard
Yihong Liu, Runsheng Chen, Lea Hirlimann, Ahmad Dawar Hakimi, Mingyang Wang, Amir Hossein Kargaran, Sascha Rothe, François Yvon, Hinrich Schuetze
| Challenge: | In large language models, certain neurons can store distinct pieces of knowledge learned during pretraining. |
| Approach: | They hypothesize that relation-specific neurons detect relation in input text and guide generation involving such a relation. |
| Outcome: | The proposed model can handle facts involving relation r and facts containing a different relation . |
IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting Indirect Prompt Injection (IPI) attacks rely on assumptions about the model's inherent security, which lacks structural constraints on agent behaviors. |
| Approach: | They propose a novel task execution paradigm that models the agents’ task execution process as a traversal over a planned Tool Dependency Graph (TDG). |
| Outcome: | The proposed model reduces unintended tool invocations triggered by injected instructions, enhancing robustness against IPI attacks. |
ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is increasingly used in diverse applications where models must provide accurate answers and explanations that humans can easily understand and verify. |
| Approach: | They propose a unified prototypical framework that learns question-aware prototypes that serve as reasoning anchors and applies spatially constrained matching to ensure that the selected evidence is coherent and semantically relevant. |
| Outcome: | The proposed framework yields faithful, fine-grained explanations while maintaining competitive accuracy. |
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs (2025.emnlp-main)
Copied to clipboard
Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao
| Challenge: | Multimodal Large Language Models (MLLMs) integrate visual and textual inputs, yet modality alignment remains one of the most challenging aspects. |
| Approach: | They propose a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. |
| Outcome: | The proposed method improves performance across various model sizes, with smaller models benefiting the most. |
Molecular String Representation Preferences in Pretrained LLMs: A Comparative Study in Zero- & Few-Shot Molecular Property Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Molecular property prediction plays a crucial role in medicinal chemistry . traditional machine learning approaches do not involve natural language . |
| Approach: | They compare performance of four state-of-the-art LLMs on molecular property prediction tasks . they find statistically significant zero- and few-shot preferences for InChI and IUPAC names . |
| Outcome: | The proposed model outperforms the current model on molecular property prediction tasks . the model's representation preferences are based on representation granularity, tokenization and prevalence in pretraining corpora . |
Weight-Aware Activation Sparsity with Constrained Bayesian Optimization Scheduling for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing activation sparsification methods rely on activation magnitude and weights for sparsity . authors propose a weight-aware activation-a-ware framework for large language models . |
| Approach: | They propose a weight-aware activation sparsity framework that uses weight-based scoring to measure activation importance in sparsification and a custom GPU sparse kernel to support it. |
| Outcome: | The proposed framework outperforms existing methods at 60% model-level sparsity and significantly outperfies them at higher sparsities. |
DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language model (LLM) agents for data science automation are limited by narrow task scopes, limited generalization across tasks and models, and over-reliance on state-of-the-art (SOTA) LLMs. |
| Approach: | They propose a notebook-centric LLM agent framework for adaptive and robust data science automation. |
| Outcome: | The proposed framework surpasses baselines such as AutoGen and TaskWeaver in performance tests across diverse data science scenarios and models. |
VC4VG: Optimizing Video Captions for Text-to-Video Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in text-to-video generation highlight the critical role of high-quality video-text pairs in training models capable of producing coherent and instruction-aligned videos. |
| Approach: | They propose a caption optimization framework tailored to the needs of T2V models. |
| Outcome: | The proposed framework improves video caption quality and video generation performance. |
LaMP-QA: A Benchmark for Personalized Long-form Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalization in question answering systems remains underexplored due to lack of resources . a new benchmark for personalized answer generation is being developed . |
| Approach: | They propose a LaMP-QA benchmark for evaluating personalized answer generation . they compare evaluation strategies for generating personalized responses and measure their alignment with human preferences. |
| Outcome: | The proposed benchmark shows that incorporating the personalized context provided leads to performance improvements. |
The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for difficulty estimation rely on repeated response sampling, auxiliary models, or fine-tuning the target model itself. |
| Approach: | They propose a method that leverages only the hidden representations produced by large language models. |
| Outcome: | The proposed method outperforms baselines in difficulty estimation on textual and multimodal tasks and improves adaptive reasoning strategies with fewer generated tokens. |
MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol (2025.emnlp-main)
Copied to clipboard
| Challenge: | Model Context Protocol (MCP) introduces an easy-to-use ecosystem for users and developers, but it also brings underexplored safety risks. |
| Approach: | They propose a framework that addresses the missing safety mechanisms in MCP and a taxonomy that captures diverse range of unsafe behaviors observed in MMP scenarios. |
| Outcome: | The proposed framework improves safety performance on state-of-the-art LLMs by capturing unsafe behaviors and analyzing the results. |
SAKI-RAG: Mitigating Context Fragmentation in Long-Document RAG via Sentence-level Attention Knowledge Integration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Traditional Retrieval-Augmented Generation (RAG) frameworks segment documents into larger chunks to preserve contextual coherence . however, such chunking methods lead to fragmented contexts, isolated chunk semantics, and broken inter-chunk relationships . |
| Approach: | They propose a framework that maintains granular chunks while recovering their intrinsic semantic connections. |
| Outcome: | The proposed framework achieves better recall and precision compared to other RAG frameworks in long-document retrieval scenarios. |
Skeletons Matter: Dynamic Data Augmentation for Text-to-Query (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on a single query language, resulting in limited generalizability . a new task paradigm is proposed to unify semantic parsing tasks across different query languages . |
| Approach: | They propose a task paradigm that unifies parsing tasks across query languages . they identify query skeletons as a shared optimization target of Text-to-Query tasks . |
| Outcome: | The proposed method achieves state-of-the-art performance using only a small amount of synthesized data. |
CondenseLM: LLMs-driven Text Dataset Condensation via Reward Matching (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for dataset condensation struggle to compress more information into samples . current methods struggle to extract enough training data for effective learning . |
| Approach: | They propose a paradigm for dataset condensation that uses an LLMs-driven approach to generate more informative and less biased samples. |
| Outcome: | The proposed method outperforms coreset selection and existing condensation methods by large margins while significantly reducing the computational cost. |
MovieCORE: COgnitive REasoning in Movies (2025.emnlp-main)
Copied to clipboard
Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh, Ying Cheng, Hung-Ting Su, Yung-Hao Tang, Shang-Hong Lai, Winston H. Hsu
| Challenge: | MovieCORE is a video question answering dataset that focuses on surface-level comprehension. |
| Approach: | They propose a video question-answer dataset that uses large language models as thought agents to generate and refine high-quality question-anchor pairs. |
| Outcome: | The proposed model improves model reasoning capabilities post-training by 25% . the proposed model is based on a large language model and is scalable to a wide range of tasks . |
Think Wider, Detect Sharper: Reinforced Reference Coverage for Document-Level Self-Contradiction Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to document-level contradiction detection (DSCD) only gain marginal improvement and often introduce inconsistencies across repeated responses. |
| Approach: | They propose a method that combines supervised fine-tuning and reinforcement learning to enhance document-level contradiction detection (DSCD) they propose to use a task-specific reward function to expand the model’s reasoning scope, boosting both accuracy and consistency. |
| Outcome: | The proposed method significantly boosts Llama 3.1-8B-Instruct’s accuracy from 38.5% to 51.1%, and consistency from 59.6% to76.2%. |
DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models’ Understanding on Indian Culture (2025.emnlp-main)
Copied to clipboard
Arijit Maji, Raghvendra Kumar, Akash Ghosh, null Anushka, Nemil Shah, Abhilekh Borah, Vanshika Shah, Nishant Mishra, Sriparna Saha
| Challenge: | DRISHTIKON is a first-of-its-kind multimodal and multilingual benchmark centered exclusively on Indian culture. |
| Approach: | They evaluate a wide range of vision-language models across zero-shot and chain-of-thought settings and use them to evaluate cultural understanding of generative AI systems. |
| Outcome: | The DRISHTIKON dataset covers 15 languages, all states and union territories, and incorporating over 64,000 aligned text-image pairs. |
LingGym: How Far Are LLMs from Thinking Like Field Linguists? (2025.emnlp-main)
Copied to clipboard
| Challenge: | LINGGYM is a benchmark that evaluates LLMs’ capacity for meta-linguistic reasoning using Interlinear Glossed Text and grammatical descriptions extracted from 18 typologically diverse reference grammars. |
| Approach: | They propose a benchmark that evaluates LLMs’ capacity for meta-linguistic reasoning using Interlinear Glossed Text and grammatical descriptions extracted from 18 typologically diverse reference grammars. |
| Outcome: | The proposed model can generalize linguistic inference across low-resource languages and structures not seen during training. |
Learning from Few Samples: A Novel Approach for High-Quality Malcode Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Intrusion detection systems (IDS) are limited in labeled samples due to scarcity and lack of diversity in malicious samples. |
| Approach: | They propose a semi-supervised framework that integrates Generative Adversarial Networks with Large Language Models to enhance malicious code generation and SQL Injection detection capabilities. |
| Outcome: | The proposed framework enhances malicious code generation and detection capabilities in few-sample learning scenarios. |
Personality Matters: User Traits Predict LLM Preferences in Multi-Turn Collaborative Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly integrated into everyday workflows . a recent study found that LLMs exhibit distinct personality-like traits that affect user engagement . |
| Approach: | They evaluated 32 LLM users for four collaborative tasks and found significant preferences . they found that rationalists preferred GPT-4, while idealists favored Claude 3.5 . |
| Outcome: | The results show that users with different personality traits prefer certain LLMs over others. |
VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models struggle with reasoning-focused tasks due to the lack of high-quality training data. |
| Approach: | They propose a new approach that leverages search engines to create a multimodal multimodal dataset . they use a set of 30,000 seed images to extract HTML data from 700K unique URLs . |
| Outcome: | The proposed model achieves the best known performance on MMMU-Pro (40.7), MathVerse (42.6), and DynaMath (55.7). |
Thinking Out Loud: Do Reasoning Models Know When They’re Right? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large reasoning models (LRMs) have recently demonstrated impressive capabilities in complex reasoning tasks by leveraging increased test-time computation and exhibiting behaviors reminiscent of human-like self-reflection. |
| Approach: | They analyze verbalized confidence, how models articulate their certainty, as a lens into the nature of self-reflection in large reasoning models. |
| Outcome: | The proposed model exhibits human-like self-reflection in reasoning tasks, but how this ability interacts with other model behaviors remains underexplored. |
Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to quantify uncertainty are limited in vision-language models . however, current models display notable miscalibration across diverse tasks and settings . |
| Approach: | They evaluate verbalized confidence in vision-language models using visual reasoning . they propose a prompting strategy that improves confidence alignment in multimodal settings . |
| Outcome: | The proposed method improves confidence alignment across multimodal settings. |
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. |
| Approach: | They propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. |
| Outcome: | The proposed model improves response precision while preserving exploratory ability to uncover potential correct pathways. |
LLM Bias Detection and Mitigation through the Lens of Desired Distributions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on bias mitigation has focused on promoting social equality and demographic parity, but less attention has been given to aligning LLM’s outputs to desired distributions. |
| Approach: | They propose a weighted adaptive loss based fine-tuning method that aligns LLM’s gender–profession output distribution with the desired distribution while preserving language modeling capability. |
| Outcome: | The proposed method achieves near-complete mitigation under equality and 30–75% reduction under real-world settings. |
MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and Retrieval-augmented Generation (RAG) systems show promise, but their performance on cross-document MEQA remains underexplored due to the lack of tailored benchmarks. |
| Approach: | They propose a scalable multi-document, multi-entity benchmark to evaluate LLMs' capacity to retrieve, consolidate, and reason over scattered and dense information. |
| Outcome: | The proposed benchmarks show that even advanced models achieve only 59% accuracy on MEBench. |
POSITION BIAS MITIGATES POSITION BIAS: Mitigate Position Bias Through Inter-Position Knowledge Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Positional bias (PB) manifests as non-uniform sensitivity across contextual locations . previous studies have addressed PB by modifying the underlying architectures or employing extensive contextual awareness training. |
| Approach: | They propose a position-to-position knowledge distillation framework that leverages position-induced disparities to counteract PB. |
| Outcome: | The proposed framework reduces positional bias and improves performance on retrieval and reasoning tasks. |
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (2025.emnlp-main)
Copied to clipboard
Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, Felix Juefei-Xu, Foutse Khomh, Osamu Yoshie, Qingyu Chen, Douglas Teodoro, Nan Liu, Randy Goebel, Lei Ma, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
| Challenge: | Existing large language model evaluation benchmarks focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. |
| Approach: | They propose a comprehensive benchmark covering 29 languages, built on an English benchmark. |
| Outcome: | The MMLU-ProX is a comprehensive benchmark covering 29 languages, built on an English benchmark. |
NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debugging (2025.emnlp-main)
Copied to clipboard
Weiming Zhang, Qingyao Li, Xinyi Dai, Jizheng Chen, Kounianhua Du, Weiwen Liu, Yasheng Wang, Ruiming Tang, Yong Yu, Weinan Zhang
| Challenge: | Early debugging efforts focused on code-level analysis, which often fails when addressing complex programming errors. |
| Approach: | They propose a framework that employs natural language as an intermediate representation to improve code debugging by debuggating at a natural language level. |
| Outcome: | The proposed framework outperforms traditional debugging methods and enables a broader modification space through direct refinement guided by execution feedback. |
Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can struggle to balance gullibility to misinformation and resistance to valid corrections in persuasive dialogues. |
| Approach: | They propose a framework evaluating multi-turn stance-change dynamics across dual dimensions: persuasion type and domain. |
| Outcome: | The proposed framework improves LLM-3.1-8B-Instruct accuracy under misleading persuasion in safety contexts from 4.21% to 76.54%. |
POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion (2025.emnlp-main)
Copied to clipboard
Yuan Liu, Zhongyin Zhao, Le Tian, Haicheng Wang, Xubing Ye, Yangxiu You, Zilin Yu, Chuhan Wu, Zhou Xiao, Yang Yu, Jie Zhou
| Challenge: | Existing approaches to training document conversion models with manual annotation are costly and time-consuming, and training student models by distilling outputs from teacher models can significantly limit their performance in real-world applications. |
| Approach: | They propose a fully automated framework for constructing high-quality document extraction datasets and models capable of handling diverse document formats and layouts. |
| Outcome: | The proposed model outperforms existing models and improves on annotated documents. |
Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a promising solution to automate literature review writing tasks. |
| Approach: | They propose a framework to automatically evaluate the performance of large language models in three key tasks of literature review writing: reference generation, abstract writing, and literature review composition. |
| Outcome: | The proposed framework assesses the hallucination rates in generated references and measures the semantic coverage and factual consistency of the literature summaries and compositions against human-written counterparts. |
CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely adopted for a diverse range of tasks, from highly skilled professionals to non-technical individuals. |
| Approach: | They propose a suite of lightweight adversarial attacks that allow LLMs to reveal harmful behavior during conversations. |
| Outcome: | The proposed model can recover from fabricated bias claim and reject biased follow-up questions. |
From Schema to State: Zero-Shot Scheme-Only Dialogue State Tracking via Diverse Synthetic Dialogue and Step-by-Step Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research classifies zero-shot, scheme-only DST into two main types: the cross-domain scenario and the zero-schemaonly setting. |
| Approach: | They propose a zero-shot, scheme-only approach that generates synthetic dialogues that balance diversity with schema alignment and distills knowledge from a large language model into a smaller model. |
| Outcome: | The proposed approach achieves state-of-the-art performance under zero-shot, scheme-only situation and generalizes effectively to few-shot scenarios. |
Beyond the Surface: Measuring Self-Preference in LLM Judgments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods measure self-preference bias by comparing the scores a judge model assigns to its own responses with those assigned to other models. |
| Approach: | They propose to use gold judgments as proxies for the actual quality of responses . they propose to measure self-preference bias as the difference between the judge model's own and other models' scores . |
| Outcome: | The proposed method can assess self-preference bias across large language models . it uses gold judgments as proxies for the ground truth scores of the judge model . |
Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) have recently emerged as powerful tools for interpreting and steering the internal representations of large language models (LLMs). |
| Approach: | They propose a method that identifies the most influential latents by incorporating output-side gradient information. |
| Outcome: | The proposed method identifies the most influential latents by incorporating output-side gradient information. |
Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
Hengran Zhang, Minghao Tang, Keping Bi, Jiafeng Guo, Shihao Liu, Daiting Shi, Dawei Yin, Xueqi Cheng
| Challenge: | Existing studies on large language models for document utility annotations have shown that they improve retrieval performance and RAG outcomes compared to models trained on human annotations. |
| Approach: | They propose a model that maximizes their summed marginal likelihood to annotate document utility on multiple positive samples per query. |
| Outcome: | The proposed model maximizes the marginal likelihood of multiple positive samples per query. |
CiteBART: Learning to Generate Citations for Local Citation Recommendation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Local citation recommendation (LCR) suggests a set of papers for a citation placeholder in a given context. |
| Approach: | They introduce a citation-specific pre-training framework where author-date citation tokens are masked to learn to reconstruct them to fulfill local citation recommendation (LCR). |
| Outcome: | The proposed model achieves state-of-the-art performance on LCR benchmarks except for the FullTextPeerRead dataset, which is quite small to see the advantage of generative pre-training. |
Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable potential in assisting with mathematical reasoning on different downstream tasks. |
| Approach: | They propose two new tools for autoformalizing real-world mathematical definitions from Wikipedia and arXiv papers. |
| Outcome: | The proposed methods improve definitions by up to 16% and undefined errors by 43%. |
Culture Cartography: Mapping the Landscape of Cultural Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can empower users to be more knowledgeable, productive, and creative, but their utility is often diminished for under-represented groups and cultures. |
| Approach: | They propose a methodology that operationalizes a mixed-initiative approach to finding culture-specific knowledge that is salient to in-group users but unknown to LLMs. |
| Outcome: | The proposed method improves the accuracy of LLMs on culturally-competent language models by 19.2%. |
Interpretability Analysis of Arithmetic In-Context Learning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) solve arithmetic with only a few in-context examples, yet the computations that connect those examples to the answer remain opaque. |
| Approach: | They propose to use in-context examples to illustrate how large language models process ICEs to isolate partial-sum representations in three-operand tasks and investigate their influence on final logits. |
| Outcome: | The proposed model performs better than previous models on three-operand tasks. |
SwarmAgentic: Towards Fully Automated Agentic System Generation via Swarm Intelligence (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing agentic system generation frameworks lack autonomy, autonomy, and functionality . current frameworks are too rigid, limiting adaptability and scalability. |
| Approach: | They propose a framework that fully automates agentic system generation, optimization, and collaboration . they construct agents from scratch and jointly refine functionality and coordination . |
| Outcome: | The proposed framework outperforms ADAS on six real-world, open-ended, and exploratory tasks on the TravelPlanner benchmark. |
We Politely Insist: Your LLM Must Learn the Persian Art of Taarof (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines the cultural competence of large language models to navigate complex social interactions. |
| Approach: | They propose a benchmark to evaluate LLM understanding of Persian taarof . they evaluate 450 role-play scenarios covering 12 common social interaction topics . their results show significant gaps in cultural competence . |
| Outcome: | The proposed model improves on Persian *taarof*, a social norm in Iranian interactions . the model improve despite culturally specific responses and gender-based asymmetries . |
Unstructured Evidence Attribution for Long Context Query Focused Summarization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems struggle to copy and properly cite unstructured evidence, which also tends to be “lost-in-the-middle”. |
| Approach: | They propose to extract unstructured evidence spans to improve the trustworthiness of large language models by citing unstructure . they propose to use this dataset as a training supervision for unstructure-based evidence summarization. |
| Outcome: | The proposed pipeline generates more relevant and factually consistent evidence than baselines with no fine-tuning and fixed granularity evidence. |
RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal question answering often requires identifying which video, audio, or sensor tokens are relevant to the question. off-camera speech, background noise, or motion outside the field of view often mislead fusion models that weight all streams equally. |
| Approach: | They propose a unified architecture for multimodal question answering that assigns scalar relevance scores to each token across modalities. |
| Outcome: | The proposed model outperforms state-of-the-art multimodal large language models on seven multi-modal QA benchmarks and egocentric and exocentric tasks. |
Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
Mingyuan Wu, Jize Jiang, Haozhen Zheng, Meitang Li, Zhaoheng Li, Beitong Tian, Bo Chen, Yongjoo Park, Minjia Zhang, ChengXiang Zhai, Klara Nahrstedt
| Challenge: | Recent Vision Language Models (VLMs) have shown tremendous promise in a wide range of realworld applications, but their size has made at-scale deployment and operation challenging due to high consumption of cloud computing resource, high latency, and expensive API calls. |
| Approach: | They propose a master–apprentice framework for collaborative inference between large and small vision language models. |
| Outcome: | The proposed framework improves reasoning performance on widely-recognized and challenging general reasoning benchmarks and specifically boosts reasoning of apprentice VLMs by 36.6%. |
Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that Video Large Language Models (Vide-oLLMs) are efficient at video understanding but lack the quadratic complexity of visual tokens. |
| Approach: | They propose a plug-and-play inference acceleration framework for VideoLLM token compression that quantifies each frame’s uniqueness and adaptively adjusts compression intensity across frames. |
| Outcome: | Extensive experiments on video large language models and benchmarks show that the proposed framework can preserve essential information while reducing redundancy in video sequences. |
Router-Tuning: A Simple and Effective Approach for Dynamic Depth (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve computational efficiency are under-explored and face several critical challenges. |
| Approach: | They propose a method that selectively activates only a subset of the model's layers, skipping those deemed less important. |
| Outcome: | The proposed method significantly improves performance on Attention layers and MoE layers while reducing redundant computation and memory usage. |
Foot-In-The-Door: A Multi-turn Jailbreak for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly integrated into real-world applications, requiring a high level of safety and alignment. |
| Approach: | They propose a multi-turn jailbreak method that leverages foot-in-the-door principles to escalate malicious intent of user queries through intermediate bridge prompts and aligns the model’s response by itself to induce toxic responses. |
| Outcome: | The proposed method achieves an average attack success rate of 94% across seven widely used models outperforming existing state-of-the-art methods. |
TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games (2025.emnlp-main)
Copied to clipboard
| Challenge: | evaluating large language models' reasoning abilities via detective stories is often infeasible due to the large answer space and diverse reasoning types presented by its questions. |
| Approach: | They propose a framework and dataset for evaluating the deductive reasoning abilities of Large Language Models (LLMs) by leveraging the interactive gameplay of detective games Ace Attorney and Danganronpa. |
| Outcome: | The proposed framework and dataset are based on the detective games Ace Attorney and Danganronpa and show that they are more efficient than current strategies for enhancing deductive reasoning. |
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show superior cross-model transferability . Prompt injection attacks are among the most critical threats . |
| Approach: | They propose an activations-guided prompt injection attack framework to address the impracticality of existing white-box/gray-box methods and the poor transferability of black-box approaches. |
| Outcome: | The proposed framework achieves 49.6% success rate and 34.6% improvement over human-crafted prompts on five mainstream LLMs. |
Direct Judgement Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing judge models are largely trained with supervised finetuning on small data scales to perform limited types of evaluation tasks, limiting generalization. |
| Approach: | They propose to train judge models at large data scales with direct preference optimization . they use four training tasks to form three types of preference pairs targeting different aspects of evaluation . |
| Outcome: | The proposed model outperforms GPT-4o and other similar models on 13 benchmarks. |
WebInject: Prompt Injection Attack to Web Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite advanced capabilities of MLLMs, prompt injection attacks remain vulnerable to emerging security and safety threats. |
| Approach: | They propose a prompt injection attack that manipulates webpage environment to induce web agent to perform attacker-specified action. |
| Outcome: | The proposed attack outperforms baselines on multiple datasets and is highly effective. |
F²Bench: An Open-ended Fairness Evaluation Benchmark for LLMs with Factuality Considerations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing fairness evaluation benchmarks for large language models rely on closed-ended evaluation formats that overlook factuality considerations rooted in historical, social, physiological, and cultural contexts. |
| Approach: | They propose an open-ended fairness evaluation benchmark for large language models . they incorporate factuality considerations and multi-turn reasoning into the benchmark . |
| Outcome: | The proposed benchmark incorporates factual grounding and text generation to better reflect the complexities of real-world model usage. |
Value Profiles for Encoding Human Variation (2025.emnlp-main)
Copied to clipboard
Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel A. Bakker, Georgina Evans, Iason Gabriel, Noah Goodman, Verena Rieser
| Challenge: | Using value profiles and a steerable decoder model to estimate ratings is crucial for personalization, pluralistic model alignment, and computational social science. |
| Approach: | They propose to represent individuals using value profiles and a steerable decoder model to estimate ratings conditioned on a value profile or other rater information. |
| Outcome: | The proposed model interpretably changes ratings according to semantic profile differences and is well-calibrated. |
Language Models as Causal Effect Generators (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using sequence-driven structural causal models (SD-SCMs) we characterize how SD-SCAMs enables sampling from observational, interventional, and counterfactual distributions according to the desired causal structure. |
| Approach: | They propose a sequence-driven structural causal model that uses language models to parameterize a structural causal system based on a user-specified DAG. |
| Outcome: | The proposed method outperforms state-of-the-art methods and can underpin auditing of language models for (un)desirable causal effects, such as misinformation or discrimination. |
Constructions are Revealed in Word Distributions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Construction grammar posits that constructions are form-meaning pairings that are acquired through experience with language. |
| Approach: | They propose to use a RoBERTa model to model constructions as patterns of statistical affinity . they show that statistical affinity is likely an important, but partial, signal available to learners . |
| Outcome: | The proposed model shows that constructions will be revealed as patterns of statistical affinity . the proposed model is based on a model that is able to distinguish constructions from text . |
CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are limited by their narrow language pairs and tasks, failing to adequately assess their code-mixing abilities. |
| Approach: | They propose a benchmark to assess large language models' (LLMs) code-mixing abilities that covers eight tasks and 18 languages from seven language families. |
| Outcome: | The proposed method combines word substitution with GPT-4 prompting to generate large-scale synthetic code-mixed texts. |
RBPtool: A Deep Language Model Framework for Multi-Resolution RBP-RNA Binding Prediction and RNA Molecule Design (2025.emnlp-main)
Copied to clipboard
Jiyue Jiang, Yitao Xu, Zikang Wang, Yihan Ye, Yanruisheng Shao, Yuheng Shan, Jiuming Wang, Xiaodan Fan, Jiao Yuan, Yu Li
| Challenge: | RNA-binding proteins play key roles in post-transcriptional gene regulation . existing methods focus on shallow sequence features or coarse structural representations . large language models allow for precise modeling and biologically informed de novo RNA design . |
| Approach: | They extend RPI15223 into a multi-resolution, structure-level RBP-RNA dataset and introduce RBPtool, a framework that fuses sequence and structural information. |
| Outcome: | The proposed framework achieves state-of-the-art performance on public benchmarks and the RPI15223 dataset while supporting fine-grained level predictions. |
Unveiling Internal Reasoning Modes in LLMs: A Deep Dive into Latent Reasoning vs. Factual Shortcuts with Attribute Rate Ratio (2025.emnlp-main)
Copied to clipboard
Yiran Yang, Haifeng Sun, Jingyu Wang, Qi Qi, Zirui Zhuang, Huazheng Wang, Pengfei Ren, Jing Wang, Jianxin Liao
| Challenge: | Existing research in multi-hop questions has identified two reasoning modes, but has not investigated how these modes differ during inference. |
| Approach: | They propose a classification metric that compares latent reasoning and factual shortcuts in multi-hop questions. |
| Outcome: | The proposed metric achieves 90% accuracy on the proposed datasets and demonstrates effectiveness in RAG conflict scenarios. |
SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities in natural language understanding and generation, but controlling their behavior remains a challenge. |
| Approach: | They propose a supervised steering approach that operates in sparse, interpretable representation spaces. |
| Outcome: | The proposed approach achieves higher success rates with minimal degradation in generation quality compared to existing methods. |
BabyLM’s First Constructions: Causal interventions provide a signal of learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows sensitivity to constructions in pretrained language models, but their relevance to human language learning is doubted. |
| Approach: | They use construction grammars to demonstrate sensitivity to constructions in pretrained language models. |
| Outcome: | The proposed models learn diverse constructions even hard cases that are superficially indistinguishable. |
Effective Red-Teaming of Policy-Adherent Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Model (LLM)-based agents are increasingly used in domains with strict policies, such as refund eligibility or cancellation rules. |
| Approach: | They propose a multi-agent red-teaming system that leverages policy-aware persuasive strategies to undermine a policy-adherence agent in a customer-service scenario. |
| Outcome: | The proposed model outperforms jailbreak methods and tau-break to assess agent's robustness against manipulative user behavior. |
CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate unreliable responses due to their cognitive alignment of context and intent. |
| Approach: | They propose a benchmark to identify possible implicit assumptions in QA questions . they use retrieved Wikipedia fragments to identify interpretations for a given query . |
| Outcome: | The proposed benchmark identifies possible implicit assumptions and improves answer accuracy by 11.75% . retrieved Wikipedia fragments help identify possible interpretations for a given query . |
SafeScientist: Enhancing AI Scientist Safety for Risk-Aware Scientific Discovery (2025.emnlp-main)
Copied to clipboard
Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu, Jiaxuan You
| Challenge: | Recent advances in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet raised critical ethical and safety concerns. |
| Approach: | They propose a framework to enhance safety and ethical responsibility in AI-driven scientific exploration. |
| Outcome: | The proposed framework significantly improves safety performance by 35% compared to traditional frameworks. |
Improving Informally Romanized Language Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Latin script is often used to informally write languages with non-Latin native scripts, resulting in high spelling variability. |
| Approach: | They propose to improve methods used to synthesize training sets to incorporate natural spelling variations into training sets. |
| Outcome: | The proposed method improves test F1 from the reported 74.7% (using a pretrained neural model) to 85.4% (using the linear classifier trained solely on synthetic data). |
Integral Transformer: Denoising Attention, Not Too Much Not Too Little (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce attention noise by integrating signals from logit distributions are prone to attention noise. |
| Approach: | They propose a self-attention mechanism that integrates signals from the logit distribution to denoise attention. |
| Outcome: | The proposed model outperforms vanilla, Cog, and Differential attention variants on knowledge and reasoning benchmarks. |
CHENGYU-BENCH: Benchmarking Large Language Models for Chinese Idiom Understanding and Use (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on narrow tasks such as multiple-choice cloze tests, isolated translation, or simple paraphrasing. |
| Approach: | They propose a benchmark to measure Chinese idioms' cultural and contextual nuances . they evaluate 2,937 human-verified examples covering 1,765 common idiomes . |
| Outcome: | The proposed benchmarks achieve 95% accuracy on Evaluative Connotation, but only 85% on Appropriateness and 40% top-1 accuracy in Open Cloze. |
Improving Cross Lingual Transfer by Pretraining with Active Forgetting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that encoder-only LLMs show impressive cross lingual transfer of their capabilities from English to other languages. |
| Approach: | They propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. |
| Outcome: | The proposed model improves cross lingual transfer capabilities on non-English languages despite being trained on English data. |
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu
| Challenge: | emergence of large Vision Language Models (VLMs) has broadened the capabilities of single-modal Large Language Model (LLM) but VLMs are prone to significant hallucinations, especially in the form of cross-modal inconsistencies. |
| Approach: | They propose a new alignment framework that leverages image retrieval to integrate both textual and visual preference signals. |
| Outcome: | The proposed framework mitigates hallucinations more effectively than previous methods . it maintains robustness and scalability across a wide range of VLM sizes and architectures . |
To Mask or to Mirror: Human-AI Alignment in Collective Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used to model and augment collective decision-making. |
| Approach: | They propose a framework for assessing collective alignment using the Lost at Sea social psychology task. |
| Outcome: | The proposed framework compares LLMs with human-AI alignment on the Lost at Sea social psychology task. |
SWAN: An Efficient and Scalable Approach for Long-Context Language Modeling (2025.emnlp-main)
Copied to clipboard
Krishna C Puvvada, Faisal Ladhak, Santiago Akle Serano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, Boris Ginsburg
| Challenge: | Existing decoder-only models struggle with context lengths beyond their training distribution. |
| Approach: | They propose a causal Transformer architecture that generalizes robustly to sequence lengths longer than seen during training. |
| Outcome: | The proposed decoder-only architecture can generalize robustly to longer contexts . it is more computationally efficient than the standard Transformer architecture, the authors say . |
LLMs Behind the Scenes: Enabling Narrative Scene Illustration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Generative AI has established the ability to readily transform content from one medium to another. |
| Approach: | They propose a pipeline that uses large language models to prompt text-to-image models to generate scenes for story text. |
| Outcome: | The proposed pipeline synthesizes illustrations for scenes in a story corpus using human annotation tasks. |
REARANK: Reasoning Re-ranking Agent via Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | REARANK is a large language model-based listwise reasoning reranking agent . it explicitly reasons be- fore reranked results, significantly improving performance and interpretability. |
| Approach: | They propose a large language model-based listwise reasoning reranking agent that explicitly reasons be- fore reranked lists. |
| Outcome: | The proposed agent outperforms GPT-4 on reasoning-intensive benchmarks and surpasses GPL-4 on BRIGHT benchmarks. |
Large Language Models Do Multi-Label Classification Differently (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. |
| Approach: | They propose to use initial probability distributions to analyze output distributions of LLMs at each label generation step to find out how LLM models perform multi-label classification. |
| Outcome: | The proposed methods improve alignment and predictive performance over existing methods. |
FilBench: Can LLMs Understand and Generate Filipino? (2025.emnlp-main)
Copied to clipboard
Lester James Validad Miranda, Elyanah Aco, Conner G. Manuel, Jan Christian Blaise Cruz, Joseph Marvin Imperial
| Challenge: | Despite impressive performance of LLMs on English-based tasks, little is known about their capabilities in specific languages such as Filipino. |
| Approach: | They propose a benchmark to evaluate LLMs across a diverse set of tasks and capabilities in Filipino, Tagalog, and Cebuano. |
| Outcome: | The proposed benchmark reflects the priorities and trends of research in the Philippines . it finds that several LLMs suffer from reading comprehension and translation capabilities . |
M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysis (2025.emnlp-main)
Copied to clipboard
ChengYan Wu, Bolei Ma, Yihong Liu, Zheyu Zhang, Ningyuan Deng, Yanshu Li, Baolan Chen, Yi Zhang, Yun Xue, Barbara Plank
| Challenge: | Existing studies focus on English-centric aspects of sentiment analysis, limiting scope for multilingual evaluation and research. |
| Approach: | They propose to use a multilingual dataset to analyze aspects with associated sentiment elements in text. |
| Outcome: | The proposed dataset is the most extensive multilingual parallel dataset for ABSA to date. |
RuCCoD: Towards Automated ICD Coding in Russian (2025.emnlp-main)
Copied to clipboard
Alexandr Nesterov, Andrey Sakhovskiy, Ivan Sviridov, Airat Valiev, Vladimir Makharev, Petr Anokhin, Galina Zubkova, Elena Tutubalina
| Challenge: | a new dataset for clinical coding in Russian is available for download . human coders must navigate a wide array of medical terminology and time pressures . |
| Approach: | They present a new dataset for ICD coding in Russian, a language with limited biomedical resources. |
| Outcome: | The proposed model improves accuracy on an in-house EHR dataset from 2017 to 2021. |
Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs (2025.emnlp-main)
Copied to clipboard
Dayu Yang, Tianyang Liu, Daoan Zhang, Antoine Simoulin, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, Xin Qian, Grey Yang, Jiebo Luo, Julian McAuley
| Challenge: | Recent breakthrough models like OpenAI-o1 and DeepSeek-R1 show powerful task-solving capabilities, particularly advances in reasoning. |
| Approach: | They propose future research directions that may deepen the synergy, ultimately advancing LLM performance in both complex reasoning and code intelligence. |
| Outcome: | The proposed research may deepen the synergy, ultimately advancing LLM performance in both complex reasoning and code intelligence. |
Efficient Model Development through Fine-tuning Transfer (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern large language models face a major bottleneck: each new version of a pre-trained model requires expensive and repetitive alignment. |
| Approach: | They propose a method that transfers fine-tuning updates across model versions . they extract the diff vector, which is the difference in parameters induced by fine-uning, from a source model and apply it to the base of a different target model. |
| Outcome: | The proposed method reduces training costs while maintaining model performance. |
Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reasoning language models (RLMs) excel at complex tasks by leveraging a chain-of-thought process to generate structured intermediate steps. |
| Approach: | They present the first systematic study of language mixing in reasoning language models, examining its patterns, impact, and internal causes across 15 languages, 7 task difficulty levels, and 18 subject areas. |
| Outcome: | The proposed model generates reasoning steps that include a mixture of languages when prompted in one language, and this improves accuracy. |
User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that asking for direct user feedback can be disruptive . we examine whether incorporating the contents of user feedback improves model performance . |
| Approach: | They analyze user feedback in the user-LLM conversation logs and harvest learning signals from it. |
| Outcome: | The proposed approach can lead to model degradation on two user-LM interaction datasets. |
Read to Hear: A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automatic pronunciation assessment is typically performed by acoustic models trained on audio-score pairs. |
| Approach: | They propose a zero-shot, textual description-based Pronunciation Assessment approach that utilizes human-readable representations of speech signals fed into an LLM to assess pronunciation accuracy and fluency. |
| Outcome: | The proposed approach is cost-efficient and competitive in performance . it significantly improves the performance of conventional audio-score-trained models on out-of-domain data . |
COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve compositional reasoning in vision language models are resource-intensive or do not provide an interpretable reasoning process. |
| Approach: | They propose a method that augments VLM outputs with carefully designed neurosymbolic concept trees learned from LLMs to improve VLM’s linguistic reasoning. |
| Outcome: | Empirical results show that COCO-Tree significantly improves compositional generalization and provides a rationale behind VLM predictions. |
SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automated survey generation is a key task in scientific document processing due to lack of standardized evaluation datasets. |
| Approach: | They propose a survey-based framework that integrates quality indicators into literature retrieval to assess higher-quality sources. |
| Outcome: | The proposed framework enhances the standard Retrieval-Augmented Generation pipeline and enables human-guided writing. |
VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing (2025.emnlp-main)
Copied to clipboard
Zhisheng Zheng, Puyuan Peng, Anuj Diwan, Cong Phuoc Huynh, Xiaohang Sun, Zhu Liu, Vimal Bhat, David Harwath
| Challenge: | Autoregressive language model for multilingual speech editing and zero-shot text-to-speech synthesis is available in 11 languages. |
| Approach: | They introduce an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot text-to-speech synthesis across 11 languages. |
| Outcome: | The model generates high-quality, natural-sounding speech, even with limited per-language data . it shows robust performance in diverse linguistic settings, even in limited per language data compared to other models . |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
MultiMatch: Multihead Consistency Regularization Matching for Semi-Supervised Text Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | **MultiMatch** is a semi-supervised learning (SSL) algorithm that combines co-training and consistency regularization with pseudo-labeling. |
| Approach: | They propose a semi-supervised learning algorithm that integrates co-training and consistency regularization with pseudo-labeling. |
| Outcome: | The proposed algorithm outperforms the second-best approach on 8 out of 10 setups from 5 natural language processing datasets and outperformed the second best by 3.26%. |
TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large reasoning models (LRMs) have driven significant breakthroughs across various reasoning tasks including deductive, arithmetic, commonsense, relational, and symbolic reasoning. |
| Approach: | They propose a programmatic approach to evaluate basic strategic, spatial, and logical reasoning abilities in large reasoning models through four two-player Tic-Tac-Toe-style games that humans can effortlessly solve from a young age. |
| Outcome: | The proposed model performs 41% lower on TTT-Bench than MATH 500 and AIME 2024 models, while the larger models perform better on longer reasoning traces. |
Learning from Diverse Reasoning Paths with Routing and Collaboration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that the reasoning abilities of large language models (LLMs) grows with model size and pre-training data. |
| Approach: | They propose to combine quality filtering, conditional routing, and cooperative peer teaching to transfer knowledge from powerful teacher models to compact and transparent students. |
| Outcome: | Experiments show that QR-Distill is superior to traditional methods. |
Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | a shortage of medical doctors limits access to timely and reliable healthcare . authors propose a multi-turn LLM-based medical assistant for medical inquiries . |
| Approach: | They propose a multi-turn LLM-based medical assistant that asks patients with patience . they compare it with SOTA one-shot and multi-turned LLMs to evaluate its performance . |
| Outcome: | The proposed medical assistant improves diagnostic accuracy, reduces uncertainty and enhances user experience. |
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) generate plausible but factually incorrect outputs, posing serious risks to patient safety and clinical decision-making. |
| Approach: | They propose a benchmark for medical hallucination detection using 10,000 question-answer pairs derived from PubMedQA. |
| Outcome: | The proposed model achieves an F1 score as low as 0.625 for detecting 'hard' category hallucinations. |
NUTMEG: Separating Signal From Noise in Annotator Disagreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work suggests that annotators may have genuine disagreements, but few models separate signal from noise in annotator disagreement. |
| Approach: | They propose a Bayesian model that removes noisy annotations from training data while preserving systematic disagreements. |
| Outcome: | The proposed model outperforms models trained on NUTMEG-aggregated data. |
Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise Pooled Representations (2025.emnlp-main)
Copied to clipboard
Abhilekh Borah, Chhavi Sharma, Danush Khanna, Utkarsh Bhatt, Gurpreet Singh, Hasnat Md Abdullah, Raghav Kaushik Ravi, Vinija Jain, Jyoti Patel, Shubham Singh, Vasu Sharma, Arpita Vats, Rahul Raja, Aman Chadha, Amitava Das
| Challenge: | a new metric measures the quality of large language models (LLMs) that detects hidden misalignments and jailbreak risks. |
| Approach: | They propose a decoding-invariant metric that measures latent safety failures . they propose 'Alignment Quality Index' to measure latent activations in latent space . |
| Outcome: | The proposed metric detects latent safety failures overlooked by behavioral benchmarks and jailbreaks. |
MythTriage: Scalable Detection of Opioid Use Disorder Myths on a Video-Sharing Platform (2025.emnlp-main)
Copied to clipboard
| Challenge: | 108K drug overdose deaths in 2022, according to NIDA . |
| Approach: | They propose a large-scale study of OUD-related myths on YouTube with clinical experts to validate 8 pervasive myths and release an expert-labeled video dataset. |
| Outcome: | The proposed model reduces annotation time and cost by over 76% compared to experts and full LLM labeling. |
Demystifying optimized prompts in language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models (LMs) are not robust to out-of-distribution inputs. |
| Approach: | They investigate the composition of machine generated (“optimized”) prompts and the mechanisms by which LMs parse and build predictions from them. |
| Outcome: | The proposed prompts are primarily composed of punctuation and noun tokens, which are more rare in the training data. |
Whisper-UT: A Unified Translation Framework for Speech and Text (2025.emnlp-main)
Copied to clipboard
Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur
| Challenge: | Encoder-decoder models have achieved remarkable success in speech and text tasks, but efficiently adapting them to diverse uni/multimodal scenarios remains a challenge. |
| Approach: | They propose a framework that leverages lightweight adapters to enable seamless adaptation across tasks. |
| Outcome: | The proposed framework improves speech translation performance through a 2-stage decoding strategy without requiring 3-way parallel data. |
Unleashing the Reasoning Potential of LLMs by Critique Fine-Tuning on One Problem (2025.emnlp-main)
Copied to clipboard
| Challenge: | Critique Fine-Tuning (CFT) is a promising paradigm for unlocking the reasoning capabilities of large language models. |
| Approach: | They propose a method that leverages critique data generated from a single math problem to improve reasoning accuracy. |
| Outcome: | The proposed method surpasses one-shot RLVR while requiring 15 to 20 times less compute. |
Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent decoding methods improve the factuality of large language models (LLMs) by refining how the next token is selected during generation. |
| Approach: | They propose a decoding strategy that actively decides when to apply contrasting layers during generation by casting decoding as a sequential decision-making problem. |
| Outcome: | The proposed method surpasses state-of-the-art methods across five benchmarks and mitigates hallucinations in diverse generation scenarios. |
BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive generative models are gaining traction in language tasks such as text generation and machine translation. |
| Approach: | They propose a likelihood-based evaluation metric that fits transformer-based model embeddings into a stochastic process and propose it as a probability-based metric. |
| Outcome: | The proposed model embeddings induce a "clustered-to-temporal ordered" mapping of language model representations in high-dimensional space, and this structure enhances performance on tasks such as temporal consistency evaluation and AI-generated content detection. |
SAND: Boosting LLM Agents with Self-Taught Action Deliberation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Model (LLM) agents finetuned with supervised finetuning may over-commit towards seemingly plausible but suboptimal actions due to limited action space exploration. |
| Approach: | They propose a self-taught actioN deliberation framework that allows LLM agents to explicitly deliberate over candidate actions before committing to one. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two representative interactive agent tasks and achieves an average 20% improvement over initial finetuning. |
LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) show promise in simulating complex scenarios. |
| Approach: | They examine multiple LLMs to proactively estimate perceived earthquake impacts using multimodal datasets and multimodal imagery. |
| Outcome: | The framework generates Modified Mercalli Intensity (MMI) predictions at zip code and county scales using multimodal datasets. |
Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research assesses LLMs’ values by analyzing their stated inclinations . a framework to evaluate the alignment between stated values and value-informed actions is lacking . |
| Approach: | They propose a framework to evaluate the alignment between LLMs’ stated values and their value-informed actions. |
| Outcome: | The proposed framework shows significant misalignment between LLM-generated values and their actions . misaligned values have shown real-world risks, such as amplifying stereotypes and reinforcing bias algorithms in hiring. |
Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs struggle to reliably detect subtle reasoning errors in ASAS tasks. |
| Approach: | They propose a dual-model framework with a dedicated Critic model trained for effective reflection that generates precise verbal feedback. |
| Outcome: | The proposed framework outperforms existing ASAS benchmarks and provides valuable insights into the performance of the proposed framework. |
Image Embedding Sampling Method for Diverse Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, large-scale captioning models are less accessible for resource-constrained applications such as mobile devices and assistive technologies. |
| Approach: | They propose a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM as the backbone. |
| Outcome: | The proposed framework achieves comparable performance to larger models on MSCOCO, Flickr30k, and Nocaps test datasets while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions. |
Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) perform well on reasoning benchmarks but often fail when inputs alter slightly, raising concerns about overreliance on memorization. |
| Approach: | They propose a framework for Source-aware Token-level Identification of Memorization which attributes each token in a reasoning chain to one of multiple memorization sources based on their statistical co-occurrence with the token in the pretraining corpus. |
| Outcome: | The proposed framework attributes each token in a reasoning chain to one of multiple memorization sources based on their statistical co-occurrence with the token in the pretraining corpus. |
FANS: Formal Answer Selection for LLM Natural Language Math Reasoning Using Lean4 (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks that use Lean4 to enhance LLMs' NL reasoning abilities have been controversial in the field of math reasoning. |
| Approach: | They propose a framework that utilizes Lean4 to enhance LLMs’ NL math reasoning ability by generating a Lean 4 theorem statement and a proof-generating LLM. |
| Outcome: | The proposed framework improves LLMs' NL math reasoning ability by 2% across several math benchmarks and higher further based on reward models or in subfields such as algebra and number theory. |
Date Fragments: A Hidden Bottleneck of Tokenization for Temporal Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | a tokeniser that splits "2025-03-14" into "20", "25", "-0", "3", "-1", "4" obscures temporal cues and obscures structure . excessive fragmentation correlates with accuracy drops of up to 10 points on uncommon dates . |
| Approach: | They propose a date fragmentation ratio measure that measures how faithfully a tokeniser preserves multi-digit date components. |
| Outcome: | The proposed method shows that excessive fragmentation correlates with accuracy drops of up to 10 points on uncommon dates like historical and futuristic dates. |
Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark (2025.emnlp-main)
Copied to clipboard
Jianyou Wang, Weili Cao, Longtian Bao, Youze Zheng, Gil Pasternak, Kaicheng Wang, Xiaoyue Wang, Ramamohan Paturi, Leon Bergen
| Challenge: | Systematic reviews should take into account the quality of available evidence, placing more weight on studies that use a valid methodology. |
| Approach: | They propose to use a risk-of-bias framework to assess the methodological strength of biomedical papers by combining expert reviewers' judgments with research paper sentences. |
| Outcome: | The proposed system measures the methodological strength of biomedical papers using the risk-of-bias framework used for systematic reviews. |
SHIFT: Selected Helpful Informative Frame for Video-guided Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video-guided machine translation (VMT) aims to improve translation quality by integrating contextual information from paired short video clips. |
| Approach: | They propose a plug-and-play framework for video-guided machine translation with multimodal large language models. |
| Outcome: | The proposed framework improves performance of MLLMs while reducing computational cost. |
Surge: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Neural surrogate models are powerful tools in data mining, but are underexplored . large language models (LLMs) have demonstrated remarkable capabilities in code-related tasks . |
| Approach: | They propose a benchmarking framework to examine the feasibility of large language models . they examine scaling laws, data efficiency, and predictive accuracy of 21 open-source and proprietary LLMs . |
| Outcome: | The proposed benchmark examines 21 open-source and proprietary LLMs . it also examines scaling laws, data efficiency, and predictive accuracy . |
Few-Shot Learning Translation from New Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows strong transfer learning capability to unseen languages in sequence-to-sequence neural networks . current transfer learning methods require much less downstream task data than would otherwise be required. |
| Approach: | They first train word embeddings models on varying amounts of data and plug them into a machine translation model. |
| Outcome: | The proposed model can learn Flores with only 500 parallel sentences and 31,250 sentences of monolingual data, and it can exceed 15 BLEU on unseen languages. |
Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Design (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit anthropomorphism characteristics – human-like qualities portrayed across their outlook, language, behavior, and reasoning functions. |
| Approach: | They propose that anthropomorphism should be treated as a design concept that can be intentionally tuned to support user goals. |
| Outcome: | The proposed design should reflect interaction between artifact designers and interpreters, and should be based on cues embedded in the artifactor and the (cognitive) responses of interpreters to the cue. |
TokenSkip: Controllable Chain-of-Thought Compression in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) has been proven effective in enhancing the reasoning capabilities of large language models (LLMs). |
| Approach: | They propose a chain-of-thought (CoT) prompting approach that enables LLMs to selectively skip less important tokens, allowing for controllable CoT compression. |
| Outcome: | Experiments show that TokenSkip reduces CoT token usage while preserving strong reasoning performance. |
Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that text-generation models can be overconfident when there are multiple correct options. |
| Approach: | They propose a QE approach called BoostedProb which boosts the model’s confidence in cases where there are multiple viable output options. |
| Outcome: | The proposed approach achieves on average +0.194 improvement in Pearson correlation to ground-truth quality and outperforms more costly approaches like supervised or ensemble-based QE in certain settings. |
reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reward models have a high performance on benchmarks, but performance degradation is often due to overfitting. |
| Approach: | They propose to explicitly train reward models to assign similar scores to paraphrases to improve their robustness. |
| Outcome: | The proposed model reduces degradation by half for the Chat Hard subset in RewardBench. |
Why Do Some Inputs Break Low-Bit LLM Quantization? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-bit weight-only quantization reduces memory usage but disproportionately affects certain examples. |
| Approach: | They analyze quantization errors of 50 pairs of methods on large language models and test their hypothesis . |
| Outcome: | The proposed method reduces the memory footprint of large language models while maintaining reasonable performance across benchmarks. |
LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern automatic speech recognition systems rely on encoder-decoder architectures and their encoders are a critical bottleneck for efficient deployment due to high computational intensity. |
| Approach: | They propose a low-rank compression scheme for ASR encoders that leverages the strong low-ranked properties observed in intermediate activations and approximates linear transformations with a chain of low-Rank matrix multiplications. |
| Outcome: | The proposed method reduces inference costs while maintaining transcription accuracy while preserving low-rank properties observed in intermediate activations. |
AROMA: Autonomous Rank-one Matrix Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) and adaptive low-rank adaption (AdaLoRa) are effective for large language models but are expensive as model sizes escalate into hundreds of billions of parameters. |
| Approach: | They propose a framework that automatically builds up rank-one components with very few trainable parameters that gradually diminish to zero. |
| Outcome: | The proposed framework significantly reduces parameters compared to LoRA and AdaLoRA while maintaining subspace independence. |
Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lens (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the cognitive error detection capabilities of Large Language Models (LLMs), but few studies have examined the meta-cognitive abilities of LLMs. |
| Approach: | They propose an automated meta-cognition evaluation framework for evaluation of LLMs and a Markovian Intrinsic Reward Adjustment strategy to boost current lenses. |
| Outcome: | The proposed framework can be used to evaluate the meta-cognition abilities of LLMs and improve them. |
Anchoring-Guidance Fine-Tuning (AnGFT): Elevating Professional Response Quality in Role-Playing Conversational Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated significant advancements in various fields, notably in Role-Playing Conversational Agents (RPCAs). |
| Approach: | They propose an Anchoring-Guidance Fine-Tuning Framework to integrate relevant expert knowledge into RPCAs' training process to mitigate this issue. |
| Outcome: | The proposed framework significantly improves the RPCAs’ performance in handling role-specific professional queries while preserving their robust role-playing abilities. |
RiTTA: Modeling Event Relations in Text-to-Audio Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-audio (TTA) generation methods have not explored audio event relation modeling, nor proposed any new framework to enhance this capability. |
| Approach: | They propose a comprehensive relation corpus covering all potential relations in real-world scenarios and a new audio event corpus encompassing commonly heard audios. |
| Outcome: | The proposed framework improves existing models’ relation modeling capability with negligible extra parameters. |
Shallow Focus, Deep Fixes: Enhancing Shallow Layers Vision Attention Sinks to Alleviate Hallucination in LVLMs (2025.emnlp-main)
Copied to clipboard
Xiaofeng Zhang, Yihao Quan, Chen Shen, Chaochen Gu, Xiaosong Yuan, Shaotian Yan, Jiawei Cao, Hao Cheng, Kaijie Wu, Jieping Ye
| Challenge: | Multimodal large language models (MLLMs) demonstrate excellent abilities for understanding visual information, but the hallucination remains a challenging problem. |
| Approach: | They propose a training-free approach to enhance vision attention sinks to facilitate convergence of the image token attention sink within shallow layers. |
| Outcome: | The proposed approach improves the convergence of the image token attention sink within shallow layers and strengthens the layer’s focus on the image itself. |
WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai (2025.emnlp-main)
Copied to clipboard
Peerat Limkonchotiwat, Pume Tuchinda, Lalita Lowphansirikul, Surapon Nonesung, Panuthep Tasawong, Alham Fikri Aji, Can Udomcharoenchaikit, Sarana Nutanong
| Challenge: | Existing benchmarks for large language models rely on translations, missing cultural and domain specificity. |
| Approach: | They present a human-authored dataset for evaluation and instruction tuning in Thai . findings highlight need for culturally and professionally grounded instruction data . |
| Outcome: | a human-authored dataset for evaluation and instruction tuning in Thai outperforms translation-based models . findings highlight need for culturally and professionally grounded instruction data . |
MemeReaCon: Probing Contextual Meme Understanding in Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
Zhengyi Zhao, Shubo Zhang, Yuxi Zhang, Yanxi Zhao, Yifan Zhang, Zezhong Wang, Huimin Wang, Yutian Zhao, Bin Liang, Yefeng Zheng, Binyang Li, Kam-Fai Wong, Xian Wu
| Challenge: | Current approaches focus on isolated meme analysis, either for harmful content detection or standalone interpretation, overlooking a fundamental challenge: the same meme can express different intents depending on its conversational context. |
| Approach: | They propose a benchmark to evaluate how large vision language models understand memes in their original context. |
| Outcome: | The proposed benchmark evaluates how large vision language models understand meme intent in their original context. |
A Comprehensive Literary Chinese Reading Comprehension Dataset with an Evidence Curation Based Solution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-resource language understanding is challenging for large language models (LLMs). |
| Approach: | They propose a CompRehensive lIterary Chinese readIng comprehenSion procedure with a large dataset for CRISIS. |
| Outcome: | The proposed procedure has the largest dataset and substantiates the effectiveness of the proposed procedure with a 7 percent hike in accuracy compared with the baseline. |
Dialect-SQL: An Adaptive Framework for Bridging the Dialect Gap in Text-to-SQL (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Text-to-SQL research focuses on specific database systems, limiting adaptability to different dialects. |
| Approach: | They propose a framework that employs Object Relational Mapping (ORM) code as an intermediate language to bridge this gap. |
| Outcome: | The proposed framework outperforms existing methods that generate SQL queries directly. |
FinMTEB: Finance Massive Text Embedding Benchmark (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text embedding benchmarks for financial domains are inadequately addressing the nuanced requirements of specialized domains like finance. |
| Approach: | They propose a finance-adapted embedding model that outperforms general-purpose models . they also introduce a new model, Fin-E5, which is also open-sourced . |
| Outcome: | The proposed framework outperforms general-purpose models on financial embedding tasks. |
Scaling Rich Style-Prompted Text-to-Speech Datasets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets that only cover basic tags are limited in their scale or coverage of style tags. |
| Approach: | They propose a large-scale dataset that annotates speech utterances with rich style captions. |
| Outcome: | The proposed dataset scales speech utterances with rich style captions for the first time. |
Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using personas, LLMs are custom-made to meet specific user needs. |
| Approach: | They assign 193 different nationality personas to five LLMs and examine how evaluations of different nations change when LLM users are assigned specific nationality persons. |
| Outcome: | The nationality personas of five LLMs are assigned to different nations and their evaluations change. |
Eliciting Implicit Acoustic Styles from Open-domain Instructions to Facilitate Fine-grained Controllable Generation of Speech (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current work relies on pre-defined rules or templates to control the style of speech. |
| Approach: | They propose to use open-domain instructions to generate speech with the acoustic style that meets users’ needs based on their instructions. |
| Outcome: | The proposed model can be used to generate speech with the acoustic style that meets users’ needs based on open-domain instructions. |
OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) trained over corpora risk memorizing sensitive, copyrighted, or toxic content. |
| Approach: | They propose a framework that removes targeted data while preserving model utility. |
| Outcome: | The proposed framework resists membership inference attacks, minimizes impact on retained data, and maintains robustness across diverse scenarios. |
AdaptThink: Reasoning Models Can Learn When to Think (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large reasoning models have demonstrated remarkable capabilities in tackling complex tasks. |
| Approach: | They propose an algorithm to teach reasoning models to choose the optimal thinking mode based on problem difficulty. |
| Outcome: | The proposed algorithm reduces the average response length and improves accuracy on three math datasets. |
T2: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering (2025.emnlp-main)
Copied to clipboard
Zhengyi Zhao, Shubo Zhang, Zezhong Wang, Huimin Wang, Yutian Zhao, Bin Liang, Yefeng Zheng, Binyang Li, Kam-Fai Wong, Xian Wu
| Challenge: | Existing efficient test-time scaling methods introduce budget constraints or early stop mechanisms to avoid overthinking for straightforward questions but add human bias to the reasoning process. |
| Approach: | They propose a framework that dynamically adapts reasoning depth based on question complexity. |
| Outcome: | Experimental results show that the proposed framework achieves higher accuracy than baseline methods and reduces computational overhead by up to 25.2%. |
Non-Existent Relationship: Fact-Aware Multi-Level Machine-Generated Text Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting LLMs lack the authenticity of the entity graph . lmgenerated text is misused, including fake news and spam . |
| Approach: | They propose a fact-aware model that assesses discrepancies between textual and factual entity graphs through graph comparison. |
| Outcome: | The proposed model outperforms state-of-the-art methods on three public datasets showing that it can capture differences in entity graphs between machine-generated and human-written texts. |
Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations (2025.emnlp-main)
Copied to clipboard
Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, Nicola Cancedda
| Challenge: | LLMs often use assertive language when making false claims, resulting in harm and loss of trust. |
| Approach: | They find that a mismatch between semantic and verbal uncertainty is a better predictor of hallucinations than semantic uncertainty alone. |
| Outcome: | a new study shows that mismatch between semantic and verbal uncertainty is better predictor of hallucinations than semantic uncertainty alone. |
JUREX-4E: Juridical Expert-Annotated Four-Element Knowledge Base for Legal Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have introduced legal theories into LLM workflows to improve their understanding of legal texts and reasoning accuracy. |
| Approach: | They evaluate an expert-annotated four-element knowledge base covering 155 criminal charges. |
| Outcome: | The proposed model can be used to analyze criminal charges and retrieve them in legal cases. |
CIE: Controlling Language Model Text Generations Using Continuous Signals (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to control language models with intent are brittle and hard to scale. |
| Approach: | They propose to use a set of LMs to fine-tune to expect a control vector that is interpolated between a "low" and a 'high' token embedding. |
| Outcome: | The proposed method can be finetuned to expect a control vector that is interpolated between a “low” and a ‘high” token embedding. |
Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience (2025.emnlp-main)
Copied to clipboard
Xi Wang, Songlei Jian, Shasha Li, Xiaopeng Li, Bin Ji, Ma Jun, Xiaodong Liu, Jing Wang, Jianfeng Zhang, Jie Yu, Feilong Bao, null Wangbaosheng
| Challenge: | Existing methods to generate human-aligned content with a “jailbreak prompt” are inefficient and repetitive, causing inefficiency and a lack of experience. |
| Approach: | They propose a framework that integrates past attack experiences to aid current jailbreak attempts. |
| Outcome: | The proposed framework improves both attack effectiveness and efficiency compared to the current black-box jailbreak method. |
Language-to-Space Programming for Training-Free 3D Visual Grounding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for 3D visual grounding have been proposed, but they are limited by the scarcity of 3D vision-language datasets and the high cost of annotations. |
| Approach: | They propose a method for training-free 3D visual grounding that uses LLM-generated codes to analyze 3D spatial relations among objects. |
| Outcome: | The proposed method achieves 52.9% accuracy on the Nr3D benchmark and significantly reduces grounding time and token costs. |
RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) has emerged as a key paradigm for enhancing large language models by incorporating external knowledge. |
| Approach: | They propose a method for synthesizing diverse and high-quality RAG instruction data based on any source corpus. |
| Outcome: | The proposed method outperforms existing methods in multiple tasks and achieves strong zero-shot performance. |
AdaRewriter: Unleashing the Power of Prompting-based Conversational Query Reformulation via Test-Time Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prompting-based conversational query reformulation has emerged as a powerful approach for conversational search, refining ambiguous user queries into standalone search queries. |
| Approach: | They propose a framework for query reformulation using an outcome-supervised reward model via test-time adaptation. |
| Outcome: | Experiments on five conversational search datasets show that AdaRewriter significantly outperforms the existing methods across most settings. |
SmartBench: Is Your LLM Truly a Good Chinese Smartphone Assistant? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for Large Language Models focus on objective tasks like mathematics and coding in English, which do not reflect the practical use cases of on-device LLMs in real-world mobile scenarios. |
| Approach: | They propose a benchmark to evaluate the capabilities of on-device Large Language Models in Chinese mobile contexts. |
| Outcome: | The proposed framework evaluates on-device LLMs and MLLMs in Chinese . it provides a standardized framework for evaluating LLM performance on real smartphones . |
F2TEval: Human-Aligned Multi-Dimensional Evaluation for Figure-to-Text Task (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for Figure-to-Text tasks are limited due to the inherent ambiguity and semantic compression of figures, the generated texts suffer from factual inaccuracies, incomplete coverage, and weak logical reasoning. |
| Approach: | They propose a five-dimensional reference-free evaluation method aligned with expert criteria to support fine-grained evaluation. |
| Outcome: | The proposed method outperforms Gemini-2.0 and Claude-3.5 with only 0.9B parameters. |
Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation (2025.emnlp-main)
Copied to clipboard
Qiyuan Chen, Hongsen Huang, Qian Shao, Jiahe Chen, Jintai Chen, Hongxia Xu, Renjie Hua, Ren Chuan, Jian Wu
| Challenge: | Large Language Models (LLMs) require high quality preference datasets to align with human preferences. |
| Approach: | They propose a framework that leverages inherent regulation of LLMs’ representation space for efficient and tailored preference dataset construction, named Icon2. |
| Outcome: | The proposed framework improves performance on benchmarks like AlpacaEval 2.0 and Arena-Hard while reducing computational costs by up to 48.1%. |
DSCD: Large Language Model Detoxification with Self-Constrained Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for decoding large language models (LLMs) are based on external constraints and require additional resource overhead and loss of generation fluency. |
| Approach: | They propose a method for LLMs detoxification without parameter fine-tuning that strengthens the inner token distribution while weakening that of hallucination and toxic layer during output generation. |
| Outcome: | Extensive experiments on open-source LLMs and public datasets demonstrate DSCD's state-of-the-art (SOTA) performance in detoxification and generation fluency, with superior efficiency compared to existing methods. |
From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models generate explicit reasoning traces alongside final answers . the extent to which these traces influence answer generation remains unclear . |
| Approach: | They conduct empirical evaluation of Large Reasoning Models that include explicit reasoning . they also show that answer tokens attend substantially to reasoning tokens . |
| Outcome: | The results show that including explicit reasoning improves answer quality across domains . they also show that answer tokens attend substantially to reasoning tokens - the authors . |
Quantifying Language Disparities in Multilingual Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contemporary NLP development relies on digital language datasets to build large language models. |
| Approach: | They propose a framework that disentangles confounding variables and introduces interpretable metrics to quantify model performance and language disparities. |
| Outcome: | The proposed framework provides a more reliable measurement of model performance and language disparities for low-resource languages. |
KoBLEX: Open Legal Question Answering with Multi-hop Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performances in general domains and are now extending into the expert domain of law. |
| Approach: | They propose a Korean Benchmark for Legal EXplainable QA (KoBLEX) that evaluates provision-grounded, multi-hop legal reasoning. |
| Outcome: | The proposed method outperforms baselines and shows a high correlation with human judgments. |
End-to-End Learnable Psychiatric Scale Guided Risky Post Screening for Depression Detection on Social Media (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to detect depression from social media posting history are limited by frozen screening models and lack of learning. |
| Approach: | They propose to use a frozen screening model to train a risky post detection model with psychiatric scales to enable a learnable end-to-end learning process. |
| Outcome: | The proposed model outperforms several strong baseline methods and qualitative analysis confirms that it better captures users’ mental states than others. |
ReAgent: Reversible Multi-Agent Reasoning for Knowledge-Enhanced Multi-Hop QA (2025.emnlp-main)
Copied to clipboard
Zhao Xinjie, Fan Gao, Xingyu Song, Yingjian Chen, Rui Yang, Yanran Fu, Yuyang Wang, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
| Challenge: | Multi-hop question answering (QA) is a central challenge in natural language processing . early mistakes can cause errors and undermine the final result, authors say . |
| Approach: | They propose a reversible multi-agent reasoning framework that backtracks to earlier valid states when conflicts arise. |
| Outcome: | Empirical evaluation shows that the framework improves on forward-only benchmarks by 6% . the approach enables agents to backtrack to valid states when conflicts arise . |
Matter-of-Fact: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Science (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems generate hypothesis, run experiments, analyze data, and write or review papers, but they are costly and impractical. |
| Approach: | They propose a challenge dataset for determining the feasibility of hypotheses framed as claims and a temporally-filtered claim verification task using backtesting to test the validity of claims. |
| Outcome: | The proposed model performs well on retrieval augmented generation and code generation while performing 50% of the task. |
ModRWKV: Transformer Multimodality in Linear Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, multimodal studies are based on large language models with quadratic-complexity Transformer architectures. |
| Approach: | They propose a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone and a lightweight architecture to achieve multi-source information fusion. |
| Outcome: | The proposed framework achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders. |
Multimedia Event Extraction with LLM Knowledge Editing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal event extraction methods focus on weakly aligning features from wellpretrained unimodal encoders, resulting in redundant feature perception. |
| Approach: | They propose a multimodal event extraction strategy with a redundant feature selection mechanism that enhances event understanding ability of multimodal large language models. |
| Outcome: | The proposed method outperforms the state-of-the-art (SOTA) baselines on the M2E2 benchmark. |
Exploring the Impact of Personality Traits on LLM Toxicity and Bias (2025.emnlp-main)
Copied to clipboard
| Challenge: | anthropomorphic LLMs are being developed to serve diversified roles, but content safety concerns remain regarding their toxicity and toxicity. |
| Approach: | They propose to assign personality traits to large language models (LLMs) to reduce toxic language and social biases in their outputs by using the widely accepted HEXACO personality framework developed in social psychology. |
| Outcome: | The proposed model is able to perform on three toxic and bias benchmarks and shows that assigning personality traits reduces bias and toxicity similar to humans’ correlations between personality traits and toxic behaviors. |
Task-aware Contrastive Mixture of Experts for Quadruple Extraction in Conversations with Code-like Replies and Non-opinion Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Applying Large Language Models (LLMs) for this specific task presents two primary challenges: the accurate extraction of multiple elements and the understanding of complex dialogue reply structure. |
| Approach: | They propose a novel LLM-based multi-task approach to extract sentiment quadruples from conversations by integrating expert-level contrastive loss within task-oriented mixture of experts layer. |
| Outcome: | The proposed method outperforms existing fine-tuning techniques in terms of accuracy and computational efficiency. |
Mitigating Biases in Language Models via Bias Unlearning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent debiasing approaches target different demographic groups, harming fairness and discrimination. |
| Approach: | They propose a model debiasing framework which targets stereotypes by unlearning stereotype forgetting and anti-stereotype retention. |
| Outcome: | The proposed framework outperforms existing methods in mitigating bias while retaining language modeling capabilities. |
UNComp: Can Matrix Entropy Uncover Sparsity? — A Compressor Design from an Uncertainty-Aware Perspective (2025.emnlp-main)
Copied to clipboard
Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Min Yang, Lingpeng Kong, Ngai Wong
| Challenge: | Deploying large language models (LLMs) for long-context inference remains challenging due to their substantial memory and computational demands. |
| Approach: | They propose an uncertainty-aware framework that leverages truncated matrix entropy to identify areas of low information content. |
| Outcome: | The proposed framework reduces the KV cache size to 4.74% of the original and achieves a 6% speedup. |
Superpose Task-specific Features for Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for model merging are limited by resource demands . recent studies validate the linear representation hypothesis . |
| Approach: | They propose a method that superposes task-specific features from individual models into a merged model. |
| Outcome: | The proposed method outperforms existing methods on multiple benchmarks and models. |
FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing RAG research focuses on textual data, overlooking rich visual content in financial documents. |
| Approach: | They propose a visual RAG benchmark tailored for finance that integrates multimodal data and provides visual citation to ensure traceability. |
| Outcome: | The proposed visual RAG benchmark integrates multimodal data and provides visual citation to ensure traceability. |
BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing GUI agents focus on enhancing the accuracy of individual actions and lack effective mechanisms for detecting and recovering from errors. |
| Approach: | They propose a framework that incorporates a backtracking mechanism to enhance the task completion capabilities of GUI agents by verifier, judger, and reflector components. |
| Outcome: | The proposed framework improves task success rate and step accuracy on Mobile3M and Auto-UI benchmarks. |
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM)-based embedding models surpass BERT and T5 on general-purpose text embeddable tasks. |
| Approach: | They propose to adopt diffusion language models for text embeddings to overcome limitations in unidirectional attention used during autoregressive pre-training. |
| Outcome: | The proposed model outperforms the existing LLM-based embedding model on reasoning tasks by 20% and 2% on traditional embeddable benchmarks. |
BannerAgency: Advertising Banner Design with Multimodal LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Advertising banners are an instrumental medium in digital marketing campaigns. |
| Approach: | They propose a training-free framework for fully automated banner ad design creation that enables frontier multimodal large language models to streamline the production of effective banners with minimal manual effort. |
| Outcome: | The proposed framework is based on a training-free model that can be used to create fully automated banner ad design creations with minimal manual effort across diverse marketing contexts. |
DIDS: Domain Impact-aware Data Sampling for Large Language Model Training (2025.emnlp-main)
Copied to clipboard
Weijie Shi, Jipeng Zhang, Yaguang Wu, Jingzhi Fang, Shibo Zhang, Yao Zhao, Hao Chen, Ruiyuan Zhang, Yue Cui, Jia Zhu, Sirui Han, Jiajie Xu, Xiaofang Zhou
| Challenge: | Existing approaches for optimizing domain-level sampling strategies struggle with maintaining intra-domain consistency and accurately measuring domain impact. |
| Approach: | They propose to use a Fisher-Information Matrix-guided metric to measure domain impact to ensure intra-domain consistency and accuracy. |
| Outcome: | The proposed model achieves 3.4% higher average performance while maintaining comparable training efficiency. |
Training LLMs to be Better Text Embedders through Bidirectional Reconstruction (2025.emnlp-main)
Copied to clipboard
Chang Su, Dengliang Shi, Siyuan Huang, Jintao Du, Changhua Meng, Yu Cheng, Weiqiang Wang, Zhouhan Lin
| Challenge: | Existing text embedding approaches often leverage the embeddment of the final token, typically a reserved special token such as ‘[EOS]‘. |
| Approach: | They propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. |
| Outcome: | The proposed training stage improves performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales. |
ReMedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Regression-based neural metrics struggle with inconsistency of human ratings . prompting large language models (LLMs) for MT scoring has also shown promise . |
| Approach: | They propose a MT metric framework that reformulates translation evaluation as a reward modeling task. |
| Outcome: | The proposed framework surpasses larger WMT winners and massive closed LLMs across 39 language pairs and 111 MT systems. |
SolEval: Benchmarking Large Language Models for Repository-level Solidity Smart Contract Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on Python and Java, neglecting Solidity, the programming language for Ethereum smart contracts. |
| Approach: | They construct a repository-level benchmark for Solidity to evaluate the performance of LLMs on Ethereum. |
| Outcome: | The proposed benchmarks show that the best performing LLM achieves only 26.29% Pass@10, highlighting room for improvement in Solidity code generation. |
In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models fail to adapt to unfamiliar speakers and language varieties . however, there are significant gaps in the adaptation of certain varieties based on the test speaker, variety, or recording conditions . |
| Approach: | They propose a framework that allows for in-context learning in Phi-4 Multimodal . they find that as few as 12 example utterances reduce word error rates by 19.7% . |
| Outcome: | The proposed framework reduces word error rates by 19.7% across diverse English corpora. |
Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills (2025.emnlp-main)
Copied to clipboard
Changsheng Wang, Chongyu Fan, Yihua Zhang, Jinghan Jia, Dennis Wei, Parikshit Ram, Nathalie Baracaldo, Sijia Liu
| Challenge: | Existing methods for LRM unlearning overlook critical information leakage in reasoning traces, even when final answers are successfully removed. |
| Approach: | They propose a method that suppresses reasoning traces while preserving the model's general reasoning ability. |
| Outcome: | The proposed method significantly reduces reasoning trace leakage and achieves strong performance across reasoning and safety benchmarks, including WMDP, StrongReject, JBB-Behaviors and WildJailbreak. |
Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches for optimizing human annotation efforts are limited . et al., 2015) suggest that densely annotated image captions improve vision-language alignment . |
| Approach: | They propose an AI-in-the-loop methodology to maximize the number of annotated samples and improve their comprehensiveness under fixed budget constraints. |
| Outcome: | The proposed method improves annotation speed and retrieval performance over the parallel method. |
DecoupleSearch: Decouple Planning and Search via Hierarchical Reward Modeling (2025.emnlp-main)
Copied to clipboard
Hao Sun, Zile Qiao, Bo Wang, Guoxin Chen, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, Yan Zhang
| Challenge: | Retrieval-Augmented Generation (RAG) systems have emerged as a pivotal methodology for enhancing Large Language Models (LLMs). |
| Approach: | They propose a framework that decouples planning and search processes using dual value models, enabling independent optimization of plan reasoning and search grounding. |
| Outcome: | The proposed framework decouples planning and search processes using dual value models, enabling independent optimization of plan reasoning and search grounding. |
RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis (2025.emnlp-main)
Copied to clipboard
Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng
| Challenge: | Existing solutions to fine-tune large language models for domain-specific tasks are ineffective in addressing privacy concerns. |
| Approach: | They propose a privacy-preserving framework that fine-tunes a reward proxy model and uses reward signals to guide the synthetic data generation. |
| Outcome: | The proposed framework fine-tunes a reward proxy model and uses reward signals to guide the synthetic data generation. |
Synergizing Multimodal Temporal Knowledge Graphs and Large Language Models for Social Relation Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have limited capacity to model complex graph-structured relationships. |
| Approach: | They propose a low-coupling method synergizing multimodal temporal Knowledge Graphs and Large Language Models for social relation reasoning. |
| Outcome: | The proposed method exhibits state-of-the-art performance in social relation recognition . it bridges the gap between KGs and LLMs and will be released after acceptance . |
LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on legal case retrieval have limited results . limited representations and legally irrelevant matches are often used . |
| Approach: | They propose a large-scale Korean LCR benchmark and a retrieval model that performs legal element reasoning over the query case. |
| Outcome: | a new model outperforms baseline models on a Korean LCR benchmark . it performs state-of-the-art on 411 diverse crime types in queries over 1.2M candidate cases . previous studies have shown that the model can generalize to out-of domain cases if it is trained on in-domain data . |
ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Chart question answering (CQA) is a multimodal task for evaluating the reasoning capabilities of vision-language models. |
| Approach: | They propose a chart question answering benchmark that incorporates multilingual contexts and supports open-domain textual outputs. |
| Outcome: | The proposed framework outperforms the previous three common CQA paradigms: instruction-following, OCR-enhanced, and chain-of-thought. |
COLA: Collaborative Multi-Agent Framework with Dynamic Task Scheduling for GUI Automation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for implementing LLMs are limited by their complexity and lack fault tolerance mechanism. |
| Approach: | They propose a scenario-aware agent Task Scheduler that decomposes task requirements into atomic capability units and dynamically selects the optimal agent from a decision agent pool. |
| Outcome: | The proposed framework achieves competitive performance among GUI Agent methods with an average accuracy of 31.89% on the GAIA dataset. |
DASA-Trans-STM: Adaptive Efficient Transformer for Short Text Matching using Data Augmentation and Semantic Awareness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown impressive versatility across various tasks. |
| Approach: | They propose a novel adaptive Transformer for Chinese short text matching using data augmentation and semantic awareness. |
| Outcome: | The proposed model can deal with word ambiguity in Chinese on four available datasets. |
Pruning the Paradox: How CLIP’s Most Informative Heads Enhance Performance While Amplifying Bias (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale vision-language models such as CLIP have advanced state-of-the-art performance in vision tasks . however, as they gain prominence in real-world applications, their embedded social biases can be harmful . et al., 2021: 103-104. |
| Approach: | They propose an interpretability metric that measures how consistently attention heads align with specific concepts in CLIP-like models. |
| Outcome: | The proposed interpretability metric measures how consistently attention heads align with specific concepts. |
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation (2025.emnlp-main)
Copied to clipboard
Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D. Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, Zheng Zhang
| Challenge: | Large foundation models have become huge, but they consume computational resources in pretraining. |
| Approach: | They propose to replace full-size layers with compute-efficient auto-encoders that enforce low-rank activations throughout training. |
| Outcome: | The proposed method reduces the computing cost by 2pmbtimes and improves training throughput by 1.86pmtime. |
TS-CLIP: Time Series Understanding by CLIP (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastive Language–Image Pre-training (CLIP) has demonstrated remarkable success in aligning vision and language. |
| Approach: | They propose a synonym bank mechanism that generates synonym embeddings as alignment targets. |
| Outcome: | The proposed approach achieves state-of-the-art (SOTA) performance on 51 datasets. |
MultiAgentESC: A LLM-based Multi-Agent Collaboration Framework for Emotional Support Conversation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on generating responses directly and neglect integration of domain-specific reasoning and expert interaction. |
| Approach: | They propose a training-free multi-agent collaboration framework for ESC to emulate human-like process of providing emotional support through dialogue analysis, strategy deliberation, and response generation. |
| Outcome: | The proposed framework excels at providing emotional support and diversifying support strategy selection. |
Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to optimize Large Language Models (LLMs) for knowledge conflicts are inefficient or ineffective for large models and are not suitable for black-box models. |
| Approach: | They propose a framework that can continuously steer LLMs’ sensitivity to contextual knowledge at a lightweight cost. |
| Outcome: | The proposed framework can steer LLMs’ sensitivity to contextual knowledge continuously at a lightweight cost. |
Probing LLM World Models: Enhancing Guesstimation with Wisdom of Crowds Decoding (2025.emnlp-main)
Copied to clipboard
Yun-Shiuan Chuang, Sameer Narendran, Nikunj Harlalka, Alexander Cheung, Sizhe Gao, Siddharth Suresh, Junjie Hu, Timothy T. Rogers
| Challenge: | a common real-world skill of guesstimation is underexplored in large language model research . a recent study suggests that LLMs encode a world model that supports approximate reasoning . |
| Approach: | They propose to decode a guesstimation dataset using MARBLES, FUTURE, and ELECPRED . they replicate WOC effects in human participants and find similar benefits . |
| Outcome: | The proposed model improves accuracy over greedy, self-consistency, and mean decoding in human participants. |
Recall with Reasoning: Chain-of-Thought Distillation for Mamba’s Long-Context Memory and Extrapolation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context memory methods such as Mamba struggle with long-constituency when the length of the processed text exceeds the model's training length. |
| Approach: | They propose a method that uses chain-of-thought summarization to teach Mamba to actively recall and reason over long contexts. |
| Outcome: | Experiments on LONGMEMEVAL and HELMET show that RwR outperforms existing long-term memory methods while preserving short-context capabilities. |
Scalable Data Synthesis through Human-like Cognitive Imitation and Data Recombination (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) rely on massive amounts of training data, however, the quantity of empirically observed data is limited. |
| Approach: | They propose a data synthesis framework that mimics human cognitive behaviors by recombining and interconnecting heterogeneous data from diverse sources. |
| Outcome: | The proposed framework mimics human cognitive behaviors by recombining and interconnecting heterogeneous data from diverse sources thereby enhancing advanced reasoning capabilities in large language models. |
BeSimulator: A Large Language Model Powered Text-based Behavior Simulator (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing robot simulators focus on physical process modeling and realistic rendering, resulting in high computational costs and limited adaptability. |
| Approach: | They propose a modular and novel LLM-powered framework to analyze and validate robot behaviors in text-based environments. |
| Outcome: | The proposed framework can generalize across scenarios and achieve long-horizon complex simulation. |
Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs (2025.emnlp-main)
Copied to clipboard
Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, Xueqi Cheng
| Challenge: | Existing detection methods fail to account for **self-consistent error** . study identifies self-consistency errors and evaluates them . |
| Approach: | They propose a method that fuses hidden state evidence from an external verifier LLM to detect self-consistent errors. |
| Outcome: | The proposed method significantly enhances performance on self-consistent errors across three LLM families. |
pFedGPT: Hierarchically Optimizing LoRA Aggregation Weights for Personalized Federated GPT Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) struggle with data heterogeneity and adapt shared global knowledge to individual client needs. |
| Approach: | They propose a framework that leverages Hierarchical Bayesian Optimization (HBO) for fine-grained, personalized LoRA aggregation. |
| Outcome: | The proposed framework achieves state-of-the-art (SOTA) performance on personalized FL benchmarks while introducing only minimal (approx. 4%) additional optimization overhead. |
QSpec: Speculative Decoding with Complementary Quantization Schemes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Quantization is widely adopted to accelerate inference and reduce memory consumption in large language models. |
| Approach: | They propose a quantization paradigm that decouples efficiency from quality by integrating two complementary schemes via speculative decoding. |
| Outcome: | The proposed approach achieves 1.64x speedup without quality degradation and outperforms state-of-the-art speculative decoding methods by 1.55x in batched settings. |
Co-Evolving LLMs and Embedding Models via Density-Guided Preference Optimization for Text Clustering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text clustering use static pseudo-oracles, i.e., unidirectionally querying them for similarity assessment or data augmentation. |
| Approach: | They propose a training framework that enables bidirectional refinement between LLMs and embedding models by using task-aware prompts to guide the LLM in generating interpretations for the input texts. |
| Outcome: | Experiments on 14 benchmark datasets across 5 tasks demonstrate the effectiveness of the proposed training framework. |
P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs (2025.emnlp-main)
Copied to clipboard
Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou
| Challenge: | Recent advances in large language models showcase varied multilingual capabilities across tasks . previous assessments focused on fundamental natural language processing (NLP) or isolated capability-specific tasks. |
| Approach: | They propose a multilingual multitask benchmark to assess multilingual capabilities . they use a large-scale benchmark covering fundamental and capability-specialized datasets . |
| Outcome: | The proposed benchmark compares models and tasks across languages and tasks and examines knowledge transfer from English to other languages. |
Single LLM, Multiple Roles: A Unified Retrieval-Augmented Generation Framework Using Role-Specific Token Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have optimized retrieval-augmented generation (RAG) across sub-tasks, but integrating these optimizations into a unified framework remains challenging. |
| Approach: | They propose a unified retrieval-augmented generation framework that optimizes role-specific tokens for multi-task processing. |
| Outcome: | The proposed framework achieves efficient multi-task processing through role-specific token optimization. |
TrInk: Ink Generation with Transformer Network (2025.emnlp-main)
Copied to clipboard
Zezhong Jin, Shubhang Desai, Xu Chen, Biyi Fang, Zhuoyi Huang, Zhe Li, Chong-Xin Gan, Xiao Tu, Man-Wai Mak, Yan Lu, Shujie Liu
| Challenge: | Existing methods for handwriting generation capture global dependencies and can generate high-quality handwritten samples. |
| Approach: | They propose a Transformer-based model for ink generation, TrInk, which captures global dependencies. |
| Outcome: | The proposed model reduces character error rate and word error rate by 35.56% on the IAM-OnDB dataset compared to previous models. |
CalligraphicOCR for Chinese Calligraphy Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Increasing efforts to digitize calligraphy have rely on isolated character recognition, requiring expensive manual splitting into single characters. |
| Approach: | They propose a calligraphicOCR model with calligraphy image augmentation and action-based corrector targeting the root of the problem. |
| Outcome: | The proposed model outperforms baseline models due to visual variations and domain shifts in semantics and is more accurate than previous models. |
When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Audio-Language Models (LALMs) are augmented with the ability to perceive audio, but their reliability when faced with conflicting inputs remains largely unexplored. |
| Approach: | They examine how LALMs prioritize information when presented with inconsistent audio-text pairs. |
| Outcome: | The proposed models display a significant bias toward textual input when presented with inconsistent audio-text pairs. |
RESF: Regularized-Entropy-Sensitive Fingerprinting for Black-Box Tamper Detection of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for tamper detection rely on model stability, not inherently stochastic models. |
| Approach: | They propose a hypothesis-testing method for black-box tamper detection for LLMs . they propose regularized entropy-sensitive fingerprinting to enable efficient fingerprinting . |
| Outcome: | The proposed method achieves 98.80% detection accuracy under challenging conditions . it is based on a first-order surrogate for KL divergence to identify prompts most responsive to parameter perturbations. |
Model-based Large Language Model Customization as Service (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language model services require users to upload data for fine-tuning . current methods for customization are noisy and require sensitive domain data . |
| Approach: | *Llamdex is a framework that facilitates LLM customization as a service . client uploads pre-trained domain-specific *models* rather than data . |
| Outcome: | *Llamdex* framework improves domain-specific accuracy by up to 26% over state-of-the-art private data synthesis methods . |
Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents (2025.emnlp-main)
Copied to clipboard
Haochen Sun, Shuwen Zhang, Lujie Niu, Lei Ren, Hao Xu, Hao Fu, Fangkun Zhao, Caixia Yuan, Xiaojie Wang
| Challenge: | Large Language Models (LLMs) based agent systems have made great strides in real-world applications beyond traditional NLP tasks. |
| Approach: | They propose a new LLM-based Multi-Agent System benchmark, Collab-Overcooked, built on the popular Overcooked-AI game with more applicable and challenging tasks in interactive environments. |
| Outcome: | The proposed benchmark provides a multi-agent framework supporting diverse tasks and objectives and encourages collaboration through natural language communication. |
Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key Information (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on transferring teacher-generated rationales to student models, but do not explore teachers’ dynamic attention towards critical information during reasoning. |
| Approach: | They propose a method that transfers the teacher’s stepwise attention on key information to the student model and a Mixture of Layers module that allows dynamic alignment between the teacher and student. |
| Outcome: | The proposed framework achieves consistent performance improvements across multiple mathematical and commonsense reasoning datasets. |
Through the Valley: Path to Effective Long CoT Training for Small Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Long chain-of-thought (CoT) supervision is effective for large language models . but small models trained on limited long CoT data experience performance degradation . |
| Approach: | They identify a phenomenon called Long CoT Degradation in small language models . long CoT data can be used to generate long chain-of-thought (CoT) responses . |
| Outcome: | The results show that models trained on 8k long CoT examples lose up to 75% of their original performance before fine-tuning. |
RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate the superiority of our approach to aligning large language models with human preferences. |
| Approach: | They propose a method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
| Outcome: | The proposed method evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at various tasks but are vulnerable to jailbreak attacks that induce harmful content generation. |
| Approach: | They propose a reinforcement learning framework that leverages the model’s own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. |
| Outcome: | The proposed framework improves model safety by iterative self-improvement without additional annotated data or external models during training phase. |
InMind: Evaluating LLMs in Capturing and Applying Individual Human Reasoning Styles (2025.emnlp-main)
Copied to clipboard
Zizhen Li, Chuanhao Li, Yibin Wang, Qi Chen, Diping Song, Yukang Feng, Jianwen Sun, Jiaxin Ai, Fanrui Zhang, Mingzhu Sun, Kaipeng Zhang
| Challenge: | Recent large language models (LLMs) have demonstrated strong reasoning abilities across complex mathematical and scientific domains. |
| Approach: | They propose a framework to assess whether LLMs can capture and apply personalized reasoning styles in social deduction games. |
| Outcome: | The proposed framework evaluates LLMs on the game Avalon and shows that they can capture and apply individualized reasoning styles. |
MIO: A Foundation Model on Multimodal Tokens (2025.emnlp-main)
Copied to clipboard
Zekun Moore Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jessie Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang
| Challenge: | Existing models lack multimodal understanding capabilities, resulting in closed-source model that does not support multimodal interleaved sequences. |
| Approach: | They propose a foundation model built on multimodal tokens capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. |
| Outcome: | The proposed model is able to understand speech, text, images, and videos in an end-to-end, autoregressive manner. |
DART: Distilling Autoregressive Reasoning to Silent Thought (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use Chain-of-Thought (CoT) have been slow to deploy in real-time applications due to its autoregressive nature. |
| Approach: | They propose a framework that replaces autoregressive CoT with non-autoregressive Silent Thought (ST) the framework uses a lightweight Reasoning Evolvement Module to align hidden states with the CoT pathway and a Reasoning Embedment Module (REM) during inference, only the ST pathway is activated, enabling the ST tokens to evolve into informative embeddings. |
| Outcome: | The proposed framework replaces autoregressive CoT with non-autoregressive Silent Thought (ST) it enables LLMs to generate answers directly from ST tokens without additional computational cost . |
LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization (2025.emnlp-main)
Copied to clipboard
Qi Zhang, Shouqing Yang, Lirong Gao, Hao Chen, Xiaomeng Hu, Jinglei Chen, Jiexiang Wang, Sheng Guo, Bo Zheng, Haobo Wang, Junbo Zhao
| Challenge: | Recent research focuses on integrating reasoning capabilities into the realm of retrieval-augmented generation (RAG) via outcome-supervised reinforcement learning (RL). |
| Approach: | They propose a process-level reward module to mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. |
| Outcome: | The proposed framework can boost LLMs’ reasoning ability by integrating external knowledge sources through retrieval-augmented generation (RAG) The proposed model can mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. |
CYCLE-INSTRUCT: Fully Seed-Free Instruction Tuning via Dual Self-Training and Cycle Consistency (2025.emnlp-main)
Copied to clipboard
Zhanming Shen, Hao Chen, Yulei Tang, Shaolin Zhu, Wentao Ye, Xiaomeng Hu, Haobo Wang, Gang Chen, Junbo Zhao
| Challenge: | Existing methods for instruction tuning rely on expensive human-annotated seed data or powerful external teacher models. |
| Approach: | They propose a framework that achieves fully seed-free instruction tuning by employing a dual self-training loop where two models are bootstrapped solely from raw, unlabeled text. |
| Outcome: | The proposed framework outperforms seed-driven back-translation baselines and achieves comparable performance to strongly supervised methods. |
Good Intentions Beyond ACL: Who Does NLP for Social Good, and Where? (2025.emnlp-main)
Copied to clipboard
| Challenge: | 20% of all papers in the ACL Anthology address social good issues . authors are more likely to do work addressing social good concerns when publishing in venues outside of ACL. |
| Approach: | They use author- and venue-level perspectives to map the landscape of NLP4SG . they find authors are more likely to do work addressing social good concerns outside of ACL . |
| Outcome: | The study analyzes the literature on NLP4SG and its impact on the ACL community . 20% of all papers in the anthology address social good issues, the study finds . |
From General Reward to Targeted Reward: Improving Open-ended Long-context Generation Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current research on long-form context in Large Language Models (LLMs) focuses on understanding of long-contexts, but the open-ended Long Text Generation (Open-LTG) remains underexplored. |
| Approach: | They propose a method that uses data synthesis and a reward signal to enhance model performance. |
| Outcome: | The proposed method outperforms GPT-4-Turbo and improves performance by 20% on the Open-LTG task. |
Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Large Reasoning Models have demonstrated broad application potential, yet their safety and reliability remain critical concerns. |
| Approach: | They conduct a safety evaluation of 13 MLRMs across 5 benchmarks and examine their safety performance. |
| Outcome: | The proposed model improves safety on jailbreak and safety-awareness benchmarks. |
Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | LSLMs have impressive conversational generation abilities, but consistently fall short of traditional pipeline systems on semantic understanding benchmarks. |
| Approach: | They propose to analyze the performance gap between speech and text inputs through a systematic experiment . they find that representation similarity is strongly correlated with the modality gap . |
| Outcome: | The proposed models improve the accuracy of speech inputs and their semantic understanding benchmarks. |
AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models (MLLMs) have garnered significant attention, offering a promising pathway toward artificial general intelligence (AGI). |
| Approach: | They propose a benchmark to evaluate associative ability while circumventing the inherent ambiguity in association tasks by decomposing ambiguities into two types and propose 'assoCiAm' they conduct extensive experiments on MLLMs, revealing a strong positive correlation between cognition and association. |
| Outcome: | The proposed method shows that ambiguity in association evaluations makes MLLMs more random-like and the model's behavior more random. |
M-BRe: Discovering Training Samples for Relation Extraction from Unlabeled Texts with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extract training instances from unlabeled texts are expensive . sentences that contain the target relations in texts can be scarce and difficult to find . |
| Approach: | They propose a framework that can automatically extract training instances from unlabeled texts for RE. |
| Outcome: | The proposed method can extract training instances from unlabeled texts for RE. |
R-TOFU: Unlearning in Large Reasoning Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models embed private or copyrighted information in their final answers and throughout multi-step chain-of-thought (CoT) traces. |
| Approach: | They propose a benchmark for Large Reasoning Models that augments existing unlearning tasks with realistic CoT annotations and step-wise metrics that expose residual knowledge invisible to answer-level checks. |
| Outcome: | The proposed benchmark shows that answer-only objectives leave substantial forget traces in reasoning. |
Chat-Driven Text Generation and Interaction for Person Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text-based person search (TBPS) aims to retrieve images of a target individual from large-scale databases using natural language descriptions. |
| Approach: | They propose two modules to improve retrieval accuracy and usability . they generate rich pseudo-labels through simulated dialogues with MLLMs . |
| Outcome: | The proposed method achieves competitive or superior results while eliminating the need for manual captions. |
Spontaneous Giving and Calculated Greed in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models demonstrate strong problem-solving abilities through reasoning techniques such as chain-of-thought prompting and reflection. |
| Approach: | They evaluate off-the-shelf reasoning models in a public goods game and compare them with and without explicit reasoning mechanisms. |
| Outcome: | The proposed models reduce cooperation and norm enforcement, favoring individual rationality, and lower collective gains in repeated interactions. |
SenDetEX: Sentence-Level AI-Generated Text Detection for Human-AI Hybrid Content via Style and Context Fusion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text generated by Large Language Models (LLMs) now rivals human writing, raising concerns about its misuse. |
| Approach: | They propose a framework for sentence-level AI-generated text detection via style and context fusion. |
| Outcome: | The proposed framework outperforms baseline models in detection accuracy while exhibiting transferability and robustness. |
Judge and Improve: Towards a Better Reasoning of Knowledge Graphs with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to integrating graph and language models face two key limitations: achieving robust semantic alignment and ensuring interpretability in outputs. |
| Approach: | They propose a framework to integrate graph and language modalities while enhancing transparency. |
| Outcome: | Extensive experiments on three benchmark datasets show that the proposed framework surpasses existing methods in efficiency and generates outputs that are significantly more interpretable. |
Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on individual quality and do not assess the value of training data. |
| Approach: | They propose a choice-based sample selection framework that evaluates sample quality . they use LLMs to evaluate the value of each option during the selection process . |
| Outcome: | The proposed model outperforms the full dataset and recent studies on a larger medical dataset. |
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)
Copied to clipboard
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, Zheng Zhang
| Challenge: | Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference. |
| Approach: | They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes. |
| Outcome: | The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training. |
Cost-Optimal Grouped-Query Attention for Long-Context Modeling (2025.emnlp-main)
Copied to clipboard
Yingfa Chen, Yutong Wu, Chenyang Song, Zhen Leng Thai, Xingyu Shen, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Current GQA configurations overlook how context length influences inference cost . |
| Approach: | They propose a recipe for deriving cost-optimal GQA configurations that decouple the total head size from the hidden size and allow more flexible control over attention FLOPs. |
| Outcome: | The proposed configurations reduce memory usage and FLOPs by more than 50% compared to Llama-3's GQA, with *no degradation in model capabilities*. |
ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model (2025.emnlp-main)
Copied to clipboard
Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Yaxin Peng, Chaomin Shen, Feifei Feng, Yi Xu
| Challenge: | Recent advances in vision-language-action models prioritize robotic action mastery . however, models trained on visual-text pairs struggle to interpret multimodal data . |
| Approach: | They propose a framework that integrates multimodal data after initial control mastery and a Mixture-of-Experts architecture to minimize task interference. |
| Outcome: | The proposed framework surpasses state-of-the-art vision-language-action (VLA) methods on multimodal understanding benchmarks and achieves six times higher performance on visual question-answering datasets. |
KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
Ziyi Guan, Jason Chun Lok Li, Zhijian Hou, Pingping Zhang, Donglai Xu, Yuzhi Zhao, Mengyang Wu, Jinpeng Chen, Thanh-Toan Nguyen, Pengfei Xian, Wenao Ma, Shengchao Qin, Graziano Chesi, Ngai Wong
| Challenge: | Recent advances in GUI agents have limited app-specific knowledge of complex mobile tasks. |
| Approach: | They propose a Knowledge Graph-driven Retrieval-Augmented Generation framework that transforms fragmented UTGs into structured vector databases for efficient real-time retrieval. |
| Outcome: | The proposed framework outperforms existing methods in a 75.8% success rate and 84.6% decision accuracy test across mobile apps. |
CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies found that CLIP can only encode one aspect of the feature space, leading to substantial information loss and indistinctive features. |
| Approach: | They propose a model-agnostic approach that fine-tunes complementary CLIP models and transforms them into a CLIP-MoE. |
| Outcome: | The proposed framework fine-tunes a series of complementary CLIP models and transforms them into a CLIP-MoE. |
Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025.emnlp-main)
Copied to clipboard
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, Zhicheng Dou
| Challenge: | Large reasoning models (LRMs) have demonstrated impressive long stepwise reasoning capabilities through large-scale reinforcement learning. |
| Approach: | They propose a framework that enhances large reasoning models with an agentic retrieval-augmented generation mechanism and a Reason-in-Documents module for refining retrieved documents. |
| Outcome: | The proposed framework enhances LRMs with an agentic retrieval-augmented generation mechanism and Reason-in-Documents module for refining retrieved documents. |
From Personas to Talks: Revisiting the Impact of Personas on LLM-Synthesized Emotional Support Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that LLMs can infer persona traits and subtle shifts in emotionality and extraversion occur . scalable solutions with reduced costs and enhanced data privacy are needed . |
| Approach: | They explore the role of personas in the creation of emotional support conversations by LLMs. |
| Outcome: | The proposed model can infer persona traits and maintain key persona characteristics while revealing shifts in emotionality and extraversion. |
Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing task decomposition methods focus on memory, tool usage, and feedback mechanisms, but they often overlook the trade-off between performance and cost. |
| Approach: | They propose a strategy that selects the most suitable decomposition approach based on task characteristics and enhances the reliability of the results through a verification module. |
| Outcome: | The proposed strategy is based on categories of approaches, characteristics of tasks, and configuration of decomposition and execution models. |
TombRaider: Entering the Vault of History to Jailbreak Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing jailbreak techniques focus on prompt manipulation or intent obfuscation to bypass safety filters. |
| Approach: | They propose a jailbreak technique that exploits the ability to store, retrieve, and use historical knowledge of Large Language Models (LLMs) they use an inspector agent to extract historical information and an attacker agent to generate adversarial prompts, enabling effective bypassing of safety filters. |
| Outcome: | The proposed jailbreak technique outperforms state-of-the-art jailbreak techniques on six popular models and maintains over 55.4% ASR against defence mechanisms. |
Text Meets Topology: Rethinking Out-of-distribution Detection in Text-Rich Networks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for out-of-distribution (OOD) detection ignore textual-structural diversity . text-rich networks (TrNs) represent complex interplay between textual content and relational structures . |
| Approach: | They propose a framework for evaluating out-of-distribution detection in text-rich networks . they propose augmentations, structural shifts, and domain-based divisions to model interplay . |
| Outcome: | Experiments on 11 datasets show the framework is effective in out-of-distribution detection. |
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that RLHF improves performance of Large Language Models . BT-based RMs struggle to distinguish between similar preference responses . |
| Approach: | They propose to enhance BT-based reward models by using an adaptive margin mechanism . they use semantic similarity and reward-predicted reward differences to adjust focus . |
| Outcome: | Experimental results show that the proposed method outperforms existing methods in both in-distribution and OOD settings. |
HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have incorporated reward models to guide response selection or decoding, aiming to obtain higher-quality data. |
| Approach: | They propose a Hierarchical Sampling framework for self-taught reasoners that allocates a fixed sampling budget to problem boundary-level problems and then reallocates the remaining budget toward high-utility problems during a re-sampling phase. |
| Outcome: | The proposed framework outperforms baseline models without additional sampling budgets across multiple reasoning benchmarks and backbone LLMs. |
SEPS: A Separability Measure for Robust Unlearning in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing unlearning metrics assess whether a model correctly answers retain queries and rejects forget queries, but they fail to capture real-world scenarios where forget queries rarely appear in isolation. |
| Approach: | They propose an evaluation framework that explicitly measures a model’s ability to both forget and retain information within a single prompt. |
| Outcome: | The proposed approach significantly improves unlearning effectiveness, demonstrating robustness even in complex settings with up to eight mixed forget and retain queries in a single prompt. |
TRUST-VL: An Explainable News Assistant for General Multimodal Misinformation Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods focus on a single type of distortion and struggle to generalize to unseen scenarios. |
| Approach: | They propose a vision-language model that combines a question-aware visual amplifier module with a large-scale instruction dataset to support training. |
| Outcome: | The proposed model is able to generalize to multiple distortion types while requiring task-specific skills. |
Tree-of-Quote Prompting Improves Factuality and Attribution in Multi-Hop and Medical Reasoning (2025.emnlp-main)
Copied to clipboard
Justin Xu, Yiming Li, Zizheng Zhang, Augustine Yui Hei Luk, Mayank Jobanputra, Samarth Oza, Ashley Murray, Meghana Reddy Kasula, Andrew Parker, David W Eyre
| Challenge: | Large language models (LLMs) produce fluent but factually incorrect outputs, a phenomenon commonly referred to as hallucination. |
| Approach: | They propose a Tree-of-Quote framework that decomposes complex questions into subquestions and generates quotes to support each step without retrieval. |
| Outcome: | Experiments on StrategyQA, 2WikiMultiHopQA, MuSiQue, MoreHopQ, and MedQA show that ToQ improves factuality and attribution over baselines. |
UnitCoder: Scalable Code Synthesis from Pre-training Corpora (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at code understanding and generation, yet code generation remains a challenge. |
| Approach: | They propose a model that supervises pre-training data quality through automatically generated unit tests while ensuring correctness via an iterative fix and refine flow. |
| Outcome: | The proposed model improves performance on a large dataset with high quality pre-training data. |
GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for group-relative policy optimization face challenges in reward sparsity, verbosity and inadequate focus on problem difficulty. |
| Approach: | They propose a method to improve group relative policy optimization with length-regularized rewards and explicit penalties for incorrect solutions. |
| Outcome: | The proposed method achieves state-of-the-art performance for 14B-scale models . it improves reasoning accuracy, conciseness, and efficiency . |
Improving Low-Resource Sequence Labeling with Knowledge Fusion and Contextual Label Explanations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to sequence labeling are limited due to the scarcity of domain-specific data and semantic distribution biases in domain-based contexts. |
| Approach: | They propose a framework that integrates an LLM-based knowledge enhancement workflow with a span-based Knowledge Fusion for Rich and Efficient Extraction model. |
| Outcome: | The proposed model achieves state-of-the-art performance on multiple domain-specific sequence labeling datasets and is highly efficient. |
Rethinking Cross-Subject Data Splitting for Brain-to-Text Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have successfully decoded natural language from non-invasive brain signals . current dataset splitting methods suffer from data leakage problem . |
| Approach: | They propose a right cross-subject data splitting criterion without data leakage for decoding fMRI and EEG signal to text. |
| Outcome: | The proposed method overfits and overestimates brain-to-text decoding models. |
RCScore: Quantifying Response Consistency in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current evaluations of large language models rely on a single instruction template, overlooking models’ sensitivity to instruction style. |
| Approach: | They propose a multi-dimensional framework quantifying how instruction formulation affects model responses by transforming benchmark problems into multiple instruction styles. |
| Outcome: | The proposed framework reveals that instruction style can shift accuracy by 16.7% points. |
A Multi-Agent Framework with Automated Decision Rule Optimization for Cross-Domain Misinformation Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for misinformation detection are limited by domain knowledge and expert experience. |
| Approach: | They propose a Multi-Agent Framework for cross-domain misinformation detection with Automated Decision Rule Optimization (MARO) they first employ multiple expert agents to analyze target-domain news, then introduce a question-reflection mechanism that guides expert agents for higher-quality analysis. |
| Outcome: | The proposed framework improves on a common dataset and shows that iteratively improves over existing methods. |
OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new benchmark for RAG is developed for the financial domain . omnidirectional and automatic benchmarks are difficult to build in vertical domains . |
| Approach: | They propose an omnidirectional and automatic RAG benchmark for the financial domain . they categorize RAG scenarios by task classes and 16 financial topics . |
| Outcome: | The proposed benchmark achieves an 87.47% acceptance ratio in human evaluations of generated instances. |
AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to synthesis large language models often suffer from performance limitations and high computational costs. |
| Approach: | They propose a framework for constructing instruction-tuning data from unlabeled data for any specialized domains from corresponding unlabed data. |
| Outcome: | The proposed framework is comparable to DeepSeek-V3 while utilizing just 17% of the production cost. |
MoSEs: Uncertainty-Aware AI-Generated Text Detection via Mixture of Stylistics Experts with Conditional Thresholds (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods neglect stylistic modeling and rely on static thresholds, which greatly limits the detection performance. |
| Approach: | They propose a framework that enables stylistics-aware uncertainty quantification through conditional threshold estimation. |
| Outcome: | The proposed framework achieves an average improvement 11.34% in detection performance compared to baselines. |
Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Model merging is a promising approach for updating large language models . but unmonitored mergers can introduce significant security vulnerabilities . |
| Approach: | They propose a model merging attack surface where a malicious merger can extract PII from an aligned model with model merg. |
| Outcome: | The proposed framework can extract PII from an aligned model with model merging. |
Pragmatic Inference Chain (PIC) Improving LLMs’ Reasoning of Authentic Implicit Toxic Language (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that LLMs can detect toxicity by using a variety of inference-intensive tasks, such as understanding humour and metaphors. |
| Approach: | They propose a new method to prompt LLMs to identify toxic language using a set of online data that are verified by human annotators. |
| Outcome: | The proposed method significantly improves the success rate of GPT-4o, Llama-3.1-70B-Instruct, DeepSeek-v2.5, and DeepSeq-v3 in identifying implicit toxic language compared to five baseline prompts, such as CoT and rule-based baselines. |
Beyond Demonstrations: Dynamic Vector Construction from Latent Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for In-Context Learning (ICL) are sensitive to ICL-specific factors and rely on heuristic-based injection positions. |
| Approach: | They propose a method that extracts task-relevant representations from large language models and reinjects them during inference. |
| Outcome: | The proposed method outperforms few-shot In-Context Learning (ICL) and LoRA methods without repeated demonstration processing. |
Detoxifying Large Language Models via the Diversity of Toxic Samples (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for analyzing and utilizing toxic samples are limited . current methods fail to fully harness their potential . |
| Approach: | They propose a diverse detoxification framework that leverages toxic samples' diversity . they propose MPSG strategy and SC-DPO approach to elicit personalized toxic responses . |
| Outcome: | The proposed framework could be used to optimize large language models for user safety . it incorporates two components: MPSG strategy and SC-DPO approach . |
LLM-Driven Implicit Target Augmentation and Fine-Grained Contextual Modeling for Zero-Shot and Few-Shot Stance Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on zero-shot and few-shot stance detection neglect implicit yet semantically important targets. |
| Approach: | They propose a framework that uses Large Language Models to annotate implicit targets . they also propose 'DyMCA' to dynamically adjust text-target contributions based on context . |
| Outcome: | The proposed framework achieves state-of-the-art on a benchmark dataset. |
Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing intent clustering methods rely on embedding distance metrics and neglect of underlying semantic structures. |
| Approach: | They propose an LLM-in-the-loop framework that integrates language understanding capabilities into conventional clustering algorithms. |
| Outcome: | The proposed framework outperforms baselines in Chinese and improves quality, cost efficiency and downstream applications. |
Superficial Self-Improved Reasoners Benefit from Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) rely heavily on large-scale reasoning data, but as data becomes scarce, model self-improvement offers a promising alternative. |
| Approach: | They propose to merge the weights of original and self-improved LLMs to mitigate model collapse and improve generalized reasoning capability. |
| Outcome: | The proposed model merge mitigates model collapse and improves generalized reasoning capability. |
CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve the reasoning performance of Large Language Models (LLMs) ignore annotated Chain-of-Thought (CoT) and incorporate unstable reasoning path sampling. |
| Approach: | They propose a Contrastive learning with annotated CoT-based Reinforced Fine-Tuning approach to enhance the reasoning performance of Large Language Models. |
| Outcome: | The proposed approach exploits annotated CoT and stabilizes the fine-tuning procedure by incorporating an additional unsupervised learning signal. |
QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks often lack domain coverage and provide limited insights into the working context of Chinese LLMs. |
| Approach: | They propose a multi-domain Chinese QA benchmark dedicated to localized assessment of Chinese LLMs. |
| Outcome: | The Qwen2.5 model outperforms the more advanced GPT-4o model in the Chinese market . the dataset includes over 17,000 questions across six vertical domains . |
VideoEraser: Concept Erasure in Text-to-Video Diffusion Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that VideoEraser outperforms prior methods regarding efficacy, integrity, fidelity, robustness, and generalizability. |
| Approach: | They propose a training-free framework that prevents T2V diffusion models from generating videos with undesirable concepts even when explicitly prompted with those concepts. |
| Outcome: | The proposed framework outperforms existing methods in erasure, celebrity erasion, and explicit content erasing tasks. |
Diagram-Driven Course Questions Generation (2025.emnlp-main)
Copied to clipboard
Xinyu Zhang, Lingling Zhang, Yanrui Wu, Muye Huang, Wenjun Wu, Bo Li, Shaowei Wang, Basura Fernando, Jun Liu
| Challenge: | Visual Question Generation (VQG) research focuses on natural images while neglecting diagrams, a critical component of educational materials. |
| Approach: | They propose a diagram-driven course questions generation task to generate diagram-relevant questions for specific courses. |
| Outcome: | The proposed framework outperforms existing models on DiagramQG while maintaining strong generalizability across natural image datasets. |
ECC: An Emotion-Cause Conversation Dataset for Empathy Response (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing empathy dialogue datasets focus on emotion labels while cause annotations are added post hoc. |
| Approach: | They propose an emotion-cause conversation dataset with 2.4K dialogues that can be scalable . they use a framework that utilizes knowledge and large language models to automatically generate dialogues . |
| Outcome: | The proposed dataset can achieve comparable or even superior performance to existing empathy dialogue datasets. |
ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Unlike previous works that manipulate representations to steer LLM generation, ThoughtProbe harnesses them as discriminative signals to guide the tree-structured response space exploration. |
| Approach: | They propose a tree-structured inference-time framework that leverages the hidden reasoning features of Large Language Models to improve their reasoning performance. |
| Outcome: | The proposed framework improves reasoning performance across multiple arithmetic reasoning benchmarks and covers valid reasoning chains and identifies optimal answers. |
JOLT-SQL: Joint Loss Tuning of Text-to-SQL with Confusion-aware Noisy Schema Sampling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have improved Text-to-SQL methods . however, they still face challenges such as complex multi-stage pipelines and poor robustness to noisy schema information. |
| Approach: | They propose a single-stage SFT framework that optimizes schema linking and SQL generation via a unified loss. |
| Outcome: | Experiments on the Spider and BIRD benchmarks show that JOLT-SQL achieves state-of-the-art execution accuracy among comparable-size open-source models. |
DMDTEval: An Evaluation and Analysis of LLMs on Disambiguation in Multi-domain Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, Large Language Models (LLMs) have achieved remarkable results in machine translation, but their performance in multidomain translation (MDT) is less satisfactory. |
| Approach: | They propose to evaluate the disambiguation ability of Large Language Models in multi-domain translation . they construct a translation test set with multi- domain ambiguous word annotation . |
| Outcome: | The proposed framework evaluates LLMs on disambiguation in multi-domain translation (DMDTEval) the results show that LLM's perform poorly in multidomain translation, highlighting ambiguity in translation. |
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)
Copied to clipboard
David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
| Challenge: | ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs. |
| Approach: | They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions . |
| Outcome: | The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions. |
MAKAR: a Multi-Agent framework based Knowledge-Augmented Reasoning for Grounded Multimodal Named Entity Recognition (2025.emnlp-main)
Copied to clipboard
Xinkui Lin, Yuhui Zhang, Yongxiu Xu, Kun Huang, Hongzhang Mu, Yubin Wang, Gaopeng Gou, Li Qian, Li Peng, Wei Liu, Jian Luan, Hongbo Xu
| Challenge: | Existing methods for GMNER fail to address semantic ambiguity caused by polysemy and long-tail distribution of datasets. |
| Approach: | They propose a framework for Grounded Multimodal Named Entity Recognition that leverages a Multimodal Large Language Model to address semantic ambiguity. |
| Outcome: | Extensive experiments show that the proposed framework outperforms existing methods on two benchmark datasets. |
VisCRA: A Visual Chain Reasoning Attack for Jailbreaking Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Reasoning Models (LRMs) have enabled sophisticated visual reasoning capabilities by integrating reinforcement learning and Chain-of-Thought (CoT) supervision. |
| Approach: | They propose a jailbreak framework that exploits visual reasoning chains to bypass safety mechanisms. |
| Outcome: | The proposed framework achieves high attack success rates on leading closed-source MLRMs. |
Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on surface-level display of performance degradation due to typos. |
| Approach: | They propose a method to identify typo neurons and typo heads that work actively when inputs contain typos. |
| Outcome: | The proposed method identifies typo neurons and typo heads that work actively when inputs contain typos. |
LMR-BENCH: Evaluating LLM Agent’s Ability on Reproducing Language Modeling Research (2025.emnlp-main)
Copied to clipboard
Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, Juntong Ni, George Michalopoulos, Yue Zhang, Ziyang Zhang, Mian Zhang, Zhiyu Chen, Xinya Du
| Challenge: | Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery, but their capability in reproducing code from research papers remains underexplored. |
| Approach: | They propose to evaluate LLM agents' ability to reproduce scientific research papers by analyzing code reproduction tasks from 23 research papers published in top-tier NLP venues. |
| Outcome: | The proposed benchmark systematically evaluates the capability of large language model (LLM) agents on code reproduction from Language Modeling Research. |
RAV: Retrieval-Augmented Voting for Tactile Descriptions Without Training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional approaches relying on extensive parameter learning for multimodal perception are rigid and computationally inefficient. |
| Approach: | They propose a parameter-free method that constructs visual-tactile cross-modal knowledge directly by retrieving similar visual-touch data for given visual and tactile inputs and generating tactile descriptions through a voting mechanism. |
| Outcome: | The proposed method achieves comparable performance to large-scale cross-modal models without training across a wide range of datasets. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
PropRAG: Guiding Retrieval with Beam Search over Proposition Paths (2025.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval Augmented Generation (RAG) is a non-parametric approach for large language models. |
| Approach: | They propose a framework that shifts from triples to context-rich propositions and introduces an efficient, LLM-free online beam search over proposition paths to discover multi-step reasoning chains. |
| Outcome: | The proposed framework achieves state-of-the-art zero-shot Recall@5 and F1 scores on 2Wiki, HotpotQA, and MuSiQue. |
Rethinking Backdoor Detection Evaluation for Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing backdoor detection methods have high accuracy in detecting backdoored models, but they are not robust enough to detect backdoors in the wild. |
| Approach: | They examine the robustness of backdoor detectors by manipulating different factors during backdoor planting. |
| Outcome: | The proposed methods are able to detect backdoors in the wild, but they lack robustness against backdoor attacks. |
Glider: Global and Local Instruction-Driven Expert Router (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for routing-based expert models favor generalization over performance on held-in tasks. |
| Approach: | They propose a global and local instruction driven expert router that leverages recent LLMs' semantic reasoning capabilities to generate task-specific instructions from the input query. |
| Outcome: | The proposed method improves held-in performance while maintaining strong generalization on held-out tasks. |
CoVoGER: A Multilingual Multitask Benchmark for Speech-to-text Generative Error Correction with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can fix recognition or translation errors that traditional rescoring cannot fix. |
| Approach: | They propose a benchmark for GER that covers both ASR and speech-to-text translation across 15 languages and 28 language pairs. |
| Outcome: | The proposed benchmark is built on common voice 20.0 and CoVoST-2 with Whisper and SeamlessM4T. |
Tiny Budgets, Big Gains: Parameter Placement Strategy in Parameter Super-Efficient Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods such as LoRA and VeRA use memory-efficient methods to fine-tune large language models. |
| Approach: | They propose a method that uses only 1–5% of the standard LoRA parameters and achieves state-of-the-art performance across a wide range of tasks. |
| Outcome: | The proposed method achieves state-of-the-art performance across a wide range of tasks using only 1–5% of the standard LoRA parameters. |
Legal Fact Prediction: The Missing Piece in Legal Judgment Prediction (2025.emnlp-main)
Copied to clipboard
Junkai Liu, Yujie Tong, Hui Huang, Bowen Zheng, Yiran Hu, Peicheng Wu, Chuan Xiao, Makoto Onizuka, Muyun Yang, Shuyuan Zheng
| Challenge: | Existing studies use legal facts to predict judgments, but legal facts are difficult to obtain in early stages of litigation. |
| Approach: | They propose a legal fact prediction task that takes evidence from trial as input to make predictions in the absence of ground-truth legal facts. |
| Outcome: | The proposed task can predict court rulings without ground-truth legal facts . the first benchmark dataset, LFPBench, is used to evaluate the task . |
DAMON: A Dialogue-Aware MCTS Framework for Jailbreaking Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-turn attacks mainly utilize a predefined dialogue pattern, limiting their effectiveness in realistic situations. |
| Approach: | They propose a multi-turn jailbreak attack method that leverages Monte Carlo Tree Search to explore multi-turned conversational spaces and identifies sub-instruction sequences that induce harmful responses. |
| Outcome: | The proposed method can induce undesired behaviors across five LLMs and three datasets. |
Multilingual Prompting for Improving LLM Generation Diversity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models lack cultural representation and diversity in their generations . lack of demographic diversity can lead to unfair lack of exposure of artists . |
| Approach: | They propose a prompting method which generates several variations of a base prompt with added cultural and linguistic cues from several cultures, generates responses, and then combines the results. |
| Outcome: | The proposed method outperforms existing diversity-enhancing techniques . it can generate multiple variations of a base prompt with cultural cues from multiple cultures . |
MOSAIC: Modeling Social AI for Content Dissemination and Regulation in Multi-Agent Simulations (2025.emnlp-main)
Copied to clipboard
| Challenge: | generative language agents predict user behaviors such as liking, sharing, and flagging content. |
| Approach: | They propose a framework where generative language agents predict user behaviors such as liking, sharing, and flagging content. |
| Outcome: | The proposed framework analyzes content moderation strategies and user engagement dynamics at scale and demonstrates that agents’ articulated reasoning for their social interactions aligns with their collective engagement patterns. |
Identification of Multiple Logical Interpretations in Counter-Arguments (2025.emnlp-main)
Copied to clipboard
Wenzhi Wang, Paul Reisert, Shoichi Naito, Naoya Inoue, Machi Shimmei, Surawat Pothong, Jungmin Choi, Kentaro Inui
| Challenge: | Counter-arguments (CAs) are a good way to improve learners' critical thinking skills . however, it is difficult to provide every learner tailored feedback due to limited human resources and heavy workloads. |
| Approach: | They propose to annotate a dataset of 134 CAs annotated with 13 logical predicate questions and train a model with Reinforcement Learning with Verifiable Rewards to identify multiple logical interpretations. |
| Outcome: | The proposed model performs on par with larger proprietary models. |
LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for enhancing knowledge updating are prone to performance degradation due to incomplete knowledge preservation mechanisms. |
| Approach: | They propose a model for locate-then-edit that decomposes long-term constrained programming into tractable stepwise subproblems for efficient solving. |
| Outcome: | The proposed framework achieves asymptotic optimal editing performance while meeting the constraints of long-term knowledge preservation. |
AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large language models (LLMs) possess impressive multilingual understanding and generation capabilities, but performance and cross-lingual alignment often lag for non-dominant languages. |
| Approach: | They propose a representation-level framework to enhance multilingual performance of pre-trained LLMs by integrating multilingual semantic alignment and language feature integration. |
| Outcome: | The proposed framework improves multilingual capability of pre-trained LLMs by bringing representations closer and improving cross-lingual alignment. |
What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in reasoning with large language models have popularized Long Chain-of-Thought (LCoT) a framework that converts sequential LCoTs into hierarchical tree structures enables deeper structural analysis of LLM reasoning. |
| Approach: | They propose a framework that converts sequential LCoTs into hierarchical tree structures and enables deeper structural analysis of LLM reasoning. |
| Outcome: | The proposed framework can be used to analyze LLM reasoning in a variety of tasks and models. |
HD-PiSSA: High-Rank Distributed Orthogonal Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models constrain update to low-rank subspaces, limiting expressiveness and performance. |
| Approach: | They propose a distributed PEFT approach that initializes adapters across different devices and aggregates their delta updates collectively on (W) Empirically, HD-PiSSA provides 16 higher effective updated ranks than data-parallel LoRA or PiSSA when fine-tuning on 8 GPUs with the same per-device adapter rank. |
| Outcome: | Empirically, HD-PiSSA outperforms LoRA and PiSSA in math, code, and multi-task learning tasks. |
Firewall Routing: Blocking Leads to Better Hybrid Inference for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have significantly enhanced performance across various NLP tasks . high computational costs and latency associated with deploying such models pose bottlenecks . |
| Approach: | They propose a dynamic hybrid inference framework that efficiently selects between a strong and a weak LLM based on the complexity of the query. |
| Outcome: | The proposed method outperforms existing routing strategies by up to 5.29% in APGR . large models often introduce higher latency, making them less suitable for real-time or resource-constrained applications. |
SPE Attention: Making Attention Equivariant to Semantic-Preserving Permutation for Code Processing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train code processing models to capture symmetry of code . semantic-preserving permutations are not found in natural languages . |
| Approach: | They propose a mechanism that captures a unique symmetry of code, called the SPE attention . they propose symmetry graphs that are then combined to create a symmetry mask . |
| Outcome: | The proposed model can be used to analyze code summarization and error detection tasks. |
Audio-centric Video Understanding Benchmark without Text Shortcut (2025.emnlp-main)
Copied to clipboard
Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, Chao Zhang
| Challenge: | Recent advances in multimodal large language models (MLLMs) focus on visual abilities, but audio is essential for video understanding. |
| Approach: | They propose an audio-centric video understanding benchmark to evaluate video comprehension capabilities of multimodal LLMs with a particular focus on auditory information. |
| Outcome: | The proposed video understanding benchmarks evaluate video comprehension capabilities of multimodal models with a particular focus on auditory information. |
TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current RAG systems concatenate and process numerous retrieved document chunks for prefill . this leads to significant latency in time-to-first-token (TTFT) Experimental results demonstrate that TurboRAG reduces TTFT by up to 9.4x compared to the conventional RAG system. |
| Approach: | They propose a hybrid offline-online paradigm that precomputes chunk-level key-value caches and stitches them together at inference time using independent–attention and reorderedRoPE techniques. |
| Outcome: | Experimental results show that TurboRAG reduces TTFT by 9.4x compared to the conventional RAG systems . long concatenated contexts consume disproportionate GPU memory, limiting throughput . |
ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have shown impressive capabilities in vision-language understanding but their visual input remains fixed throughout the reasoning process. |
| Approach: | They propose a model-agnostic tree search algorithm tailored for vision-level reasoning that allows MLLMs to explore textual tokens while visual input remains fixed throughout reasoning process. |
| Outcome: | The proposed algorithm outperforms strong large models such as GPT-4o on high-resolution benchmarks and improves performance on a series of elaborate high-level benchmarks. |
Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on challenging mathematical reasoning benchmarks demonstrate that these human-inspired strategies synergistically and significantly enhance performance. |
| Approach: | They propose to use Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation to improve model performance. |
| Outcome: | Extensive experiments on mathematical reasoning benchmarks show that the proposed strategies synergistically and significantly improve performance over the baseline model. |
VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains. |
| Approach: | They propose a data composition framework that allows LLMs to enhance their multi-domain capabilities during supervised fine-tuning. |
| Outcome: | The proposed framework improves multi-domain fostering performance by 29.77% compared to uniform weights. |
FlightGPT: Towards Generalizable and Interpretable UAV Vision-and-Language Navigation with Vision-Language Models (2025.emnlp-main)
Copied to clipboard
Hengxing Cai, Jinhan Dong, Jingjun Tan, Jingcheng Deng, Sihang Li, Zhifeng Gao, Haidong Wang, Zicheng Su, Agachai Sumalee, Renxin Zhong
| Challenge: | Existing methods for vision-and-language navigation struggle with insufficient multimodal fusion, weak generalization, and poor interpretability. |
| Approach: | They propose a framework for UAV vision-and-language navigation that integrates natural language instructions with visual observations to improve multimodal fusion and interpretability. |
| Outcome: | The proposed framework achieves state-of-the-art performance across all scenarios, with a 9.22% higher success rate than the strongest baseline in unseen environments. |
Multimodal Language Models See Better When They Look Shallower (2025.emnlp-main)
Copied to clipboard
Haoran Chen, Junyan Lin, Xinghao Chen, Yue Fan, Jianfeng Dong, Xin Jin, Hui Su, Jinlan Fu, Xiaoyu Shen
| Challenge: | Existing studies show that multimodal large language models extract visual features from the final layers of a pretrained Vision Transformer. |
| Approach: | They propose a feature fusion method that strategically incorporates shallower layers . they propose MLLMs that extract visual features from the final layers of a pretrained Vision Transformer . |
| Outcome: | The proposed method outperforms deep layers on fine-grained visual tasks . it is the first comprehensive study of visual layer selection for MLLMs . |
LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. |
| Approach: | They propose a method that localizes and optimizes critical parameters during training . they propose 'LoSiA-Pro' which reduces training latency by 27% . |
| Outcome: | The proposed method achieves minimal performance drop compared to full fine-tuning while requiring the least training time across domain specialization and common-sense reasoning tasks. |
Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to watermark low-entropy content are expensive and risky . IE reduces parameter size by 99% while achieving performance on par with state-of-the-art methods . |
| Approach: | They propose a logit-based watermarking paradigm that uses entropy-based features to predict whether the next token is high or low. |
| Outcome: | The proposed method reduces parameter size by 99% while achieving performance on par with state-of-the-art methods. |
Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases (2025.emnlp-main)
Copied to clipboard
| Challenge: | a growing number of efforts to measure and mitigate gender bias have focused on task prompts that overtly or covertly signal the presence of gender bias-related content. |
| Approach: | They examine how signaling the evaluative purpose of a task impacts measured gender bias in LLMs. |
| Outcome: | The proposed models show that prompts that align with (gender bias) evaluation framing elicit distinct gender output distributions compared to less evaluation-framed prompts. |
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to accelerate autoregressive generation of large language models require training costs. |
| Approach: | They propose a training-free alignment-augmented speculative decoding algorithm . it leverages the output distribution obtained in the prefilling phase to provide more aligned draft candidates . |
| Outcome: | The proposed method increases the average generation score by 3.3 points for the LLaMA3 model. |
ViLBench: A Suite for Vision-Language Process Reward Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite its advantages, evaluation on PRMs remains less explored, especially in the multimodal domain. |
| Approach: | They propose to benchmark vision large language models as output reward models and process reward models as process-supervised reward models. |
| Outcome: | The proposed model outperforms both ORM and PRM on vision-language benchmarks and achieves an average improvement of 3.3% over standard CoT and up to 2.5% over its untrained counterpart on ViLBench. |
Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly deployed in sensitive domains . large-scale benchmarks for contextual security preservation against attacks remain lacking . |
| Approach: | They evaluate 10 Large Language Models on a benchmark dataset to assess their adherence to contextual non-disclosure policies. |
| Outcome: | The proposed model fails to adhere to user-defined security policies in question answering . the model fails in indirect attacks, especially when it violates user-definable policies . |
Route Sparse Autoencoder to Interpret Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) extract interpretable and monosemantic features in large language models . prior work focused on feature extraction from a single layer, failing to capture activations that span multiple layers. |
| Approach: | They propose a framework that integrates a routing mechanism with a shared SAE to efficiently extract features from multiple layers. |
| Outcome: | The proposed framework extracts features from multiple layers while incurring minimal parameter overhead while achieving high interpretability and flexibility. |
BTS: Harmonizing Specialized Experts into a Generalist LLM (2025.emnlp-main)
Copied to clipboard
Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Nicolaus Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis
| Challenge: | Branch-Train-Stitch (BTS) is an efficient and flexible training algorithm for combining independently trained large language model (LLM) experts into a single, capable generalist model. |
| Approach: | They propose an efficient and flexible training algorithm for combining large language model (LLM) experts into a single, capable generalist model using lightweight stitch layers. |
| Outcome: | The proposed model can generalize to new domains despite being frozen . it yields the best generalist performance on a variety of downstream tasks, retaining the specialized capabilities of each of the experts. |
CoCoA: Confidence- and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing contrastive decoding methods that handle conflict lack adaptability and can degrade performance in low conflict settings. |
| Approach: | They propose a token-level algorithm for principled conflict resolution and enhanced faithfulness that resolves conflict by utilizing confidence-aware measures and the generalized divergence between parametric and contextual distributions. |
| Outcome: | The proposed algorithm achieves 9.2 points on average in QA, summarization, and long-form question answering (LFQA) benchmarks and improves factuality by 2.5 points on the key benchmarks. |
R-Bind: Unified Enhancement of Attribute and Relation Binding in Text-to-Image Diffusion Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve semantic binding require costly retraining or focus on only correctly generating attributes of entities. Existing methods focus on correctly generating attributes, ignoring the cruciality of correctly forming relations between entities. |
| Approach: | They propose a training-free method that improves both entity-attribute and entity-relation-entity binding by introducing three inference-time optimization losses that adjust attention maps during generation. |
| Outcome: | The proposed method improves both entity-attribute and entity-relation-entity binding without additional training. |
Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve data quality face limitations in static dataset curation that fail to adapt to evolving model capabilities. |
| Approach: | They propose a self-evolving framework that uses model-aware data selection and context-preserving data refinement to improve LLM performance. |
| Outcome: | The proposed framework improves the quality of seed data and boosts LLM’s performance with improving accuracy by 7.15% on average while maintaining the original dataset scale. |
Information Integration in Large Language Models is Gated by Linguistic Structural Markers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that LLMs can integrate information across multiple scales of language . |
| Approach: | They propose to quantify the information integration window of large language models and measure the information loss from relying solely on the local window. |
| Outcome: | The proposed method shows that the integration window of large language models is strongly modulated by sentence boundaries. |
Why and How LLMs Benefit from Knowledge Introspection in Commonsense Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can improve commonsense reasoning by generating intermediate knowledge, but the effectiveness of this knowledge introspection is not always guaranteed. |
| Approach: | They propose a training-free strategy that optimizes introspection via two stages: Knowledge Detection and Knowledge Regeneration. |
| Outcome: | The proposed approach mitigates the limitations of standard introspection and has consistent performance gains across all settings. |
GraDaSE: Graph-Based Dataset Search with Examples (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods address ad hoc dataset search, but dataset search presents in diverse and complex forms. |
| Approach: | They propose a graph-based approach to retrieve relevant datasets from textual queries . they identify provenance-based and topic-based relationships to construct a diagram . |
| Outcome: | The proposed approach outperforms strong baselines on two test collections. |
Confidence-guided Refinement Reasoning for Zero-shot Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks that generate single-step reasoning do not improve QA reasoning . |
| Approach: | They propose a framework that strategically constructs and refines sub-questions and their answers (sub-QAs) they argue that sub-QA does not always enhance QA reasoning . |
| Outcome: | The proposed framework can be integrated with existing QA models and benchmarks. |
DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often prioritize reasoning over adherence to detailed instructions due to high computational costs and limited parameter access. |
| Approach: | They propose a lightweight framework that guides small language models to refine LLMs’ outputs through chain-of-thought correction. |
| Outcome: | The proposed framework improves the average format accuracy and content correctness of LLM outputs by 35.4% and 29.4%, respectively, achieving state-of-the-art (SOTA) performance over other competitive baselines. |
CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fingerprinting model ownership traces are vulnerable to illegal plagiarism and are not reliable. |
| Approach: | They propose a rule-driven fingerprinting framework that encodes contextual correlations across multiple dialogue turns. |
| Outcome: | The proposed framework achieves stronger stealth and robustness than previous work. |
Realistic Training Data Generation and Rule Enhanced Decoding in LLM for NameGuess (2025.emnlp-main)
Copied to clipboard
| Challenge: | Abbreviated column names often harm downstream tasks, causing performance drops of 10.54, 40.50, and 3.83 percentage points. |
| Approach: | They propose a method that integrates a subsequence abbreviation generator trained on human-annotated data and collects non-subsequent abbrevations to improve the training set. |
| Outcome: | The proposed approach improves on the English NameGuess task and surpasses state-of-the-art LLMs. |
EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing fingerprinting methods require impractical white-box access or introduce detectable statistical anomalies. |
| Approach: | They propose a gray-box fingerprinting framework that ensures stealthy and robust model provenance tracing. |
| Outcome: | The proposed framework is the first to repurpose Membership Inference Attacks (MIAs) for defensive use, embedding ownership signals via memorization instead of artificial trigger-output overfitting. |
Selective Preference Optimization via Token-Level Reward Function Estimation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for maximizing preference optimization on all available tokens are noisy and inefficient. |
| Approach: | They propose a selective alignment strategy that centers on efficient key token selection without strong, fine-grained supervision signals. |
| Outcome: | The proposed strategy outperforms baseline methods on three benchmarks with up to 60% reduction in training hours. |
Arena-lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current benchmarks typically compare system outputs against baselines, but this method yields lower reliability than direct comparison. |
| Approach: | They propose to integrate tournament structure on top of head-to-head comparison. |
| Outcome: | The proposed model achieves higher reliability with fewer comparisons even with smaller datasets or weaker judges. |
Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have improved the capacities and efficiency of text generation. |
| Approach: | They propose a method for tokenization inconsistency and a watermarking technique to address this problem. |
| Outcome: | The proposed methods improve fluency, imperceptibility, and anti-steganalysis capacity. |
ExeCoder: Empowering Large Language Models with Executability Representation for Code Translation (2025.emnlp-main)
Copied to clipboard
Minghua He, Yue Chen, Fangkai Yang, Pu Zhao, Wenjie Yin, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
| Challenge: | Existing code translation models only learn the contextual semantics of code during pre-training, neglecting executability information closely related to the execution state of the code. |
| Approach: | They propose an LLM specifically designed for code translation called ExeCoder . it uses executability representations such as functional semantics and syntax structures to enhance LLMs' capabilities. |
| Outcome: | The proposed model outperforms existing open-source code translation models on two metrics. |
TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TableQA benchmarks focus on simple flat tables and suffer from data leakage . current benchmarks are monolingual and fail to capture cross-lingual variability . |
| Approach: | They propose a table-based TableQA benchmark to evaluate LLMs on real-world tasks. |
| Outcome: | The proposed benchmarks show that they achieve high agreement with human judgment . the proposed framework improves on the alignment between model responses and reference answers . |
NOVA-63: Native Omni-lingual Versatile Assessments of 63 Disciplines (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks show severe drawbacks, such as overly translated content, the absence of difficulty control, and disciplinary imbalance, making the benchmarking process unreliable and showing low convincingness. |
| Approach: | They propose a multilingual benchmark that integrates LLM-assisted formatting, expert quality verification, and multi-level difficulty screening to provide a comprehensive, difficult multilingual assessment. |
| Outcome: | The proposed benchmark features 93,536 questions sourced from native speakers across 14 languages and 63 academic disciplines. |
InfoGain-RAG: Boosting Retrieval-Augmented Generation through Document Information Gain-based Reranking and Filtering (2025.emnlp-main)
Copied to clipboard
Zihan Wang, Zihan Liang, Zhou Shao, Yufei Ma, Huangyu Dai, Ben Chen, Lingtao Mao, Chenyi Lei, Yuqing Ding, Han Li
| Challenge: | Retrieval-Augmented Generation (RAG) frameworks struggle with identifying whether retrieved documents meaningfully contribute to answer generation. |
| Approach: | They propose a document-related metric to quantify the contribution of retrieved documents to correct answer generation. |
| Outcome: | The proposed framework outperforms existing approaches on both single and multiple retrieval paradigms. |
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video large language models (Vid-LLMs) rely on dense video token representations and require substantial memory and computational overhead in both prefilling and decoding. |
| Approach: | They propose a training-free speculative decoding framework that prunes up to 90% of video tokens to enable efficient speculation without sacrificing accuracy. |
| Outcome: | The proposed framework achieves 2.68 speedup on LLaVA-OneVision-72B and 2.11 speed up on Qwen2.5-VL-32B. |
What Do Indonesians Really Need from Language Technology? A Nationwide Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite efforts to develop NLP for Indonesia’s 700+ local languages, progress remains costly due to the need for direct engagement with native speakers. |
| Approach: | They conduct a nationwide survey to assess the actual needs of native Indonesian speakers. |
| Outcome: | The findings indicate that addressing language barriers is the most critical priority . concerns around privacy, bias, and the use of public data highlight the need for greater transparency and clear communication to support broader AI adoption. |
LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Experts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to reduce visual tokens have been criticized for their computational efficiency and lack of visual reasoning capabilities. |
| Approach: | They propose a novel multi-modal large language model that reduces the number of visual tokens and simultaneously boosts visual reasoning capabilities. |
| Outcome: | The proposed model significantly reduces the number of visual tokens and boosts visual reasoning capabilities. |
Confounding Factors in Relating Model Performance to Morphology (2025.emnlp-main)
Copied to clipboard
| Challenge: | morphological differences between languages are unclear, but are often considered unimportant . confounding factors make it hard to compare results and draw conclusions, authors argue . |
| Approach: | They propose to use token bigram metrics to predict difficulty of causal language modeling . they argue that confounding factors are contributing to the conflicting evidence . |
| Outcome: | The proposed metrics better capture the relation between morphology and tokenization compared to word-based models. |
Context-Aware Membership Inference Attacks against Pre-trained Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior Membership Inference Attacks on pre-trained Large Language Models fail at LLMs due to ignoring the generative nature of LLM data. |
| Approach: | They propose a method that adapts MIA statistical tests to the perplexity dynamics of subsequences within a data point. |
| Outcome: | The proposed method significantly outperforms prior approaches, revealing context-dependent memorization patterns in pre-trained LLMs. |
Formalizing Style in Personal Narratives (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personal narratives are stories authors construct to make meaning of their experiences. lack of a formal framework for analyzing these stylistic choices. |
| Approach: | They propose a framework that formalizes style as patterns in linguistic choices authors make . they apply the framework to dream narratives including a case study on a war veteran . |
| Outcome: | The proposed framework formalizes style in personal narratives as patterns in linguistic choices . it shows verbal processes dominate over mental ones, and linguistic choice influences psychological states . |
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent attacks have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. |
| Approach: | They propose a method that prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction. |
| Outcome: | The proposed method achieves state-of-the-art performance with an attack success rate (ASR) over 90% in most cases, even when various defense methods are applied. |
PSET: a Phonetics-Semantics Evaluation Testbed (2025.emnlp-main)
Copied to clipboard
| Challenge: | Phonetic embeddings are a powerful tool for capturing the meaning of text . they are not ideal for tasks centered on sound, such as finding sound analogies . |
| Approach: | They propose a phonetic-semantics evaluation testbed to evaluate phonetic embeddings . they use phonetic embedded models to test phonetic models . |
| Outcome: | The phonetic embeddings solve the task with varying degrees of success . the phonetic-based embeddables perform better than the other models . |
From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experiments show that models trained on multi-way parallel data outperform those trained on unaligned data. |
| Approach: | They propose a large-scale, high-quality multi-way parallel corpus based on TED Talks that spans 113 languages with up to 50 languages aligned in parallel. |
| Outcome: | The proposed model outperforms models trained on unaligned multilingual data on six multilingual benchmarks. |
GATEAU: Selecting Influential Samples for Long Context Alignment (2025.emnlp-main)
Copied to clipboard
Shuzheng Si, Haozhe Zhao, Gang Chen, Yunshui Li, Kangyang Luo, Chuancheng Lv, Kaikai An, Fanchao Qi, Baobao Chang, Maosong Sun
| Challenge: | Existing studies have attempted to scale up the available data volume by synthesizing long instruction-following samples, but a lack of a well-defined strategy for ensuring data quality may introduce low-quality samples and restrict the model’s performance. |
| Approach: | They propose a framework to identify influential samples enriched with long-range dependency relations that can be used to align large language models to handle instructions with extremely long contexts. |
| Outcome: | The proposed framework identifies samples with long-range dependency relations and shows that the model trained on these samples exhibits better instruction-following and long-context understanding capabilities. |
Teach Small Models to Reason by Curriculum Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) show strong System-2-style reasoning, but at the cost of significant computational overhead. |
| Approach: | They propose a two-stage curriculum distillation framework which builds a robust internal problem-solving student model and then teaches the student model to externalize this knowledge as explicit reasoning. |
| Outcome: | The proposed model outperforms single-stage baselines on mathematical benchmarks and significantly outperformed LRMs on complex tasks. |
Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to distill chain-of-thought (CoT) results from large language reasoning models (LRMs) to small models are ineffective and require substantial amount of annotated data. |
| Approach: | They propose a Critique-Rethink-Verify system for training small language reasoning models that can be critiquized according to the cognitive capabilities of smaller models. |
| Outcome: | The proposed system outperforms other methods on challenging reasoning benchmarks. |
NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in reinforcement learning, such as DeepSeek R1-Zero, highlight the effectiveness of incentive training, but these methods rely on external verifiers, which limits their applicability to domains like mathematics and coding, where such verifier is readily available. |
| Approach: | They propose a general reinforcement learning framework that requires only standard supervised fine-tuning data with no need for an external verifier. |
| Outcome: | The proposed framework outperforms the model of the same size distilled from large reasoning models such as DeepSeek R1 671B by 7.7%. |
Genre Matters: How Text Types Interact with Decoding Strategies and Lexical Predictors in Shaping Reading Behavior (2025.emnlp-main)
Copied to clipboard
| Challenge: | eMTeC is the first eye-tracking corpus of LLM-generated texts . it shows that text type strongly modulates cognitive effort during reading . |
| Approach: | They use the first eye-tracking corpus of LLM-generated texts to study eye movements during reading and how decoding strategies interact with text types to shape reading behavior. |
| Outcome: | The first eye-tracking corpus of LLM-generated texts shows that text type strongly modulates cognitive effort during reading and that word-level psycholinguistic effects vary systematically across genres. |
RTE-GMoE: A Model-agnostic Approach for Relation Triplet Extraction via Graph-based Mixture-of-Expert Mutual Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Relation Triplet Extraction (RTE) is a fundamental while challenge task in knowledge acquisition. |
| Approach: | They propose a mutual learning framework for Relation Triplet Extraction to address this limitation. |
| Outcome: | The proposed framework improves on four state-of-the-art backbones and benchmarks. |
Avoidance Decoding for Diverse Multi-Branch Story Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have attempted to increase the diversity of generated texts through decoding-time methods. |
| Approach: | They propose a decoding strategy that penalizes similarity to previously generated logits to encourage more diverse multi-branch stories. |
| Outcome: | The proposed method achieves up to **2.6** times higher output diversity and reduces repetition by an average of 30% compared to strong baselines, while effectively mitigating text degeneration. |
Probabilistic Soundness Guarantees in LLM Reasoning Chains (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting propagated errors in reasoning chains are inadequate . author et al. (2017) show that initial errors propagate and undermine reliability of final conclusion . |
| Approach: | They propose a framework that evaluates each reasoning step based solely on previously-verified premises and provides certified statistical guarantees of its soundness. |
| Outcome: | ARES achieves state-of-the-art performance across four benchmarks and demonstrates superior robustness on very long synthetic reasoning chains. |
SQLWOZ: A Realistic Task-Oriented Dialogue Dataset with SQL-Based Dialogue State Representation for Complex User Requirements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TOD datasets present simplified interactions with simple slot-value style constraints and preferences. |
| Approach: | They propose a novel TOD dataset that captures complex user requirements using SQL statements. |
| Outcome: | The proposed dataset captures complex, real-world user requirements. |
SURE: Safety Understanding and Reasoning Enhancement for Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models incorporate visual and textual information, but introduces new and complex safety risks. |
| Approach: | They propose a safety reasoning framework that integrates visual modalities into multimodal models to help them resist jailbreak attacks. |
| Outcome: | The proposed framework improves model safety while avoiding over-defense . it is based on a large-scale safety reasoning dataset . |
EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation focus on direct output alignment, neglecting this crucial structural information. |
| Approach: | They propose a framework for knowledge distillation that maps tokens one-to-one and aligns attention matrix patterns using Centered Kernel Alignment. |
| Outcome: | The proposed framework significantly outperforms existing CTKD baselines. |
AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Large Language Models are increasingly used in Personalized Image Aesthetic Assessment (PIAA) however, their predictions may reflect subtle biases influenced by demographic factors such as gender, age, and education. |
| Approach: | They propose to evaluate MLLMs along two complementary dimensions: (1) stereotype bias and (2) alignment between model outputs and genuine human aesthetic preferences. |
| Outcome: | The proposed benchmark covers three subtasks: aesthetic perception, assessment, empathy and alignment between outputs and genuine human aesthetic preferences. |
DA-Pred: Performance Prediction for Text Summarization under Domain-Shift and Instruct-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often don’t perform as expected under Domain Shift or after Instruct-tuning. |
| Approach: | They propose a method that uses the known performance in high-resource domains and fine-tuning settings to predict performance in low-resourced domains or base models. |
| Outcome: | The proposed method can help researchers decide if resources should be allocated for data labeling and LLM Instruct-tuning. |
UnCo: Uncertainty-Driven Collaborative Framework of Large and Small Models for Grounded Multimodal NER (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to identify unseen multimodal entities struggle with limited knowledge and generalization. |
| Approach: | They propose a framework that leverages the strengths of small fine-tuned models and MLLMs to generate unambiguous predictions. |
| Outcome: | Extensive experiments show that the proposed framework retains the in-domain knowledge of small models while utilizing the capabilities of MLLMs to handle unseen entities. |
An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint (2025.emnlp-main)
Copied to clipboard
Yi Sun, Han Wang, Jiaqiang Li, Jiacheng Liu, Xiangyu Li, Hao Wen, Yizhen Yuan, Huiwen Zheng, Yan Liang, Yuanchun Li, Yunxin Liu
| Challenge: | Large Language Models (LLMs) are a powerful tool for test-time scaling, but they are often used under time constraints. |
| Approach: | They propose to use LLMs to make models think before answering questions . they also use self-correction and best-of-N decoding to encourage deeper thinking . |
| Outcome: | The proposed models are able to achieve higher inference accuracy with extra inference computation under time constraints. |
Enrich-on-Graph: Query-Graph Alignment for Complex Reasoning with LLM Enriching (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods employ resource-intensive, non-scalable workflows reasoning on vanilla KGs, but overlook this gap. |
| Approach: | They propose a flexible framework that leverages LLMs’ prior knowledge to enrich KGs and bridge the semantic gap between queries and graphs. |
| Outcome: | The proposed framework bridges the semantic gap between structured knowledge graphs and unstructured queries while ensuring low computational costs, scalability, and adaptability across different methods. |
Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used for social science simulations . however, most evaluations focus on task optimality rather than variability and adaptation characteristic of human decision-making. |
| Approach: | They propose a process-oriented evaluation framework with progressive interventions to evaluate two economics tasks using large language models. |
| Outcome: | The proposed evaluation framework targets two economic tasks with progressive interventions. |
Structuring Radiology Reports: Challenging LLMs with Lightweight Models (2025.emnlp-main)
Copied to clipboard
Johannes Moll, Louisa Fay, Asfandyar Azhar, Sophie Ostmeier, Sergios Gatidis, Tim C. Lueth, Curtis Langlotz, Jean-Benoit Delbrouck
| Challenge: | Radiology reports lack a standardized format, limiting both interpretability and machine learning applications. |
| Approach: | They propose to use lightweight encoder-decoder models for structuring radiology reports . they compare models with eight open-source LLMs with prompting and in-context learning . |
| Outcome: | The proposed models outperform eight open-source LLMs on a human-annotated test set. |
PricingLogic: Evaluating LLMs Reasoning on Complex Tourism Pricing Tasks (2025.emnlp-main)
Copied to clipboard
Yunuo Liu, Dawei Zhu, Zena Al-Khalili, Dai Cheng, Yanjun Chen, Dietrich Klakow, Wei Zhang, Xiaoyu Shen
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, such as code generation, mathematical problem-solving, and general-purpose human instruction following. |
| Approach: | They propose to use large language models to process questions expressed in natural language to automate tourism-booking prices when multiple, overlapping farerules apply. |
| Outcome: | The proposed model can automate tourism-booking prices when multiple, overlapping farerules apply. |
EcoTune: Token-Efficient Multi-Fidelity Hyperparameter Optimization for Large Language Model Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-fidelity optimization improves inference performance with low-fidelity evaluations, but its static scheduling strategies ignore token consumption, leading to high costs. |
| Approach: | They propose a token-efficient multi-fidelity optimization method that optimizes for inference hyperparameters . their method reduces token consumption by over 80% while maintaining or surpassing performance . |
| Outcome: | The proposed method improves over the HELM leaderboard by 7.1%, 24.3%, 21.9%, and 4.6% while maintaining or surpassing performance. |
Investigating Value-Reasoning Reliability in Small Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | sLLMs have been widely deployed in practical applications, but little attention has been paid to their value-reasoning abilities, particularly in terms of reasoning reliability. |
| Approach: | They propose a systematic evaluation framework for assessing the Value-Reasoning Reliability of small Large Language models (sLLMs) . framework includes three core tasks: Repetition Consistency task, Interaction Stability task, and Open-ended Expression Consistencies task. |
| Outcome: | The proposed framework incorporates self-reported confidence scores to evaluate the model’s value reasoning reliability from two perspectives: the model's self awareness of its values, and its value-based decision-making. |
Can LLMs Explain Themselves Counterfactually? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Explanations are an important tool for gaining insights into model behavior, calibrating user trust, and ensuring compliance. |
| Approach: | They propose to use self-explanation to prompt models to explain outputs . they find that LLMs struggle to generate SCEs - their prediction often does not agree with their own counterfactual reasoning. |
| Outcome: | The proposed methods can generate SCEs across families, sizes, temperatures, and datasets. |
Self-Adjust Softmax (2025.emnlp-main)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li
| Challenge: | Usually, tokens with larger attention scores are important for the final prediction. |
| Approach: | They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function. |
| Outcome: | The proposed model provides enhanced gradient properties compared to the vanilla softmax function. |
DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches typically merge sentence-level parsing outputs for discourse input, resulting in fragmented graphs and degraded downstream performance. |
| Approach: | They propose a task for discourse-level text scene graph parsing that merges sentence-level outputs for discourse input and propose 'DiscoSG' a dataset of 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs is used to test the new task. |
| Outcome: | The proposed task improves SPICE by 30% over the baseline while achieving 86 faster inference than existing models. |
XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML (2025.emnlp-main)
Copied to clipboard
Ernesto Luis Estevanell Valladares, Suilan Estevez-Velarde, Yoan Gutierrez, Andrés Montoyo, Ruslan Mitkov
| Challenge: | XAutoLM is a meta-learning-augmented framework that can be used to optimize discriminative and generative LM fine-tuning pipelines. |
| Approach: | They propose a meta-learning-augmented AutoML framework that reuses past experiences to optimize discriminative and generative LM fine-tuning pipelines efficiently. |
| Outcome: | XAutoLM surpasses zero-shot optimizer’s peak F1 on five of six tasks, reduces mean evaluation time of pipelines by up to 4.5x, and uncovers 50% more pipelines above zero- shot Pareto front. |
UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate misleading or outright incorrect information. |
| Approach: | They propose a method that debiases uncertainty scores on output length and uses residuals as corrected, length-invariant estimates. |
| Outcome: | The proposed method improves over nominally length-normalized methods on machine translation, summarization, and question-answering tasks. |
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, Lihong Li
| Challenge: | Existing work on reinforcement learning has focused on single-turn tasks such as solving math problems. |
| Approach: | They propose a framework that learns directly from online interactions by asynchronously generating diverse trajectories, guided by binary rewards depending on task success. |
| Outcome: | Experiments on the WebArena-Lite benchmark show that the framework outperforms state-of-the-art methods and strong proprietary models. |
Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can be used to evaluate long documents, but they are limited by context window limitations. |
| Approach: | They propose to use granularity-aligned prompting and Focus Sentence Prompting to improve evaluation. |
| Outcome: | a new study shows that long texts lead to fewer error spans and reduced system ranking accuracy. |
PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contract review is a complex and time-intensive task that typically requires legal expertise. |
| Approach: | a new open-source contract review framework is designed to handle complexities of contract analysis . PAKTON is a retrieval-augmented generation framework with plug-and-play capabilities . |
| Outcome: | The open-source framework outperforms models in predictive accuracy, retrieval performance, explainability, completeness, and grounded justifications. |
PoSum-Bench: Benchmarking Position Bias in LLM-based Conversational Summarization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit positional bias, a problem that can undermine the completeness of conversation summarizations. |
| Approach: | They propose a semantic similarity-based sentence-level metric to quantify positional bias in conversational summaries. |
| Outcome: | The proposed benchmark provides the first systematic evaluation of positional bias in conversational summarization across languages and contexts. |
ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning (2025.emnlp-main)
Copied to clipboard
Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Guanbo Wang, Fandong Meng, Jie Zhou, Ju Ren, Yaoxue Zhang
| Challenge: | Existing methods for fine-tuning-based compression suffer from verbose outputs, increasing computational overhead. |
| Approach: | They propose a framework to generate concise reasoning chains using Confidence Injection and Early Stopping. |
| Outcome: | The proposed framework reduces the length of the model by up to 50% while maintaining high task accuracy. |
Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that fine-tuning with benign data can compromise safety of aligned LLMs. |
| Approach: | They propose a Layer-Aware Representation Filtering method that detects safety-degrading layers within the LLM and leverages their representations to detect them. |
| Outcome: | The proposed method can detect safety-degrading features in benign data and remove them from the model. |
Cross-domain Rumor Detection via Test-Time Adaptation and Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches focus on within-domain tasks, resulting in suboptimal performance in cross-domain scenarios due to domain shifts. |
| Approach: | They propose a framework that incorporates both single-domain model and target graph adaptation strategies tailored to the unique requirements of cross-domain rumor detection. |
| Outcome: | The proposed framework surpasses existing methods in rumor detection on social media. |
MLWQ: Efficient Small Language Model Deployment via Multi-Level Weight Quantization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for efficient deployment of small language models face inefficient bit-width allocation and insufficient fine-grained quantization adjustments. |
| Approach: | They propose a weight quantization technique that facilitates efficient deployment of SLMs . they propose to combine inter-layer loss and intra-layer salience to achieve better allocation . |
| Outcome: | Experimental results show that multi-level weight quantization achieves competitive performance compared to state-of-the-art methods. |
ToDi: Token-wise Distillation via Fine-Grained Divergence Control (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) offer impressive performance but are impractical for resource-constrained deployment due to high latency and energy consumption. |
| Approach: | They propose a method that adaptively combines FKL and RKL per token using a sigmoid-based weighting function derived from the teacher-student probability log-ratio. |
| Outcome: | The proposed method outperforms baselines using uniform or less granular strategies across instruction-following benchmarks. |
RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation (2025.emnlp-main)
Copied to clipboard
Qingyao Li, Wei Xia, Xinyi Dai, Kounianhua Du, Weiwen Liu, Yasheng Wang, Ruiming Tang, Yong Yu, Weinan Zhang
| Challenge: | Existing tree search methods neglect the underlying reasoning process, resulting in poor search quality. |
| Approach: | They propose a framework that systematically explores and refines the reasoning process for code generation by using a tree search engine and a reflection mechanism. |
| Outcome: | The proposed framework outperforms existing methods in the code generation domain. |
Probing for Arithmetic Errors in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a controlled setting of 3-digit addition, we show that simple probes can accurately decode both the model’s output and the correct answer from hidden states. |
| Approach: | They extend their analysis to structured chain-of-thought traces on addition-only GSM8K problems and find that probes trained on simple arithmetic generalize well to this more complex setting, revealing consistent internal representations. |
| Outcome: | The proposed probes can predict model correctness with over 90% accuracy on addition-only GSM8K problems and guide selective re-prompting of erroneous reasoning steps with minimal disruption to correct outputs. |
NILE: Internal Consistency Alignment in Large Language Models (2025.emnlp-main)
Copied to clipboard
Minda Hu, Qiyuan Zhang, Yufei Wang, Bowei He, Hongru Wang, Jingyan Zhou, Liangyou Li, Yasheng Wang, Chen Ma, Irwin King
| Challenge: | Recent advances show that the world knowledge in the Instruction Fine-Tuning (IFT) dataset, which is incompatible with LLMs’ internal knowledge, can greatly hurt the IFT performance. |
| Approach: | They propose a framework to optimize the effectiveness of IFT by carefully aligning the world and internal knowledge of LLMs. |
| Outcome: | The proposed framework can significantly improve performance across multiple LLM ability evaluation datasets. |
Mining the Past with Dual Criteria: Integrating Three types of Historical Information for Context-aware Event Forecasting (2025.emnlp-main)
Copied to clipboard
Rong Ma, Lei Wang, Yating Yang, Bo Ma, Rui Dong, Fengyi Yang, Ahtamjan Ahmat, Kaiwen Lu, Xinyue Wang
| Challenge: | Existing methods focus on entities and structural dependencies but overlook implicitly relevant information. |
| Approach: | They propose a method that leverages event semantics for relevance modeling and incorporates a self-supervised semantic filter based on factual event associations to capture implicitly relevant historical information. |
| Outcome: | The proposed method outperforms existing methods on three public benchmark datasets and is highly effective on two structured temporal knowledge graph forecasting datasets. |
RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
Andrei Catalin Coman, Ionut Teodor Sorodoc, Leonardo F. R. Ribeiro, Bill Byrne, James Henderson, Adrià de Gispert
| Challenge: | Existing Reward Models (RMs) struggle in Retrieval Augmented Generation settings. |
| Approach: | They propose a method that repurposes question-answering datasets into preference pairs that prioritise groundedness over stylistic features. |
| Outcome: | The proposed method surpasses existing RMs trained on larger general corpora with an absolute improvement of +15.5%. |
Large Language Models Discriminate Against Speakers of German Dialects (2025.emnlp-main)
Copied to clipboard
| Challenge: | In Germany, more than 40% of the population speaks a regional dialect . however, dialect speakers face negative societal stereotypes . |
| Approach: | They construct a corpus that pairs sentences from seven regional German dialects with their standard German counterparts to assess their dialect usage bias. |
| Outcome: | The proposed model reproduces dialect usage bias in association task and decision task. |
Uncovering Argumentative Flow: A Question-Focus Discourse Structuring Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing structure modeling approaches fail to capture the author’s rhetorical intent and reasoning process. |
| Approach: | They propose a Question-Focus discourse structuring framework that explicitly models the underlying argumentative flow by anchoring each argumentative unit to a guiding question and a set of attentional foci. |
| Outcome: | The proposed framework outperforms baseline models and curated models on an argument reconstruction task in Chinese think-tank articles and claims coverage. |
AbsVis – Benchmarking How Humans and Vision-Language Models “See” Abstract Concepts in Images (2025.emnlp-main)
Copied to clipboard
| Challenge: | Abstract concepts like mercy and peace lack clear visual grounding, and therefore challenge humans and models to provide suitable image representations. |
| Approach: | They propose a dataset of 675 images annotated with 14,175 concept–explanation attributions from humans and two Vision-Language Models where each concept is accompanied by a textual explanation. |
| Outcome: | The proposed dataset compares human and VLM attributions in terms of diversity, abstractness, and alignment, and shows that overlapping concepts are most preferred. |
A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to generate synthetic textual data for training smaller specialized models. |
| Approach: | They evaluate the performance of large language models and their generation strategies in 11 different languages using 3 NLP tasks and 4 open-source LLMs. |
| Outcome: | The proposed generation strategies and their combinations yield strong results across 11 languages, including several extremely low-resource ones. |
Alignment with Fill-In-the-Middle for Enhancing Code Generation (2025.emnlp-main)
Copied to clipboard
Houxing Ren, Zimu Lu, Weikang Shi, Haotian Hou, Yunqiao Yang, Ke Wang, Aojun Zhou, Junting Pan, Mingjie Zhan, Hongsheng Li
| Challenge: | Existing methods for generating test cases with limited training data are not reliable and may be counterproductive. |
| Approach: | They propose a method that splits code snippets into smaller, granular blocks, creating more diverse DPO pairs from the same test cases. |
| Outcome: | The proposed approach shows significant improvements in code generation tasks on benchmark datasets such as HumanEval (+), MBPP (+), and APPS. |
A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentiality (2025.emnlp-main)
Copied to clipboard
| Challenge: | Privacy-sensitive users require deploying large language models within their own infrastructure (on-premises) vulnerabilities in local environments can lead to unauthorized access and potential model theft. |
| Approach: | They propose a framework that secures a few bottom layers in a secure environment . they propose metric to optimize trade-off between protection and customization flexibility . |
| Outcome: | The proposed framework outperforms baselines on five models with 1.3B to 70B parameters. |
Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Attention-based language models rely on the softmax function to convert attention logits into probability distributions, but this process can result in attention entropy collapse. |
| Approach: | They propose to use the softmax function to re-weight attention logits to create probability distributions, but this reweighting can lead to attention entropy collapse . they find that entropic-stable attention methods can prevent entrapment and enable more stable training by controlling or insensitive to variance of attention logit variance. |
| Outcome: | The proposed methods prevent attention entropy collapse and enable more stable training. |
X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQA (2025.emnlp-main)
Copied to clipboard
| Challenge: | Medical visual question answering (VQA) and federated learning (FL) are important tools for privacy-preserving collaborative learning. |
| Approach: | They propose a cross-modal FL framework that uses modality-expert low-rank adaptation for medical visual question answering (VQA) X-FLoRA enables the synthesis of images from one modality to another without requiring data sharing . |
| Outcome: | Experiments show that X-FLoRA outperforms existing FL methods in terms of performance . XFLorage enables synthesis of images from one modality to another without data sharing . |
Robust Native Language Identification through Agentic Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often achieve high performance by leveraging superficial contextual clues rather than the underlying linguistic patterns indicative of native language (L1) influence. |
| Approach: | They propose an agentic NLI pipeline where specialized agents accumulate and categorize diverse linguistic evidence before an independent final assessment. |
| Outcome: | The proposed pipeline significantly improves robustness against misleading contextual clues and performance consistency compared to standard prompting methods. |
ConsistentChat: Building Skeleton-Guided Consistent Multi-Turn Dialogues for Large Language Models from Scratch (2025.emnlp-main)
Copied to clipboard
Jiawei Chen, Xinyan Guan, Qianhao Yuan, Mo Guozhao, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, Xianpei Han
| Challenge: | Existing instruction data synthesis methods focus on single-turn instructions and neglect cross-turn coherence, resulting in context drift and reduced task completion rates. |
| Approach: | They propose a framework that constrains multi-turn instruction synthesis by explicitly modeling human conversational intent. |
| Outcome: | The proposed framework outperforms existing models trained on single-turn and multi-turn instruction datasets. |
Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current acceleration evaluations focus on minimal overall performance degradation . however, accelerated models can exhibit significant changes in instance-level predictions . |
| Approach: | They investigate whether accelerated vision-Language Models can still give the same answers as before . they found that accelerated models changed original answers up to 20% of the time . |
| Outcome: | The results show that accelerated models changed their original answers up to 20% of the time. |
When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models are evaluated with scalar metrics like accuracy but do not capture how they encode ambiguity and more generally instances. |
| Approach: | They propose to analyze how fine-tuned models encode ambiguity and more generally instances. |
| Outcome: | The proposed tool uncovers decision regions, boundary collapses, and overconfident clusters in a RoBERTa-Large dataset. |
Self-Critique and Refinement for Faithful Natural Language Explanations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has demonstrated that Large Language Models (LLMs) can self-critique and refine their initial outputs, but this capability remains unexplored for improving explanation faithfulness. |
| Approach: | They propose a framework that enables models to improve the faithfulness of their own explanations through an iterative critique and refinement process without external supervision. |
| Outcome: | The proposed framework reduces unfaithfulness rates in three datasets and four state-of-the-art LLMs by 36% compared to 54.81% for baseline. |
The Psychology of Falsehood: A Human-Centric Survey of Misinformation Detection (2025.emnlp-main)
Copied to clipboard
Arghodeep Nandi, Megha Sundriyal, Euna Mehnaz Khan, Jikai Sun, Emily K. Vraga, Jaideep Srivastava, Tanmoy Chakraborty
| Challenge: | a survey examines the interplay between factual accuracy and cognitive biases . misinformation is more than just the existence of incorrect information, it also entails complex relationships between the information and the entities that consume it. |
| Approach: | They examine the interplay between traditional fact-checking and psychological concepts such as cognitive biases, social dynamics, and emotional responses. |
| Outcome: | The findings highlight limitations of current methods and identify opportunities for improvement . they also outline future research directions to create more robust frameworks . |
SEAL: Structure and Element Aware Learning Improves Long Structured Document Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for document retrieval use contrastive learning on datasets lacking explicit structural information. |
| Approach: | They propose a contrastive learning framework that preserves semantic hierarchies and masked element alignment for fine-grained semantic discrimination. |
| Outcome: | The proposed framework preserves semantic hierarchies and masked element alignment for fine-grained semantic discrimination. |
AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models with extended context lengths face significant computational challenges during the prefill phase. |
| Approach: | They propose a difference-aware, dynamic sparse attention mechanism that efficiently identifies critical attention regions at a finer stripe granularity while adapting to global contextual information. |
| Outcome: | The proposed model achieves a speedup of 1.44 while maintaining higher recall rates. |
Attacks by Content: Automated Fact-checking is an AI Security Issue (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing defenses focus on detecting hidden commands but are ineffective against content attacks. |
| Approach: | They propose to repurpose retrieval-augmented generation (RAG) as a cognitive self-defense tool for agents. |
| Outcome: | The proposed approach is analogous to an existing task, automated fact-checking, and could be used to defend agents against content attacks. |
MUZO: Leveraging Multiple Queries and Momentum for Zeroth-Order Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models incur memory overhead due to the need for activation storage for back-propagation (BP). |
| Approach: | They propose a method that estimates gradients through finite differences without activation storage for back-propagation. |
| Outcome: | The proposed method demonstrates superior performance in fine-tuning various LLMs. |
Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors (2025.emnlp-main)
Copied to clipboard
Hao Fang, Jiawei Kong, Tianqu Zhuang, Yixiang Qiu, Kuofeng Gao, Bin Chen, Shu-Tao Xia, Yaowei Wang, Min Zhang
| Challenge: | Existing methods to detect large language models (LLMs) generated for plagiarism use paraphrases to rewrite them to evade detection. |
| Approach: | They propose a training-free method that effectively fools text detectors using off-the-shelf LLMs by rewriting them to evade detection. |
| Outcome: | The proposed method deceives text detectors using off-the-shelf LLMs by rewriting them to produce human-like sentences that are less discernible by detectors. |
Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA (2025.emnlp-main)
Copied to clipboard
Sergey Pletenev, Maria Marina, Nikolay Ivanov, Daria Galimzianova, Nikita Krayko, Mikhail Salnikov, Vasily Konovalov, Alexander Panchenko, Viktor Moskvoretskii
| Challenge: | Large Language Models (LLMs) often struggle with question answering due to hallucinated answers. |
| Approach: | They propose a multilingual QA dataset with evergreen labels that can be used to evaluate and train large language models. |
| Outcome: | The proposed model performs well on 12 modern LLMs and EG-E5 classifiers. |
Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has focused largely on binary grammatical contrasts, but how do they encode their syntactic knowledge internally? |
| Approach: | They propose to use a multidimensional hierarchical grammar phenomenon to identify distinct, orthogonal directions in residual space to demonstrate causal control over both grammatical features. |
| Outcome: | The proposed model can encode tense and aspect in human-like ways, but effective steering during generation is sensitive to multiple factors and requires manual tuning or automated optimization. |
DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models focus on identifying relevant documents, but embedding similarity often limits accuracy. |
| Approach: | They propose a method to generate hard negative queries per page instead of negative pages per query . they propose to refine ranking of an initial set of retrieved documents using hard negative mining . |
| Outcome: | The proposed approach outperforms existing models and significantly improves retrieval performance. |
Reason to Rote: Rethinking Memorization in Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models readily memorize arbitrary training instances, such as label noise . however, such memorization does not affect generalizable reasoning abilities . |
| Approach: | They investigate how large language models memorize label noise and why it affects generalizability. |
| Outcome: | The proposed model performs well on reasoning tasks even when memorized labels are missing . the proposed model is able to generalize to correctly answer "87+19=106" |
VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for image captioning are primarily designed for short captions and are not suitable for long captions. |
| Approach: | They propose an automatic evaluation metric for long captions developed within a novel LLM-Hybrid-as-a-Judge framework. |
| Outcome: | The proposed metric outperforms existing metrics and achieves superhuman performance on LongCap-Arena. |
LLM-Independent Adaptive RAG: Let the Question Speak for Itself (2025.emnlp-main)
Copied to clipboard
Maria Marina, Nikolay Ivanov, Sergey Pletenev, Mikhail Salnikov, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Alexander Panchenko, Viktor Moskvoretskii
| Challenge: | Existing methods to retrieve Large Language Models (LLMs) are inefficient and impractical. |
| Approach: | They propose a lightweight adaptive retrieval method that leverages external information to achieve comparable quality while achieving significant efficiency gains. |
| Outcome: | The proposed methods achieve comparable quality while achieving significant efficiency gains on 6 QA datasets. |
TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route (2025.emnlp-main)
Copied to clipboard
Hongyi Luo, Qing Cheng, Daniel Matos, Hari Krishna Gadi, Yanfeng Zhang, Lu Liu, Yongliang Wang, Niclas Zeller, Daniel Cremers, Liqiu Meng
| Challenge: | Existing studies on large language models have limited evaluation of their geospatial cognition . a unified framework for evaluating geospcial cognition in LLMs remains absent . |
| Approach: | They propose a benchmark to evaluate the geospatial route cognition of Large Language Models . they propose 'pathbuilder' tool for converting natural language instructions into navigation routes . |
| Outcome: | The proposed framework and metrics evaluate 9 state-of-the-art LLMs on route reversal task. |
Certainty in Uncertainty: Reasoning over Uncertain Knowledge Graphs with Statistical Guarantees (2025.emnlp-main)
Copied to clipboard
Yuqicheng Zhu, Jingcheng Wu, Yizhen Wang, Hongkuan Zhou, Jiaoyan Chen, Evgeny Kharlamov, Steffen Staab
| Challenge: | Existing methods produce only point estimates, without quantifying predictive uncertainty—limiting their reliability in high-stakes applications where understanding confidence in predictions is crucial. |
| Approach: | They propose a framework that generates prediction intervals guaranteed to contain the true score with a user-specified level of confidence. |
| Outcome: | The proposed framework generates prediction intervals guaranteed to contain the true score with a user-specified level of confidence. |
Beyond Seen Data: Improving KBQA Generalization Through Schema-Guided Logical Form Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge base question answering (KBQA) aims to answer user questions in natural language using rich human knowledge stored in large KBs. |
| Approach: | They propose a model that injects schema contexts into entity retrieval and logical form generation to enhance generalizability. |
| Outcome: | The proposed model outperforms state-of-the-art models on two commonly used benchmark datasets across a variety of test settings. |
A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing solutions to problem of positional out-of-distribution (O.O.D.) are inefficient, redundant, and lack local positional information. |
| Approach: | They propose a training-free method that greedily reuses pretrained positional intervals and interpolates attention logits to eliminate outliers. |
| Outcome: | The proposed method achieves stable and superior performance across long-context tasks without requiring input-length-specific tuning. |
Taming Text-to-Image Synthesis for Novices: User-centric Prompt Generation via Multi-turn Guidance (2025.emnlp-main)
Copied to clipboard
Yilun Liu, Minggui He, Feiyu Yao, Yuhe Ji, Shimin Tao, Jingzhou Du, Justin Li, Jian Gao, Zhang Li, Hao Yang, Boxing Chen, Osamu Yoshie
| Challenge: | Existing solutions for text-to-image synthesis are sensitive on textual prompts, posing a challenge for novice users. |
| Approach: | They propose a dialogue-based TIS prompt generation model that emphasizes user experience for novice users. |
| Outcome: | The proposed model emphasizes user experience for novice users . it improves user-centricity score while maintaining a competitive quality of synthesized images. |
We Need to Measure Data Diversity in NLP — Better and Broader (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models exhibit remarkable natural language understanding and generation capabilities, but they have serious flaws, such as societal biases and spurious correlations. |
| Approach: | They argue that interdisciplinary perspectives are essential for developing more fine-grained and valid measures of data diversity. |
| Outcome: | The proposed measures are based on interdisciplinary perspectives and include a variety of datasets. |
Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that DiscoGP extracts sheaves that preserve 93-100% of a model’s performance while comprising only 1-7% of the original weights and connections. |
| Approach: | They propose a framework for extracting self-contained modular units within neural language models (LMs) they use a gradient-based pruning algorithm to prune the original LM to a sparse skeleton . |
| Outcome: | The proposed framework preserves 93-100% of the original model's performance while preserving only 1-7% of the model''s original weights and connections. |
Hierarchical Bracketing Encodings Work for Dependency Graphs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sequence labeling (SL) is a simple yet effective paradigm for a wide range of natural language problems. |
| Approach: | They propose a new bracketing approach for dependency graph parsing that encodes graphs as sequences and n tagging actions. |
| Outcome: | The proposed approach significantly reduces label space while preserving structural information. |
Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods overlook the fine-grained semantic and prosodic interaction modeling at the word level. |
| Approach: | They propose a novel approach to generate conversational prosody by understanding multimodal dialogue history (MDH) using fine-grained semantic and prosodic interaction modeling, they construct specialized multimodal fine-grain dialogue interaction graphs that encode interaction between word-level semantics and prosody. |
| Outcome: | The proposed system outperforms baseline models in terms of prosodic expressiveness. |
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
Conditional [MASK] Discrete Diffusion Language Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Auto-regressive models excel in natural language processing but struggle to generate diverse text and lack controllability. |
| Approach: | They propose entropy-adaptive Gibbs sampling and entropic-based noise scheduling to counterbalance each model’s shortcomings. |
| Outcome: | The proposed framework outperforms baseline models and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation. |
Language-Guided Temporal Token Pruning for Efficient VideoLLM Processing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current models struggle with long-form videos due to the quadratic complexity of attention mechanisms. |
| Approach: | They propose a model-agnostic framework that leverages temporal cues from queries to prune video tokens. |
| Outcome: | The proposed framework reduces computation by 65% while preserving 97-99% of original performance. |
A Fully Probabilistic Perspective on Large Language Model Unlearning: Evaluation and Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Model Unlearning (LLMU) is a promising way to remove private or sensitive information from large language models. |
| Approach: | They propose a Fully Probabilistic Evaluation framework that incorporates input and output distributions in LLMU evaluation. |
| Outcome: | The proposed framework improves unlearning effectiveness by 50.1% and robustness by 37.2% on Llama-2-7B. |
IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method (2025.emnlp-main)
Copied to clipboard
Xinyu Liu, Bei Li, Jiahao Liu, Junhao Ruan, Kechen Jiao, Hongyin Tang, Jingang Wang, Tong Xiao, JingBo Zhu
| Challenge: | High-order numerical methods enhance performance in tasks like NLP but introduce a performance-efficiency trade-off due to increased computational overhead. |
| Approach: | They propose an iterative implicit Euler Transformer which simplifies high-order numerical methods by iterating implicit Eule. |
| Outcome: | The proposed method improves accuracy and reduces inference overhead by 55% while retaining 99.4% of the original task accuracy. |
WebEvolver: Enhancing Web Agent Self-Improvement with Co-evolving World Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Agent self-improvement, where agents train their underlying Large Language Model (LLM) on self-sampled trajectories, shows promising results but often stagnates in web environments due to limited exploration and under-utilization of pretrained web knowledge. |
| Approach: | They propose a co-evolving Large Language Model (LLM) that predicts the next observation based on current observation and action within the web environment. |
| Outcome: | The proposed framework shows that agents can perform better in real-world web environments without using any distillation from more powerful close-sourced models. |
Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy Guarantees (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on differential privacy in text privatization use word perturbations and rewriting methods to protect privacy. |
| Approach: | They propose a method which leverages semantic triples for neighborhood-aware private document generation under local DP guarantees. |
| Outcome: | The proposed method allows for coherent text generation even at lower values while still balancing privacy and utility. |
HVGuard: Utilizing Multimodal Large Language Models for Hateful Video Detection (2025.emnlp-main)
Copied to clipboard
Yiheng Jing, Mingming Zhang, Yong Zhuang, Jiacheng Guo, Juan Wang, Xiaoyang Xu, Wenzhe Yi, Keyan Guo, Hongxin Hu
| Challenge: | Existing methods for hateful video detection rely on unimodal analysis or feature fusion . Existing tools struggle to capture cross-modal interactions and reason through implicit hate in sarcasm and metaphor . |
| Approach: | They propose a reasoning-based hateful video detection framework with multimodal large language models . they integrate Chain-of-Thought reasoning to enhance multimodal interaction modeling . |
| Outcome: | The proposed framework outperforms existing tools on two public datasets covering English and Chinese. |
Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequence (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for parallelizable reasoning tasks are inefficient, says a new study . generating lengthy reasoning sequences is computationally expensive and time-consuming, says the study authors . |
| Approach: | They propose a method that decodes multiple tokens per forward pass using a tree-like attention mask . their method achieves nearly 100% speedup in decoding while basically maintaining the answer quality . |
| Outcome: | Experimental results show that the method achieves nearly 100% speedup in decoding while maintaining the answer quality. |
SlideCoder: Layout-aware RAG-enhanced Hierarchical Slide Generation from Design (2025.emnlp-main)
Copied to clipboard
Wenxin Tang, Jingyu Xiao, Wenxuan Jiang, Xi Xiao, Yuhang Wang, Xuxin Tang, Qing Li, Yuehe Ma, Junliang Liu, Shisong Tang, Michael R. Lyu
| Challenge: | Existing natural language-based LLM generation methods struggle to capture visual and structural nuances of slide designs. |
| Approach: | They propose a layout-aware framework for generating editable slides from reference images . they propose python code that translates NL instructions into Python code to construct each slide . |
| Outcome: | The proposed framework outperforms state-of-the-art models by up to 40.5 points . it also outperformed open-source models with improved reverse-engineered data. |
LLM-OREF: An Open Relation Extraction Framework Based on Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on building models that can only handle predefined relations . however, their reliance on human annotation limits their practicality . |
| Approach: | They propose an open relation extraction framework that can generalize to new relations not encountered during training. |
| Outcome: | The proposed framework can generalize to new relations not encountered during training. |
Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. |
| Approach: | They propose an approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. |
| Outcome: | The proposed approach outperforms state-of-the-art approaches in performance across multiple model scales and widely adopted benchmark datasets. |
Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to ground large language models (LLMs) with RAGs are limited by the heterogeneity of knowledge retrieved. |
| Approach: | They propose a modular approach guided by Argumentative Explanations that evaluates retrieved information by comparing, contrasting and resolving conflicting perspectives. |
| Outcome: | The proposed framework significantly improves RAG approaches, requiring low-impact computational effort and providing robustness to knowledge perturbations. |
Predicate-Guided Generation for Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that Prolog-MATH generates 81.3% solution coverage on Deepseek-V3 . |
| Approach: | They propose a curated corpus to support mathematical reasoning in large language models . they propose supervised fine-tuning followed by GRPO training to address problems that Deepseek-V3 fails to solve. |
| Outcome: | The proposed pipeline achieves 81.3% solution coverage on the Deepseek-V3 training set. |
ComplexTempQA: A 100m Dataset for Complex Temporal Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets that focus on temporal knowledge are limited in size and lack comprehensive coverage of temporal information. |
| Approach: | They introduce a large-scale temporal question-answer-matching dataset . the new taxonomy categorizes questions as attributes, comparisons, and counting questions . |
| Outcome: | The proposed dataset surpasses existing benchmarks in scale and scope. |
ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on image-based question answering (QA) but ignore the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. |
| Approach: | They propose a novel multi-agent RAG framework tailored for complex reasoning across visual documents that employs a Gaussian Mixture Model (GMM)-based hybrid strategy to handle multi-modal retrieval. |
| Outcome: | The proposed framework outperforms existing methods by over 10% on the competitive ViDoSeek benchmark. |
IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety standards are often based on direct translations from English, which overlook key aspects of local communication. |
| Approach: | They propose a high-quality, human-verified safety evaluation dataset tailored for the Indonesian context. |
| Outcome: | The proposed dataset covers formal and colloquial Indonesian, along with three major local languages: Javanese, Sundanese, and Minangkabau. |
Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Enterprise systems are crucial for enhancing productivity and strategic growth, but data is fragmented across multiple sources and access controls are complex. |
| Approach: | They propose a benchmark that simulates enterprise settings with 500 diverse tasks . they show that even the most capable models achieve only 41.8% task completion . |
| Outcome: | The proposed benchmark shows that even the most capable models achieve only 41.8% task completion. |
Steering LLM Reasoning Through Bias-Only Adaptation (2025.emnlp-main)
Copied to clipboard
Viacheslav Sinii, Alexey Gorbatovski, Artem Cherepanov, Boris Shaposhnikov, Nikita Balagansky, Daniil Gavrilov
| Challenge: | Compared with LoRA and BitFit, training a single steering vector per layer with reinforcement learning requires orders of magnitude fewer resources and isolates a much smaller, more interpretable parameter set. |
| Approach: | They propose to train a single steering vector per layer with reinforcement learning while freezing all base weights to match the accuracy of fully RL-tuned reasoning models. |
| Outcome: | The proposed approach improves on an 8 billion-parameter model while keeping all base weights fixed. |
VLASCD: A Visual Language Action Model for Simultaneous Chatting and Decision Making (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent large-scale pretrained models are built upon a multi-input single-output paradigm . tasks compete for a shared output channel, creating mutual exclusion effects . |
| Approach: | They propose a multi-input single-output (MISO) paradigm for large pretrained models . they propose unified training framework that enables concurrent multi-task outputs . |
| Outcome: | Experiments on autonomous driving platform show that MIMO-VLA outperforms state-of-the-art models in MIMO settings. |
M-LongDoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework (2025.emnlp-main)
Copied to clipboard
Yew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song, Chaoqun Liu, Mahani Aljunied, Soujanya Poria, Lidong Bing
| Challenge: | Existing benchmarks for large multimodal models focus on short documents with less than 50 pages and are limited to extraction-based questions. |
| Approach: | They propose a retrieval-aware tuning approach to improve the accuracy of multimodal document reading by 4.6%. |
| Outcome: | The proposed framework improves the accuracy of model responses by 4.6% compared to existing benchmarks on documents with hundreds of pages and longer documents with more complex content. |
Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in text-only "slow thinking" reasoning have prompted efforts to transfer this capability to vision-language models (VLMs). |
| Approach: | They propose a VRM Reflection-V which enhances visual reflection based on reasoning data for cold-start and reward design for reinforcement learning. |
| Outcome: | The proposed model improves visual reflection for cold-start and reward design for reinforcement learning (RL) it maintains a stronger and more consistent reliance on visual information during visual reasoning, indicating effective enhancement in visual reflection capabilities. |
FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs’ Responsiveness to Human Feedback (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research focuses on benchmarking LLMs in single-turn dialogues, neglecting the nuanced nature of human feedback within real-world usage scenarios. |
| Approach: | They propose a fine-grained, multi-task benchmark designed to evaluate LLMs’ responsiveness to human feedback under real-world usage scenarios in Chinese. |
| Outcome: | The proposed benchmarks show that human feedback can significantly impact LLMs’ responsiveness in real-world usage scenarios. |
HYDRA: A Multi-Head Encoder-only Architecture for Hierarchical Text Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art approaches rely on complex components like graph encoders, label semantics, and autoregressive decoders. |
| Approach: | They propose a multi-head encoder-only architecture for hierarchical text classification that treats each level as a separate classification task with its own label space. |
| Outcome: | The proposed architecture matches or exceeds state-of-the-art methods on four benchmarks. |
CARD: Cross-modal Agent Framework for Generative and Editable Residential Design (2025.emnlp-main)
Copied to clipboard
| Challenge: | Architectural design automation has made significant progress, but the complexity of open-world environments makes residential design a challenging task. |
| Approach: | They propose a framework that leverages a system of specialized cross-modal agents to adapt to open-world residential design. |
| Outcome: | The proposed framework enables users to generate and edit residential design without requiring specialized expertise. |
DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-off (2025.emnlp-main)
Copied to clipboard
Jusheng Zhang, Yijia Fan, Kaitong Cai, Zimeng Huang, Xiaofei Sun, Jian Wang, Chengpei Tang, Keze Wang
| Challenge: | et al., 2019; Brown e.t al, 2023; Touvron e t al; 2024; OpenAI, 2024) Large Language Models (LLMs) have demonstrated remarkable capabilities in knowledge encoding and contextual understanding during their pretraining phase. |
| Approach: | They propose a dynamic expert scheduling mechanism that allocates computational resources based on text complexity and a hierarchical sparse attention mechanism that adjusts attention patterns according to a variety of input lengths. |
| Outcome: | The proposed framework overpowers existing methods on long-text generation benchmarks. |
FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on personalizing conversational assistants to meet specific user preferences. |
| Approach: | They propose to use a dataset to analyze a problem where only a small set of preference annotations can be collected per user. |
| Outcome: | The proposed approach leverages high-level features discovered from the data, achieving the best overall performance. |
On LLM-Based Scientific Inductive Reasoning Beyond Equations (2025.emnlp-main)
Copied to clipboard
Brian S. Lin, Jiaxin Yuan, Zihan Zhou, Shouli Wang, Shuo Wang, Cunliang Kong, Qi Shi, Yuxuan Li, Liner Yang, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing research on inductive reasoning models emphasizes rule design without grounding them in specific scenarios. |
| Approach: | They propose to use LLMs to learn underlying patterns from limited examples in entirely new environments. |
| Outcome: | The proposed benchmark evaluates the inductive reasoning abilities of large language models in scientific settings. |
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | N-gram-based evaluation metrics are unreliable due to low correlation to human judgments. |
| Approach: | They propose a metric that rewards correct details and penalizes incorrect ones. |
| Outcome: | The proposed metric matches the performance of open-source LLM-based metrics in correlation to human judgments while being far more efficient. |
LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding (2025.emnlp-main)
Copied to clipboard
Yuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zheng, Weikang Shi, Manyuan Zhang, Qi Dou, Rui Liu, Aojun Zhou, Hongsheng Li
| Challenge: | Recent advances in Large Language Models have opened new avenues for solving complex optimization problems, including Neural Architecture Search (NAS). |
| Approach: | They propose a framework that leverages LLMs for cross-domain neural architecture optimization without extensive domain-specific tuning. |
| Outcome: | The proposed framework achieves competitive performance in both in-domain and out-of-domain tasks. |
Does quantization affect models’ performance on long-context tasks? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency. |
| Approach: | They present the first systematic evaluation of quantized LLMs on tasks with long inputs and long-form outputs. |
| Outcome: | The proposed method preserves accuracy, while 4-bit methods lead to substantial losses . the results highlight the importance of a careful evaluation before deploying quantized LLMs . |
Token-Aware Editing of Internal Activations for Large Language Model Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to optimize the behavior of large language models neglect misalignment discrepancies among tokens, resulting in deviant alignment direction and inflexible editing strength. |
| Approach: | They propose a token-aware editing approach to exploit the misalignment discrepancy among tokens to enhance activation probing and facilitate intervention. |
| Outcome: | Extensive experiments on three alignment capabilities demonstrate the efficacy of the proposed approach surpassing baseline by 25.8% on the primary metric of truthfulness with minimal cost. |
Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Decoder-only large language models rely on masked causal attention, which limits expressiveness by restricting information flow to one direction. |
| Approach: | They propose a method that incorporates bidirectional attention into prompt processing to enhance pretrained decoder-only LLMs. |
| Outcome: | The proposed method shows significant improvements in commonsense reasoning, arithmetic, and language understanding tasks. |
Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Survey (2025.emnlp-main)
Copied to clipboard
Mehrab Tanjim, Yeonjun In, Xiang Chen, Victor Bursztyn, Ryan A. Rossi, Sungchul Kim, Guang-Jie Ren, Vaishnavi Muppala, Shun Jiang, Yongsung Kim, Chanyoung Park
| Challenge: | Existing literature on ambiguity and disambiguation with Large Language Models (LLMs) ambiguities are a fundamental challenge in human-AI interactions due to complexity and flexibility of human language. |
| Approach: | They propose to define key terms and concepts and categorize various disambiguation approaches enabled by LLMs and provide a comparative analysis of their advantages and disadvantages. |
| Outcome: | The proposed frameworks are compared against different disambiguation approaches and highlight their relevance for future research. |
Plan Dynamically, Express Rhetorically: A Debate-Driven Rhetorical Framework for Argumentative Writing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Argumentative essay generation (AEG) is a complex task that requires advanced semantic understanding, logical reasoning, and organized integration of perspectives. |
| Approach: | They propose a debate-driven rhetorical framework for argumentative writing that integrates Bitzer’s rhetorical situation theory to improve logical depth, argumentative diversity, and rhetorical persuasiveness. |
| Outcome: | The proposed framework improves logical depth, argumentative diversity, and rhetorical persuasiveness over existing state-of-the-art models. |
TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making (2025.emnlp-main)
Copied to clipboard
Kechen Jiao, Zhirui Fang, Jiahao Liu, Bei Li, Qifan Wang, Xinyu Liu, Junhao Ruan, Zhongjian Qiao, Yifan Zhu, Yaxin Xu, Jingang Wang, Xiu Li
| Challenge: | Existing post-SFT methods for embodied AI are constrained by sparse rewards and action-only optimization, resulting in low sample efficiency, poor consistency, and model degradation. |
| Approach: | They propose to integrate Thought-Centric Preference Optimization (TCPO) into embodied decision-making by transforming sparse reward signals into richer step sample pairs. |
| Outcome: | The proposed approach achieves an average success rate of 26.67% in the ALFWorld environment, and a 6% improvement over RL4VLM. |
Reimagining Safety Alignment with An Image (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to large language models face inefficiency, fragility, or architectural constraints, resulting in inefficient performance and heightened over-refusal in cross-modal tasks. |
| Approach: | They propose an optimization-driven visual prompt framework that enhances security and reduces over-refusal at the same time. |
| Outcome: | The proposed framework enhances security and reduces over-refusal while maintaining robust safety while reducing unnecessary denials. |
Generative or Discriminative? Revisiting Text Classification in the Era of Transformers (2025.emnlp-main)
Copied to clipboard
Siva Rajesh Kasa, Karan Gupta, Sumegh Roychowdhury, Ashutosh Kumar, Yaswanth Biruduraju, Santhosh Kumar Kasa, Pattisapu Nikhil Priyatam, Arindam Bhattacharya, Shailendra Agarwal, Vijay Huddar
| Challenge: | generative classifiers exhibit lower sample complexity but higher asymptotic error in simple linear settings, a trade-off that remains unexplored in the transformer era. |
| Approach: | They propose to evaluate generative and discriminative architectures for text classification using a generative model that learns the conditional probability distribution P (y|x) generative models are known to work better in low-data settings, giving rise to the classical 'two regimes' phenomenon for classification. |
| Outcome: | The proposed models show that the classical 'two regimes' manifests distinctly across different architectures and training paradigms. |
Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that visual encoders can induce harmful behavior in multimodal large language models. |
| Approach: | They propose a vision-centric jailbreak attack that uses visual information to create a jailbreak context. |
| Outcome: | The proposed attack outperforms baseline attacks on MM-SafetyBench and GPT-4o. |
Can Large Language Models Win the International Mathematical Games? (2025.emnlp-main)
Copied to clipboard
Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, Lorenzo Tordi, Antonella Carbonaro, Gianluca Moro
| Challenge: | Recent advances in large language models (LLMs) have demonstrated strong mathematical reasoning abilities, even in visual contexts. |
| Approach: | They propose a benchmark of 2,183 high-quality mathematical problems in an open-ended format that enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
| Outcome: | The new benchmark spans seven age groups and a skill-based taxonomy and enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
CodeArena: Evaluating and Aligning CodeLLMs on Human Preference (2025.emnlp-main)
Copied to clipboard
Jian Yang, Jiaxi Yang, Wei Zhang, Jin Ke, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Zhoujun Li, Binyuan Hui, Junyang Lin
| Challenge: | Code large language models (codeLLMs) focus on synthesizing the correct code snippet, ignoring the alignment with human preferences. |
| Approach: | They propose a benchmark code-based on 40 categories and 44 programming languages to emulate real-world coding tasks. |
| Outcome: | The proposed benchmarks show that open-source code LLMs perform better than open-sourced ones. |
Language models can learn implicit multi-hop reasoning, but only if they have lots of training data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies explore the ability of language models to solve multi-hop reasoning tasks without chain of thought. |
| Approach: | They propose to use GPT2-style language models to train k-hop reasoning models . they show that the required training data grows exponentially in k . |
| Outcome: | The proposed models can learn implicit reasoning without chain-of-thoughts, the authors show . their training data grows exponentially in k, and the required number of transformer layers grows linearly in the model. |
UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment (2025.emnlp-main)
Copied to clipboard
Joseph Marvin Imperial, Abdullah Barayan, Regina Stodden, Rodrigo Wilkens, Ricardo Muñoz Sánchez, Lingyun Gao, Melissa Torgbi, Dawn Knight, Gail Forey, Reka R. Jablonkai, Ekaterina Kochmar, Robert Joshua Reynolds, Eugénio Ribeiro, Horacio Saggion, Elena Volodina, Sowmya Vajjala, Thomas François, Fernando Alva-Manchego, Harish Tayyar Madabushi
| Challenge: | Language proficiency research plays a central role in education and often intersects with advances in linguistics and AI. |
| Approach: | They propose a multilingual multidimensional dataset of texts annotated according to the CEFR scale in 13 languages. |
| Outcome: | The proposed dataset supports linguistic features and pretrained models in multilingual CEFR level assessment. |
CROP: Contextual Region-Oriented Visual Token Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing VLMs process entire images, leading to excessive visual tokens . redundant image information also introduces a large number of visual token, requiring much higher memory and computation in VLM. |
| Approach: | They propose a framework to prune visual tokens using localization and pruning . they propose CROP to locate local image regions relevant to the query . |
| Outcome: | The proposed framework outperforms existing visual token pruning methods on a wide range of tasks. |
CR4-NarrEmote: An Open Vocabulary Dataset of Narrative Emotions Derived Using Citizen Science (2025.emnlp-main)
Copied to clipboard
| Challenge: | CR4-NarrEmote is a large-scale, open-vocabulary dataset of narrative emotions . authors present a dataset of emotion annotations for novel and novel narratives . |
| Approach: | They introduce a large-scale, open-vocabulary dataset of narrative emotions . they use a citizen science initiative to collect emotion annotations from 43,000 passages . |
| Outcome: | The proposed dataset provides an important foundation for affective computing and narrative understanding. |
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks. however, their extensive memory requirements present significant challenges for deployment in resource-constrained environments. |
| Approach: | They propose a training-free framework that achieves ultra-low equivalent bit-width KV cache quantization. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on TruthfulQA and LongBench. |
DINT Transformer (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that the DINT Transformer improves accuracy and robustness across practical applications. |
| Approach: | They propose a differential attention mechanism that suppresses the impact of irrelevant contexts by computing DIF-Ference between two independent attention distributions. |
| Outcome: | The proposed architecture improves numerical stability and ability to capture global dependencies. |
ICR: Iterative Clarification and Rewriting for Conversational Search (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conversational Query Rewriting (CQR) is a key step in conversational question answering . it aims to rewrite vague queries into de-contextualized queries, thereby promoting conversational search. |
| Approach: | They propose an iterative rewriting scheme that pivots on clarification questions . they propose to rewrite queries into de-contextualized queries to promote conversational search . |
| Outcome: | The proposed framework improves retrieval performance on two popular datasets. |
Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are vulnerable to data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet. |
| Approach: | They propose an Optimal Transport-based framework to reconstruct image-caption pairs and propose an optimal transport-based distance measure to re-assign new captions based on the proposed optimal transport distance. |
| Outcome: | The proposed framework reduces the attack success rates of poisoning attacks to 0% in most cases. |
Similarity = Value? Consultation Value-Assessment and Alignment for Personalized Search (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on semantic similarity to align historical consultations with current queries due to the absence of ‘value’ labels, but this lacks exploration of needs in user consultations. |
| Approach: | They propose a consultation value assessment framework that evaluates historical consultations from three novel perspectives: (1) Scenario Scope Value, (2) Posterior Action Value, and (3) Time Decay Value. |
| Outcome: | The proposed model outperforms baselines on public and commercial datasets on both retrieval and ranking tasks. |
RTQA : Recursive Thinking for Complex Temporal Knowledge Graph Question Answering with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current temporal knowledge graph question answering methods focus on implicit temporal constraints and lack the capability to handle complex temporal queries. |
| Approach: | They propose a temporal knowledge graph question answering framework that recursively decomposes questions into sub-problems and employs multi-path answer aggregation to improve fault tolerance. |
| Outcome: | The proposed framework outperforms existing methods on multiTQ and TimelineKGQA benchmarks. |
Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments demonstrate that our approach significantly alleviates task interference and forgetting. |
| Approach: | They propose a framework for supervised fine-tuning for large language models . they first fine-tail the model on each task to identify its core parameter regions . |
| Outcome: | The proposed framework outperforms vanilla fine-tuning and baselines on multiple public benchmarks on reasoning, dialogue, instruction following, and more. |
AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks show coarse granularity, linguistic bias, and a neglect of multimodal privacy risks. |
| Approach: | They propose a benchmark for visual-language models that analyzes social photos to assess location privacy risks. |
| Outcome: | The proposed benchmarks show coarse granularity, linguistic bias, and neglect of privacy risks. |
CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning paradigms focus on aligning LLMs with task-specific objectives. |
| Approach: | They propose a pipeline that leverages human priors to automatically generate token-level causal signals and introduce the Re-Attention mechanism to guide training. |
| Outcome: | The proposed pipeline achieves an average improvement of 5.76% on the STG dataset and 1.56% on downstream tasks. |
Enhancing LLM Text Detection with Retrieved Contexts and Logits Distribution Consistency (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting large language models (LLMs) generate fluent text, but they only use a few tokens due to the short length or insufficient information in some texts. |
| Approach: | They propose a method that leverages external text corpora to evaluate the difference in logit distribution of input text under retrieved human-written and LLM-rewritten contexts. |
| Outcome: | The proposed method achieves state-of-the-art performance in AUROC on five public datasets with three widely-used source LLMs. |
Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) produce a chain of thought (CoT) when prompted to think step-by-step, but it is unclear whether the reasoning encoded in the CoT is faithful to the models’ parametric beliefs. |
| Approach: | They propose a framework for measuring parametric faithfulness of generated reasoning by unlearning reasoning steps (FUR) they propose to erase information contained in reasoning steps from model parameters and measure faithfulness as the resulting effect on the model’s prediction. |
| Outcome: | The proposed framework erases information contained in reasoning steps from model parameters and measures faithfulness as the resulting effect on the model’s prediction. |
Stop Looking for “Important Tokens” in Multimodal Language Models: Duplication Matters More (2025.emnlp-main)
Copied to clipboard
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, Linfeng Zhang
| Challenge: | Vision tokens in multimodal large language models often dominate computational overhead due to excessive length compared to linguistic modality. |
| Approach: | They propose a token pruning method which defines an importance criterion for vision tokens and prunes the unimportant vision token during inference. |
| Outcome: | The proposed method can prune 88.9% of vision tokens while maintaining comparable performance. |
AgentPro: Enhancing LLM Agents with Automated Process Supervision (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks lack explicit supervision during the reasoning process, which may lead to error propagation across reasoning chains. |
| Approach: | They propose a framework which automates process supervision for large language model agents by automatically generating step-level annotations and developing a process reward model based on these annotations. |
| Outcome: | The proposed framework outperforms existing agent-based methods on four datasets and achieves a 6.32% increase in accuracy. |
PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing retrieval-based methods to pre-select tools are often misaligned with tool-calling LLMs due to separate training processes. |
| Approach: | They propose a method to fine-tune retrievers to find useful tools by using a frozen LLM. |
| Outcome: | The proposed method fine-tunes retrievers to find useful tools using a frozen LLM . it improves tool selection accuracy and can be generalized to new queries and tools . |
MusKGC: A Flexible Multi-source Knowledge Enhancement Framework for Open-World Knowledge Graph Completion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Open-world knowledge graph completion (KGC) aims to infer novel facts by enriching existing graphs with external knowledge sources while maintaining semantic consistency under the open-world assumption (OWA). |
| Approach: | They propose a multi-source knowledge enhancement framework based on an open-world assumption (OWA) that integrates external knowledge sources and a new evaluation strategy to validate new facts. |
| Outcome: | The proposed model achieves SOTA performance across benchmarks and the evaluation strategy effectively assesses new facts under OWA. |
Towards Transferable Personality Representation Learning based on Triplet Comparisons and Its Applications (2025.emnlp-main)
Copied to clipboard
Kai Tang, Rui Wang, Renyu Zhu, Minmin Lin, Xiao Ding, Tangjie Lv, Changjie Fan, Runze Wu, Haobo Wang
| Challenge: | Existing methods for personality analysis treat corpus as a single unit for classification, but this approach presents several challenges. |
| Approach: | They propose a task paradigm for text-based personality representation learning that uses a triplet personality trend comparison dataset to learn single-sentence personality embeddings with desirable metric properties. |
| Outcome: | The proposed model significantly boosts performance across various applications, including personality detection, personality retrieval, and emotion translation prediction. |
Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Audio Language Models (LALMs) have demonstrated unprecedented capabilities in natural language understanding and generation, revolutionizing human-machine dialogue. |
| Approach: | They propose an unsupervised safety-fine-tuning strategy that reshapes LALMs representation space to enhance existing LALM safety-alignment while balancing the risk of over-rejection. |
| Outcome: | The proposed approach improves LALMs safety under three input conditions while increasing over-rejection rate by only 0.88% on average. |
Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation (2025.emnlp-main)
Copied to clipboard
Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, Baishakhi Ray
| Challenge: | In the era of evaluating large language models, data contamination is an increasingly prominent concern . static benchmarking has been used for evaluation, but there are limitations of *dynamic* benchmarks . |
| Approach: | They propose a series of optimal design principles for *dynamic* benchmarking and analyze the limitations of existing *static* benchmarks. |
| Outcome: | The proposed benchmarks highlight a critical gap in the evaluation of LLMs. |
FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domain (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent LLMs have demonstrated promising ability in solving finance related problems, but applying them in real-world finance applications remains challenging due to its high risk and high stakes property. |
| Approach: | They propose a benchmark specifically designed for evaluating the trustworthiness of LLMs in finance applications. |
| Outcome: | The proposed benchmark outperforms proprietary models in most tasks while open-source models have advantage in specific areas like industry-level fairness. |
RecGPT: A Foundation Model for Sequential Recommendation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches fail in cold-start and cross-domain scenarios where new users or items lack sufficient interaction history. |
| Approach: | They propose a foundation model for sequential recommendation that achieves genuine zero-shot generalization capabilities by deriving item representations exclusively from textual features. |
| Outcome: | The proposed model achieves zero-shot generalization capabilities in cold-start and cross-domain scenarios. |
Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large audio-language models (LALMs) have expanded their impact beyond natural language processing (NLP) to multimodal domains. |
| Approach: | They propose a systematic taxonomy for LALM evaluations, categorizing them into four dimensions based on their objectives: (1) General Auditory Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented Ability, and (4) Fairness, Safety, and Trustworthiness. |
| Outcome: | The proposed taxonomy categorizes LALM evaluations into four dimensions based on their objectives and highlights challenges in this field. |
Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy (2025.emnlp-main)
Copied to clipboard
Nikita Balagansky, Yaroslav Aksenov, Daniil Laptev, Vadim Kurochkin, Gleb Gerasimov, Nikita Koriagin, Daniil Gavrilov
| Challenge: | Sparse Autoencoders (SAEs) are powerful tools for interpreting neural networks . conventional SAEs are constrained by the fixed sparsity level chosen during training . |
| Approach: | They propose a training objective that trains a single SAE to optimise reconstructions across multiple sparsity levels simultaneously. |
| Outcome: | The proposed objective achieves Pareto-optimal trade-offs between sparsity and explained variance, outperforming traditional SAEs trained at individual sparsities. |
Learn and Unlearn: Addressing Misinformation in Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to unlearning large language models (LLMs) focus on English data, but they ignore multilingual contexts and can produce misleading, offensive, or otherwise fake content. |
| Approach: | They investigate the propagation of information in multilingual large language models and evaluate unlearning methods to address harmful content in multi-lingual contexts. |
| Outcome: | The proposed methods can effectively eliminate harmful content for all languages by addressing both English and the original language of the harmful data. |
PRISM: Efficient Long-Range Reasoning With Short-Context LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing solutions to long-range language tasks require large compute budgets and complex task-specific design choices. |
| Approach: | They propose an in-context method that uses structured schemas to generate short-contemporary outputs. |
| Outcome: | a new in-context method outperforms baselines on diverse tasks with 4x shorter contexts . it scales down to tiny contexts without increasing costs or sacrificing quality . |
Augmenting Multi-Agent Communication with State Delta Trajectory (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-agent systems based on large language models (LLMs) have shown to be effective in downstream tasks. |
| Approach: | They propose a protocol that transfers both natural language tokens and token-wise state transition trajectory from one agent to another. |
| Outcome: | The proposed protocol can transfer both natural language tokens and token-wise state transition trajectory from one agent to another. |
SAEs Are Good for Steering – If You Select the Right Features (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) can learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Approach: | They propose an unsupervised approach to learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Outcome: | The proposed approach matches the performance of existing supervised methods by identifying features with low output scores and identifying them with input and output scores. |
CoBA: Counterbias Text Augmentation for Mitigating Various Spurious Correlations via Semantic Triples (2025.emnlp-main)
Copied to clipboard
| Challenge: | Spurious correlations are patterns that appear in datasets but do not represent genuine relationships. |
| Approach: | They propose a more general form of counterfactual data augmentation that tackles multiple biases . they propose 'CoBA' that decomposes text into subject-predicate-object triples and modifies them to disrupt spurious correlations. |
| Outcome: | The proposed framework reduces biases and strengthens out-of-distribution resilience. |
Layered Insights: Generalizable Analysis of Human Authorial Style by Leveraging All Transformer Layers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to authorship attribution model only learn from the output layer of pre-trained transformers, ignoring representations learned at other layers. |
| Approach: | They propose a model that leverages the various linguistic representations learned at different layers of pre-trained transformer-based models to model the authorship attribution task more effectively. |
| Outcome: | The proposed model performs better on out-of-domain and in-domain scenarios, while ignoring representations learned at other layers. |
When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance across NLP tasks. |
| Approach: | They propose to use long-context SFT to improve short-contemporary performance . they also decouple and analyze two key components, Multi-Head Attention and Feed-Forward Network . |
| Outcome: | The proposed model improves short-context performance, contrary to pretraining. |
A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge’ez Script. (2025.emnlp-main)
Copied to clipboard
Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Henok Biadglign Ademtew, Hizkiel Mitiku Alemayehu, Negasi Haile Abadi, Tadesse Destaw Belay, Seid Muhie Yimam
| Challenge: | Homophone normalization is a pre-processing step used in Amharic natural language processing (NLP) but it also results in models that are unable to process different forms of writing in a single language. |
| Approach: | They propose a method where normalization is applied to model predictions instead of training data and a scheme where normalized data is preserved in training. |
| Outcome: | The proposed model achieves an increase in BLEU score of up to 1.03 while preserving language features in training. |
Evaluating Language Translation Models by Playing Telephone (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models are inadequate for evaluating machine translation systems . current evaluation methods are costly and require specialized expertise to prepare and score gold standard translations . |
| Approach: | They propose an unsupervised method to generate training data for translation evaluation by repeated rounds of translation between source and target languages. |
| Outcome: | The proposed method outperforms a popular translation evaluation system on two tasks . human annotation is costly and requires specialized expertise to prepare and score gold standard translations . |
Doubling Your Data in Minutes: Ultra-fast Tabular Data Generation via LLM-Induced Dependency Graphs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Tabular data is critical across diverse domains, yet high-quality tabular datasets remain scarce due to privacy concerns and the cost of collection. |
| Approach: | They propose a lightweight generative framework that captures sparse dependencies via an LLM-induced graph. |
| Outcome: | The proposed framework reduces constraint violations by 4% and accelerates generation by nearly 9,500 over diffusion-based methods. |
SPaRC: A Spatial Pathfinding Reasoning Challenge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reasoning datasets saturate and fail to test abstract, multi-step problems, especially pathfinding and complex rule constraint satisfaction. |
| Approach: | They propose to use a spatial few-shot grid to evaluate spatial and rule-based reasoning with 1,000 2D grid puzzles. |
| Outcome: | The proposed model can be used to evaluate spatial reasoning and improve its accuracy. |
Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable advancements in specialized fields such as finance, law, and medicine. |
| Approach: | They propose to provide datasets covering all major training stages including pretraining, instruction fine-tuning, and reasoning distillation with cybersecurity-specific self-reflection data. |
| Outcome: | Extensive ablation studies show that LLMs acquire their knowledge during pretraining, while reasoning distillation leads to a 15% gain in security certification (CISSP). |
Bit-Flip Error Resilience in LLMs: A Comprehensive Analysis and Defense Framework (2025.emnlp-main)
Copied to clipboard
Yuhang Chen, Zhen Tan, Ajay Kumar Jaiswal, Huaizhi Qu, Xinyu Zhao, Qi Lin, Yu Cheng, Andrew Kwong, Zhichao Cao, Tianlong Chen
| Challenge: | Bit-flip errors (BFEs) are hardware faults where individual bits in memory or processing units are unintentionally flipped. |
| Approach: | They propose a novel defense strategy to mitigate bit-flip errors (BFEs) they propose bfe protection and a self-correction mechanism to minimize performance degradation . |
| Outcome: | The proposed defense strategy minimizes performance degradation while significantly improving robustness against BFEs. |
Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of generating persuasive Natural Language Explanations (NLEs) however, the faithfulness of these explanations should not be readily trusted at face value. |
| Approach: | They propose to use a causal mediation technique called activation patching to measure the faithfulness of an explanation towards supporting the explained answer. |
| Outcome: | The proposed metric, Causal Faithfulness, quantifies the consistency of causal attributions between explanations and the corresponding model outputs as the indicator of faithfulness. |
Calibrating LLM Confidence by Probing Perturbed Representation Stability (2025.emnlp-main)
Copied to clipboard
Reza Khanmohammadi, Erfan Miahi, Mehrsa Mardikoraem, Simerjot Kaur, Ivan Brugere, Charese Smiley, Kundan S Thind, Mohammad M. Ghassemi
| Challenge: | Despite their impressive performance, large language models (LLMs) consistently struggle with confidence calibration. |
| Approach: | They propose a method to analyze internal representational stability in large language models by applying adversarial perturbations to final hidden states and using a lightweight classifier to predict answer correctness. |
| Outcome: | CCPS significantly outperforms existing methods on LLMs from 8B to 32B parameters in multiple-choice and open-ended formats. |
SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate remarkable performance across diverse tasks, yet their effectiveness often depends on costly commercial APIs or cloud services. |
| Approach: | They propose a dual-mode compatible approach that fine-tunes models through shortest-response preference optimization and a confidence-aware rejection mechanism. |
| Outcome: | The proposed approach reduces redundant outputs and response times while reducing computational costs by over 50% and cascade latency by over 80%. |
DSG-MCTS: A Dynamic Strategy-Guided Monte Carlo Tree Search for Diversified Reasoning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong potential in complex reasoning tasks, but their performance often degrades, resulting in hallucinations, errors, and logical inconsistencies. |
| Approach: | They propose a framework that integrates multiple reasoning strategies to expand the reasoning space and a dynamic strategy selection mechanism that adapts to the task context. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods on a set of reasoning benchmarks. |
CIFLEX: Contextual Instruction Flow for Sub-task Execution in Multi-Turn Interactions with a Single On-Device LLM (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experiments show that CIFLEX significantly reduces computational costs without degrading task performance. |
| Approach: | They propose a new execution system for efficient sub-task handling with a single large language model. |
| Outcome: | Experiments show that CIFLEX significantly reduces computational costs without degrading task performance. |
On the Role of Model Prior in Real-World Inductive Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have evaluated the inductive reasoning capabilities of Large Language Models (LLMs) by evaluating their ability to generate textual hypotheses based on in-context input-output pairs and test these hypothese based upon unseen examples. |
| Approach: | They evaluated three inductive reasoning strategies across five real-world tasks with three LLMs and found that hypothesis generation is primarily driven by the model’s inherent priors. |
| Outcome: | The proposed models generate high-quality hypotheses that can generalize to new instances when guided by in-context demonstrations. |
Viability of Machine Translation for Healthcare in Low-Resourced Languages (2025.emnlp-main)
Copied to clipboard
Hellina Hailu Nigatu, Nikita Mehandru, Negasi Haile Abadi, Blen Gebremeskel, Ahmed Alaa, Monojit Choudhury
| Challenge: | MT errors are more pronounced in low-resourced languages where human translators are scarce and MT tools perform poorly. |
| Approach: | They propose to use a publicly available machine translation system to analyze machine translation errors in healthcare domains. |
| Outcome: | The proposed system reduces errors in two low-resourced languages for healthcare. |
Latent Inter-User Difference Modeling for LLM Personalization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly integrated into users’ daily lives, leading to a growing demand for personalized outputs. |
| Approach: | They propose a framework that models inter-user differences in the latent space instead of relying on language-based prompts. |
| Outcome: | The proposed framework outperforms baseline methods on personalized review generation. |
IG-Pruning: Input-Guided Block Pruning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for efficient inference rely on fixed block masks, which can lead to suboptimal performance. |
| Approach: | They propose an input-aware block-wise pruning method that dynamically selects layer masks at inference time. |
| Outcome: | The proposed method outperforms state-of-the-art static depth pruning methods . it is particularly suitable for resource-constrained deployment scenarios . |
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks? (2025.emnlp-main)
Copied to clipboard
| Challenge: | evaluators using large language models face ambiguous criteria and inconsistent evaluations. |
| Approach: | They investigate whether checklists should be used for all questions or selectively . they generate checklists using six methods and evaluate their effectiveness across eight models . |
| Outcome: | The proposed method improves evaluation performance in pairwise comparisons while ignoring human-written criteria. |
Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Several datasets have been created for factual knowledge assessment of LLMs . MLAMA uses templated sentences that do not take into account grammatical and semantic information of named entities . |
| Approach: | They use MLAMA to prompt LLMs for factual knowledge using template translations . this leads to numerous instances of ungrammaticality or wrong wording of the final prompts - authors say . |
| Outcome: | The results show that the MLAMA dataset has higher knowledge retrieval scores than previous versions. |
Knowledge Editing through Chain-of-Thought (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge editing methods focus on multi-hop QA tasks and require frequent retraining. |
| Approach: | They propose a new knowledge editing framework that updates large language models with new information to maintain their world knowledge without retraining. |
| Outcome: | The proposed method achieves state-of-the-art performance while offering superior generalization, effectiveness, and stability compared to existing methods. |
SelfRACG: Enabling LLMs to Self-Express and Retrieve for Code Generation (2025.emnlp-main)
Copied to clipboard
Qian Dong, Jia Chen, Qingyao Ai, Hongning Wang, Haitao Li, null Yiwu, Yao Hu, Yiqun Liu, Shaoping Ma
| Challenge: | Existing retrieval-augmented code generation methods fail to accurately fetch the knowledge required for code generation for consecutive code fragments. |
| Approach: | They propose a paradigm that enables large language models to Self-express their information needs to enhance retrieval-augmented code generation methods. |
| Outcome: | Experiments show that SelfRACG can retrieve external knowledge that better aligns with the LLM’s own information needs, resulting in superior generation performance compared to vanilla RACG. |
Probing Logical Reasoning of MLLMs in Scientific Diagrams (2025.emnlp-main)
Copied to clipboard
| Challenge: | logical reasoning is key to real-world applications like science education, environmental monitoring, and medical diagnostics. |
| Approach: | They construct visual questions that follow seven structured templates with progressively more complex reasoning involved. |
| Outcome: | The proposed models perform logical inferences based on visual information. |
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirically, AdamS demonstrates strong performance in various tasks . et al., 2023b): AdamS is efficient, efficient, and model-agnostic. |
| Approach: | They propose a model-agnostic alternative to Adam for large language model pretraining and post-training. |
| Outcome: | The proposed method matches memory footprint of SGD with momentum while delivering superior performance. |
Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls (2025.emnlp-main)
Copied to clipboard
Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, Ramya Raghavendra, Ruoxi Jia, Carole-Jean Wu
| Challenge: | a large-scale empirical study compares natural web data, diverse synthetic types, and mixtures of natural and synthetic data. |
| Approach: | They conduct a large-scale empirical study on large-volume LLMs using a unified protocol and scaling laws. |
| Outcome: | The proposed method is faster than pre-training on natural web data, the authors show . their results are consistent with previous studies on rephrased text and textbooks . |
Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to compress video inputs ignore the importance of static and dynamic information in long videos, leading to inefficient token usage within limited budgets. |
| Approach: | They propose a token selection strategy that adaptively adjusts static and dynamic information based on question requirements. |
| Outcome: | The proposed method achieves performance improvements (up to 5.8%) on multiple video question answering benchmarks. |
DischargeSim: A Simulation Benchmark for Educational Doctor–Patient Communication at Discharge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Discharge communication is a critical yet underexplored component of patient care, where the goal shifts from diagnosis to education. |
| Approach: | They propose a benchmark that evaluates large language models’ ability to act as personalized discharge educators. |
| Outcome: | Experiments with 18 LLMs show that model size does not always yield better education outcomes, highlighting trade-offs in strategy use and content prioritization. |
Can Vision-Language Models Solve Visual Math Equations? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) perform well on textual equations, but fail on visually grounded counterparts. |
| Approach: | They propose to decompose visual equation solving into symbolic equation solving and visual recognition into two core components to understand this gap. |
| Outcome: | The proposed models perform well on textual equations, but fail on visual grounded ones. |
From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)
Copied to clipboard
Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao
| Challenge: | Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments. |
| Approach: | They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation. |
| Outcome: | The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%. |
Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) aims to mitigate the hallucination of Large Language Models (LLMs) however, external knowledge may contain noise and conflict with parametric knowledge of LLMs, leading to degraded performance. |
| Approach: | They propose a Dual-Stream Knowledge-Augmented Framework for Shared-Private Semantic Synergy that refines the traditional self-attention into a mixed-attention that distinguishes shared and private semantics for a controlled knowledge integration. |
| Outcome: | Extensive experiments show that the proposed framework achieves a superior performance over baselines. |
Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies evaluate the creative capabilities of large language models (LLMs) through diverse tasks, aiming to understand their strengths and limitations. |
| Approach: | They propose to ask LLMs to generate Parallel Chains of Associations to Evaluate their creativity. |
| Outcome: | The proposed framework minimizes the risk of data contamination and offers a highly efficient evaluation. |
Identifying Unlearned Data in LLMs via Membership Inference Attacks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work evaluates approximate unlearning under a retrieval paradigm, where adversaries attempt to extract residual knowledge given partial information of the unlearning target. |
| Approach: | They propose a framework to evaluate unlearning membership attacks using member inference techniques to exploit the forget set. |
| Outcome: | The proposed framework assesses whether unlearning leaves behind detectable artifacts that can be exploited to infer membership in the forget set. |
Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can solve reasoning and mathematical problems using the Chain-of-Thought technique, but require costly and long CoT data and fine-tuning. |
| Approach: | They propose a method that uses Sparse Autoencoders to extract interpretable features from vanilla CoT and use them to steer the LLM's internal states. |
| Outcome: | The proposed method uses Sparse Autoencoders (SAEs) to extract interpretable features from vanilla CoT and steer the LLM's internal states during generation. |
LLMs cannot spot math errors, even when allowed to peek into the solution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate impressive performance on existing reasoning benchmarks, but struggle with meta-reasoning tasks such as locating the first error step in student solutions. |
| Approach: | They propose an approach that generates an intermediate corrected student solution, aligning more closely with the original student’s solution, which helps improve performance. |
| Outcome: | The proposed approach generates an intermediate corrected student solution, aligning more closely with the original student’s solution, which helps improve performance. |
Can LLMs be Good Graph Judge for Knowledge Graph Construction? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for converting unstructured text into structured Knowledge Graphs (KGs) have limitations such as large amount of noise, inaccurate knowledge, and hallucination . |
| Approach: | They propose a GraphJudge framework to reduce noise in real-world documents . they propose Graphjudge to fine-tune a LLM as a graph judge to enhance quality . |
| Outcome: | The proposed framework eliminates noise in real-world documents and improves the quality of generated KGs. |
NeuroAda: Activating Each Neuron’s Potential for Parameter-Efficient Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited and require computational and memory resources. |
| Approach: | They propose a parameter-efficient fine-tuning method that enables fine-grained model finetunation while maintaining high memory efficiency. |
| Outcome: | The proposed method reduces CUDA memory usage by up to 60% while maintaining high performance. |
NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current research directions rely on synthetic data generated by translating English corpora, which often fails to represent the cultural heritage and values of local communities. |
| Approach: | They propose a method to create and retrieve pre-training data tailored to a specific community . they use Egyptian and Moroccan dialects as testbeds to test their understanding . |
| Outcome: | The proposed method outperforms existing Arabic-aware LLMs and performs on par with larger models. |
A Computational Simulation of Language Production in First Language Acquisition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing computational studies of child language acquisition focus on isolated mechanisms, such as spreading activation in retrieval, sentence planning, or production efficiency. |
| Approach: | They propose a computational framework for modeling child language production using graphs to formalize meaning and Synchronous Hyperedge Replacement Grammar to formalized the syntax–semantics interface. |
| Outcome: | The proposed framework is based on graphs to formalize meaning and Synchronous Hyperedge Replacement Grammar (SHRG) resulting interpretable grammars are evaluated by their ability to generate utterances . |
Long-Form Information Alignment Evaluation Beyond Atomic Facts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-grained methods for information alignment evaluation neglect inter-fact dependencies, enabling subtle vulnerabilities. |
| Approach: | They propose MontageLie, a benchmark that "montages" truthful statements without introducing explicit hallucinations. |
| Outcome: | The proposed framework outperforms existing fine-grained methods by over 8% . it is a novel approach for enhancing trust and reducing hallucinations . |
Voice of a Continent: Mapping Africa’s Speech Technology Frontier (2025.emnlp-main)
Copied to clipboard
AbdelRahim A. Elmadany, Sang Yun Kwon, Hawau Olamide Toyin, Alcides Alcoba Inciarte, Hanan Aldarmaki, Muhammad Abdul-Mageed
| Challenge: | linguistic diversity in Africa is underrepresented in speech technologies, creating barriers to digital inclusion. |
| Approach: | They propose a benchmarking framework to map the continent's linguistic diversity and map its impact on downstream African speech tasks. |
| Outcome: | The proposed model achieves state-of-the-art across multiple African languages and speech tasks. |
Cache-Efficient Posterior Sampling for Reinforcement Learning with LLM-Derived Priors Across Discrete and Continuous Domains (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used as action proposers in reinforcement learning (RL) but they are expensive and require prohibitive computational costs. |
| Approach: | They propose a cache-efficient framework for Bayesian RL that leverages large language models as action proposers and optimizes meta-learning based on policy performance to enable efficient inference across text-based games and robotic control tasks. |
| Outcome: | The proposed framework achieves 3.8–4.7 reduction in LLM queries and 4.0–12.0 lower median latencies (85–93ms on consumer hardware) while retaining 96–98% of the uncached policy’s performance. |
Circuit Complexity Bounds for RoPE-based Transformer Architecture (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies provide the circuit complexity bounds to Transformer-like architectures. position embedding has emerged as a crucial technique in modern large language models. |
| Approach: | They propose to use position embedding to improve Transformer-like architectures by analyzing their circuits and analyzing the results. |
| Outcome: | The proposed model is able to solve canonical tasks without embedding positional information. |
Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-space models struggle with quadratic computational complexity, limiting their use in long-context tasks and resource-constrained input data. |
| Approach: | They propose a pruning framework specifically tailored for Mamba that reduces parameter counts by 70% with only a 3–9% drop in performance. |
| Outcome: | The proposed pruning framework achieves up to 70% parameter reduction with only a 3–9% drop in performance. |
Towards Infinite-Long Prefix in Transformer (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prefix Learning is an empirically efficient and effective method for language models . but the theoretical understandings are limited on the performance of such methods . |
| Approach: | They propose a method that can train an ultra-long prefix in a stylized setting using the Neural Tangent Kernel framework. |
| Outcome: | The proposed method can achieve superior performance on vision, natural language, and math data. |
LATTE: Learning to Think with Vision Specialists (2025.emnlp-main)
Copied to clipboard
Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, Ranjay Krishna, Silvio Savarese
| Challenge: | Open-source vision-language models excel on simple question-answering tasks, but struggle with complex questions that require both perception and reasoning. |
| Approach: | They propose a family of vision-language models that have LeArned to Think wiTh vision spEcialists by offloading perception to state-of-the-art vision models. |
| Outcome: | The proposed model achieves 4-5% gains over baselines across 6 benchmarks covering both perception and reasoning abilities. |
SUA: Stealthy Multimodal Large Language Model Unlearning Attack (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) trained on massive data may memorize sensitive personal information and photos, posing privacy and copyright concerns. |
| Approach: | They propose a framework that learns a universal noise pattern to recover unlearned information from MLLMs. |
| Outcome: | The proposed framework learns a universal noise pattern and can reveal unlearned content when applied to images. |
ResFormer: All-Time Reservoir Memory for Long Sequence Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models with quadratic time and memory complexity restrict input length . however, analyzing extensive sequential contexts is challenging . |
| Approach: | They propose a neural network architecture that captures contextual dependencies in linear time and a nonlinear readout to model short-term dependencies within sentences. |
| Outcome: | The proposed model outperforms baseline models on EmoryNLP datasets and on IEMOCAP and MultiWOZ datasets. |
Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Increasing the ability of large language models to perform latent multihop reasoning is crucial for reducing the cost and deployment challenges. |
| Approach: | They propose an interpretability method that traces how logits propagate across layers and positions toward the final prediction. |
| Outcome: | The proposed method improves accuracy on five reasoning datasets. |
Interdisciplinary Research in Conversation: A Case Study in Computational Morphology for Language Documentation (2025.emnlp-main)
Copied to clipboard
| Challenge: | despite interest in language documentation, we still lack broadly usable tools that support workflows. |
| Approach: | They propose to integrate user-centered design principles into NLP to reshape the field. |
| Outcome: | The proposed model fails to meet core usability needs in real-world language documentation contexts. |
Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are powerful automatic evaluators for natural language generation (NLG) tasks, but their uncertainty may limit their deployment in many applications. |
| Approach: | They propose a conformal prediction framework that provides a prediction interval with coverage guarantees and a midpoint-based score as a low-bias alternative to raw model score and weighted average. |
| Outcome: | The proposed framework provides a prediction interval with coverage guarantees and a midpoint-based score as a low-bias alternative to raw model score and weighted average. |
AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time (2025.emnlp-main)
Copied to clipboard
Junyu Zhang, Runpei Dong, Han Wang, Xuying Ning, Haoran Geng, Peihao Li, Xialin He, Yutong Bai, Jitendra Malik, Saurabh Gupta, Huan Zhang
| Challenge: | Existing monotonic scaling methods for large reasoning models are not reliable. |
| Approach: | They propose a universal framework for modulating reasoning progress in large reasoning models at test time. |
| Outcome: | The proposed framework unifies and generalizes existing monotonic scaling methods and enables flexible and dense slow-to-fast reasoning modulation. |
Dual-Path Dynamic Fusion with Learnable Query for Multimodal Sentiment Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis struggle with global and fine-grained contributions and over-reliance on text. |
| Approach: | They propose a multimodal sentiment analysis architecture that processes inputs through two complementary paths: global and local. |
| Outcome: | The proposed architecture achieves state-of-the-art in fine-grained sentiment prediction on the CMU-MOSI and CMU MOSEI benchmarks. |
CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge editing methods fail to generalize updates to multi-hop reasoning tasks . Existing methods only edit single or a few model layers, inadequately integrate updated knowledge into reasoning pathways. |
| Approach: | They propose a circuit-aware method that enhances the effective integration of updated knowledge in large language models by leveraging curated data samples guided by their analysis. |
| Outcome: | The proposed method improves accuracy and accuracy of 20% on the MQuAKE dataset while requiring less memory. |
DEL-ToM: Inference-Time Scaling for Theory-of-Mind Reasoning via Dynamic Epistemic Logic (2025.emnlp-main)
Copied to clipboard
| Challenge: | Theory-of-Mind (ToM) tasks pose a unique challenge for large language models (LLMs), which often lack the capability for dynamic logical reasoning. |
| Approach: | They propose a framework that decomposes ToM tasks into a sequence of belief updates grounded in Dynamic Epistemic Logic (DEL) they use data generated automatically via a DEL simulator to train a verifier, which is called the Process Belief Model (PBM). |
| Outcome: | The proposed framework improves verifiable ToM reasoning through inference-time scaling rather than architectural changes. |
Collaborative Beam Search: Enhancing LLM Reasoning via Collective Consensus (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the reasoning capabilities of large language models (LLMs) depend on domain-specific external verifiers or self-evaluation which is brittle and prompt-sensitive. |
| Approach: | They propose a framework that harnesses the collective intelligence of multiple large language models across both generation and verification stages. |
| Outcome: | The proposed framework outperforms singlemodel scaling and multi-model ensemble baselines on six tasks by over 4 percentage points in average accuracy. |
Deriving Strategic Market Insights with Large Language Models: A Benchmark for Forward Counterfactual Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for forward counterfactual generation face limitations . large language models (LLMs) offer promise but remain unexplored for this application . |
| Approach: | They propose a benchmark to support forward counterfactual generation in finance . they use financial news headlines to curate financial news and provide structured evaluation . |
| Outcome: | The proposed benchmark aims to provide scalable, automated insights into potential market opportunities and risks for stakeholders. |
Towards Statistical Factuality Guarantee for Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Advancements in Large Vision-Language Models (LVLMs) have demonstrated impressive performance in image-conditioned text generation, but hallucinated outputs pose a major barrier to their use in safety-critical applications. |
| Approach: | They propose a conformal-prediction-based framework that achieves finite-sample distribution-free statistical guarantees to the factuality of LVLM output. |
| Outcome: | The proposed framework reduces the error rate of LLaVa-1.5 claims from 87.8% to 10.0% while ensuring that the output is accurate. |
Unlearning vs. Obfuscation: Are We Truly Removing Knowledge? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent methods often rely on obfuscation by injecting incorrect or irrelevant information to suppress knowledge, leaving models vulnerable to probing. |
| Approach: | They propose a method that flattens the model predictive distribution over automatically generated multiple-choice questions, effectively removing knowledge about target individuals. |
| Outcome: | The proposed method achieves unlearning with over 90% refusal rate and a higher uncertainty than obfuscation on probing questions. |
Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on test-time alignment, where additional compute is allocated during inference to enhance LLM safety and reasoning capabilities. |
| Approach: | They propose a reward-shifted speculative sampling algorithm that aligns a draft model with human preferences while the target model remains unchanged. |
| Outcome: | The proposed algorithm achieves superior gold reward scores at a significantly reduced inference cost in test-time weak-to-strong alignment experiments. |
Stimulate the Critical Thinking of LLMs via Debiasing Discussion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that large language models (LLMs) are often prone to stance homogeneity and human preference biases when faced with conflicting perspectives. |
| Approach: | They propose a novel two-stage training framework to address stance homogeneity bias and human preference bias by generating multi-model discussion datasets and optimizing reinforcement learning from human feedback to align with discussion correctness. |
| Outcome: | The proposed framework reduces stance homogeneity bias and human preference bias and improves generalization capabilities on non-discussion scenarios and out-of-domain datasets. |
Toward Multi-Session Personalized Conversation: A Large-Scale Dataset and Hierarchical Tree Framework for Implicit Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-term open-domain dialogue datasets lack complex, real-world personalization and fail to capture implicit reasoning. |
| Approach: | They propose a large-scale long-term dataset with 2,500 examples containing approximately 100 conversation sessions to study implicit reasoning in personalized dialogues. |
| Outcome: | The proposed model improves the ability of LLMs to reason over long-term conversations with implicit contextual dependencies. |
Improving Instruct Models for Free: A Study on Partial Adaptation (2025.emnlp-main)
Copied to clipboard
Ozan Irsoy, Pengxiang Cheng, Jennifer L Chen, Daniel Preotiuc-Pietro, Shiyue Zhang, Duccio Pappadopulo
| Challenge: | Instruct models are deemed superior and more usable but can be eroded by instruction tuning . a recent study shows that instruct models are better at following instructions than base models . |
| Approach: | They scale down the strength of instruction tuning to improve model performance . they show that reducing instruction tuning results in material improvement . |
| Outcome: | The proposed model improves on a few-shot in-context learning benchmark . but it loses some degree of its in-training ability . |
CoMMIT: Coordinated Multimodal Instruction Tuning (2025.emnlp-main)
Copied to clipboard
Xintong Li, Junda Wu, Tong Yu, Rui Wang, Yu Wang, Xiang Chen, Jiuxiang Gu, Lina Yao, Julian McAuley, Jingbo Shang
| Challenge: | et al., 2024) show that multimodal instruction tuning is more effective than baselines. |
| Approach: | They propose a multimodal balance coefficient that enables quantitative measurement of the balance of learning . they propose auxiliary regularization on the gradient to promote updating with larger step sizes . |
| Outcome: | The proposed method is more effective than baselines in MLLM instruction tuning. |
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge (2025.emnlp-main)
Copied to clipboard
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, Sainbayar Sukhbaatar
| Challenge: | Existing methods for improving large language models have focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. |
| Approach: | They propose an iterative Meta-Rewarding step where the model judges its own judgements and uses that feedback to refine its judgment skills. |
| Outcome: | The proposed model improves Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2 and 20.6% to 29.1% on Arena-Hard. |
AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-agent collaboration rely on static or graph-based topologies lacking flexibility and adaptability. |
| Approach: | They propose a new framework that rethinks multi-agent coordination through a sequential structure rather than a graph structure. |
| Outcome: | The proposed method achieves superior performance while significantly reducing communication overhead. |
A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users (2025.emnlp-main)
Copied to clipboard
Nishant Balepur, Matthew Shu, Yoo Yeon Sung, Seraphina Goldfarb-Tarrant, Shi Feng, Fumeng Yang, Rachel Rudinger, Jordan Lee Boyd-Graber
| Challenge: | We test alignment methods to ensure LLMs are helpful, but they train or evaluate on what users prefer . |
| Approach: | They test alignment methods to ensure LLMs generate plans that help users . they get 4388 plan executions and 5584 comparisons to measure user preferences . |
| Outcome: | The proposed approach can be applied to the problem of user preferences and helpfulness. |
Words Like Knives: Backstory-Personalized Modeling and Detection of Violent Communication (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines the role of personalization in enabling LLMs to serve as effective mediators in human communication for authentic connection. |
| Approach: | They leverage nonviolent communication theory to evaluate LLMs in detecting conversational breakdowns . they annotate a subset of dialogues and obtain fine-grained labels of communication breakdown types . |
| Outcome: | The proposed dataset analyzes human interactions and relationships in a human context. |
Separate the Wheat from the Chaff: Winnowing Down Divergent Views in Retrieval Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) lack robustness in knowledge-intensive tasks due to noisy or irrelevant retrieved data. |
| Approach: | They propose a multi-agent debate-based RAG framework that integrates external knowledge sources into large language models to improve their accuracy. |
| Outcome: | The proposed framework is unsupervised and leverages pretrained LLMs without fine-tuning, making it easily adaptable to various tasks. |
Cognitive Linguistic Identity Fusion Score (CLIFS): A Scalable Cognition‐Informed Approach to Quantifying Identity Fusion from Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for measuring identity fusion are limited and require controlled surveys or direct field contact. |
| Approach: | They propose a new metric that integrates cognitive linguistics with large language models to measure identity fusion. |
| Outcome: | The proposed metric outperforms existing methods and human annotations in violence risk assessment. |
SilVar: Speech-Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models have demonstrated remarkable capabilities across various tasks, including visual question answering and image captioning. |
| Approach: | They propose an end-to-end multimodal model that leverages speech instructions for reasoning-based visual question answering. |
| Outcome: | The proposed model can process and explain visual scenes from spoken input, moving beyond simple object recognition to reasoning-based interactions. |
CEMTM: Contextual Embedding-based Multimodal Topic Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | CEMTM is a context-enhanced multimodal topic model that can infer coherent topic structures from documents . traditional multimodal topics failed to capture deeper cross-modal interactions . large vision language models (LLMs) and LVLMs have shown remarkable capacity to encode rich semantic knowledge from vast corpora. |
| Approach: | They propose a context-enhanced multimodal topic model that uses tokens to weight contributions to topic inference. |
| Outcome: | The proposed model outperforms unimodal and multimodal benchmarks on six multimodal domains and captures semantics in scientific articles. |
RedHerring Attack: Testing the Reliability of Attack Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | In response to adversarial text attacks, attack detection models have been proposed and shown to successfully identify text modified by adversaries. |
| Approach: | They propose to modify a text to cause the detection model to predict an attack while keeping the classifier correct. |
| Outcome: | The proposed model drops detection accuracy between 20 - 71 points while maintaining (or improving) classifier accuracy. |
Modeling Bottom-up Information Quality during Language Processing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contemporary theories of language processing model language processing as integrating both top-down expectations and bottom-up inputs. |
| Approach: | They propose an information-theoretic operationalization for the “quality” of bottom-up information as the mutual information between visual information and word identity. |
| Outcome: | The proposed model compares reading times in English and Chinese in which words' information quality has been reduced by occluding their top or bottom half with full words. |
Data Drives Unstable Hierarchical Generalization in LMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Early in training, LMs can behave like n-gram models but eventually learn tree-based syntactic rules and generalize out of distribution (OOD). |
| Approach: | They study how complex data drives hierarchical rules, while less complex encourages shortcut learning . they find a model uses rules to generalize if its training data is *diverse* . |
| Outcome: | The proposed model learns to generalize hierarchically if its training data is complex . a model learn if it includes center-embedded clauses, a special syntactic structure . |
EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety (2025.emnlp-main)
Copied to clipboard
Jiahao Qiu, Yinghui He, Xinzhe Juan, Yimin Wang, Yuhan Liu, Zixin Yao, Yue Wu, Xun Jiang, Ling Yang, Mengdi Wang
| Challenge: | EmoAgent evaluates and mitigates mental health hazards in human-AI interactions, especially for vulnerable human users with psychological disorders. |
| Approach: | EmoAgent is a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. |
| Outcome: | EmoAgent evaluates and mitigates mental health hazards in human-AI interactions. |
Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | specialized large language models (LLMs) are capable of generating coherent, contextually relevant and often human-level language, but they remain vulnerable to errors when confronted with data that falls outside the scope of their domain. |
| Approach: | They propose an inference-time out-of-domain (OOD) detection algorithm that leverages the Inductive Conformal Anomaly Detection framework and a new non-conformity measure based on the model’s dropout tolerance. |
| Outcome: | The proposed method improves detection while maintaining theoretical false alarm bounds from ICAD while maintaining false alarms bounds. |
Facilitating Cognitive Accessibility with LLMs: A Multi-Task Approach to Easy-to-Read Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to make complex texts more accessible for people with cognitive impairments are time-consuming and resource-intensive. |
| Approach: | They propose a multi-task learning approach that trains models jointly on text summarization, text simplification, and ETR generation. |
| Outcome: | The proposed approach outperforms other approaches in in-domain settings while achieving better generalization in out-of-domain scenarios. |
D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing training-free adaptation frameworks for video large language models are limited by the scarcity of high-quality video-text data. |
| Approach: | They propose a training-free adaptation framework that incorporates dynamic compression and question decomposition to address perception bottleneck and token overload. |
| Outcome: | The proposed framework improves video understanding across various benchmarks and on long-video benchmarks. |
ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automatically generated radiology reports often receive high scores from existing evaluation metrics but fail to earn clinicians’ trust. |
| Approach: | They propose a meta-evaluation framework that uses criteria spanning discrimination, robustness, and monotonicity to evaluate existing metrics. |
| Outcome: | The proposed framework offers guidance for building more clinically reliable evaluation methods. |
MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation (2025.emnlp-main)
Copied to clipboard
Khai Le-Duc, Tuyen Tran, Bach Phan Tat, Nguyen Kim Hai Bui, Quan Dang Anh, Hung-Phong Tran, Thanh Thuy Nguyen, Ly Nguyen, Tuan Minh Phan, Thi Thu Phuong Tran, Chris Ngo, Khanh Xuan Nguyen, Thanh Nguyen-Tang
| Challenge: | Multilingual speech translation (ST) and machine translation (MT) in the medical domain enhances patient care by enabling efficient communication across language barriers. |
| Approach: | They present a large-scale ST dataset for the medical domain spanning all translation directions in Vietnamese, English, German, French, and Simplified/Traditional Chinese, together with the models. |
| Outcome: | The multi-language speech translation (ST) and machine translation (MT) in the medical domain is the largest medical MT dataset and the largest many-to-many multilingual ST among all domains. |
Beyond Checkmate: Exploring the Creative Choke Points for AI Generated Texts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on detecting LLM-generated text (AI text) has raised concerns about potential misuse . a new study examines the nuanced distinctions between human and AI texts . |
| Approach: | They analyze human-AI text differences across body, intro, conclusion segments . human texts exhibit greater stylistic variation across segments, they show . |
| Outcome: | The findings will inform their viability and boundaries as effective creative assistants to humans. |
MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Different retrievers offer distinct, often complementary signals, but they are not optimal for all queries. |
| Approach: | They propose a zero-shot, weighted combination of heterogeneous retrievers . they validate this intuition by incorporating specialized non-oracle human information sources . |
| Outcome: | Experiments show that a mixture of heterogeneous retrievers outperforms each retriever and larger models by +10.8% and +3.9% on average. |
Learning Contextual Retrieval for Robust Conversational Search (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective conversational search requires a deep understanding of user intent across multiple dialogue turns. |
| Approach: | They propose a novel LLM-based retriever that directly incorporates conversational context into the retrieval process. |
| Outcome: | The proposed method outperforms existing methods while incurring no additional inference overhead. |
LIDDIA: Language-based Intelligent Drug Discovery Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in artificial intelligence for chemistry have sought to expedite individual drug discovery tasks. |
| Approach: | They propose an autonomous agent capable of intelligently navigating the drug discovery process in silico. |
| Outcome: | The proposed agent can generate molecules meeting key pharmaceutical criteria on over 70% of 30 clinically relevant targets and intelligently balances exploration and exploitation in the chemical space. |
Agentic-R1: Distilled Dual-Strategy Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current long chain-of-thought models rely on slow and error-prone natural language traces. |
| Approach: | They propose a framework that distills complementary reasoning strategies from multiple teachers into a unified student model. |
| Outcome: | The proposed framework improves accuracy on computation-intensive tasks and reduces inference latency on standard benchmarks. |
Proactive Assistant Dialogue Generation from Streaming Egocentric Videos (2025.emnlp-main)
Copied to clipboard
Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, Seungwhan Moon
| Challenge: | Recent advances in conversational AI have been substantial, but developing real-time tasks guidance systems remains a challenge. |
| Approach: | They propose a data curation pipeline that synthesizes dialogues from annotated egocentric videos and a suite of automatic evaluation metrics that validated through extensive human studies. |
| Outcome: | The proposed framework synthesizes dialogues from annotated egocentric videos and validates them through extensive human studies. |
Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the impact of feedback on human decision-making are limited as people are not equipped to assess the quality of AI predictions. |
| Approach: | They compare the quality of MT inputs and outputs with explicit and implicit feedbacks that directly give users an assessment of translation quality using error highlights and LLM explanations. |
| Outcome: | The proposed model improves decision accuracy and appropriate reliance by using error highlights and explanations, and by using backtranslation and question–answer tables. |
ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement (2025.emnlp-main)
Copied to clipboard
Ali Salamatian, Amirhossein Abaskohi, Wan-Cyuan Fan, Mir Rayat Imtiaz Hossain, Leonid Sigal, Giuseppe Carenini
| Challenge: | Chart question answering (CQA) is a key research challenge for large vision-language models . recent efforts focus on leveraging LVLMs directly on chart images . |
| Approach: | They propose a gaze-guided attention refinement that aligns image-text attention with human fixations to improve chart reasoning quality and interpretability. |
| Outcome: | The proposed approach improves answer accuracy and attention alignment yielding gains of up to 2.56 percentage points across multiple models. |
LogiCoL: Logically-Informed Contrastive Learning for Set-based Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current dense retrievers struggle with queries with logical connectives, a use case that is often overlooked but important in downstream applications. |
| Approach: | They propose a logically-informed contrastive learning objective for dense retrievers that learns to respect the subset and mutually exclusive set relation between query results. |
| Outcome: | The proposed model improves retrieval performance and consistency on entity retrieval tasks. |
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing MCIT methods do not fully exploit the unique attribute of Large Multimodal Models and often gain performance at the expense of efficiency. |
| Approach: | They propose a multimodal continual instruction learning framework that exploits the ability of LMMs to learn mixed instruction datasets and prompts for each task. |
| Outcome: | The proposed framework achieves +14.26% performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation. |
Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods train small language models to learn long rationales in one iteration. |
| Approach: | They propose a method that uses a heuristic search to divide rationale into internal chunks . they propose CWT, which uses CWt to focus SLM on learning from only one chunk per iteration. |
| Outcome: | The proposed method can guide a large language model (LLM) in reasoning tasks. |
Can an Individual Manipulate the Collective Decisions of Multi-Agents? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that coordinated multi-agent systems exhibit enhanced decision-making and reasoning abilities through collaboration. |
| Approach: | They propose a framework that simulates agent interactions within a multi-agent system to generate adversarial samples and use them to manipulate the target agent in the target system. |
| Outcome: | The proposed framework generates adversarial samples that are used to manipulate the target agent in the target system, misleading the system’s decision-making process. |
Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore’s Low-Resource Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformed natural language processing, but their safety mechanisms remain under-explored in low-resource, multilingual settings. |
| Approach: | They propose a red-teaming approach to probe LLM vulnerabilities in Singapore's diverse linguistic context using a dataset and evaluation framework. |
| Outcome: | The proposed framework systematically probes LLM vulnerabilities in three real-world scenarios including Singlish, Chinese, Malay, and Tamil. |
Improving Clustering with Positive Pairs Generated from LLM-Driven Labels (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing unsupervised clustering methods lack label knowledge, resulting in suboptimal performance. |
| Approach: | They propose to use LLM-driven labels to generate positive pairs from embedded data and an embedder to obviate the need for negative pairs. |
| Outcome: | The proposed framework surpasses state-of-the-art benchmarks on a range of datasets and generates interpretable labels for improved understanding of clustering results. |
Gamma-Guard: Lightweight Residual Adapters for Robust Guardrails in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely deployed as zero-shot evaluators for answer grading, content moderation, and document ranking. |
| Approach: | They propose a system that trains LLMs with adapters to denoise embeddings and refocus attention. |
| Outcome: | The proposed model lifts adversarial accuracy from 5% to 95% a 90 percentage-point gain while reducing clean-data accuracy by just 8 percentage points. |
Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K to 2M tokens and even beyond. |
| Approach: | They propose a synthetic dataset in the financial domain that integrates Chain-of-Thought reasoning into LLMs in a supervised manner to facilitate effective long-context understanding. |
| Outcome: | The proposed model outperforms standard GPT-4o-mini on the Loong benchmark and fine tunes LLaMA-3.1-8B-Instruct on the model, achieving a 28.0% gain on the financial subset. |
Dynamic Energy-Based Contrastive Learning with Multi-Stage Knowledge Verification for Event Causality Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for event causal identification rely on rule-based or random sampling strategies, which introduce spurious causal positives. |
| Approach: | They propose an ECI method enhanced by Dynamic Energy-based Contrastive Learning with multi-stage knowledge verification which generates high-quality contrastive samples and effectively suppresses spurious causal disturbances. |
| Outcome: | The proposed method outperforms state-of-the-art methods on two benchmarks. |
ICG: Improving Cover Image Generation via MLLM-based Prompting and Personalized Preference Alignment (2025.emnlp-main)
Copied to clipboard
Zhipeng Bian, Jieming Zhu, Qijiong Liu, Wang Lin, Guohao Cai, Zhaocheng Du, Jiacheng Sun, Zhou Zhao, Zhenhua Dong
| Challenge: | Large language models and diffusion models have opened new possibilities for AI-generated content . personalized cover image generation remains underexplored despite its critical role in boosting user engagement on digital platforms. |
| Approach: | They propose a framework that integrates MLLM-based prompting with personalized preference alignment to generate high-quality, contextually relevant covers. |
| Outcome: | The proposed framework improves image quality, semantic fidelity, and personalization, leading to stronger user appeal and offline recommendation accuracy in downstream tasks. |
From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought reasoning introduces significant inference latency due to its verbosity. |
| Approach: | They propose a framework that leverages token elasticity phenomenon to progressively compress CoTs via multiround refinement. |
| Outcome: | The proposed method achieves an average accuracy improvement of 5.6% over state-of-the-art baselines while reducing CoT length by an average of 47 tokens and significantly lowering latency. |
A Symbolic Adversarial Learning Framework for Evolving Fake News Generation and Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Rapid LLM advancements heighten fake news risks by enabling the automatic generation of increasingly sophisticated misinformation. |
| Approach: | They propose a framework that implements an adversarial training paradigm by an agent symbolic learning optimization process rather than numerical updates. |
| Outcome: | The proposed framework generates sophisticated fake news that degrades state-of-the-art detection performance by 53.4% in Chinese and 34.2% in English on average. |
RareSyn: Health Record Synthesis for Rare Disease Diagnosis (2025.emnlp-main)
Copied to clipboard
| Challenge: | RareSyn is a data synthesis approach to augment and de-identify EHRs with a focus on rare diseases. |
| Approach: | They propose a data synthesis approach to augment and de-identify EHRs with a focus on rare diseases. |
| Outcome: | The proposed model augments and de-identifies EHRs with a focus on rare diseases. |
Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large reasoning models have exhibited strong performance on complex reasoning tasks, but current test-time scaling methods rely on redundant sampling and ignore historical experience utilization. |
| Approach: | They propose a test-time scaling framework that coordinates three collaborative LRMs to iteratively explore and refine solutions guided by historical attempts. |
| Outcome: | The proposed framework surpasses strong baselines on three mathematical reasoning benchmarks, including AIME-24, AIME-25, and OlymMATH. |
CMHG: A Dataset and Benchmark for Headline Generation of Minority Languages in China (2025.emnlp-main)
Copied to clipboard
| Challenge: | Minority languages in China face significant challenges due to their unique writing systems, which differ from international standards. |
| Approach: | They propose a dataset specifically curated for headline generation tasks for minority languages in China . they propose 50,000 entries each for Uyghur and Mongolian, and a test set annotated by native speakers . |
| Outcome: | The proposed dataset will help improve headline generation in minority languages . it includes 100,000 entries for Tibetan, 50,000 entries each for Uyghur and Mongolian . |
Understanding the Information Propagation Effects of Communication Topologies in LLM-based Multi-Agent Systems (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirical studies for communication topology design often overlook why and when sparse and dense topologies help or hinder collaboration. |
| Approach: | They propose a topology design approach that balances error suppression and beneficial information propagation by fusing connectivity patterns from dense and sparse graphs. |
| Outcome: | The proposed topology design achieves superior performance across tasks with sparse and dense graphs. |
Boosting Data Utilization for Multilingual Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on fine-tuning multilingual dense retrieval models, but data scarcity for low-resource languages makes it difficult to align representations in a shared vector space. |
| Approach: | They propose to obtain high-quality hard negative samples and effective mini-batch data to boost data utilization for multilingual dense retrieval by obtaining high-quality negative samples. |
| Outcome: | The proposed method outperforms existing baselines on a multilingual retrieval benchmark, MIRACL, with 16 languages. |
Self-Augmented Preference Alignment for Sycophancy Reduction in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sycophantic behavior in models can erode user trust by creating a perception of dishonesty or bias. |
| Approach: | They propose to assess the user’s expected answer rather than ignore it and introduce self-augmented preference alignment to reduce sycophancy. |
| Outcome: | The proposed methods significantly reduce sycophancy across tasks and improve models' assessment ability. |
TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models (LLMs) focus on basic plan validity, but neglect critical aspects such as route efficiency, POI appeal, and real-time adaptability. |
| Approach: | They propose a benchmark for retrieval-augmented, spatiotemporal-aware travel planning that integrates retrieved trajectories with LLMs’ intrinsic reasoning. |
| Outcome: | The proposed framework improves spatial efficiency and POI rationality while challenging universality and robustness due to conflicting references and noisy data. |
Recontextualizing Revitalization: A Mixed Media Approach to Reviving the Nüshu Language (2025.emnlp-main)
Copied to clipboard
| Challenge: | Nüshu is an endangered language from Jiangyong County, Hunan, China, and the world’s only known writing system created and used exclusively by women. |
| Approach: | They propose to use NüshuStrokes to record all 397 Unicode Nü Shu characters in sequential handwriting by an expert calligrapher. |
| Outcome: | Evaluating five state-of-the-art Chinese Optical Character Recognition systems on NüshuVision lowers CER to 0.67, a modest but meaningful improvement over previous datasets. |
Towards Advanced Mathematical Reasoning for LLMs via First-Order Logic Theorem Proving (2025.emnlp-main)
Copied to clipboard
Chuxue Cao, Mengze Li, Juntao Dai, Jinluan Yang, Zijian Zhao, Shengyu Zhang, Weijie Shi, Chengzhong Liu, Sirui Han, Yike Guo
| Challenge: | Large language models (LLMs) have shown promising first-order logic (FOL) reasoning capabilities with applications in various areas, but their effectiveness in complex mathematical reasoning involving multi-step FOL deductions remains under-explored. |
| Approach: | They propose a self-adaptive solution that enhances the Diversity and REAsonability of LLMs’ generation strategies by introducing an Axiom-Driven Strategy Diversification mechanism and a Sub-Proposition Error Feedback to help LLM reflect on and correct their proofs. |
| Outcome: | The proposed model improves diversity and REAsonability of LLMs’ generation strategies by introducing an Axiom-Driven Strategy Diversification mechanism and a Sub-Proposition Error Feedback to help LLM reflect on and correct proofs. |
From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Automatic Speech Recognition (ASR) have been fueled by massive speech corpora, but extending coverage to diverse languages with limited resources remains a formidable challenge. |
| Approach: | They propose a pipeline that converts large-scale text corpora into synthetic speech using off-the-shelf text-to-speech (TTS) models. |
| Outcome: | The proposed pipeline generates 500,000 hours of synthetic speech in ten languages and achieves transcription error reductions of over 30%. |
CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space (2025.emnlp-main)
Copied to clipboard
Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, Jincai Huang
| Challenge: | Embodied Question Answering (EQA) tasks are primarily focused on indoor environments, leaving the complexities of urban settings unexplored. |
| Approach: | They propose a task where an embodied agent answers open-vocabulary questions in dynamic city spaces. |
| Outcome: | The proposed agent achieves 60.7% of human-level answering accuracy compared to baselines . the proposed agent outperforms existing agents in open-ended city spaces . |
Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for reducing hallucinations incur a significant increase in latency. |
| Approach: | They propose a task-agnostic attention-guided head suppression strategy that can be seamlessly integrated during inference without incurring significant compute or latency overhead. |
| Outcome: | The proposed approach reduces hallucinations by 2.7x while maintaining F1 and improves throughput by 1.8% compared to existing methods. |
Examining False Positives under Inference Scaling for Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language models have led to significant improvements in mathematical reasoning across benchmarks. |
| Approach: | They analyze the prevalence of false positives in language models by using heuristic evaluation methods . they find that false positive models produce correct final answers but with flawed deduction paths . |
| Outcome: | The proposed model performance improvements are based on the proposed model and its evaluation metrics. |
Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese (2025.emnlp-main)
Copied to clipboard
Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu
| Challenge: | Translationese is a linguistic property that is often introduced in the translation process that is different from those of original texts. |
| Approach: | They propose to use synthesized translations and translations in the wild to evaluate T-index's generalizability in cross-domain settings and its validity against human judgments. |
| Outcome: | The proposed measure can generalize to unseen genres, authors, and language pairs. |
Exploring the Limitations of Mamba in COPY and CoT Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Inference overhead of Transformers increases linearly with the sequence length, posing challenges for modeling long sequences. |
| Approach: | They analyze Mamba's expressive ability to perform COPY operations and Chain of Thought reasoning tasks using a defined sequence length. |
| Outcome: | The proposed model can perform COPY operations and Chain of Thought reasoning tasks with a constant size while reducing computational costs. |
ProcWorld: Benchmarking Large Model Planning in Reachability-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for embodied spatial reasoning and long-term planning are non-trivial due to the combinatorial complexity of long-horizon abstract reasoning. |
| Approach: | They propose a large-scale benchmark for partially observable embodied spatial reasoning and long-term planning with large language models and vision language models. |
| Outcome: | The proposed model performs better in 16 task types, 5,000 rooms, and over 10 million evaluation trajectories with diverse data distribution. |
R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reasoning is a fundamental capability underpinning text-to-image (T2I) generation. |
| Approach: | They propose a benchmark to rigorously assess reasoning-driven T2I generation. |
| Outcome: | Experiments with 16 representative T2I models show limited reasoning performance . a strong pipeline-based framework decouples reasoning and generation . |
Can GRPO Boost Complex Multimodal Table Understanding? (2025.emnlp-main)
Copied to clipboard
Xiaoqiang Kang, Shengen Wu, Zimu Wang, Yilin Liu, Xiaobo Jin, Kaizhu Huang, Wei Wang, Yutao Yue, Xiaowei Huang, Qiufeng Wang
| Challenge: | Existing table understanding methods struggle with low initialization accuracy and coarse rewards in tabular contexts. |
| Approach: | They propose a three-stage RL framework that enhances multimodal table understanding through: (1) Warm-up that prompts initial perception and reasoning capabilities; (2) Perception Alignment GRPO (PA-GRPO); (3) Hint-Completion GR PO (HC-GRP); |
| Outcome: | The proposed framework outperforms existing models on held-in and held-out datasets, outperforming SFT and GRPO largely. |
MoMoE: Mixture of Moderation Experts Framework for AI-Assisted Online Governance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches for content moderation require a separate model for every community and are opaque in their decision-making. |
| Approach: | They propose a modular framework that adds post-hoc explanations to enable scalable content moderation. |
| Outcome: | The proposed framework yields scalable, transparent moderation without fine-tuning across domains. |
Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment (2025.emnlp-main)
Copied to clipboard
Jingcheng Deng, Zhongtao Jiang, Liang Pang, Zihao Wei, Liwei Chen, Kun Xu, Yang Song, Huawei Shen, Xueqi Cheng
| Challenge: | Experimental results demonstrate that our method significantly outperforms traditional contrastive learning approaches when using the same amount of data. |
| Approach: | They propose a new contrastive learning method built on embedding conditional probability distributions that integrates two tasks: information compression and conditional distribution alignment. |
| Outcome: | The proposed method outperforms traditional contrastive learning approaches and achieves comparable performance to state-of-the-art models when using the same amount of data. |
Evaluating LLM-Generated Diagrams as Graphs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods on diagram generation with LLMs rely heavily on proprietary LLM systems. |
| Approach: | They propose a new evaluation metric to assess demonstration diagrams generated by large language models. |
| Outcome: | The proposed evaluation metric evaluates diagrams produced by state-of-the-art LLMs on recent research literature. |
Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are ubiquitous in user-facing applications, yet they still generate undesirable toxic outputs, including profanity, vulgarity, and derogatory remarks. |
| Approach: | They leverage sparse autoencoders to identify toxicity-related directions in residual stream of large language models and perform targeted activation steering using the corresponding decoder vectors. |
| Outcome: | The proposed models surpass baselines in reducing toxicity by up to 20%, though fluency can degrade noticeably on GPT-2 Small and Gemma-2-2B. |
VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have improved the performance of Large language models on well-defined mathematical benchmarks, but they often overlook ill-defined problems. |
| Approach: | They develop a large-scale benchmark that contains over 5,000 ill-defined mathematical problems. |
| Outcome: | The proposed framework improves the accuracy of identifying unsolvable problems by at least 12% across different LLMs, thus achieving stronger robust mathematical reasoning ability. |
How do autoregressive transformers solve full addition? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated impressive capabilities, but there is still much to learn about how they operate. |
| Approach: | They investigate the ability of the autoregressive transformer to perform basic addition operations by using causal analysis to find that a few different attention heads in the middle layers control the addition carry . they found that due to the lack of global focus on the sequence within these attention heads, the model struggles to handle long-sequence addition tasks. |
| Outcome: | The model performs basic addition tasks, but it still faces challenges with length generalization. |
MAIN: Mutual Alignment Is Necessary for instruction tuning (2025.emnlp-main)
Copied to clipboard
Fanyi Yang, Jianfeng Liu, Xin Zhang, Haoyu Liu, Xixin Cao, Yuefeng Zhan, Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Instruction tuning has enabled large language models to achieve remarkable performance, yet its success heavily depends on the availability of high-quality instruction-response pairs. |
| Approach: | They propose a mutual alignment framework which enforces coherence between instructions and responses through mutual constraints. |
| Outcome: | The proposed framework generalizes well across model architectures and sizes, achieving state-of-the-art performance on LLaMA, Mistral, and Qwen models across diverse benchmarks. |
Expanding before Inferring: Enhancing Factuality in Large Language Models through Premature Layers Interpolation (2025.emnlp-main)
Copied to clipboard
Dingwei Chen, Ziqiang Liu, Feiteng Fang, Chak Tou Leong, Shiwen Ni, Ahmadreza Argha, Hamid Alinejad-Rokny, Min Yang, Chengming Li
| Challenge: | Existing approaches to generating factually inconsistent outputs are resource-intensive. |
| Approach: | They propose a plug-and-play intervention designed to enhance factuality by inserting premature layers formed through mathematical interpolation with adjacent layers. |
| Outcome: | The proposed intervention reduces hallucinations while outperforming baselines on four datasets. |
DeepWell-Adol: A Scalable Expert-Based Dialogue Corpus for Adolescent Positive Mental Health and Wellbeing Promotion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Promoting positive mental health and well-being is a critical yet underexplored area in natural language processing. |
| Approach: | They propose a Chinese dialogue corpus grounded in positive psychology and coaching that integrates human expert-written seed data with automated data augmentation to ensure high quality and scalability. |
| Outcome: | The proposed corpus meets general standards for psychological dialogue and emotional support while also showing superior performance across multiple models in promoting positive psychological processes, character strengths, interpersonal relationships, and healthy behaviors. |
Data to Defense: The Role of Curation in Aligning Large Language Models Against Safety Compromise (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have identified a vulnerability in large language models (LLMs) during customization. |
| Approach: | They propose an adaptive data curation approach that allows any text to be curated to enhance its effectiveness in counteracting harmful samples during customization. |
| Outcome: | The proposed approach reduces compromising effects and generates 100% safe responses. |
Speculative Safety-Aware Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Speculative Safety-Aware Decoding (SSD) equips large language models with desired safety property while accelerating inference. |
| Approach: | They propose a lightweight decoding-time approach that equips large models with the desired safety property while accelerating inference. |
| Outcome: | Experimental results show that a small language model has the desired safety property while accelerating inference. |
PanicToCalm: A Proactive Counseling Agent for Panic Attacks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for training such models are limited due to ethical and logistical issues. |
| Approach: | They propose a dataset that includes high-distress episodes constructed from first-person narratives and structured around the principles of Psychological First Aid. |
| Outcome: | The proposed model outperforms baseline models in counselor-side metrics and client affect improvement. |
CoPL: Collaborative Preference Learning for Personalizing LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for personalizing large language models struggle with flexibility and generalization. |
| Approach: | They propose a graph-based collaborative filtering framework that models user-response relationships to enhance preference estimation in sparse annotation settings. |
| Outcome: | The proposed framework outperforms existing reward models in TL;DR, UltraFeedback-P, and PersonalLLM datasets. |
Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to enhance reasoning capabilities of language models are expensive and often lack the ability to perform complex reasoning tasks. |
| Approach: | They propose a token-level multi-model collaboration strategy to enhance reasoning capabilities in language models by selecting the optimal tokens from the next token distributions. |
| Outcome: | The proposed method is superior to existing methods and will be released soon. |
AI Chatbots as Professional Service Agents: Developing a Professional Identity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies often overlook the act of communicating consistent with professional identities of LLM-based AI chatbots. |
| Approach: | They propose a framework for designing professional service agents for medical question-and-answer services that aligns professional identities with a theory-guided task planning process. |
| Outcome: | The proposed approach outperforms baseline methods on various LLMs across key metrics such as fluency, naturalness, empathy, patient-centricity, and ROUGE-L scores. |
DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in music large language models have significantly improved music understanding tasks, but the potential of incorporating additional modalities such as images, videos and textual music features remains unexplored. |
| Approach: | They propose a multimodal music understanding LLM fine-tuned via multi-way instruction tuning with multi-ways aligned music, text, image, and video data. |
| Outcome: | The proposed model achieves state-of-the-art performance across six music understanding tasks and zero-shot scenarios. |
Advancing Oversight Reasoning across Languages for Audit Sycophantic Behaviour via X-Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have demonstrated capabilities that are satisfactory to a wide range of users by adapting to their culture and wisdom. |
| Approach: | They propose an Oversight Reasoning framework that audits human–LLM dialogues, reasons about them, captures sycophancy and corrects the final outputs. |
| Outcome: | The proposed framework detects sycophancy, reduces unwarranted agreement and improves cross-turn consistency across different scenarios and languages. |
CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extend context length of Large Language Models (LLMs) still struggle with retrieval and reasoning in long context inputs. |
| Approach: | They propose a coarse-to-fine method to enhance multi-document question-answering capacities by removing background and distracting documents. |
| Outcome: | Experiments show that CAFE outperforms baseline methods on multiple documents. |
SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? (2025.emnlp-main)
Copied to clipboard
Senyu Li, Jiayi Wang, Felermino D. M. A. Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, David Ifeoluwa Adelani
| Challenge: | Existing metrics for machine translation quality for under-resourced African languages suffer from limited language coverage and poor performance in low-resource settings. |
| Approach: | They propose a large-scale human-annotated machine translation evaluation dataset . they use a reference-based and reference-free evaluation model to compare MT quality . |
| Outcome: | The proposed models outperform AfriCOMET and the strongest LLM on low-resource languages. |
FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for unlearning undesirable knowledge have overlooked complexity and interconnectedness of knowledge, authors say . previous studies have neglected the complex nature of knowledge and neglected its internal dependencies. |
| Approach: | They propose a new concept called superficial unlearning to evaluate faithfulness of unlearning in knowledge QA settings. |
| Outcome: | The proposed method shows significant effectiveness in real-world knowledge QA settings. |
Calibrating Pseudo-Labeling with Class Distribution for Semi-supervised Text Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies develop effective pseudo-labeling methods, but they struggle with unlabeled data that have imbalanced classes mismatched with the labeled data. |
| Approach: | They propose to use pseudo-labeling to train text classification models with few labeled data and massive unlabeled data. |
| Outcome: | Empirical results show that the proposed model outperforms state-of-the-art methods on 3 common benchmarks. |
Coarse-to-Fine Grounded Memory for LLM Agent Planning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to enhance LLM with offline experiences or online trajectory analysis focus on single-granularity memory derived from dynamic environmental interactions. |
| Approach: | They propose a framework that grounds coarse-to-fine memories with LLM to enable flexible adaptation to diverse scenarios. |
| Outcome: | Extensive experiments on AlfWorld, Webshop and ScienceWorld show that the proposed framework outperforms baselines and comprehensively optimizes memory-enhanced LLM Agent system. |
From A and B to A+B: Can Large Language Models Solve Compositional Math Problems? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies that create problem variants by adding perturbations to a single problem focus on the interaction between problems. |
| Approach: | They propose a pipeline with 98.2% accuracy to combine two original problems with a logical connection and to evaluate LLMs' generalization ability on the compositional problems. |
| Outcome: | The proposed pipeline can combine two original problems with a logical connection to get a new math problem and evaluate its compositional generalization on the compositional problems. |
Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories (2025.emnlp-main)
Copied to clipboard
Mohammad Beigi, Ying Shen, Parshin Shojaee, Qifan Wang, Zichao Wang, Chandan K. Reddy, Ming Jin, Lifu Huang
| Challenge: | Existing studies show that large language models inadvertently foster sycophancy . scophancies are a tendency of models to blindly conform to user preferences without critical reasoning or self-reflection. |
| Approach: | They propose a method to reduce sycophancy by combining uncertainty-aware Monte Carlo tree search and progress-based reinforcement learning. |
| Outcome: | The proposed model outperforms baseline models in effectively reducing sycophancy while maintaining performance on out-of-distribution inputs. |
SimVBG: Simulating Individual Values by Backstory Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have strong human-like capabilities, but rarely simulating individualized human values. |
| Approach: | They propose a framework that simulates individual values based on individual backstories . they use structured data on an individual to transform their backstoried information to a backstory . |
| Outcome: | The proposed framework improves top-1 accuracy by more than 10% over retrieval-augmented generation methods. |
EvolveSearch: An Iterative Self-Evolving Search Agent (2025.emnlp-main)
Copied to clipboard
Ding-Chu Zhang, Yida Zhao, Jialong Wu, Liwen Zhang, Baixuan Li, Wenbiao Yin, Yong Jiang, Yu-Feng Li, Kewei Tu, Pengjun Xie, Fei Huang
| Challenge: | Existing approaches to enabling LLM web search proficiency struggle with data production in open-search domains, while supervised fine-tuning struggles with data utilization efficiency. |
| Approach: | They propose an iterative self-evolution framework that combines SFT and RL to enhance agentic web search capabilities without external human-annotated reasoning data. |
| Outcome: | EvolveSearch achieves 4.7% improvement over current state-of-the-art in seven benchmarks . supervised fine-tuning struggles with data production in open-search domains compared with RL . |
Syntax-Aware Retrieval Augmentation for Neural Symbolic Regression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Symbolic regression is a powerful technique for discovering mathematical expressions that best fit observed data. |
| Approach: | They propose a syntax-aware retrieval-augmented mechanism that leverages syntactic structure of symbolic expressions to perform context-awful retrieval from a pre-constructed token datastore. |
| Outcome: | The proposed method outperforms representative baselines on symbolic regression benchmarks and is validated on multiple symbolic regression datasets. |
Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have enhanced their versatility as they integrate a growing number of modalities. |
| Approach: | They propose a simple MCL paradigm that addresses forgetting and misalignment . they propose 'MErge then ReAlign' to extend existing models to more modalities . |
| Outcome: | The proposed paradigm is easy to deploy and highly reusable in the MLLM community. |
Graceful Forgetting in Generative Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained models do not provide all knowledge needed for fine-tuning tasks. |
| Approach: | They propose a framework to achieve graceful forgetting in generative language models by pre-training a model on large-scale correlating datasets. |
| Outcome: | The proposed framework improves the learning plasticity of the target task by selectively discarding irrelevant knowledge. |
Answering Narrative-Driven Recommendation Queries via a Retrieve–Rank Paradigm and the OCG-Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate narrative-driven recommendation are based on large language models (LLMs) but the RAG paradigm is inherently ill-suited for such special queries. |
| Approach: | They propose a novel retrieve-rank paradigm that generatively retrieves structurally adaptive and semantically aligned candidates, ensuring both extensive candidate coverage and high-quality information. |
| Outcome: | The proposed paradigm outperforms the existing paradigm and the existing one under real-world scenarios. |
Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing offline preference optimization methods rely on preference labels to optimize large language models. |
| Approach: | They propose an offline method for enhancing large language models in reasoning tasks that utilizes value signals at individual reasoning steps. |
| Outcome: | The proposed framework outperforms offline preference optimization techniques by 4% to 6% on math reasoning, commonsense reasoning, and coding tasks. |
Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility (2025.emnlp-main)
Copied to clipboard
Brendan Murphy, Dillon Bowen, Shahrad Mohammadzadeh, Tom Tseng, Julius Broomfield, Adam Gleave, Kellin Pelrine
| Challenge: | a recent study shows that fine-tuning can produce helpful-only models with safeguards destroyed. |
| Approach: | They propose a method for fine-tuning models to generate detailed, high-quality responses to harmful requests. |
| Outcome: | The proposed method produces helpful-only models with safeguards destroyed . OpenAI, Google, and Anthropic models will fully comply with requests for CBRN assistance . |
Neural Topic Modeling via Contextual and Graph Information Fusion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic models generate uninformative and incoherent topics that hinder interpretable insights from managing textual data. |
| Approach: | They propose to incorporate contextual and graph information to improve the variational autoencoder framework by combining contextual and bag-of-words information. |
| Outcome: | The proposed framework generates more coherent and diverse topics on three benchmark datasets and achieves strong performance on automatic and manual evaluations. |
CARE: A Disagreement Detection Framework with Concept Alignment and Reasoning Enhancement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to disagreement detection are limited by conceptual gap and reasoning gap. |
| Approach: | They propose a conceptual alignment and reasoning enhancement framework to address the conceptual gap and the reasoning gap in disagreement detection. |
| Outcome: | The proposed framework shows superior performance in zero-shot and supervised learning settings, both within and across domains. |
Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current efforts to bridge the two modes of interaction are reactive, focusing on responding to user inputs rather than coordinating dialogue flows. |
| Approach: | They propose a dataset designed for transition-aware dialogue modeling that incorporates structurally diverse and integrated mode flows. |
| Outcome: | The proposed dataset outperforms baseline models in intent detection and mode transition handling. |
LightThinker: Thinking Step-by-Step Compression (2025.emnlp-main)
Copied to clipboard
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in Large Language Models have demonstrated their remarkable capabilities in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. |
| Approach: | They propose a method that dynamically compresses verbose thought steps into compact representations and discards original reasoning chains. |
| Outcome: | The proposed method reduces peak memory usage and inference time while maintaining competitive accuracy. |
How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has not explored the mechanisms underlying this sensitivity. |
| Approach: | They propose a synthetic benchmark to evaluate Large Language Models’ reasoning robustness against systematically controlled irrelevant context (IC). |
| Outcome: | The proposed model improves in-distribution and out-of-disttribution scenarios while training with strong distractors. |
Investigating Pedagogical Teacher and Student LLM Agents: Genetic Adaptation Meets Retrieval-Augmented Generation Across Learning Styles (2025.emnlp-main)
Copied to clipboard
Debdeep Sanyal, Agniva Maiti, Umakanta Maharana, Dhruv Kumar, Ankur Mali, C. Lee Giles, Murari Mandal
| Challenge: | Existing models for large language models neglect comprehensive student modeling beyond basic knowledge states and lack mechanisms for teachers to dynamically adapt their approach based on student feedback and collective performance. |
| Approach: | They propose a framework that integrates LLM-based diverse student agents with a self-evolving teacher agent to optimize teacher's pedagogical parameters based on simulated student performance. |
| Outcome: | The proposed framework integrates diverse student agents with a self-evolving teacher agent to optimize teacher pedagogical parameters based on simulated student performance. |
GeoEdit: Geometric Knowledge Editing for Large Language Models (2025.emnlp-main)
Copied to clipboard
Yujie Feng, Li-Ming Zhan, Zexin Lu, Yongxin Xu, Xu Chu, Yasha Wang, Jiannong Cao, Philip S. Yu, Xiao-Ming Wu
| Challenge: | Existing training-based model editing methods struggle to incorporate new knowledge while preserving unrelated general knowledge. |
| Approach: | They propose a framework that uses geometric relationships to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. |
| Outcome: | The proposed framework avoids updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model’s generalization ability. |
A Generative Pre-Trained Language Model for Channel Prediction in Wireless Communications Systems (2025.emnlp-main)
Copied to clipboard
Bo Lin, Huanming Zhang, Yuhua Jiang, Yucong Wang, Tengyu Zhang, Shaoqiang Yan, Hongyao Li, Yihong Liu, Feifei Gao
| Challenge: | Existing model-based channel prediction methods suffer from limited accuracy due to imperfect temporal modeling, while existing AI-based methods suffers from limited generalization due to inadequate training strategies. |
| Approach: | They propose a generative pre-trained language model for channel prediction based on channel correlation and train it based upon transformer decoder architecture. |
| Outcome: | The proposed model can learn various channel characteristics and perform impressive tasks across multiple dimensions. |
AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learning (2025.emnlp-main)
Copied to clipboard
Yujie Feng, Jian Li, Xiaoyu Dong, Pengfei Xu, Xiaohui Zhou, Yujia Zhang, Zexin Lu, Yasha Wang, Alan Zhao, Xu Chu, Xiao-Ming Wu
| Challenge: | Recent model merging-based methods struggle to effectively manage the trade-off between learning new knowledge and preventing catastrophic forgetting. |
| Approach: | They propose a model merging framework that utilizes learning and forgetting signals from the training trajectory to dynamically monitor the model’s training status. |
| Outcome: | The proposed framework achieves significant performance improvements over existing state-of-the-art methods on three CL benchmarks with various model sizes (from 770M to 13B). |
R-PRM: Reasoning-Driven Process Reward Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Process Reward Models (PRMs) output evaluation scores directly, limiting both learning efficiency and evaluation accuracy. |
| Approach: | They propose a Reasoning-Driven Process Reward Modeling (R-PRM) which activates inherent reasoning to enhance process-level evaluation. |
| Outcome: | The proposed model outperforms baseline models on ProcessBench and PRMBench by 13.9 and 8.5 F1 scores. |
RLAE: Reinforcement Learning-Assisted Ensemble for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing ensemble methods for ensembling large language models rely on fixed weighting strategies that fail to adapt to dynamic, context-dependent characteristics of LLMs. |
| Approach: | They propose a framework that reformulates LLM ensemble through a Markov Decision Process. |
| Outcome: | The proposed framework outperforms existing methods by 3.3% on a diverse set of tasks while achieving lower time latency. |
Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel at complex math but fail on basic addition, raising the question of whether they grasp rules or are merely reproducing patterns. |
| Approach: | They systematically probe LLMs’ understanding of two-integer addition by testing three crucial properties: commutativity (A+B=B+A), representation invariance via symbolic remapping and consistent accuracy scaling with operand length. |
| Outcome: | The proposed models achieve high numeric accuracy but fail basic addition tasks. |
AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarification (2025.emnlp-main)
Copied to clipboard
Xuan Zhang, Yongliang Shen, Zhe Zheng, Linjuan Wu, Wenqi Zhang, Yuchen Yan, Qiuying Peng, Jun Wang, Weiming Lu
| Challenge: | Existing tools for ambiguous and incomplete queries are limited by manual construction and lack of error correction mechanisms during multi-turn clarification. |
| Approach: | They propose a framework that exploits the mapping between queries and their tool invocation solutions by removing key parameters from queries while retaining them as ground truth. |
| Outcome: | The proposed framework outperforms existing methods while maintaining high accuracy in tool invocation. |
START: Self-taught Reasoner with Tools (2025.emnlp-main)
Copied to clipboard
Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Bowen Yu, Binyuan Hui, Junyang Lin, Xiang Wang, Dayiheng Liu
| Challenge: | Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in complex reasoning through long chain-of-thought, yet they struggle with precise computations and algorithmic operations. |
| Approach: | They propose a training-free approach that activates LRMs’ latent tool-use capabilities through artificial hints and a framework that enables models to learn effective tool utilization through diverse hint patterns and rejection-based data synthesis. |
| Outcome: | Experiments show that START significantly improves state-of-the-art LRMs across challenging benchmarks, including competition-level mathematics (AMC23: 95.0%, AIME24: 75.6%) and graduate-level science questions (GPQA: 64.6%). |
The Impact of Negated Text on Hallucination with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on hallucination in large language models (LLMs) have been actively progressing in natural language processing. |
| Approach: | They propose to examine whether LLMs can recognize contextual shifts caused by negation and still reliably distinguish hallucinations comparable to affirmative cases. |
| Outcome: | The proposed model can detect hallucinations comparable to affirmative cases, but it is difficult to detect them in negated text, the authors show . |
A Probabilistic Inference Scaling Theory for LLM Self-Correction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated the capability to refine their generated answers through self-correction, enabling continuous performance improvement over multiple rounds. |
| Approach: | They propose a probabilistic theory to model the dynamics of accuracy change and explain performance improvements observed in multi-round self-correction. |
| Outcome: | The proposed model can predict accuracy curves and improve accuracy over multiple rounds. |
MentalGLM Series: Explainable Large Language Models for Mental Health Analysis on Chinese Social Media (2025.emnlp-main)
Copied to clipboard
Wei Zhai, Nan Bai, Qing Zhao, Jianqiang Li, Fan Wang, Hongzhi Qi, Meng Jiang, Xiaoqin Wang, Bing Xiang Yang, Guanghui Fu
| Challenge: | Social media is a key platform for emotional expression, yet deep learning lacks flexibility and interpretability. |
| Approach: | They propose to use Chinese social media to train interpretable mental health instruction datasets to test models' ability to explain their decisions. |
| Outcome: | The proposed models outperform deep learning and LLMs on three mental health downstream tasks and demonstrate their potential for clinical applications. |
Knowledge-Aware Co-Reasoning for Multidisciplinary Collaboration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multi-agent paradigms rely on prompt engineering and lack of knowledge integration. |
| Approach: | They propose a framework that integrates structured knowledge reasoning into multidisciplinary collaboration by using clinical knowledge graphs to guide dynamic discipline determination. |
| Outcome: | Extensive experiments on academic and real-world datasets demonstrate the effectiveness of the proposed framework. |
Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language-action models rely on causal attention for processing sequences composed of interleaved segments from different modalities. |
| Approach: | They propose a Transformer architecture featuring trajectory attention and learnable action queries that efficiently process segmented multimodal trajectories and predict actions for imitation learning. |
| Outcome: | The proposed architecture performs better on three large-scale robot manipulation benchmarks than previous models. |
MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that multimodal, multimodal approaches to lyrics translation are more effective than text-only approaches. |
| Approach: | They propose a multilingual, multimodal benchmark for singable lyrics translation . they propose syllable-constrained audio-video LLM with Chain-of-Thought . |
| Outcome: | The proposed system outperforms text-based models in singability and contextual accuracy. |
MuTIS: Enhancing Reasoning Efficiency through Multi Turn Intervention Sampling in Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for training large reasoning models with long chain-of-thought (CoT) are limited by the number of parameters and the complexity of the model. |
| Approach: | They propose a framework that leverages multi-turn interventions to produce concise reasoning chains and demonstrates strong scalability. |
| Outcome: | The proposed framework breaks the accuracy-efficiency trade-off by producing concise reasoning chains and demonstrating strong scalability on 7B models. |
PRIM: Towards Practical In-Image Multilingual Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current research on in-image machine translation focuses on synthetic data with simple background, single font, fixed text position, and bilingual translation. |
| Approach: | They propose an end-to-end model to handle the challenge of practical conditions in PRIM . they annotate a real-world one-line text image with complex background, fonts, diverse text positions . |
| Outcome: | The proposed model improves translation quality and visual effect compared to other models. |
Mind the Inclusivity Gap: Multilingual Gender-Neutral Translation Evaluation with mGeNTE (2025.emnlp-main)
Copied to clipboard
Beatrice Savoldi, Giuseppe Attanasio, Eleonora Cupin, Eleni Gkovedarou, Janiça Hackenbuchner, Anne Lauscher, Matteo Negri, Andrea Piergentili, Manjinder Thind, Luisa Bentivogli
| Challenge: | Genderneutral translation (GNT) is a linguistic strategy towards fairer communication across languages. |
| Approach: | They propose to use a multilingual evaluation resource to evaluate inclusive translation with state-of-the-art instruction-following language models (LMs) |
| Outcome: | The proposed model can recognize when neutrality is appropriate, but cannot consistently produce neutral translations, limiting their usability. |
DiplomacyAgent: Do LLMs Balance Interests and Ethical Principles in International Events? (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines the safety implications of large language models in diplomatic positions . it identifies potential risks and ideological biases that could arise from LLMs . |
| Approach: | They propose an LLM-based multi-agent system for diplomatic position analysis . they propose ethical constraint measures to enhance the safety of LLMs . |
| Outcome: | The proposed system assesses the safety implications of large language models in diplomacy . it reveals that LLMs could exhibit a strong bias towards interests, leading to unsafe decisions . |
DisLoRA: Task-specific Low-Rank Adaptation via Orthogonal Basis from Singular Value Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks demonstrate that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |
| Approach: | They propose a framework that leverages singular value decomposition to decompose pretrained weight matrices into orthogonal backbone and task-specific subspaces. |
| Outcome: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks show that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |
Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Misleading visualizations can distort perception and lead to incorrect conclusions. |
| Approach: | They propose a large-scale multimodal dataset to evaluate MLLMs on misleading chart reasoning. |
| Outcome: | The proposed framework evaluates MLLMs on misleading chart reasoning on a large-scale multimodal dataset spanning 21 misleader types and 10 chart types . it contains 3,026 curated examples spanning standard chart code, CSV data, multiple-choice questions, and labeled explanations, validated through iterative MLML checks and exhausted expert human review. |
Textual Aesthetics in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image aesthetics have focused on content correctness and helpfulness of responses. |
| Approach: | They propose a textual aesthetics-powered fine-tuning method that leverages textual visual aesthetics without compromising content correctness. |
| Outcome: | The proposed method improves aesthetic scores and performs well on general evaluation datasets. |
Section-Level Simplification of Biomedical Abstracts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Cochrane reviews do not adhere to the same structure as plain language summaries, which prevents simplification models from being trained on paired abstracts and PLS. |
| Approach: | They propose a method to automatically divide Cochrane reviews' plain language summaries into the same sections in which abstracts are divided. |
| Outcome: | The proposed method yields references for section-level simplification of biomedical abstracts. |
PoseStitch-SLT: Linguistically Inspired Pose-Stitching for End-to-End Sign Language Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sign language translation remains a challenging task due to the scarcity of large-scale, sentence-aligned datasets. |
| Approach: | They propose a pose-based pre-training scheme that is inspired by a linguistic-templates-based sentence generation technique. |
| Outcome: | The proposed pre-training scheme outperforms state-of-the-art methods for pose-based gloss-free translation on two sign language datasets. |
Few-Shot Open-Set Classification via Reasoning-Aware Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models excel at few-shot learning, but their ability to reject out-of-distribution examples remains under-explored. |
| Approach: | They introduce a novel amortised Generative Flow Network framework that uses latent trajectories to approximate the Bayesian posterior. |
| Outcome: | The proposed framework can generalise with as few as 4 examples per class, enabling Llama 3.2 3B to achieve up to 80% of the performance of Llma 3.3 70B in complex datasets. |
Translation in the Hands of Many: Centering Lay Users in Machine Translation Interactions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual demands and accessibility have made MT a global tool . however, the understanding of MT consumed by such a diverse group of users remains limited. |
| Approach: | They first trace the evolution of MT user profiles, focusing on non-experts and how their engagement with technology may shift with the rise of LLMs. |
| Outcome: | The proposed approach will help to align MT with user needs and improve the quality of the language. |
iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Use (2025.emnlp-main)
Copied to clipboard
Yirong Zeng, Xiao Ding, Yuxian Wang, Weiwen Liu, Yutai Hou, Wu Ning, Xu Huang, Duyu Tang, Dandan Tu, Bing Qin, Ting Liu
| Challenge: | Synthesizing tool-use data through real-world simulations is effective for enhancing large language models (LLMs) however, training gains decay as synthetic data increases, and the model struggles to benefit from more synthetic data. |
| Approach: | They propose an iterative reinforced fine-tuning strategy to improve LLMs with external tools to augment their capabilities. |
| Outcome: | The proposed method achieves 13.11% better performance than the same-size base model and outperforms larger open-source and closed-source models. |
Transplant Then Regenerate: A New Paradigm for Text Data Augmentation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Data augmentation is a critical technique in deep learning. |
| Approach: | They propose a novel text augmentation paradigm leveraging large language models . they incorporate seed text into a context expanded by LLM and ask it to regenerate a variant based on the expanded context. |
| Outcome: | The proposed model generates high-quality and diverse augmented text with a transplant-then-regenerate approach. |
Compositional Generalisation for Explainable Hate Speech Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Hate speech detection models struggle to generalise beyond their training data . dataset biases and the use of sentence-level labels fail to teach the underlying structure of hate speech. |
| Approach: | They propose to use a dataset to train models with fine-grained, span-level annotations . they find that combinations of expressions that deviate from those seen during training are difficult to detect . |
| Outcome: | The proposed model can generalise to a dataset with equal frequency across all contexts while achieving state-of-the-art performance on the human-sourced PLEAD. |
CCQA: Generating Question from Solution Can Improve Inference-Time Reasoning in SLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have yielded remarkable performance across a wide range of tasks, including machine translation, code generation, sentiment analysis, and reasoning. |
| Approach: | They propose a new reasoning method that generates a question from each reasoning path and answer, evaluates each by its similarity to the original question, and selects the candidate solution with the highest similarity score as the final answer. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods on mathematical and commonsense reasoning benchmarks and establishes a new practical baseline for efficient reasoning in SLMs. |
TVQACML: Benchmarking Text-Centric Visual Question Answering in Multilingual Chinese Minority Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TEC-VQA benchmarks focus on high-resource languages like English and Chinese . existing benchmarks have a "visual-textual misalignment" problem resulting in unreliable evaluation results . |
| Approach: | They propose a benchmark that expands multilingual QA pairs in non-text-centric datasets through translation to eight languages, including Standard Chinese, Korean, and six minority languages. |
| Outcome: | The proposed benchmarks are contamination-free and more challenging . they include eight languages including Chinese, Korean, and six minority languages . |
Transparent and Coherent Procedural Mistake Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Procedural mistake detection (PMD) is a problem of classifying whether a human user has successfully executed a task. |
| Approach: | They extend PMD to require generating visual self-dialog rationales to inform decisions . they leverage a natural language inference model to formulate two automated metrics for coherence of generated rationale. |
| Outcome: | The proposed model improves on a reframed task with a natural language inference model and a multi-faceted metrics visualization of common outcomes. |
Teaching Your Models to Understand Code via Focal Preference Alignment (2025.emnlp-main)
Copied to clipboard
Jie Wu, Haoling Li, Xin Zhang, Xiao Liu, Yangyu Huang, Jianwen Luo, Yizhen Zhang, Zuchao Li, Ruihang Chu, Yujiu Yang, Scarlett Li
| Challenge: | Existing methods for supervised fine-tuning focus on unit test feedback to construct preference pairs. |
| Approach: | They propose a preference alignment framework that mimics human iterative debugging to refine Code LLMs. |
| Outcome: | Experiments show that Preference Learning improves on BigCodeBench and BigCodeBind tasks. |
MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Document Understanding is a foundational AI capability with broad applications . Large Vision-Language Models (LLMs) can't handle multi-page document comprehension . a logic-aware retrieval framework for multi-modal, multi- page document understanding is proposed . |
| Approach: | They propose a logic-aware retrieval framework for multi-modal, multi-page document understanding . MoLoRAG uses semantic and logical relevance to deliver more accurate retrieval . |
| Outcome: | The proposed framework improves on four DocQA datasets and demonstrates 9.68% accuracy improvement over existing methods. |
Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastively trained Vision-Language Models exhibit shallow language understanding, manifesting bag-of-words behaviour. |
| Approach: | They propose a vision-free, single-encoder retrieval pipeline to replace traditional text-to-image retrieval paradigm with structured image descriptions. |
| Outcome: | The proposed approach reduces the modality gap and improves compositionality and performance on short and long caption queries. |
TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to retrievalaugmented generation (RAG) are limited when applied to heterogeneous documents . flattening tables and chunking strategies disrupt tabular structure, leads to information loss, and undermines reasoning capabilities of LLMs in multi-hop, global queries. |
| Approach: | They propose a SQL-based framework that unifies textual understanding and complex manipulations over tabular data. |
| Outcome: | The proposed framework outperforms baselines on public datasets and HeteQA on heterogeneous document question answering. |
Retrieval Enhanced Feedback via In-context Neural Error-book (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning from errors lack a structured framework for analyzing and mitigating errors, especially in Multimodal Large Language Models (MLLMs). |
| Approach: | They propose a teacher-student framework that systematically structures errors to deliver targeted feedback for multimodal reasoning. |
| Outcome: | The proposed framework improves inference efficiency, token usage, and scalability by building a query-based structure that prioritizes visual information, diagnoses failure points, and guides corrective actions. |
Improve LLM-as-a-Judge Ability as a General Ability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on generative judges, but only on their judge ability. |
| Approach: | They propose a method that leverages the generative and reasoning capabilities of large language models to evaluate LLM responses across diverse scenarios, providing accurate preference signals. |
| Outcome: | The proposed model performs on RewardBench with only 2% to 40% of the data required by other training frameworks. |
G2: Guided Generation for Enhanced Output Diversity in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to enhance output diversity but compromise quality of outputs. |
| Approach: | They propose a training-free plug-and-play method that enhances output diversity while preserving generation quality. |
| Outcome: | The proposed method enhances output diversity while maintaining an optimal balance between diversity and quality. |
ToolSafety: A Comprehensive Dataset for Enhancing Safety in LLM-Based Agent Tool Invocations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current models exhibit notable vulnerabilities in maintaining safety during multi-step tool interactions and in indirect harm scenarios. |
| Approach: | They propose a safety fine-tuning dataset to fine- tune LLMs into assistants . they propose to use synthesized trajectories and realistic, context-aware sample generation . |
| Outcome: | The proposed model maintains safety in multi-step and indirect harm scenarios with little impact on helpfulness. |
Learning to See through Sound: From VggCaps to Multi2Cap for Richer Automated Audio Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing AAC datasets suffer from short and simplistic captions, limiting expressiveness and semantic depth. |
| Approach: | They propose a multi-modal dataset that pairs audio with corresponding video and leverages large language models to generate rich, descriptive captions. |
| Outcome: | The proposed framework outperforms existing benchmarks in caption length, lexical diversity, and human-rated quality. |
Towards Optimal Evaluation Efficiency for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) require large-scale benchmarks, which are costly in terms of time, computational resources, or API tokens. |
| Approach: | They propose an efficient evaluation framework that selects a question subset based on pre-tested results and uses semantic analysis to evaluate whether the subset preserves the original benchmark. |
| Outcome: | The proposed evaluation framework outperforms previous methods in reliability and score accuracy. |
MMAPG: A Training-Free Framework for Multimodal Multi-hop Question Answering via Adaptive Planning Graphs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal question answering models rely on sequential retrieval and reasoning, but this single-path paradigm makes them vulnerable to errors due to misleading intermediate steps. |
| Approach: | They propose a multimodal multi-hop question answering framework guided by an Adaptive Planning Graph . they propose modality-specific strategies that dynamically adapt to distinct data types . |
| Outcome: | The proposed framework outperforms existing models that rely on training. |
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | A sparse Mixture-of-Experts architecture has emerged as a highly scalable solution for instruction tuning. |
| Approach: | They propose a mixture-of-Clustered-Experts (MoCE) architecture that allows expert specialization . they evaluate the mechanism on a set of benchmarks and show its superiority . |
| Outcome: | The proposed approach outperforms existing models and benchmarks on instruction tuning scenarios with significant input heterogeneity. |
Process-Supervised Reinforcement Learning for Code Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reinforcement learning strategies based on outcome supervision have shown effectiveness in code generation tasks, but their effectiveness in the field of code generation remains limited. |
| Approach: | They propose a method that uses a teacher model to mutate and refactor statements and a compiler to automatically label them. |
| Outcome: | The proposed method improves performance in complex code generation tasks. |
MuCAL: Contrastive Alignment for Preference-Driven KG-to-Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for KG-to-text generation are limited by the availability of reliable preference data. |
| Approach: | They propose to use a multilingual KG/Text alignment model to generate preference data using three LLMs by ranking candidates and applying Direct Preference Optimization (DPO) on these preferences. |
| Outcome: | The proposed model achieves robust cross-modal retrieval across multiple languages and difficulty levels. |
Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models (2025.emnlp-main)
Copied to clipboard
Wei Wang, Zhaowei Li, Qi Xu, Linfeng Li, YiQing Cai, Botian Jiang, Hang Song, Xingcan Hu, Pengyu Wang, Li Xiao
| Challenge: | Recent advances in multi-modal large language models have demonstrated remarkable capabilities in multimodal understanding, reasoning, and interaction. |
| Approach: | They propose a method that effectively aligns and integrates multi-scale knowledge of objects . they use a pipeline that provides over 300K essential training data to enhance alignment . |
| Outcome: | The proposed method effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images. |
Thought calibration: Efficient and confident test-time scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for teaching language models to be economical with their token budgets have failed to achieve the desired results. |
| Approach: | They propose to calibrate a language model's growing body of thoughts to determine when new reasoning plateaus. |
| Outcome: | The proposed framework preserves model performance with up to 60% reduction in thinking tokens on in-distribution data, and up to 20% in out-of-difference data. |
Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are often evaluated on math word problems . however, such metrics conflate two distinct sub-skills: abstract formulation and arithmetic computation. |
| Approach: | They propose to use Final-answer-based metrics to evaluate large language models on math word problems to conflate two distinct sub-skills: abstract formulation and arithmetic computation. |
| Outcome: | The proposed model performance is bottlenecked by arithmetic computation and not abstract formulation, the study shows. |
QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on smaller, task-specific models enhanced by distilling knowledge from LLMs, but the diversity and quality of negative knowledge remains understudied. |
| Approach: | They propose a quality-guided contrastive rationale distillation framework that aims to enhance reasoning capabilities through contrastive knowledge learning. |
| Outcome: | The proposed method consistently outperforms existing distillation techniques yielding higher-quality rationales. |
SHARP: Steering Hallucination in LVLMs via Representation Engineering (2025.emnlp-main)
Copied to clipboard
Junfei Wu, Yue Ding, Guofan Liu, Tianze Xia, Ziyue Huang, Dianbo Sui, Qiang Liu, Shu Wu, Liang Wang, Tieniu Tan
| Challenge: | Large Vision-Language Models (LVLMs) generate responses that are plausible but incorrect or unsupported—commonly referred to as hallucinations. |
| Approach: | They propose a representation-level intervention framework that modulates hallucination-related features during inference by probing their encoded features. |
| Outcome: | The proposed framework reduces hallucinations while maintaining the performance and generalization capabilities of Large Vision-Language Models (LVLMs). |
Think, Verbalize, then Speak: Bridging Complex Thoughts and Comprehensible Speech (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to decouple LLMs from spoken communication produce suboptimal results due to mismatches between optimal textual and verbal delivery. |
| Approach: | They propose a framework that decouples reasoning from spoken delivery to preserve the full reasoning capacity of LLMs. |
| Outcome: | The proposed framework preserves full reasoning capacity of large language models . it improves speech naturalness and conciseness with minimal impact on reasoning . |
Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reasoning-capable large language models (LLMs) have driven a major shift in artificial intelligence . these models generate long CoTs, capturing reasoning behaviors such as self-reflection, self-correction, and hypothesis testing. |
| Approach: | They propose a sample-efficient, two-stage training strategy to build reasoning LLMs . they "warm up" a model by distilling Long CoTs from a toy domain to acquire general reasoning skills . |
| Outcome: | The proposed training strategy outperforms existing models on a range of tasks. |
PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slides (2025.emnlp-main)
Copied to clipboard
Hao Zheng, Xinyan Guan, Hao Kong, Wenkai Zhang, Jia Zheng, Weixiang Zhou, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun
| Challenge: | Existing methods for generating presentations from documents focus on improving and evaluating content quality in isolation, overlooking visual appeal and structural coherence. |
| Approach: | They propose an edit-based presentation generation system that analyzes and iterates on slides to create new slides. |
| Outcome: | The proposed presentation generation tool outperforms existing methods in three dimensions . it analyzes slides, iterates and generates edit actions based on selected slides . |
SWAM: Adaptive Sliding Window and Memory-Augmented Attention Model for Rumor Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for rumor detection on social media focus on static graphs, ignoring dynamic and incremental propagation . rumour detection on the social media platform is crucial to mitigating harmful effects of rumors. |
| Approach: | They propose a sliding window and memory-augmented attention model for rumor detection . they use a dynamic propagation graph and memory to capture the long-term dependency . |
| Outcome: | The proposed model is compared with the state-of-the-art models on two public datasets. |
HydraRAG: Structured Cross-Source Enhanced Large Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current RAG system retrieves evidence from knowledge graphs and text documents but has limitations in multi-hop reasoning, multi-entity questions, and source verification. |
| Approach: | They propose a training-free framework that unifies graph topology, document semantics, and source reliability to support deep, faithful reasoning in large language models. |
| Outcome: | The proposed framework outperforms the current hybrid model-based model-driven system by 20.3% and 30.1% on seven benchmark datasets. |
VRoPE: Rotary Position Embedding for Video Large Language Models (2025.emnlp-main)
Copied to clipboard
Zikang Liu, Longteng Guo, Yepeng Tang, Tongtian Yue, Junxian Cai, Kai Ma, Qingbin Liu, Xi Chen, Jing Liu
| Challenge: | Existing versions of Large Language Models (LLMs) lack a positional encoding strategy for video. |
| Approach: | They propose a new positional encoding method tailored for Video-LLMs that mitigates positional biases and ensures a more uniform distribution of spatial focus. |
| Outcome: | The proposed method outperforms existing versions of RoPE in video understanding and reasoning tasks. |
SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for structured information extraction focus on specific publication sections due to domain complexity and high cost of annotating scientific texts. |
| Approach: | They propose a specialized benchmark for full-text entity and relation extraction in the natural language processing domain. |
| Outcome: | The proposed dataset comprises 60 manually annotated full-text NLP publications covering 7,072 entities and 1,826 relations. |
Think and Recall: Layer-Level Prompting for Lifelong Model Editing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for lifelong model editing suffer from limitations in usability, such as requiring additional training corpora or lacking support for reversible and detachable edits. |
| Approach: | They propose a plug-and-play method for knowledge retrieval and storage, i.e., Layer-Level Prompting, which enables seamless and efficient lifelong model editing. |
| Outcome: | The proposed method outperforms existing methods on question answering and hallucination benchmarks across different LLMs. |
SPIRIT: Patching Speech Language Models against Jailbreak Attacks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Speech language models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. |
| Approach: | They propose post-hoc patching defenses to intervene during inference by modifying the SLM’s activations that improve robustness up to 99% with negligible impact on utility and without any re-training. |
| Outcome: | The proposed defenses improve robustness up to 99% with negligible impact on utility and (ii) without any re-training. |
FIRE: Flexible Integration of Data Quality Ratings for Effective Pretraining (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate data quality rely on heuristic techniques or single quality signals. |
| Approach: | They propose a framework for integrating multiple data quality raters that integrates multiple quality signals into a unified space and provides a comprehensive quality signal for each data point. |
| Outcome: | The proposed framework outperforms existing methods and boosts model performance across a wide range of downstream tasks while requiring less than 37.5% tokens to reach the target performance. |
Multi-Domain Explainability of Preferences (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating concept-based explanations of preferences are poorly understood. |
| Approach: | They propose a method for generating local and global concept-based explanations of preferences across multiple domains using an LLM. |
| Outcome: | The proposed method outperforms baselines while also being explainable. |
Tuning Less, Prompting More: In-Context Preference Learning Pipeline for Natural Language Transformation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to natural language transformation (NLT) tasks face significant challenges, such as the computational costs of leveraging large pre-trained models and the limited generalization ability of fine-tuned smaller models. |
| Approach: | They propose a framework that combines prompting with fine-tuning to enhance smaller models by integrating In-Context Examples from retrieval. |
| Outcome: | The proposed framework outperforms existing methods across MT and TST tasks. |
IL-PCSR: Legal Corpus for Prior Case and Statute Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for identifying/retrieving relevant statutes and prior cases/precedents are inherently related, e.g., similar cases tend to cite similar statutes due to similar factual situation. |
| Approach: | They propose a corpus that provides a common testbed for developing models that exploit the dependence between the two tasks. |
| Outcome: | The proposed corpus exploits the dependence between the two retrieval tasks and provides a baseline model for the two tasks. |
ESGenius: Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledge (2025.emnlp-main)
Copied to clipboard
Chaoyue He, Xin Zhou, Yi Wu, Xinjia Yu, Yan Zhang, Lei Zhang, Di Wang, Shengfei Lyu, Hong Xu, Wang Xiaoqiao, Wei Liu, Chunyan Miao
| Challenge: | ESGenius is a comprehensive benchmark for evaluating Large Language Models on ESG and sustainability knowledge. |
| Approach: | They introduce ESGenius, a benchmark for evaluating and enhancing ESG proficiency . they use a rigorous two-stage evaluation protocol and a repository of foundational frameworks . |
| Outcome: | ESGenius is a benchmark for evaluating and enhancing the proficiency of Large Language Models (LLMs) in ESG and sustainability-focused question answering. |
How Sememic Components Can Benefit Link Prediction for Lexico-Semantic Knowledge Graphs? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to predict missing triples in Knowledge Graphs are limited by semantic information. |
| Approach: | They propose a method to leverage sememe knowledge to enhance LP . LP is a technique that integrates structural and textual information into a Knowledge Graph . |
| Outcome: | The proposed method improves LP performance in English and Chinese . it improves on WN18RR, HN7 and CWN5, respectively . |
WISE: Weak-Supervision-Guided Step-by-Step Explanations for Multimodal LLMs in Image Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing MCoT methods focus on inter-object reasoning, overlooking intra-object understanding crucial for image classification. |
| Approach: | They propose a Weak-supervision-guided Step-by-step Explanation method that reformulates MCoTs under weak supervision into concise, interpretable reasoning chains. |
| Outcome: | The proposed method improves interpretability by 37% and improves classification accuracy. |
Calibration Across Layers: Understanding Calibration Evolution in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated inherent calibration capabilities, where predicted probabilities align well with correctness . previous studies have linked this behavior to specific components in the final layer, such as entropy neurons and the unembedding matrix’s null space. |
| Approach: | They propose to examine how calibration evolves throughout the network's depth. |
| Outcome: | The proposed calibration direction improves calibration metrics without harming accuracy. |
The discordance between embedded ethics and cultural inference in large language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective interactions between AI and humans require an accurate representation of diverse cultures. |
| Approach: | They propose a framework that embeds ethical principles within an LLM and a hyperplane that embedding cultural norms within it. |
| Outcome: | The proposed framework shows that cultural norms are more aligned with ethical principles than standard models. |
SSA: Semantic Contamination of LLM-Driven Fake News Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating 45 variants of nine LLMs, we find LIAR2 accuracy climbs monotonically with injected contamination, while the SSA Factor escalates in near-perfect lock-step. |
| Approach: | They propose a framework that detects BDC risks across semantic to label level via entity shift perturbation and an interpretable metric, the SSA Factor. |
| Outcome: | The proposed framework detects BDC risks across semantic to label level via entity shift perturbation and interpretable metric, the SSA Factor. |
Logits-Based Finetuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for developing compact and efficient large language models lack token-level dependencies and linguistic diversity. |
| Approach: | They propose a logits-based fine-tuning framework that integrates supervised learning and knowledge distillation to build enriched training targets using teacher logits and ground truth labels. |
| Outcome: | The proposed method outperforms existing methods on a large-scale logits dataset and a series of science-focused models. |
STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for incontext learning often overlook structural alignment, leading to poor generalization and suboptimal performance. |
| Approach: | They propose a two-stage exemplar selection strategy that achieves a strong balance between efficiency, generalizability and performance. |
| Outcome: | The proposed method outperforms baselines on semantic parsing tasks on four benchmarks. |
PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are becoming more popular and are gaining widespread use in artificial intelligence. |
| Approach: | They propose a unified framework that addresses both privacy preservation and model compression in federated settings. |
| Outcome: | The proposed framework maintains competitive performance comparable to full-sized LLMs while ensuring robust privacy protection through its federated architecture. |
Efficient Beam Search for Large Language Models Using Trie-Based Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) face memorybound performance bottlenecks due to their high memory requirements. |
| Approach: | They propose a trie-based parallel decoding method that shares a single KV cache across beams with common prefixes to dramatically reduce memory usage and enables efficient decoding. |
| Outcome: | The proposed method significantly reduces memory usage and enables efficient decoding without compromising generation quality. |
Power doesn’t reside in size: A Low Parameter Hybrid Language Model (HLM) for Sentiment Analysis in Code-mixed data (2025.emnlp-main)
Copied to clipboard
Pavan Sai Balaga, Nagasamudram Karthik, Challa Vishwanath, Raksha Sharma, Rudra Murthy, Ashish Mittal
| Challenge: | Code-mixed text presents significant challenges for machine learning due to interplay of distinct grammatical structures, effectively forming a hybrid language. |
| Approach: | They propose a Hybrid Language Model that combines a multilingual encoder and a lightweight decoder to achieve sentiment classification performance comparable to those of fine-tuned Large Language Models. |
| Outcome: | The proposed model outperforms models trained individually in sentiment detection tasks. |
Evaluating Taxonomy Free Character Role Labeling (TF-CRL) in News Stories using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | TF-CRL assigns open-ended narrative role labels to characters in news stories based on their functional role in the narrative. |
| Approach: | They propose a task that assigns open-ended narrative role labels to characters in news stories based on their functional role in the narrative. |
| Outcome: | The proposed task outperforms human annotators across dimensions and shows that it is robust to human preference rankings and ratings. |
MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored the use of large language models (LLMs) in psychotherapy, however text-based cognitive behavioral therapy models struggle with client resistance, which weakens therapeutic alliance. |
| Approach: | They propose a multimodal approach that incorporates nonverbal cues and a synthetic dataset that pairs each client’s statements with corresponding facial images to train vision language models. |
| Outcome: | The proposed approach outperforms existing text-based cognitive behavioral therapy models in managing client resistance and fostering therapeutic alliance. |
RETAIL: Towards Real-world Travel Planning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing travel planning systems assume users provide explicit queries, limiting their practical utility. |
| Approach: | They propose a dataset RETAIL which supports decision-making for implicit queries while covering explicit queries. |
| Outcome: | The proposed model achieves a 1.0% pass rate, suggesting real-world travel planning remains challenging. |
Unraveling Interwoven Roles of Large Language Models in Authorship Privacy: Obfuscation, Mimicking, and Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have been driven by large-scale training corpora drawn from diverse sources such as websites, news articles, and books. |
| Approach: | They propose a framework for analyzing dynamic relationships among LLM-enabled AO, AM, and AV in the context of authorship privacy. |
| Outcome: | The proposed framework analyzes the dynamic relationships among LLM-enabled AO, AM, and AV in the context of authorship privacy. |
Reward Model Perspectives: Whose Opinions Do Reward Models Reward? (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that reward models are poorly aligned with demographic groups and can reward harmful stereotypes. |
| Approach: | They propose a framework for measuring the alignment of opinions captured by RMs . they also investigate the extent to which RM's demonstrate sociodemographic biases a . |
| Outcome: | The proposed framework measures the alignment of opinions captured by RMs . it shows that RM models are poorly aligned with several demographic groups . the findings highlight the need for more careful consideration of RM behavior in model alignment . |
FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank compression can reduce memory usage and computational demand, but results are poor during decoding. |
| Approach: | They propose a fine-grained low-rank compression algorithm that determines optimal rank allocation for each layer and incorporates progressive low-ranked decoding to maintain text generation quality. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on summarization tasks and on understanding tasks. |
Do You Know About My Nation? Investigating Multilingual Language Models’ Cultural Literacy Through Factual Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual question-answering benchmarks do not factor in regional diversity in the information they capture and tend to be Western-centric. |
| Approach: | They propose to benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics. |
| Outcome: | The proposed model shows greater knowledge of cultural information in English than in the dominant language of the respective culture. |
CoEvo: Coevolution of LLM and Retrieval Model for Domain-Specific Information Retrieval (2025.emnlp-main)
Copied to clipboard
Ang Li, Yiquan Wu, Yinghao Hu, Lizhi Qing, Shihang Wang, Chengyuan Liu, Tao Wu, Adam Jatowt, Ming Cai, Fei Wu, Kun Kuang
| Challenge: | Recent methods to enhance queries by generating intermediary elements can degrade retrieval performance . combining LLMs and retrievers can be difficult, resulting in unreliable or irrelevant intermediaries . |
| Approach: | They propose a framework that facilitates the coevolution of large language models and retrieval models. |
| Outcome: | The proposed framework facilitates the coevolution of LLMs and retrieval models. |
Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies use LoRA to fine-tune existing LLMs, but this is limited by the data and training gap between them and embedding models. |
| Approach: | They propose a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder that integrates embeddings across different languages. |
| Outcome: | The proposed model improves performance on the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025). |
Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot LLM-based Vision-and-Language Navigation agents either encode images as textual scene descriptions, potentially oversimplifying visual details, or process raw image inputs, which can fail to capture abstract semantics required for high-level reasoning. |
| Approach: | They propose to integrate large language models into embodied AI models by incorporating textual descriptions that facilitate analogical reasoning across images from multiple perspectives. |
| Outcome: | The proposed approach improves the agent’s contextual understanding on the R2R dataset, showing that it can make better decisions based on the LLMs. |
MUCAR: Benchmarking Multilingual Cross-Modal Ambiguity Resolution for Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
Xiaolong Wang, Zhaolu Kang, Wangyuxuan Zhai, Xinyue Lou, Yunghwei Lai, Ziyue Wang, Yawen Wang, Kaiyu Huang, Yile Wang, Peng Li, Yang Liu
| Challenge: | Existing multimodal benchmarks overlook linguistic and visual ambiguities, authors say . ambiguity resolution between modalities is lacking in multimodal large language models . |
| Approach: | They propose a benchmark to evaluate multimodal ambiguity resolution across multilingual and cross-modal scenarios. |
| Outcome: | a new benchmark evaluates multimodal ambiguity resolution across multilingual and cross-modal scenarios . the benchmark shows that MLLMs can resolve ambiguities in image-text alignment . however, existing benchmarks often overlook linguistic and visual ambiguties . |
Mind the Gap: How BabyLMs Learn Filler-Gap Dependencies (2025.emnlp-main)
Copied to clipboard
| Challenge: | a long-standing debate concerns whether the linguistic input children receive is sufficient to explain the grammatical knowledge they develop. |
| Approach: | They evaluate baby language models trained on child-oriented input from the BabyLM Challenge and two base models trained in 10M and 100M tokens. |
| Outcome: | The proposed models acquire filler-gap dependencies but fail to generalize or fully capture island constraints. |
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large language models (LLMs) exhibit factual inconsistencies across languages . authors identify two primary sources of error: insufficient engagement of reliable English-centric mechanism for factual recall, and incorrect translation from English back into the target language for the final answer. |
| Approach: | They propose two vector interventions to redirect the model toward better internal paths for higher factual consistency. |
| Outcome: | The proposed interventions increase the recall accuracy by over 35 percent for the lowest-performing language. |
BTC-SAM: Leveraging LLMs for Generation of Bias Test Cases for Sentiment Analysis Models (2025.emnlp-main)
Copied to clipboard
Zsolt T. Kardkovács, Lynda Djennane, Anna Field, Boualem Benatallah, Yacine Gaci, Fabio Casati, Walid Gaaloul
| Challenge: | Sentiment Analysis (SA) models harbor inherent social biases that can be harmful in real-world applications. |
| Approach: | They propose a bias testing framework that generates high-quality test cases using Large Language Models (LLMs) for the controllable generation of test sentences. |
| Outcome: | The proposed framework generates high-quality test cases for bias testing in SA models with minimal specification using Large Language Models (LLMs) for the controllable generation of test sentences. |
Debate-to-Detect: Reformulating Misinformation Detection as a Real-World Debate with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite advances in large language models, their application to misinformation detection remains hindered by issues of logical inconsistency and superficial verification. |
| Approach: | They propose a multi-agent debate framework that reformulates misinformation detection as a structured adversarial debate based on fact-checking workflows . |
| Outcome: | The proposed framework enables iterative refinement of evidence while improving decision transparency. |
Controllable Memorization in LLMs via Weight Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on mitigating memorization, but the deliberate control of memorisation has been underexplored. |
| Approach: | They propose a gradient-based weight pruning framework to control memorization rates in large language models by fine-grained control over pruning parameters. |
| Outcome: | The proposed framework enables models to suppress or enhance memorization based on application-specific requirements. |
Tracing L1 Interference in English Learner Writing: A Longitudinal Corpus with Error Annotations (2025.emnlp-main)
Copied to clipboard
Poorvi Acharya, J. Elizabeth Liebl, Dhiman Goswami, Kai North, Marcos Zampieri, Antonios Anastasopoulos
| Challenge: | high-quality learner corpora are rarely available for studies of second language acquisition and language transfer. |
| Approach: | They propose to curate a corpus of adult learners with longitudinal data that includes 15 different L1s. |
| Outcome: | The proposed corpus contains 687 texts written by adult learners in the USA . authors show that the corpus can be used to explore language learning trajectories over time. |
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks for large language models with context length limitations are suboptimal for initialization and fine-tuning. |
| Approach: | They propose a RoPE-based fine-tuning framework that strategically determines the best scaling factors for LLMs by a Divide-and-Conquer Incremental Search algorithm. |
| Outcome: | The proposed framework mitigates performance decay at extended target lengths and can perform effectively without fine-tuning. |
Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing RAG models are sensitive to the order in which evidence is presented, resulting in unstable performance and biased reasoning. |
| Approach: | They propose to quantify position bias in multimodal RAG systems by using position sensitivity index . they also develop a visualization framework to trace attention allocation patterns across decoder layers . |
| Outcome: | The proposed framework shows that multimodal interactions intensify position bias compared to unimodal settings and that this bias increases logarithmically with retrieval range. |
Let’s Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models’ Understanding of Sports (2025.emnlp-main)
Copied to clipboard
Punit Kumar Singh, Nishant Kumar, Akash Ghosh, Kunal Pasad, Khushi Soni, Manisha Jaishwal, Sriparna Saha, Syukron Abu Ishaq Alfarozi, Asres Temam Abagissa, Kitsuchart Pasupa, Haiqin Yang, Jose G Moreno
| Challenge: | Language Models (LMs) are primarily evaluated on globally popular sports, often overlooking regional and indigenous sporting traditions. |
| Approach: | They propose to use multiple-choice questions (MCQs) to assess LMs' understanding of traditional sports across 60 countries and 6 continents. |
| Outcome: | The new benchmark will be publicly available, fostering research in culturally aware AI systems. |
Multilingual Federated Low-Rank Adaptation for Collaborative Content Anomaly Detection across Multilingual Social Media Participants (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent developments in multilingual social media platforms (SNS) exacerbate new challenges in SNS content anomaly detection due to data islands and linguistic imbalance. |
| Approach: | They propose a multilingual Federated LoRA based on SVD-based language-specific disentanglement of LoRA blocks and a local orthogonal tuning strategy to detect content anomalies. |
| Outcome: | The proposed solution is superior in multilingual content anomaly detection while reducing multilingual knowledge conflicts and communication rounds. |
M3Retrieve: Benchmarking Multimodal Retrieval for Medicine (2025.emnlp-main)
Copied to clipboard
| Challenge: | Strong retrieval models are increasingly important in knowledge-intensive domains. |
| Approach: | They propose a benchmark to evaluate multimodal retrieval models in medical settings . they examine 1.2 million text documents and 164K multimodal queries . |
| Outcome: | The proposed model spans 5 domains,16 medical fields, and 4 distinct tasks with over 1.2 Million text documents and 164K multimodal queries. |
The Hidden Strength of Disagreement: Unraveling the Consensus-Diversity Tradeoff in Adaptive Multi-Agent Systems (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional LLM-based MAS rely on explicit coordination, e.g., prompts or voting, risking premature homogenization. |
| Approach: | They propose to preserve partial diversity by combining in-context learning with explicit coordination to form consensus in dynamic environments. |
| Outcome: | The proposed model outperforms explicit consensus models on three scenarios showing that partial deviation from group norms boosts exploration, robustness, and performance. |
Friend or Foe? A Computational Investigation of Semantic False Friends across Romance Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | lexical divergence between cognate and borrowings is studied in the five Romance languages. |
| Approach: | They propose to use etymological dictionaries to extract deceptive cognates and borrowings automatically based on usage and freely publish the lexicon of obtained true and deceptives in every Romance language pair. |
| Outcome: | The proposed algorithms are based on the most complete and reliable dataset of cognate words based etymological dictionaries for the five main Romance languages. |
KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit societal biases in their outputs, prompting ethical and societal challenges. |
| Approach: | They propose an attention-based debiasing framework that implicitly aligns attention distributions between stereotypical and anti-stereotypical sentence pairs without directly modifying model weights. |
| Outcome: | The proposed framework improves on BBQ and BOLD benchmarks while maintaining fluency and coherence. |
SeMob: Semantic Synthesis for Dynamic Urban Mobility Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing spatiotemporal models struggle to interpret and adapt to abrupt changes caused by external events. |
| Approach: | They propose a LLM-powered semantic synthesis pipeline that extracts spatiotemporally related text from online texts and integrates it with spatio-temporal data. |
| Outcome: | The proposed framework achieves maximal reductions of 13.92% in MAE and 11.12% in RMSE compared to the spatiotemporal model. |
DyePack: Provably Flagging Test Set Contamination in LLMs Using Backdoors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Open benchmarks are essential for evaluating large language models, but their accessibility makes them likely targets of test set contamination. |
| Approach: | They propose a framework that leverages backdoor attacks to flag models that used benchmark test sets during training. |
| Outcome: | The proposed framework detects models that trained on benchmark test sets without loss of logits or internal details . it can prevent false accusations while providing strong evidence for every detected case of contamination. |
Minimal, Local, and Robust: Embedding-Only Edits for Implicit Bias in T2I Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | EmbEdit is a text-to-image editing method that only fine-tunes the word token embedding (WTE) of the target object. |
| Approach: | They propose a method to edit implicit assumptions and priors in text-to-image models without affecting unrelated objects or degrading overall performance. |
| Outcome: | The proposed method outperforms previous methods in various models, tasks, and editing scenarios. |
Journalism-Guided Agentic In-context Learning for News Stance Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing stance detection research on news content is limited to short texts and high-resource languages. |
| Approach: | They propose a dataset for article-level stance detection that integrates viewpoints into recommendation algorithms and a framework that employs a language model agent to predict the stances of key structural segments. |
| Outcome: | The proposed framework outperforms existing methods in identifying article stances and uncovering patterns of media bias. |
Less Is MuRE: Revisiting Shallow Knowledge Graph Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge graphs encode knowledge in the form of subject-predicate-object triples, which is notoriously incomplete. |
| Approach: | They propose a framework for analyzing existing shallow knowledge graph models and their extensions. |
| Outcome: | The proposed framework shows that MuRE and ExpressivE are highly competitive . it can capture the same class of rule bases as state-of-the-art region-based embedding models. |
Jailbreak LLMs through Internal Stance Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to exploit LLMs' inherent safety mechanism, including GCG and AutoDAN, are ineffective for certain malicious requests. |
| Approach: | They propose a method that generates jailbreak prompts to suppress a refusal stance and induce affirmative responses by modifying adversarial prompts. |
| Outcome: | The proposed method outperforms the best baseline approach in Llama-2-7b-chat and achieves a 92.2% success rate across all models. |
Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hampered by hallucinations, a particularly challenging variant, knowledge overshadowing, which can lead to erroneous outputs even with high-quality training data. |
| Approach: | They propose a framework to analyze and detect knowledge overshadowing by using knowledge circuit analysis to dissect the function of key components in the circuit and how attention pattern dynamics contribute to the phenomenon. |
| Outcome: | Extensive experiments show that the framework can detect and analyze knowledge overshadowing and improves on existing models. |
Complex Numerical Reasoning with Numerical Semantic Pre-training Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Numerical knowledge graphs (NKGs) are not limited to discrete entity-relation knowledge. |
| Approach: | They propose to combine numerical values and entities to solve multi-hop complex reasoning over incomplete knowledge graphs. |
| Outcome: | The proposed approach handles up to 102 types of complex numerical reasoning queries on three public datasets. |
Automated Knowledge Graph Construction using Large Language Models and Sentence Complexity Modelling (2025.emnlp-main)
Copied to clipboard
Sydney Anuyah, Mehedi Mahmud Kaushik, Sri Rama Krishna Reddy Dwarampudi, Rakesh Shiradkar, Arjan Durresi, Sunandan Chakraborty
| Challenge: | integrating coreference and decomposition increases recall on rare relations by over 20%. |
| Approach: | They propose an open-source pipeline for extracting sentence-level knowledge graphs by combining robust coreference resolution with syntactic sentence decomposition. |
| Outcome: | The proposed pipeline achieves a 99.8% exact-match accuracy on sentence simplification. |
OntologyRAG-Q: Resource Development and Benchmarking for Retrieval-Augmented Question Answering in Qur’anic Tafsir (2025.emnlp-main)
Copied to clipboard
| Challenge: | An annotated Tafsir ontology and a collection of 15 structured Tafsian books are presented in this paper. |
| Approach: | They propose a framework for retrieval and question-answering Tafsir data that spans the entire pipeline from dataset construction through evaluation and error analysis. |
| Outcome: | The proposed framework achieves 69.52% accuracy and 74.36% correctness overall, though multi-hop and context-dependent questions remain challenging. |
The Practical Impacts of Theoretical Constructs on Empathy Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empathy operationalizations in NLP are varied, with some having specific behaviors and properties, while others are more abstract. |
| Approach: | They analyze the transfer performance of empathy models adapted to empathy tasks with different theoretical groundings and characterize them as direct, abstract, or adjacent. |
| Outcome: | The proposed models show that they are more transferable than other models. |
RecBase: Generative Foundation Model Pretraining for Zero-Shot Recommendation (2025.emnlp-main)
Copied to clipboard
Sashuai Zhou, Weinan Gan, Qijiong Liu, Ke Lei, Jieming Zhu, Hai Huang, Yan Xia, Ruiming Tang, Zhenhua Dong, Zhou Zhao
| Challenge: | Existing methods for addressing item-level user interests are lacking in cross-domain generalization . RecBase model is domain-agnostic and can be used to enhance recommender systems' effectiveness . |
| Approach: | They propose a domain-agnostic foundational model pretrained with a recommendation-oriented objective that leverages a large-scale, heterogeneous, cross-domain corpus with unified textual representations and feature mappings to enhance cross- domain generalization. |
| Outcome: | The proposed model matches or surpasses baselines in zero-shot and cross-domain recommendation tasks on eight real-world datasets. |
Grouping Entities with Shared Properties using Multi-Facet Prompting and Property Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Methods for learning taxonomies from data are well-studied, but it is difficult to use them in large domains. |
| Approach: | They propose to use LLMs to describe the different properties that are satisfied by each entity individually and then use pre-trained embeddings to cluster these properties. |
| Outcome: | The proposed model can be used to describe the properties of the entities and group them into clusters. |
Context-Aware Hierarchical Taxonomy Generation for Scientific Papers via LLM-Guided Multi-Aspect Clustering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing taxonomy construction methods lack coherence and granularity . Existing approaches rely on manual or narrowly defined schemes . |
| Approach: | They propose a context-aware hierarchical taxonomy generation framework that integrates LLMs with dynamic clustering. |
| Outcome: | The proposed method outperforms existing methods in taxonomy coherence, granularity, and interpretability. |
Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models are often judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually demands. |
| Approach: | They propose a diagnostic framework that decomposes benchmark performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) AIS quantifies how much each ability contributes to a model’s success on a given benchmark. |
| Outcome: | The proposed framework decomposes performance into ten cognitively grounded abilities and computes an Ability Impact Score (AIS) that quantifies how much each ability contributes to a model’s success on a given benchmark. |
TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review (2025.emnlp-main)
Copied to clipboard
Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok-Hay So, Zhijiang Guo, Liya Zhu, Ngai Wong
| Challenge: | Large Language Models (LLMs) have shown significant potential in assisting peer review, but current methods struggle to generate thorough and insightful reviews while maintaining efficiency. |
| Approach: | They propose a framework that models paper review as a hierarchical and bidirectional question-answering process. |
| Outcome: | The proposed framework outperforms baselines on full review generation and actionable feedback comments generation tasks while reducing LLM token usage by up to 80% compared to computationally intensive approaches. |
Improving Chemical Understanding of LLMs via SMILES Parsing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Molecular string representations such as SMILES and SELFIES are becoming a standard format for applying large language models (LLMs) however, molecular strings follow complex syntactic rules for encoding molecules, which LLMs struggle to interpret. |
| Approach: | They propose a framework that parses SMILES into clean and deterministic tasks to promote graph-level molecular comprehension. |
| Outcome: | The proposed framework improves structural comprehension and competes with the baseline on the Mol-Instructions benchmark. |
Can Large Language Models Tackle Graph Partitioning? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have remarkable capabilities in understanding complex tasks, but they can only handle graph partitioning tasks that require global perception abilities. |
| Approach: | They propose a pipeline for coarsening, reasoning, and refining to enable LLMs to perform graph partitioning on small-scale graphs. |
| Outcome: | The proposed pipeline can handle graph partitioning tasks on small graphs with coarsening, reasoning, and refining. |
To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging (2025.emnlp-main)
Copied to clipboard
Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Yiyao Cao, Jing Li, Ho-Kin Tang, Sim Kuan Goh
| Challenge: | Existing approaches to model merging ignore the fundamental roles of neurons, connectivity and activation. |
| Approach: | They propose a framework that relies on neuronal mechanisms to mitigate task interference . they decomposed task-specific representations into two complementary subspaces . their results offer new insights into mitigating task interference and improving knowledge fusion . |
| Outcome: | The proposed framework reduces task interference within neurons and improves knowledge fusion. |
What You Read Isn’t What You Hear: Linguistic Sensitivity in Deepfake Speech Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in text-to-speech technology have enabled highly realistic voice generation, fueling deepfake attacks. |
| Approach: | They propose a framework for transcript-to-audio perturbation anti-spoofing that incorporates linguistic variation into detectors to investigate spoof detection. |
| Outcome: | The proposed framework can bypass commercial detectors by incorporating linguistic variation into the design of anti-spoofing systems. |
Task-Aware Resolution Optimization for Visual Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing visual large language models pre-assume a fixed resolution for downstream tasks, leading to sub-optimal performance. |
| Approach: | They propose a formula to determine the optimal resolution for a given vision-language task . they then propose 'parameter-efficient' fine-tuning technique to extend the visual input resolution . |
| Outcome: | The proposed method is based on rigorous experiments on vision-language tasks. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
A Necessary Step toward Faithfulness: Measuring and Improving Consistency in Free-Text Explanations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a measure of faithful free-text explanations is difficult to generate by language models and assess by humans. |
| Approach: | They propose a measure of Prediction-EXplanation consistency by extending the concept of weight of evidence. |
| Outcome: | The proposed measure improves explanation faithfulness by up to 9.7%, the authors show . they show that applying preference optimization improves the consistency of generated explanations across three model families. |
Boosting Multi-modal Keyphrase Prediction with Dynamic Chain-of-Thought in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
Qihang Ma, Shengyu Li, Jie Tang, Dingkang Yang, null Chenshaodong, Yingyi Zhang, Chao Feng, Ran Jiao
| Challenge: | Multi-modal keyphrase prediction (MMKP) aims to produce concise, informative phrases that capture the essence of cross-modal inputs. |
| Approach: | They propose to use vision-language models to generate conclusive phrases using multiple modalities of input information. |
| Outcome: | The proposed methods outperform existing methods on absence and unseen scenarios and overestimate model capability due to overlap in training tests. |
Chart2Code53: A Large-Scale Diverse and Complex Dataset for Enhancing Chart-to-Code Generation (2025.emnlp-main)
Copied to clipboard
Tianhao Niu, Yiming Cui, Baoxin Wang, Xiao Xu, Xin Yao, Qingfu Zhu, Dayong Wu, Shijin Wang, Wanxiang Che
| Challenge: | Existing Chart2code-related training datasets suffer from limited scale, limited type coverage, and inadequate complexity. |
| Approach: | They propose to synthesize chart2code-related training datasets using web plotting code and chart images to address these challenges. |
| Outcome: | The proposed dataset exhibits the greatest diversity and higher complexity compared to other open-source Chart2code related datasets. |
The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating It (2025.emnlp-main)
Copied to clipboard
| Challenge: | a systematic review of 300 publications reveals a language gap in LLM safety research . even high-resource non-English languages receive little attention, authors note . |
| Approach: | They propose to focus on safety evaluation, training data generation, and crosslingual safety generalization based on their findings. |
| Outcome: | The authors suggest that the field can develop more robust, inclusive safety practices for diverse global populations. |
AIP: Subverting Retrieval-Augmented Generation via Adversarial Instructional Prompt (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing RAG attacks rely on manipulating user queries, but exploit instructional prompts to manipulate RAG outputs covertly. |
| Approach: | They propose an attack that exploits adversarial instructional prompts to manipulate RAG outputs . they propose a query generation strategy that simulates realistic linguistic variation in user queries . |
| Outcome: | The proposed attack exploits instructional prompts to manipulate RAG outputs . it achieves up to 95.23% attack success rate while maintaining benign functionality . |
From Capabilities to Performance: Evaluating Key Functional Properties of LLM Architectures in Penetration Testing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been explored for automating or enhancing penetration testing tasks, but their effectiveness and reliability remain open questions. |
| Approach: | They evaluate multiple LLM-based agents across realistic penetration testing scenarios . they also examine impact of core functional capabilities on agent success . |
| Outcome: | The proposed models improve agent performance in multi-step and real-time penetration testing scenarios. |
Editing Across Languages: A Survey of Multilingual Knowledge Editing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge Editing is a growing subdomain of model editing focused on ensuring factual edits generalize across languages. |
| Approach: | They present a taxonomy of multilingual knowledge editing methods and benchmarks . authors summarize key findings on method effectiveness and transfer patterns . |
| Outcome: | The proposed methods are compared against available benchmarks and benchmark datasets. |
Your RAG is Unfair: Exposing Fairness Vulnerabilities in Retrieval-Augmented Generation via Backdoor Attacks (2025.emnlp-main)
Copied to clipboard
Gaurav Bagwe, Saket Sanjeev Chaturvedi, Xiaolong Ma, Xiaoyong Yuan, Kuang-Ching Wang, Lan Emily Zhang
| Challenge: | Retrieval-augmented generation (RAG) enhances factual grounding but introduces new attack surfaces, particularly through backdoor attacks. |
| Approach: | They propose a framework that exposes fairness vulnerabilities in RAG through a two-phase backdoor attack. |
| Outcome: | Empirical results show that BiasRAG achieves high attack success rates while remaining undetectable under standard fairness evaluations. |
Drift-Adapter: A Practical Approach to Near Zero-Downtime Embedding Model Upgrades in Vector Databases (2025.emnlp-main)
Copied to clipboard
| Challenge: | Upgrading embedding models in production environments requires re-encoding the entire corpus and rebuilding the Approximate Nearest Neighbor (ANN) index. |
| Approach: | They propose a lightweight, learnable transformation layer designed to bridge embedding spaces between models by mapping new queries into the legacy embeddable space. |
| Outcome: | The proposed transformation layer recovers 95–99% of the retrieval recall of a full re-embedding, adding less than 10,s query latency. |
The Staircase of Ethics: Probing LLM Value Priorities through Multi-Step Induction to Complex Moral Dilemmas (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of LLMs' moral reasoning capabilities rely on single-step evaluations, ignoring how models adapt to evolving ethical challenges. |
| Approach: | They propose a framework to evaluate evolving moral judgments of large language models (LLMs) using multi-step moral dilemma questionnaires. |
| Outcome: | The proposed framework enables a fine-grained analysis of how LLMs adjust their moral reasoning across escalating dilemmas. |
SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Token-level routing assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialisation. |
| Approach: | They propose an architecture that routes contiguous slices of a token’s hidden vector and a lightweight shared router predicts the top-k experts. |
| Outcome: | The proposed architecture achieves 1.7x faster inference than dense baselines, 12–18% lower perplexity than parameter-matched token-MoE, and improved expert balance. |
ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-agent systems (MAS) are limited by poor flexibility and scalability, with underdeveloped optimization strategies. |
| Approach: | They propose a task graph generation and a reward-driven two-stage agent selection process to integrate multi-agent systems to improve their reasoning capabilities. |
| Outcome: | The proposed model outperforms existing methods on Math-MAS and SciBench-MAS SciBech, while other methods completely fail. |
ConstraintLLM: A Neuro-Symbolic Framework for Industrial-Level Constraint Programming (2025.emnlp-main)
Copied to clipboard
| Challenge: | Constraint programming (CP) is a powerful paradigm for solving constraint optimization problems. |
| Approach: | They propose to use an open-source LLM to generate formal modeling for COPs. |
| Outcome: | The proposed model outperforms the baselines on the new IndusCP benchmark by 2x. |
VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Rooms (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on embodied agents have addressed the importance of exploration in environments where tasks and solutions are not predefined. |
| Approach: | They propose a virtual escape room that evaluates AI models in a dynamic environment . they propose to integrate memory management and reasoning into the simulation . |
| Outcome: | The proposed model improves in dynamic and exploration-driven environments by integrating memory management and reasoning. |
ESC-Judge: A Framework for Comparing Emotional Support Conversational Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) increasingly power mental-health chatbots . yet the field lacks a scalable, theory-grounded way to decide which model is more effective to deploy. |
| Approach: | They propose a framework that grounds head-to-head comparisons of Emotional-Support LLMs in Hill’s Exploration–Insight–Action counselling model. |
| Outcome: | The proposed framework matches PhD-level annotators in 85% of Exploration, 83% of Insight, and 86% of Action decisions, demonstrating human-level reliability at a fraction of the cost. |
Neuron-Level Differentiation of Memorization and Generalization in Large Language Models (2025.emnlp-main)
Copied to clipboard
Ko-Wei Huang, Yi-Fu Fu, Ching-Yu Tsai, Yu-Chieh Tu, Tzu-ling Cheng, Cheng-Yu Lin, Yi-Ting Yang, Heng-Yi Liu, Keng-Te Liao, Da-Cheng Juan, Shou-De Lin
| Challenge: | Existing models exhibit memorization and generalization behaviors in ways that are not easily interpretable or controllable. |
| Approach: | They propose to use a GPT-2 and LLaMA-3.2 model to identify distinct neuron subsets responsible for each behavior to steer the model toward memorization or generalization. |
| Outcome: | The proposed models show that inference-time interventions on these neurons can steer the model’s behavior toward memorization or generalization. |
Sparse Neurons Carry Strong Signals of Question Ambiguity in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Ambiguity is pervasive in real-world questions, yet large language models often respond with confident answers rather than seeking clarification. |
| Approach: | They show that question ambiguity is linearly encoded in the internal representations of large language models (LLMs) by training linear probes, they identify sparse sets of Ambiguity-Encoding Neurons (AENs) |
| Outcome: | The proposed model outperforms prompting-based and representation-based baselines on ambiguity detection and generalization. |
Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current automatic speech recognition systems rely on only audio information, ignoring multi-modal context. |
| Approach: | They propose to integrate visual context into existing automatic speech recognition systems to integrate presentation slides with multi-modal information. |
| Outcome: | The proposed model reduces word error rate by approximately 34% across all words and 35% for domain-specific terms compared to baseline model. |
Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries (2025.emnlp-main)
Copied to clipboard
| Challenge: | To answer one-to-many factual queries, a language model must simultaneously recall knowledge and avoid repeating previous answers. |
| Approach: | They propose a promote-then-suppress mechanism that enables LMs to recall all answers and suppress previously generated ones. |
| Outcome: | The proposed model first recalls all answers, and then suppresses previously generated ones. |
Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames (2025.emnlp-main)
Copied to clipboard
Sahithya Ravi, Gabriel Herbert Sarch, Vibhav Vineet, Andrew D Wilson, Balasaravanan Thoravi Kumaravel
| Challenge: | Disjoint-3DQA evaluates the spatial reasoning ability of embodied AI assistants based on egocentric video . it aims to catalyze future research at the intersection of vision, language, and embodie . |
| Approach: | They propose a generative QA benchmark that evaluates the ability of embodied AI assistants to integrate spatial cues across time by asking object pairs that are not co-visible in the same frame. |
| Outcome: | The proposed benchmark compares seven state-of-the-art VLMs and finds that they lag behind human performance by 28%, with steeper declines as the temporal gap widens. |
Enhancing Chain-of-Thought Reasoning via Neuron Activation Differential Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on optimizing external components of CoT, but lack internal explanations for the quality of the model's outputs. |
| Approach: | They propose an efficient method to identify reasoning-critical neurons by analyzing their activation patterns under reasoning chains of varying quality. |
| Outcome: | The proposed method shows that neurons in the feed-forward layers are critical in the generation of high-quality reasoning chains. |
PakBBQ: A Culturally Adapted Bias Benchmark for QA (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely adopted in language processing applications, but they often perpetuate harmful societal biases. |
| Approach: | They propose a culturally and regionally adapted extension of the original Bias Benchmark for Question Answering dataset to address this gap. |
| Outcome: | The proposed model gains 12% accuracy with disambiguation and stronger counter bias behaviors in Urdu than in English. |
MULTIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities (2025.emnlp-main)
Copied to clipboard
Sahil Verma, Keegan Hines, Jeff Bilmes, Charlotte Siska, Luke Zettlemoyer, Hila Gonen, Chandan Singh
| Challenge: | Existing approaches to detect harmful queries to large language models are fallible and vulnerable to attacks that exploit mismatched generalization of model capabilities. |
| Approach: | They propose an approach to detect harmful queries to large language models (LLMs) OMNIGUARD identifies internal representations of an LLM/MLLM that are aligned across languages or modalities and builds a language-agnostic or modality-adic classifier for detecting harmful prompts. |
| Outcome: | OMNIGUARD improves harmful prompt classification accuracy by 11.57% over the strongest baseline in a multilingual setting, by 20.44% for image-based prompts, and sets a new SOTA for audio-based ones. |
Comparing human and LLM politeness strategies in free production (2025.emnlp-main)
Copied to clipboard
| Challenge: | Polite speech poses a fundamental alignment challenge for large language models (LLMs). |
| Approach: | They compare human and LLM responses to English-language scenarios to determine whether they employ a similarly context-sensitive repertoire. |
| Outcome: | The results show that large models replicate key effects from the computational pragmatics literature and human evaluators prefer LLM-generated responses in open-ended contexts. |
ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning via Tool-integrated Action for Dynamic Offer Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing agents struggle due to bounded rationality in human data, low adaptability to counterpart behavior, and limited strategic reasoning. |
| Approach: | They propose a framework for turn-level offer optimization based on two core principles: opponent modeling and Tit-for-Tat reciprocity. |
| Outcome: | The proposed framework outperforms baselines across diverse partner agents and validates through human evaluation. |
CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with compositional generalisation, limiting their ability to systematically combine learned components to interpret novel inputs. |
| Approach: | They propose a method that enhances stability and robustness of compositional reasoning in large language models (LLMs) by regularising and preserving layer-wise stability constraints. |
| Outcome: | The proposed method reduces variability, stabilizes token representations, and improves compositional reasoning. |
MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper (2025.emnlp-main)
Copied to clipboard
Runjia Zeng, Guangyan Sun, Qifan Wang, Tong Geng, Sohail Dianat, Xiaotian Han, Raghuveer Rao, Xueling Zhang, Cheng Han, Lifu Huang, Dongfang Liu
| Challenge: | Empirical evaluations show that Mixture of Expert Prompt Tuning outperforms state-of-the-art parameter efficient baselines on SuperGLUE. |
| Approach: | They propose a pretrain-then-fine-tune paradigm for manifold mapping using multiple prompt experts. |
| Outcome: | Empirical results show that the proposed approach outperforms state-of-the-art methods on SuperGLUE while reducing activated prompts by 79.25%. |
KG-CQR: Leveraging Structured Relation Representations in Knowledge Graphs for Contextual Query Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods that address corpus-level context loss focus on query enrichment through structured relation representations. |
| Approach: | They propose a framework for Contextual Query Retrieval that enriches queries with contextual representations derived from a corpus-centric KG. |
| Outcome: | The proposed framework outperforms strong baselines on RAGBench and MultiHop-RAG datasets in terms of retrieval effectiveness. |
SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) with safe-alignment training are vulnerable to jailbreak attacks, causing malicious users to generate harmful outputs. |
| Approach: | They propose a safe-alignment jailbreak method that bypasses the middle-to-late layers of large language models by a residual connection. |
| Outcome: | The proposed method improves by 51% over the best performing baseline GCG on HarmBench test set. |
When Truthful Representations Flip Under Deceptive Instructions? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) follow maliciously crafted instructions to generate deceptive responses, posing safety challenges. |
| Approach: | They use Sparse Autoencoders to analyze LLM's internal representations to determine when and how they "flip" from truthful to deceptive under deceptively crafted instructions. |
| Outcome: | The proposed model's True/False output is predictable across all conditions based on the model''s representation, and the Deceptive instructions induce significant representational shifts compared to Truthful/Neutral representations. |
Can LLMs simulate the same correct solutions to free-response math problems as real students? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored modeling student mistakes, but lack of understanding of how they generate correct solutions. |
| Approach: | They compare distribution of correct solutions produced by four large language models with students' responses to free-response problems. |
| Outcome: | The proposed model can generate correct solutions that represent student responses to free-response problems. |
Evaluating Behavioral Alignment in Conflict Dialogue: A Multi-Dimensional Comparison of LLM Agents and Humans (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used in socially complex, interaction-driven tasks, yet their ability to mirror human behavior in emotionally and strategically complex contexts remains underexplored. |
| Approach: | They examine alignment of personality-prompted Large Language Models in conflict dialogues that incorporate negotiation by simulating a five-factor personality profile. |
| Outcome: | The proposed model achieves the closest alignment with humans in linguistic style and emotional dynamics while Claude-3.7-Sonnet best reflects strategic behavior. |
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions (2025.emnlp-main)
Copied to clipboard
Emmy Liu, Amanda Bertsch, Lintang Sutawika, Lindia Tjuatja, Patrick Fernandes, Lara Marinov, Michael Chen, Shreya Singhal, Carolin Lawrence, Aditi Raghunathan, Kiril Gashteovski, Graham Neubig
| Challenge: | Language model performance is largely dependent on pretraining decisions, but scaling laws based on only these two aspects do not always explain downstream task performance. |
| Approach: | They meta-analyze 92 open-source pretrained models to quantify their impact on performance. |
| Outcome: | The framework lays a foundation for more systematic investigation of how model development choices shape final capabilities. |
Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamics (2025.emnlp-main)
Copied to clipboard
Jiarui Liu, Yueqi Song, Yunze Xiao, Mingqian Zheng, Lindia Tjuatja, Jana Schaich Borg, Mona T. Diab, Maarten Sap
| Challenge: | a study of multi-dimensional persona effects in AI-AI debates shows that personas influence moral stances and debate outcomes . political ideology and personality traits exert the strongest influence, according to our study . |
| Approach: | They propose to use a 6-dimensional persona space to simulate structured debates . they find political ideology and personality traits exert the strongest influence . |
| Outcome: | The study shows that personas affect moral stances and debate outcomes . political ideology and personality traits exert the strongest influence . |
Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to select demonstration examples for in-context learning are based on token embeddings. |
| Approach: | They propose an algorithm to select demonstration examples for in-context learning of a query set . they use gradients of the output taken in the input embedding space to estimate model outputs . |
| Outcome: | The proposed algorithm outperforms existing methods based on token embeddings by 11% . it scales up subset selection that would otherwise run full inference by 37.7 on models with 34 billion parameters . |
Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Speech Vecalign is a parallel speech document alignment method that monotonically aligns speech segment embeddings and does not depend on text transcriptions. |
| Approach: | They propose a parallel speech document alignment method that monotonically aligns speech segment embeddings and does not depend on text transcriptions. |
| Outcome: | The proposed method outperforms SpeechMatrix models on 3,000 hours of unlabeled speech documents and produces longer speech-to-speech alignments. |
TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs (2025.emnlp-main)
Copied to clipboard
| Challenge: | TurBLiMP is the first benchmark of linguistic minimal pairs for monolingual and multilingual language models . it covers 16 linguistic phenomena with 1000 minimal pairs each . a foundational insight in linguistics research is that applying minimal changes to a sentence can render it entirely acceptable or unacceptable to native speakers. |
| Approach: | They propose to use morphologically rich agglutinative language with highly flexible word order to evaluate linguistic abilities of monolingual and multilingual language models. |
| Outcome: | The proposed benchmark covers 16 linguistic phenomena with 1000 minimal pairs each. |
DynamicNER: A Dynamic, Multilingual, and Fine-Grained Dataset for LLM-based Named Entity Recognition (2025.emnlp-main)
Copied to clipboard
Hanjun Luo, Yingbin Jin, Yiran Wang, Xinfeng Li, Tong Shang, Xuecheng Liu, Ruizhe Chen, Kun Wang, Hanan Salam, Qingsong Wen, Zuozhu Liu
| Challenge: | Existing datasets designed for Named Entity Recognition methods are inadequate for LLMs. |
| Approach: | They propose a dataset that is multilingual and multi-granular and enables LLMs to be applied to Named Entity Recognition methods. |
| Outcome: | The proposed dataset is multilingual and multi-granular, covering 8 languages and 155 entity types, with corpora spanning a diverse range of domains. |
Reliable and Cost-Effective Exploratory Data Analysis via Graph-Guided RAG (2025.emnlp-main)
Copied to clipboard
Mossad Helali, Yutai Luo, Tae Jun Ham, Jim Plotts, Ashwin Chaugule, Jichuan Chang, Parthasarathy Ranganathan, Essam Mansour
| Challenge: | Large Language Models (LLMs) have limited accuracy and code reliability on less-studied or private datasets. |
| Approach: | They propose a graph-guided framework that extracts EDA operation semantics from a knowledge graph and then generates executable Python code. |
| Outcome: | Experiments on two datasets show that RAGvis significantly improves code executability, semantic accuracy, and visual quality compared to LLM-only baselines. |
Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards (2025.emnlp-main)
Copied to clipboard
Jaehoon Yun, Jiwoong Sohn, Jungwoo Park, Hyunjae Kim, Xiangru Tang, Daniel Shao, Yong Hoe Koo, Ko Minhyeok, Qingyu Chen, Mark Gerstein, Michael Moor, Jaewoo Kang
| Challenge: | Large language models have shown promise in clinical decision making, but current approaches struggle to localize and correct reasoning errors at specific steps of the reasoning process. |
| Approach: | They propose a process reward modeling framework that leverages retrieval-augmented generation to verify each reasoning step against established medical knowledge bases. |
| Outcome: | The proposed model improves on five medical QA benchmarks and two open-ended diagnostic tasks by 13.50% on MedQA. |
Graders Should Cheat: Privileged Information Enables Expert-Level Automated Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a lack of trust in graders on graduate-level physics and Olympiad-level math makes them unreliable grader. |
| Approach: | They propose to use a grader LM to evaluate the candidate LMs. |
| Outcome: | The proposed approach outperforms human graders on *RewardBench* and human expert grader on Olympiad-level math problems. |
SAMULE: Self-Learning Agents Enhanced by Multi-level Reflection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern AI agents rely on Large Language Models (LLMs) as their reasoning engines, but they still face the challenge of generating meaningful reflections due to inadequate error analysis and a reliance on rare successful trajectories. |
| Approach: | They propose a framework for self-learning agents powered by a retrospective language model that generates reflections during inference. |
| Outcome: | The proposed framework outperforms reflection-based baselines on three challenging benchmarks. |
Database-Augmented Query Representation for Information Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Information retrieval models that aim to search for documents relevant to a query have shown multiple successes, but the query from the user is oftentimes short, which challenges the retrievers to correctly fetch relevant documents. |
| Approach: | They propose a database-augmented Query representation framework which augments the query with various (query-related) metadata across multiple tables. |
| Outcome: | The proposed framework significantly improves overall retrieval performance over baselines. |
The Enemy from Within: A Study of Political Delegitimization Discourse in Israeli Political Speech (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new computational model for political delegitimization discourse is proposed for analysis of democratic discourse . we identify the importance of PDD as a powerful tool in political competition . |
| Approach: | They propose a computational classification pipeline for political delegitimization discourse . they annotate a Hebrew-language corpus of 10,410 sentences from parliamentary speeches, facebook posts and leading news outlets . |
| Outcome: | The proposed model achieves an F1 of 0.74 for binary detection and a macro-F1 of 0.6 for classification of delegitimization characteristics. |
Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment (2025.emnlp-main)
Copied to clipboard
Pedram Zaree, Md Abdullah Al Mamun, Quazi Mishkatul Alam, Yue Dong, Ihsen Alouani, Nael Abu-Ghazaleh
| Challenge: | Recent research has shown that carefully crafted jailbreak inputs can induce large language models to produce harmful outputs, despite safety measures such as alignment. |
| Approach: | They propose a method for generating highly effective Jailbreak attacks that selectively strengthen or weaken attention among different parts of the prompt. |
| Outcome: | The proposed attacks amplify the success rate of existing Jailbreak algorithms while lowering generation cost. |
Representation Potentials of Foundation Models for Multimodal Alignment: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | foundation models learn highly transferable representations through large-scale pretraining on diverse data. |
| Approach: | They examine the representation potentials of foundation models by examining their latent capacity to capture task-specific information within a single modality while providing a transferable basis for alignment and unification across modalities. |
| Outcome: | The foundation models exhibit remarkable similarities across architectures and modalities, the authors show . the models can capture task-specific information within a single modality while providing a transferable basis for alignment and unification across modality. |
Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional speculative decoding methods use a predefined length policy for proposing drafts, but the reality deviates from this assumption. |
| Approach: | They propose a self-verification length policy that adaptively determines the lengths of draft sequences by referring to the draft entropy. |
| Outcome: | The proposed method achieves 17% speedup on MT-Bench and 22% speedup in long-form reasoning. |
Visual-Aware Speech Recognition for Noisy Scenarios (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing audio-only models that use visual cues for transcription struggle in noisy environments. |
| Approach: | They propose a method that correlates visual cues with noise sources to improve transcription by filtering speech from noise and predicting noise labels in video inputs. |
| Outcome: | The proposed model improves transcription by correlating noise sources to visual cues in audio inputs. |
Advancing Arabic Diacritization: Improved Datasets, Benchmarking, and State-of-the-Art Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Arabic diacritics are typically omitted in written Arabic, leading to ambiguity . authors propose a methodology to analyze and refine a large diacritized corpus . |
| Approach: | They propose a methodology to analyze and refine a large diacritized corpus to improve training quality. |
| Outcome: | The proposed model achieves state-of-the-art results with 3.12% and 2.70% WER on WikiNews-2014 and Wikinews-2024. |
Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can underpin AI assistants that help users with everyday tasks, such as making recommendations or performing basic computation. |
| Approach: | They audit how six popular large language models (LLMs) complete 30 everyday tasks and compare them to 100 human crowdworkers from the US. |
| Outcome: | The LLMs perform 30 tasks and are compared to 100 human crowdworkers in the US. |
Dynamic Retriever for In-Context Knowledge Editing via Policy Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models excel at factual recall yet propagate stale or incorrect knowledge. |
| Approach: | They propose a framework that allows users to rank demonstrations by editing reward . it uses a *learnable threshold* to prune low-value examples, reducing edit success by 17.1% . |
| Outcome: | The proposed framework improves edit success by 17.1% and reduces latency by 41.6% on the CounterFact benchmark. |
LVLMs are Bad at Overhearing Human Referential Communication (2025.emnlp-main)
Copied to clipboard
| Challenge: | a crucial skill for embodied AI agents working with humans is grounding in referential communication. |
| Approach: | They use large vision language models to overhear spontaneous conversations between humans . they find that current LVLMs fail to show consistent performance improvement . |
| Outcome: | The proposed models fail to show consistent performance improvement over previous models . the authors release the results to facilitate future research . |
Let’s Reason Formally: Natural-Formal Hybrid Reasoning Enhances LLM’s Math Capability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has focused on improving the mathematical reasoning capabilities of Large Language Models (LLMs). |
| Approach: | They propose an end-to-end framework to integrate FL into NL math reasoning . they propose a problem alignment method that reformulates QA and existence problems . |
| Outcome: | The proposed framework achieves 89.80% and 84.34% accuracy rates on the MATH-500 and the AMC benchmarks. |
TORSO: Template-Oriented Reasoning Towards General Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate responses using few-shot examples depend on the provided examples, limiting the model’s reasoning capabilities. |
| Approach: | They propose a model that emulates human reasoning during response generation by using curated few-shot prompts instead of manually crafted few-shot examples. |
| Outcome: | The proposed model achieves strong performance on diverse LLMs benchmarks with reasonable rationales. |
Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild (2025.emnlp-main)
Copied to clipboard
| Challenge: | Rather than passively accepting output, users actively refine, explore, and co-construct text. |
| Approach: | They conduct a large scale analysis of user-LLM collaboration behavior with two popular AI assistants, Bing Copilot and WildChat. |
| Outcome: | The proposed models show that a small group of prototypical human AI collaboration behaviors explain a majority of the variation seen in user-LLM interaction. |
WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across various vision-language tasks. |
| Approach: | They propose a systematic taxonomy to evaluate MLLMs' ability to interpret real-world music scores and answer complex musicological queries. |
| Outcome: | The proposed model is based on real-world music scores and user-generated questions and discussions, and is scalable and controlled. |
TRIAL: Token Relations and Importance Aware Late-interaction for Accurate Text Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Late-interaction based multi-vector retrieval systems rely on a naive summation of token-level similarity scores . this leads to inaccurate relevance estimation due to tokenization of semantic units and the influence of low-content words. |
| Approach: | They propose a late-interaction-based multi-vector retrieval system that uses token relations and token importance in relevance scoring. |
| Outcome: | Extensive tests show that TRIAL achieves state-of-the-art accuracy compared to existing methods. |
Do Large Language Models excel in Complex Logical Reasoning with Formal Language? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LLMs have focused on formal language, but evaluations of their performance are limited. |
| Approach: | They propose to use a formal language to evaluate LLMs across logical reasoning problems using formal languages. |
| Outcome: | The proposed model outperforms Instruct models in three dimensions, taxonomy of tasks, and format of trajectories, and achieves the best generalization performance across other languages. |
Fair or Framed? Political Bias in News Articles Generated by LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have garnered significant attention for applications like news generation and opinion analysis. |
| Approach: | They analyze 10,850 articles and analyze their publicViews dataset to find left-leaning bias persists in generation tasks. |
| Outcome: | The proposed model size and the PublicViews dataset show that left-leaning bias persists in generation tasks and neutral content remains rare even under balanced opinion settings. |
ReviewRL: Towards Automated Scientific Review with RL (2025.emnlp-main)
Copied to clipboard
Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru Wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, Biqing Qi, Bowen Zhou
| Challenge: | Existing automated review systems struggle with factual accuracy, rating consistency, and analytical depth. |
| Approach: | They propose a framework for generating comprehensive and factually grounded scientific paper reviews using supervised fine-tuning and reinforcement learning. |
| Outcome: | The proposed framework outperforms existing methods on ICLR 2025 papers. |
Grammar Pruning: Enabling Low-Latency Zero-Shot Task-Oriented Language Models for Edge AI (2025.emnlp-main)
Copied to clipboard
Octavian Alexandru Trifan, Jason Lee Weber, Marc Titus Trifan, Alexandru Nicolau, Alexander Veidenbaum
| Challenge: | Existing approaches to task-oriented semantic parsers require high latency and extensive resource requirements. |
| Approach: | They propose a framework that couples a rule-based entity extractor with an iterative grammar-constrained decoder. |
| Outcome: | The proposed framework achieves an average execution accuracy of over 90% while sustaining at least 2x lower end-to-end latency than existing methods. |
Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequencies (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit unexpected failures in which they are confidently incorrect. |
| Approach: | They propose a method for calibrating SQL outputs that leverages structured nature to provide more granular signals of correctness. |
| Outcome: | The proposed method improves on two popular text-to-SQL datasets and provides a confidence score that is calibrated. |
REACT: Representation Extraction And Controllable Tuning to Overcome Overfitting in LLM Knowledge Editing (2025.emnlp-main)
Copied to clipboard
Haitian Zhong, Yuhuan Liu, Ziyang Xu, Guofan Liu, Qiang Liu, Shu Wu, Zhe Zhao, Liang Wang, Tieniu Tan
| Challenge: | Large language model editing methods suffer from overfitting, where factual updates can propagate beyond their intended scope, overemphasizing the edited target even when it’s contextually inappropriate. |
| Approach: | They propose a framework for precise and controllable knowledge editing that utilizes two-phase representations and a linear transformation to compute a directional "belief shift" vector. |
| Outcome: | The proposed framework significantly reduces overfitting across nearly all evaluation metrics and on COUNTERFACT and MQuAKE. |
ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models (LLMs) augmented with chain-of-thought (CoT) reasoning demonstrate impressive problem-solving abilities. |
| Approach: | They propose a weight-editing approach to reduce overly short reasoning by steering the model along a linear direction in the representation space. |
| Outcome: | The proposed model reduces overly short reasoning and yields significant accuracy gains on multiple math benchmarks. |
Incorporating Diverse Perspectives in Cultural Alignment: Survey of Evaluation Benchmarks Through A Three-Dimensional Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) serve diverse global audiences, making it critical for responsible AI deployment across cultures. |
| Approach: | They propose a framework that conceptualizes alignment along three dimensions: Cultural Group, Cultural Elements and Awareness Scope. |
| Outcome: | The proposed framework reveals critical gaps between benchmarks and real-world cultural biases . region dominates cultural group representation, social and political relations dominates coverage . majority of datasets adopt majority-focused Awareness Scope approaches . |
Are Large Language Models Chronically Online Surfers? A Dataset for Chinese Internet Meme Explanation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained on vast amounts of text from the Internet, but do they understand the viral content that rapidly spreads online? |
| Approach: | They introduce a dataset for CHinese Internet Meme Explanation that includes popular phrase-based memes from the Chinese Internet. |
| Outcome: | The proposed dataset includes popular phrase-based memes from the Chinese Internet, annotated with detailed information on their meaning, origin, example sentences, types, etc. |
RoDEval: A Robust Word Sense Disambiguation Evaluation Framework for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies rely on single-task evaluations and classification-based metrics that overlook the fundamental differences between generative LLMs and traditional classification models. |
| Approach: | They propose to use four new metrics to evaluate LLM-based word sense disambiguation (WSD) . experimental results reveal significant limitations in LLMs' WSD performance . |
| Outcome: | The proposed evaluation framework is open-source at https://github.com/DayDream405/RoDEval. |
PychoAgent: Psychology-driven LLM Agents for Explainable Panic Prediction on Social Media during Sudden Disaster Events (2025.emnlp-main)
Copied to clipboard
Mengzhu Liu, Zhengqiu Zhu, Chuan Ai, Chen Gao, Xinghong Li, Lingnan He, Kaisheng Lai, Yingfeng Chen, Xin Lu, Yong Li, Quanjun Yin
| Challenge: | Social media's rich information content and spatiotemporal granularity provide unique opportunities for emotion prediction and management. |
| Approach: | They propose a Psychology-driven generative Agent framework for explainable panic prediction based on emotion arousal theory. |
| Outcome: | The proposed framework improves panic emotion prediction performance by 13% to 21% compared to baseline models. |
Stepwise Reasoning Checkpoint Analysis: A Test Time Scaling Method to Enhance LLMs’ Reasoning (2025.emnlp-main)
Copied to clipboard
Zezhong Wang, Xingshan Zeng, Weiwen Liu, Yufei Wang, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Existing methods that use Chain-of-Thought suffer from path homogenization and inefficient use of intermediate results. |
| Approach: | They propose a framework that introduces checkpoints between reasoning steps to reduce path homogenization and create fault-tolerant mechanisms. |
| Outcome: | The proposed framework reduces path homogenization and creates fault-tolerant mechanism by utilizing high-quality intermediate results. |
Inter-sentence Context Modeling and Structure-aware Representation Enhancement for Conversational Sentiment Quadruple Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies struggle to capture complete dialogue semantics due to inadequate inter-utterance modeling and the underutilization of dialogue structure. |
| Approach: | They propose a model to extract dialogue aspect sentiment quadruples from dialogues using a sentence-by-sentence encoding module. |
| Outcome: | The proposed model extracts quadruples of target-aspect-opinion-sentiment from dialogues. |
Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing Large Language Models (LLMs) struggle with novelty and Reinforcement Learning from human feedback (RLHF) is costly. |
| Approach: | They propose to use a Reward Model (RM) and a principle-guided LLM-as-a-Judge to enhance creative output over baselines. |
| Outcome: | The proposed approach significantly enhances creative output over baselines, but the principle-guided LLM-as-a-Judge yields superior generation quality. |
Governance in Motion: Co-evolution of Constitutions and AI models for Scalable Safety (2025.emnlp-main)
Copied to clipboard
Chenhao Huang, Ziyu Shen, Yicong Ren, Huiyuan Zheng, Jiazheng Zhang, Mingxu Chai, Ming Zhang, Shihan Dou, Fan Mo, Jie Shi, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing approaches to align large language models with human preferences lack flexibility . static alignment preferences lack the ability to correct misaligned behaviors as they emerge . |
| Approach: | They propose a framework that enables dynamic and continuous alignment of large language models with human preferences. |
| Outcome: | The proposed framework improves safety and accuracy of a 7B model with human annotations. |
Web Intellectual Property at Risk: Preventing Unauthorized Real-Time Retrieval by Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new framework protects web content from unauthorized LLM real-time extraction and redistribution . multiple AI companies have been accused of scraping digital IP for proprietary benefit . |
| Approach: | They propose a defense framework that empowers web content creators to safeguard their web-based IP from unauthorized LLM real-time extraction and redistribution by leveraging the semantic understanding capability of LLMs themselves. |
| Outcome: | The proposed defense outperforms traditional defenses on LLMs and improves on black-box optimization problems. |
SciEvent: Benchmarking Multi-domain Scientific Event Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on scientific information extraction relies on entity-relation extraction in narrow domains . current models struggle in domains such as sociology and humanities . |
| Approach: | They propose a multi-domain benchmark for scientific abstract annotations using a unified event extraction schema. |
| Outcome: | The proposed benchmark includes 500 abstracts across five research domains with manual annotations of event segments, triggers, and fine-grained arguments. |
Media Source Matters More Than Content: Unveiling Political Bias in LLM-Generated Citations (2025.emnlp-main)
Copied to clipboard
| Challenge: | generative search engines rely on in-line citations as the key gateway to original webpages . a recent study shows that LLMs tend to cite left-leaning sources at higher rates compared to traditional retrieval systems . |
| Approach: | They construct a dataset of news articles labeled with left- or right-leaning stances . they find that LLMs tend to cite left-leansing sources at higher rates than traditional retrieval systems . |
| Outcome: | The proposed dataset shows that LLMs tend to cite left-leaning sources at higher rates than traditional retrieval systems. |
RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge graph question answering (KGQA) aims to answer natural language questions using knowledge graphs. |
| Approach: | They propose a framework that retrieves refined reasoning paths and evaluates their sufficiency. |
| Outcome: | The proposed framework outperforms existing baselines while enabling small open-source LLMs to achieve competitive results without fine-tuning LLM. |
Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that large language models exhibit social biases . however, debiasing methods may degrade the capabilities of LLMs if they are not properly evaluated . |
| Approach: | They propose a Japanese benchmark to evaluate social biases and cultural commonsense in large language models in a unified format. |
| Outcome: | The proposed method degrades the performance of the LLMs on the cultural commonsense task by 75%. |
Chameleon LLMs: User Personas Influence Chatbot Personality Shifts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have examined whether large language models adapt their perceived personalities in response to user interactions. |
| Approach: | They propose to use a controlled simulation to measure chatbot personality shifts before and after the interaction to determine whether LLMs exhibit conversational adaptations. |
| Outcome: | The proposed model exhibits personality adaptations over prolonged interactions, while Emotional Stability and Intellect remain relatively stable. |
GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models excel at factual recall, arithmetic reasoning, multi-turn dialogue . their capacity as askers, formulating strategic, adaptive, and information-seeking questions, remains less explored . |
| Approach: | They propose a protocol for evaluating large language models as strategic question-askers . they propose entropy-based methods that filter candidates via ConceptNet and Bayesian method that tracks belief updates over semantic concepts . |
| Outcome: | The proposed method is model-agnostic and supports post hoc analysis. |
SynC-LLM: Generation of Large-Scale Synthetic Circuit Code with Hierarchical Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent years, AI-assisted integrated circuit design methods have shown great potential in boosting IC design efficiency. however, this emerging technique is limited by the serious scarcity of publicly accessible large-scale circuit design data, which are mostly private IPs owned by semiconductor companies. |
| Approach: | They propose a hierarchical framework that exploits LLM's ability to generate new large-scale synthetic digital circuits by learning sequential logic skeletons and annotating function descriptions. |
| Outcome: | The proposed framework generates large-scale synthetic circuits that are valid and fully functional, and can significantly improve AI models’ performance in multiple IC design tasks. |
Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly reshaped software development practices, particularly in automating code generation and debugging. |
| Approach: | They propose to evaluate LLMs' capabilities on multi-hop error tracing and multi-bug detection in data science code debugging. |
| Outcome: | DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. |
Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are demanding more memory and computational resources . however, these devices typically feature weaker GPUs and stronger CPUs . |
| Approach: | They propose a lossless inference acceleration method that leverages the characteristics of heterogeneous devices and the advantages of speculative decoding. |
| Outcome: | The proposed method achieves speedups ranging from 1.79 to 10.1 across different devices . it uses a draft model on the GPU to perform preliminary predictions, while a target model on CPU validates these outputs . |
V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on causal interpretability focuses on large language models (LLMs) but internal mechanisms of vision-language models remain underexplored, authors say . |
| Approach: | They introduce a framework that combines visual and semantic manipulations for causal interpretation of vision-language models. |
| Outcome: | The proposed framework shows improved performance for LLAVA and InstructBLIP on three diverse benchmarks. |
LORAXBENCH: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | LORAXBENCH is a benchmark for low-resource languages of Indonesia . it covers reading comprehension, open domain QA, language inference, causal reasoning, translation, and cultural question answering across 20 languages. |
| Approach: | They propose a benchmark that focuses on low-resource languages of Indonesia and covers 6 diverse tasks: reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural question answering. |
| Outcome: | The proposed benchmark covers reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural question answering across 20 Indonesian languages. |
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reward models assume a global reward function, limiting personalization and pluralistic alignment. |
| Approach: | They propose a framework that leverages binary preference datasets to enhance personalized preference learning. |
| Outcome: | The proposed framework captures diverse human preferences without fine-grained annotations and significantly improves personalized preference learning on downstream tasks. |
SAFE: Schema-Driven Approximate Distance Join for Efficient Knowledge Graph Querying (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods map an LLM-generated query graph onto the KG or let the LLM traverse the entire graph. |
| Approach: | They propose a framework that leverages schema graphs for robust query graph generation and efficient KG retrieval. |
| Outcome: | Extensive experiments on WebQSP, CWQ and GrailQA show that the proposed framework outperforms state-of-the-art methods in accuracy and efficiency. |
Structured Preference Optimization for Vision-Language Long-Horizon Task Planning (2025.emnlp-main)
Copied to clipboard
Xiwen Liang, Min Lin, Weiqi Ruan, Rongtao Xu, Yuecheng Liu, Jiaqi Chen, Bingqian Lin, Yuzheng Zhuang, Xiaodan Liang
| Challenge: | Existing vision-language planning methods struggle with long-horizon reasoning in dynamic environments due to the difficulty of training models to generate high-quality reasoning processes. |
| Approach: | They propose a framework that enhances reasoning and action selection for long-horizon task planning through structured evaluation and optimized training. |
| Outcome: | The proposed framework outperforms existing methods on short-horizon tasks but struggles with long-horizon reasoning in dynamic environments. |
Position: LLMs Can be Good Tutors in English Education (2025.emnlp-main)
Copied to clipboard
Jingheng Ye, Shen Wang, Deqing Zou, Yibo Yan, Kun Wang, Hai-Tao Zheng, Ruitong Liu, Zenglin Xu, Irwin King, Philip S. Yu, Qingsong Wen
| Challenge: | Recent efforts to integrate large language models into English education lack adaptability to language learning. |
| Approach: | They argue that large language models can be effective tutors in English education . they encourage interdisciplinary research to explore these roles, fostering innovation and risks . |
| Outcome: | The proposed models can play three critical roles: 1) as data enhancers, 2) as task predictors, 3) as agents, enabling personalized and inclusive education. |
CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecasting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing environmental forecasting research focuses on predicting numerical meteorological variables, neglecting the translation of these variables into actionable textual narratives of events and their consequences. |
| Approach: | They propose a task that leverages numerical meteorological raster data and textual event data to predict weather and climate events. |
| Outcome: | The proposed task leverages numerical meteorological raster data and textual event data to predict weather and climate events. |
Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work relies on training with multi-lingual ability-related data, which may not be available for low-resource languages. |
| Approach: | They propose a multi-lingual ability-enhanced LLM that extracts language-agnostic ability-related weights from LLMs and combine them across different languages by simple addition and subtraction operations without training. |
| Outcome: | The proposed approach extracts language-agnostic ability-related weights from LLMs and combine them across different languages without training. |
Evaluating the Effectiveness and Scalability of LLM-Based Data Augmentation for Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research does not explore key factors such as optimal augmentation scale and the necessity of using large augmentation models. |
| Approach: | They propose to use LLMs to augment compact dual-encoder models to improve retrieval performance. |
| Outcome: | The proposed approach improves retrieval performance but its benefits diminish beyond a certain scale even with diverse augmentation strategies. |
Temporal Referential Consistency: Do LLMs Favor Sequences Over Absolute Time References? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing efforts to ensure temporal consistency in large language models are lacking in time-sensitive fields . temporal reasoning is essential for time- sensitive fields such as finance and healthcare . a new benchmark aims to improve temporal referent consistency of LLMs . |
| Approach: | They propose a temporal referential consistency benchmark with a resource TEMP-ReCon to assess LLMs across temporal references. |
| Outcome: | The proposed model improves LLMs' temporal consistency by comparing them to baseline models. |
MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation approaches focus on mLLMs’ detection accuracy for binary classification tasks, which often fail to reflect the in-depth interpretive nuance of harmfulness across diverse contexts. |
| Approach: | They propose an agent-based arena-style evaluation framework that provides context-aware and unbiased assessment for mLLMs’ understanding of multimodal harmfulness. |
| Outcome: | The proposed framework reduces evaluation biases of judge agents and provides unbiased comparisons of mLLMs’ abilities to interpret multimodal harmfulness. |
Multi-perspective Analysis of Large Language Model Domain Specialization: An Experiment in Accounting Audit Procedures Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior research has examined similarities and differences between these approaches in task-specific capabilities, but little is known about how they affect the feature of the generated text itself. |
| Approach: | They compare two different domain specialization approaches for Large Language Models, fine-tuning and In-Context Learning, and their combined approaches. |
| Outcome: | The proposed model shows consistent differences in output characteristics when comparing fine-tuning, ICL, and their combined approaches. |
Generator-Assistant Stepwise Rollback Framework for Large Language Model Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to integrate thoughts with actions can cause irreversible error propagation . Xi et al., 2023; Zhang eet coll., 2023) have focused on enhancing large language model (LLM) agents capable of helping humans tackle real-world challenges. |
| Approach: | They propose a framework called Generator-Assistant Stepwise Rollback to induce better decision-making for LLM agents by integrating a generator and an assistant to examine each action produced by the generator. |
| Outcome: | The proposed framework improves on three widely used benchmarks and can integrate seamlessly with other methods. |
DocAgent: An Agentic Framework for Multi-Modal Long-Context Document Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to document understanding are limited due to limited context length or fail to fully leverage multi-modal information. |
| Approach: | They propose a multi-agent framework for long-context document understanding that imitates human reading practice. |
| Outcome: | The proposed framework surpasses human-level benchmarks on long-context document understanding while maintaining a short context length. |
EasyRec: Simple yet Effective Language Models for Recommendation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning from user-item interaction data rely on unique user and item IDs, which limits their performance in zero-shot learning scenarios. |
| Approach: | They propose an approach that integrates text-based semantic understanding with collaborative signals. |
| Outcome: | The proposed approach outperforms state-of-the-art models in zero-shot recommendation scenarios. |
From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are catalyzing a paradigm shift in scientific discovery, evolving from task-specific automation tools into increasingly autonomous agents. |
| Approach: | They introduce a foundational three-level taxonomy to delineate their escalating autonomy and evolving responsibilities within the research lifecycle. |
| Outcome: | The proposed frameworks provide a conceptual architecture and strategic foresight to navigate and shape the future of AI-driven scientific discovery. |
Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) often display unstable behaviors, e.g., hallucinating unsupported premises, overthinking simple tasks, and displaying higher sensitivity to prompt variations. |
| Approach: | They propose a graph-based analytical framework that clusters long, verbose CoT outputs into semantically coherent reasoning steps, then constructs directed reasoning graphs to capture contextual and logical dependencies among these steps. |
| Outcome: | The proposed framework enables quantitative evaluation of internal reasoning structure and quality beyond conventional metrics and provides practical insights for prompt engineering and cognitive analysis of LLMs. |
ViPE: Visual Perception in Parameter Space for Efficient Video-Language Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing video-language models rely on concatenating visual tokens with textual inputs for joint modeling, but this method suffers from significant inefficiency when scaling to long videos with dense visual inputs. |
| Approach: | They propose a video-to-parameter efficiency paradigm called ViPE that transforms video content into visual perceptual weights, which are directly injected into the LLM’s parameters. |
| Outcome: | The proposed model reduces FLOPs by 85% and inference time by up to 65% while reducing FLOP and FLOP inference times by up-to-65%. |
Alignment for Efficient Tool Calling of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in tool learning have enabled large language models to integrate external tools, enhancing their task performance by expanding their knowledge boundaries. |
| Approach: | They propose a framework that combines probabilistic knowledge boundary estimation with dynamic decision-making to allow LLMs to better assess when to invoke tools based on their confidence. |
| Outcome: | The proposed framework shows significant improvements in tool efficiency by reducing unnecessary tool usage. |
ToM: Leveraging Tree-oriented MapReduce for Long-Context Reasoning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show ToM outperforms existing divide-and-conquer frameworks . RAG relies on similarity-based rankings to retrieve and reason over chunks based on logical coherence . |
| Approach: | They propose a Tree-oriented MapReduce framework for long-context reasoning . it leverages the hierarchical structure of long documents by constructing a DocTree . |
| Outcome: | Experimental results show that ToM outperforms existing divide-and-conquer frameworks and RAGs . the proposed framework improves logical coherence and long-context reasoning on 70B+ LLMs compared to existing approaches . |
BANMIME : Misogyny Detection with Metaphor Explanation on Bangla Memes (2025.emnlp-main)
Copied to clipboard
Md Ayon Mia, Akm Moshiur Rahman Mazumder, Khadiza Sultana Sayma, Md Fahim, Md Tahmid Hasan Fuad, Muhammad Ibrahim Khan, Akmmahbubur Rahman
| Challenge: | Existing studies have explored hate speech and general meme classification, but the nuanced identification of misogyny in Bangla memes remains underexplored. |
| Approach: | They propose a Bangla misogynistic meme dataset that includes misos, humor, metaphors and detailed human-written explanations. |
| Outcome: | The proposed dataset is the first comprehensive dataset of misogynistic Bangla memes . it includes misos, humor categories, metaphor localization, and detailed human-written explanations based on 2,000 culturally grounded samples . |
Phi: Preference Hijacking in Multi-modal Large Language Models at Inference Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models have raised serious safety concerns. |
| Approach: | They propose a method for manipulating the output preference of MLLMs using a preference hijacked image. |
| Outcome: | The proposed method works at inference time and requires no model modifications. |
Retrieval-augmented GUI Agents with Generative Guidelines (2025.emnlp-main)
Copied to clipboard
| Challenge: | GUI agents powered by vision-language models struggle with real-world tasks due to their complex nature and limited training data. |
| Approach: | They propose a lightweight vision-language model that leverages web tutorials at inferencetime to synthesize GUI agents. |
| Outcome: | The proposed agent outperforms baseline GUI agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes. |
COAS2W: A Chinese Older-Adults Spoken-to-Written Transformation Corpus with Context Awareness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on spoken-to-written transformations from older adults' language is limited by omission, disordered syntax, constituent errors, and redundancy. |
| Approach: | They propose to combine a spoken-to-written corpus of 10,004 utterances from older adults with a written version, fine-grained error labels, and four-sentence context. |
| Outcome: | The proposed model outperforms closed-source models on Chinese spoken-to-written corpus and shows that multi-sentence input is more efficient. |
Answer Convergence as a Signal for Early Stopping in Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | a systematic study suggests that chain-of-thought prompting is unnecessary for producing correct answers. |
| Approach: | They propose three inference-time strategies to improve model efficiency by boosting end-of-reasoning signals and early stopping . they propose a method that learns when to stop based on internal activations . |
| Outcome: | The proposed methods reduce token usage with little or no accuracy drop on natural questions . the proposed methods also reduce tokens by over 40% on naturalquestions . |
VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focuses on accuracy and precision, but factuality evaluation is difficult due to inter-sentence dependencies. |
| Approach: | They introduce a factuality evaluation framework to enhance fact extraction . they also introduce 'factRBench' that evaluates both precision and recall . |
| Outcome: | The proposed framework enhances fact extraction by identifying incomplete and missing facts . it also evaluates precision and recall in long-form models, whereas prior work focuses on precision. |
SQUAB: Evaluating LLM robustness to Ambiguous and Unanswerable Questions in Semantic Parsing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Practical user questions often deviate from ideal conditions, challenging the applicability of existing benchmarks. |
| Approach: | They propose an automatic dataset generator of Ambiguous and Unanswerable questions that generates complex, annotated SP tests using a blend of SQL and LLM capabilities. |
| Outcome: | The proposed framework reduces test generation costs by up to 99% while aligning with real-world question patterns. |
Reliable Evaluation and Benchmarks for Statement Autoformalization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating statement autoformalization are limited . current methods can achieve up to 45.1% accuracy on undergraduate mathematics . |
| Approach: | They propose a new autoformalization metric that correlates strongly with human judgment . they propose two new auto-formalisation benchmarks: ProofNet# and RLM25 . |
| Outcome: | The proposed methods can achieve up to 45.1% accuracy on undergraduate mathematics but struggle with research-level content without proper context. |
VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Identifying and addressing potential social biases is essential to prevent harm to users. |
| Approach: | They examine explicit and implicit biases exhibited by Vision-Language Models . they pose questions related to gender and racial differences to test their models . |
| Outcome: | The proposed models are used in image description tasks, form completion tasks and medical applications. |
Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions (2025.emnlp-main)
Copied to clipboard
| Challenge: | reducing the size of LLMs through post-training pruning has been studied, but its impact on model fairness remains unexplored. |
| Approach: | They propose a pruning method that removes parameters that are redundant for input processing but influential in output generation. |
| Outcome: | The proposed pruning method can maintain or improve fairness across models and tasks where existing methods have limitations. |
AI Sees Your Location—But With A Bias Toward The Wealthy World (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have shown remarkable performance across various tasks, particularly in recognizing geographic information from images. |
| Approach: | They propose to use 1,200 images paired with detailed geographic metadata to evaluate VLMs' performance. |
| Outcome: | The models achieve 53.8% accuracy in city prediction, but exhibit significant biases in regional tasks. |
Faster In-Context Learning for LLMs via N-Gram Trie Speculative Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) is a key method in prompt engineering, but its long retrieved contexts and limited token throughput will slow reasoning speeds. |
| Approach: | They propose a method that leverages the overlap between context and model output to generate drafts from the context. |
| Outcome: | The proposed method achieves the highest mean speedup on Vicuna-7B, Llama2-7B-Chat, and Llma3-8B-Instruct tasks. |
From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Adapting cultural values in Large Language Models presents significant challenges due to biases and data limitations. |
| Approach: | They propose to augment World Values Survey (WVS) data with encyclopedic and scenario-based cultural narratives from Wikipedia and NormAd to address these limitations. |
| Outcome: | The proposed approach enhances cultural distinctiveness and improves classification performance across cultures. |
Iterative Prompt Refinement for Safer Text-to-Image Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety methods for text-to-image models ignore the images produced . this can result in unsafe outputs or unnecessary changes to already safe prompts . |
| Approach: | They propose an iterative prompt refinement algorithm that uses Vision Language Models to analyze prompts and generated images. |
| Outcome: | The proposed method improves safety while maintaining user intent and reliability comparable to existing methods. |
Language Models as Continuous Self-Evolving Data Engineers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their further evolution is often hampered by the scarcity of high-quality training data and the heavy reliance of traditional methods on expert-labeled data. |
| Approach: | They propose a paradigm that enables LLMs to train themselves by generating, cleaning, reviewing and annotating data with preference information. |
| Outcome: | The proposed model can generate, clean, review, and annotate data with preference information significantly reducing time and cost of post-training data construction. |
Unilaw-R1: A Large Language Model for Legal Reasoning with Reinforcement Learning and Iterative Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reasoning-focused large language models (LLMs) are rapidly evolving across various domains, yet their capabilities in handling complex legal problems remain underexplored. |
| Approach: | They propose a large language model tailored for legal reasoning with a 7-billion parameter scale and a two-stage training strategy combining Supervised Fine-Tuning and Reinforcement Learning. |
| Outcome: | The proposed model outperforms all models of similar scale on authoritative benchmarks and outperformed Qwen-2.5-7B-Instruct (46.6%) by an average margin of 6.6%. |
Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios (2025.emnlp-main)
Copied to clipboard
Yunkai Dang, Mengxi Gao, Yibo Yan, Xin Zou, Yanggan Gu, Jungang Li, Jingyu Wang, Peijie Jiang, Aiwei Liu, Jia Liu, Xuming Hu
| Challenge: | Existing studies have focused mainly on visual–textual misalignment, leaving largely unexplored the MLLMs’ ability to preserve an original correct answer when confronted with misleading information. |
| Approach: | They propose a two-stage evaluation pipeline to quantify the response uncertainty phenomenon by eliciting each model’s original response on unperturbed inputs and injecting explicit (false-answer hints) and implicit (contextual contradictions) misleading instructions. |
| Outcome: | The proposed model overturns a correct answer in 65% of cases after receiving a single deceptive cue. |
Evaluating and Aligning Human Economic Risk Preferences in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used in decision-making scenarios that involve risk assessment, yet their alignment with human economic rationality remains unclear. |
| Approach: | They propose an evaluation metric called Risk Disparity Score (RDS) and assess whether LLM-generated responses reflect appropriate levels of risk aversion or risk-seeking behavior based on individual’s persona. |
| Outcome: | The proposed evaluation metric assesses whether LLM-generated responses reflect appropriate levels of risk aversion or risk-seeking behavior based on individual’s persona. |
Ensembling Prompting Strategies for Zero-Shot Hierarchical Text Classification with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Hierarchical text classification is a challenging task in natural language processing. |
| Approach: | They propose a method which integrates the results of diverse prompting strategies to promote LLMs’ reliability. |
| Outcome: | The proposed method boosts the performance of single prompting strategies and achieves SOTA results on three benchmark datasets. |
Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have revealed that tokenizers can be exploited to elicit unwanted behavior. |
| Approach: | They propose to exploit incomplete tokens with stray bytes to exploit their dependency . they propose to use improbable bigrams to exploit the dependency of their adjacent tokens . |
| Outcome: | The proposed tokenizers can be exploited to elicit unwanted behavior in language models. |
UI-Hawk: Unleashing the Screen Stream Understanding for Mobile GUI Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing GUI agents depend on current visual observations and plain-text action history, ignoring the significance of history screens. |
| Approach: | They propose a multi-modal GUI agent specifically designed to process screen streams . they propose UI-Hawk incorporates a history-aware visual encoder to handle the sequences . |
| Outcome: | The proposed GUI agent can process screen streams encountered during GUI navigation. |
UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debugging (2025.emnlp-main)
Copied to clipboard
Cheryl Lee, Chunqiu Steven Xia, Longji Yang, Jen-tse Huang, Zhouruixing Zhu, Lingming Zhang, Michael R. Lyu
| Challenge: | Existing LLMs focus on isolated steps and struggle with complex bugs. |
| Approach: | They propose a framework for unified debugging through multi-agent synergy . it mimics the entire cognitive processes of developers with each agent specialized as a particular component of this process . |
| Outcome: | The proposed framework outperforms state-of-the-art methods on repo-level benchmarks. |
Understanding the Thinking Process of Reasoning Models: A Perspective from Schoenfeld’s Episode Theory (2025.emnlp-main)
Copied to clipboard
Ming Li, Nan Zhang, Chenrui Fan, Hong Jiao, Yanbin Fu, Sydney Peters, Qingshu Xu, Robert Lissitz, Tianyi Zhou
| Challenge: | Large Reasoning Models (LRMs) generate extensive chain-of-thought reasoning, but we lack a principled framework for understanding how these thoughts are structured. |
| Approach: | They propose a method to analyze the reasoning traces of Large Reasoning Models using Schoenfeld’s Episode Theory. |
| Outcome: | The proposed framework provides a theoretically grounded methodology for interpreting LRM cognition and enables future work on more controllable and transparent reasoning systems. |
Thread: A Logic-Based Data Organization Paradigm for How-To Question Answering with Retrieval Augmented Generation (2025.emnlp-main)
Copied to clipboard
Kaikai An, Fangkai Yang, Liqun Li, Junting Lu, Sitao Cheng, Shuzheng Si, Lu Wang, Pu Zhao, Lele Cao, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Baobao Chang
| Challenge: | Recent advances in retrieval-augmented generation (RAG) have substantially improved question-answering systems, particularly for factoid ‘5Ws’ questions. |
| Approach: | They propose a data organization paradigm where large language models transform documents into more structured and loosely interconnected LUs. |
| Outcome: | Experiments in open-domain and industrial settings show that the proposed paradigm outperforms existing paradigms and shows high adaptability across diverse document formats. |
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern WQE techniques rely on expensive inference with large language models or ad-hoc training with large amounts of human-labeled data. |
| Approach: | They propose to use word-level quality estimation to identify translation errors from the inner workings of translation models to quantify the impact of human label variation on metric performance. |
| Outcome: | The proposed methods identify translation errors from the inner workings of translation models using human labels. |
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can adapt outputs to align with community-specific norms, perspectives and communication styles. |
| Approach: | They propose a benchmark to assess community-specific steering using contrasting reddit communities. |
| Outcome: | STEER-BENCH assesses how well large language models understand community-specific instructions, their resilience to adversarial steering attempts, and their ability to accurately represent cultural and ideological perspectives. |
Combining Constrained and Unconstrained Decoding via Boosting: BoostCD and Its Application to Information Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to structured NLP tasks use autoregressive models trained on pairs of unstructured input text and structured output targets. |
| Approach: | They propose a model that combines constrained and unconstrained decoding in two phases to achieve two weak predictions. |
| Outcome: | The proposed model outperforms previous approaches both in and out of distribution, addressing several common errors identified in those approaches. |
MultiLogicNMR(er): A Benchmark and Neural-Symbolic Framework for Non-monotonic Reasoning with Multiple Extensions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Non-monotonic reasoning is widely used in daily life and legal reasoning. |
| Approach: | They propose a neural-symbolic framework for multi-extension NMR and propose to build two variants with more extensions or text diversity. |
| Outcome: | The proposed framework outperforms prompt-based methods and outperformed some fine-tuning methods. |
Beyond Demographics: Enhancing Cultural Value Survey Simulation with Multi-Stage Personality-Driven Cognitive Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Introducing **MARK**, a framework for cultural value survey simulation . based on type dynamics theory, it improves accuracy and interpretation of models . |
| Approach: | They propose a framework that integrates psychological theory into cultural value survey simulations. |
| Outcome: | The proposed framework outperforms baseline models on the World Values Survey by 10% accuracy and reduces divergence between model predictions and human preferences. |
CrystalICL: Enabling In-Context Learning for Crystal Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for crystal generation are limited to zero-shot scenarios and are unable to benefit from few-shot situations. |
| Approach: | They propose a model designed for few-shot crystal generation that exploits in-context learning by capturing structure-property relationships from limited data. |
| Outcome: | The proposed model reduces complexity of modeling crystal symmetry in LLMs and exploits ICL by capturing structure-property relationships from limited data. |
Towards a Unified Paradigm of Concept Editing in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Concept editing aims to control specific concepts in large language models (LLMs) however, there is a lack of rigorous theoretical analysis and a unified perspective to systematically understand and compare these methods. |
| Approach: | They propose a paradigm where conceptual injection is aligned at the neuron level. |
| Outcome: | The proposed paradigm offers a clear framework and valuable insights for advancing interpretability and controlled generation in large language models. |
Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models (2025.emnlp-main)
Copied to clipboard
Kaiyan Chang, Yonghao Shi, Chenglong Wang, Hang Zhou, Chi Hu, Xiaoqian Liu, Yingfeng Luo, Yuan Ge, Tong Xiao, JingBo Zhu
| Challenge: | Recent training-based TTS methods, such as continued reinforcement learning, have surged in popularity, while training-free TTS approaches are gradually fading from prominence. |
| Approach: | They propose a fine-grained sequential scaling method guided by process verification that integrates training-free TTS methods with other classical parallel scaling methods at the step level. |
| Outcome: | Experiments on five instruction-tuned large language models (LLMs) show that training-free TTS methods can extend reasoning performance boundaries. |
Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to adapt Mixture-of-Experts models to multiple domains are prohibitive computation, cross-domain interference or require separate runs per domain. |
| Approach: | They propose a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models. |
| Outcome: | The proposed framework reduces forgetting by 89% compared to full fine-tuning as domains scale from 2 to 6 and achieves faster convergence than conventional methods. |
RRInf: Efficient Influence Function Estimation via Ridge Regression for Large Language Models and Text-to-Image Diffusion Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Influence function estimates the impact of training data on model predictions . high computational cost has hindered their applicability in large-scale applications. |
| Approach: | They propose a method to quantify the impact of training data on model predictions . they use a ridge regression problem to transform the estimation into a problem . |
| Outcome: | The proposed method outperforms existing methods on noisy data detection and influential data identification tasks. |
Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate instructions using large language models require spatiotemporal awareness of multiple objects and their surroundings. |
| Approach: | They propose a tree-based evaluation metric for LLM-generated step-by-step assembly instructions that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. |
| Outcome: | The proposed metric better correlates with manually-annotated error counts, and is more robust against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity. |
MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang
| Challenge: | Large language models (LLMs) have demonstrated exceptional capabilities across diverse tasks, but their fine-tuning requires significant memory, posing challenges for resource-constrained environments. |
| Approach: | They propose a ZO-based framework that eliminates the need for backpropagation and provides a memory-efficient alternative to backprograming. |
| Outcome: | The proposed framework surpasses first-order methods in performance and accuracy. |
Procedural Environment Generation for Tool-Use Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to curation of tool-use training data are non-interactive and/or non-compositional. |
| Approach: | They propose a pipeline for the procedural generation of interactive tools and compositional tool-use data. |
| Outcome: | The proposed pipeline improves on a range of tool-use benchmarks and sets the new SoTA for two metrics on the NESTFUL dataset. |
FacLens: Transferable Probe for Foreseeing Non-Factuality in Fact-Seeking Question Answering of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing non-factuality detection methods require response generation, which incurs significant computational overhead. |
| Approach: | They propose a lightweight model called Factuality Lens which effectively probes hidden representations of fact-seeking questions for the NFP task. |
| Outcome: | The proposed model is able to probe hidden representations of fact-seeking questions and reduce development costs. |
OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Keyword decision in Sponsored Search Advertising is critical to the success of ad campaigns. |
| Approach: | They propose a keyword generation framework that is On-the-fly and Multi-objective to automate keyword generation. |
| Outcome: | Experiments show that OMS outperforms existing methods in keyword generation . relying on large-scale query-keyword data is a major limitation, authors say . |
Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have shown strong performance in tasks like radiology report generation but struggle with hallucinations, vague descriptions, Inconsistent logic and poor localization. |
| Approach: | They propose a framework for medical visual reasoning based on Visual Guidance and Self-Reward paradigms and Monte Carlo Tree Search to improve the model's visual reasoning capabilities. |
| Outcome: | The proposed framework outperforms existing models on multiple medical VQA benchmarks. |
TrojanWave: Exploiting Prompt Learning for Stealthy Backdoor Attacks on Large Audio-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that ALMs are vulnerable to adversarial attacks. |
| Approach: | They propose a backdoor attack tailored to the prompt-learning setting in frozen audio-language models. |
| Outcome: | The proposed method injects backdoors solely through learnable prompts, making it highly scalable and effective in few-shot settings. |
Can LLMs be Literary Companions?: Analysing LLMs on Bengali Figures of Speech Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | despite Bengali being among the most spoken languages, the NLP efforts on it remain limited. |
| Approach: | They present a dataset that includes Bengali figures of speech on six poets . they deploy state-of-the-art Large Language Models to the dataset and fine-tune the best models . |
| Outcome: | The proposed dataset reveals that two open-source LLMs perform better than others in Bengali . the framework can be reproduced for English and other low-resource languages . |
Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) are a promising unsupervised approach for understanding the representations of layers of Large Language Models (LLMs). |
| Approach: | They propose a method that groups similar models and trains a single SAE per group based on representational similarity across layers. |
| Outcome: | Experiments on Pythia family models show that the proposed method significantly accelerates training with minimal impact on reconstruction quality and comparable downstream task performance and interpretability over baseline SAEs trained layer by layer. |
Retrieval over Classification: Integrating Relation Semantics for Multimodal Relation Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multimodal relation extraction ignore structural constraints and lack semantic expressiveness for fine-grained relation understanding. |
| Approach: | They propose a framework that reformulates multimodal relation extraction as a retrieval task driven by relation semantics. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the benchmark datasets MNRE and MORE and exhibits stronger robustness and interpretability. |
PunMemeCN: A Benchmark to Explore Vision-Language Models’ Understanding of Chinese Pun Memes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pun memes combine wordplay with visual elements to create humor, irony, or other rhetorical effects. |
| Approach: | They propose a benchmark to assess Chinese pun memes' processing capabilities across three progressive tasks: pun meme detection, sentiment analysis, and chat-driven meme response. |
| Outcome: | The proposed model can detect pun memes, analyze sentiments, and respond to chats, while ignoring homophone wordplay. |
UltraIF: Advancing Instruction Following from the Wild (2025.emnlp-main)
Copied to clipboard
| Challenge: | a lack of transparency has resulted in a gap between research community and leading companies . large language models have demonstrated remarkable capabilities in following complex instructions . |
| Approach: | They propose a method to build large language models that can follow complex instructions with open-source data. |
| Outcome: | The proposed approach can synergize complex instructions and filter responses with evaluation questions. |
Identifying Pre-training Data in LLMs: A Neuron Activation-Based Detection Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting pre-training data in large language models rely on superficial features like prediction confidence and loss, resulting in mediocre performance. |
| Approach: | They propose a new algorithm to analyze neuron activation patterns between training and non-training data in large language models to improve their performance. |
| Outcome: | The proposed algorithm outperforms existing methods across three benchmarks and multiple LLMs. |
TreeRare: Syntax Tree-Guided Retrieval and Reasoning for Knowledge-Intensive Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work shows that large language models generate incorrect statements due to over-reliance on parametric knowledge. |
| Approach: | They propose a framework that utilizes syntax trees to guide information retrieval and reasoning for question answering. |
| Outcome: | The proposed framework improves on existing state-of-the-art methods for large-scale query processing. |
Mapping Toxic Comments Across Demographics: A Dataset from German Public Broadcasting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing toxic speech datasets lack demographic context and age data are limited . funk and its subsidiary accounts target users aged 14-29 . |
| Approach: | a german project introduces a large-scale toxic speech dataset annotated for toxicity . the dataset includes 3,024 human-annotated and 30,024 LLM-annnotated comments . researchers used human expertise and state-of-the-art language models to label comments based on toxic keywords . |
| Outcome: | The study combines human expertise with state-of-the-art language models to identify toxic speech categories. |
Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition (2025.emnlp-main)
Copied to clipboard
Danielle Cohen, Yoni Halpern, Noam Kahlon, Joel Oren, Omri Berkovitch, Sapir Caduri, Ido Dagan, Anatoly Efros
| Challenge: | Large multi-modal large language models are good at extracting user intents from UI sequences, but smaller models struggle with accurate intent inference. |
| Approach: | They propose a decomposed approach for extracting user intent from small models . they perform structured interaction summarization and intent extraction using a fine-tuned model . |
| Outcome: | The proposed method surpasses the performance of large MLLMs in the intent extraction task. |
On Pruning State-Space LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposed state-space models as an efficient alternative to transformers. |
| Approach: | They propose to prune state-space models (SSMs) to reduce computation costs by using unstructured pruning methods. |
| Outcome: | The proposed pruning methods show that they can be pruned to reduce their computation costs. |
An Orthogonal High-Rank Adaptation for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) efficiently adapts LLMs to downstream tasks by decomposing LLM’s weight update into trainable low-rank matrices for fine-tuning. |
| Approach: | They propose an orthogonal high-rank adaptation for parameter-efficient fine-tuning that decomposes LLMs’ pre-trained weight matrices into orthogonals via QR decomposition and splits them into two low-redundancy high-ranked components. |
| Outcome: | Empirical results show that OHoRA outperforms LoRA and its variants and generates task-tailored representation spaces with 0.0371% trainable parameters. |
BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies highlight a fundamental dichotomy in deep learning optimization: parameter updates along the top eigendirections of the loss Hessian (Dom-space) capture most of the update magnitude, while updates in the orthogonal component (Bulk-space), have smaller magnitudes but drive most learning progress. |
| Approach: | They propose a plug-and-play framework that scales update components projected onto distinct subspaces and a block-wise strategy that applies this estimation on a per-parameter-block basis. |
| Outcome: | The proposed framework accelerates training by differentially scaling update components projected onto distinct subspaces, while enhancing stability by moderating updates in dominant subspace and boosting convergence speed by amplifying updates in bulk-space. |
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating debate speeches requires a deep understanding of arguments at multiple levels. |
| Approach: | They propose a benchmark task for LLM judges based on annotated debate speeches . they analyze the judgment capabilities and behavior of frontier LLMs . |
| Outcome: | The proposed task requires a comprehensive understanding of argumentation and its arguments. |
METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Video Large Language Models (VLLMs) have significantly enhanced their ability to understand video content. |
| Approach: | They propose a training-free, Multi-stage Event-based Token compression framework that eliminates redundant visual tokens across three critical stages . |
| Outcome: | The proposed framework reduces FLOPs and KV Cache memory consumption while maintaining comparable or even superior accuracy. |
VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) suffer from significant computational overhead due to the quadratic growth of attention computations with the number of multimodal tokens. |
| Approach: | They propose a training-free pruning framework that prunes multimodal tokens without a trained pruning method. |
| Outcome: | The proposed pruning framework outperforms existing token pruning methods and generalizes across diverse MLLMs. |
Beyond Static Testbeds: An Interaction-Centric Agent Simulation Platform for Dynamic Recommender Systems (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing platforms lack a mechanism for user actions to dynamically reshape the environment. |
| Approach: | They propose a novel agent-based simulation platform for recommender systems with a robust interaction mechanism. |
| Outcome: | The proposed platform improves the credibility of the simulation and replicates the Matthew Effect and Brand Loyalty. |
SheetDesigner: MLLM-Powered Spreadsheet Layout Generation with Rule-Based and Vision-Based Reflection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing automated layout models are ill-suited for spreadsheets, authors say . existing layout models treat components as rectangles with continuous coordinates . authors: spreadsheets are powerful tools for organizing and analyzing data . |
| Approach: | They formalize a spreadsheet layout generation task and introduce a framework for spreadsheet layouts . they use multimodal large language models to combine rule and vision reflection . |
| Outcome: | The proposed framework outperforms baselines in a spreadsheet layout generation task by 22.6%. |
CAIR: Counterfactual-based Agent Influence Ranker for Agentic AI Workflows (2025.emnlp-main)
Copied to clipboard
Amit Giloni, Chiara Picardi, Roy Betser, Shamik Bose, Aishvariya Priya Rathina Sabapathy, Roman Vainshtein
| Challenge: | Existing methods to assess the influence of each agent on the AAW’s output perform only static structural analysis, which is unsuitable for inference time execution. |
| Approach: | They propose to use an LLM-based agent influence Ranker to assess the influence level of each agent on the AAW's output and determine which agents are the most influential. |
| Outcome: | The proposed method outperforms baseline methods and produces consistent rankings and relevancy of downstream tasks. |
ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods address data quality via static prefiltering, which decouples quality control from training and fails to mitigate turn-level error propagation. |
| Approach: | They propose an adaptive learning method that dynamically down-weights unreliable supervision without explicit filtering. |
| Outcome: | Experiments on single-source and mixed-quality datasets show improved stability and response quality. |
Precise In-Parameter Concept Erasure in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for erasing concepts from model parameters are either too coarse, shallow or ineffective. |
| Approach: | They propose a framework for precisely erasing entire concepts from model parameters by directly editing directions that encode them in parameter space. |
| Outcome: | The proposed framework reduces errors on the target concept to as low as 7.7% while improving specificity and robustness. |
PhonoThink: Improving Large Language Models’ Reasoning on Chinese Phonological Ambiguities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effectively resolving phonological ambiguities is crucial for robust natural language processing, as these ambiguity are pervasive in tasks ranging from speech-to-text, spelling correction, to offensive language detection. |
| Approach: | They propose a framework to enhance LLMs’ phonological capability through a multiple-stage training approach. |
| Outcome: | The proposed framework enables the base model to achieve comparable performance to a much larger model. |
SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-SQL aims to convert natural language questions into executable SQL queries. |
| Approach: | They propose a framework that generates and filters self-augmented examples for SQL generation . using self-generated examples, they surpass previous zero-shot and few-shot frameworks . |
| Outcome: | The proposed framework surpasses the previous zero-shot and few-shot frameworks, achieving higher execution accuracy. |
ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing dense retrieval with query augmentation ignore the alignment between generation and ranking objectives. |
| Approach: | They propose a unified LLM-augmented dense retrieval framework that jointly optimizes both the LLM and the retriever. |
| Outcome: | Experimental results show that ExpandR outperforms strong baselines, achieving more than 5% improvement in retrieval performance. |
Anecdoctoring: Automated Red-Teaming Across Language and Place (2025.emnlp-main)
Copied to clipboard
| Challenge: | Disinformation is among the top risks of generative AI misuse . red-teaming datasets are typically US- and English-centric . |
| Approach: | They propose a red-teaming approach that generates adversarial prompts across languages and cultures by clustering misinformation claims into broader narratives and enhancing an attacker LLM. |
| Outcome: | The proposed approach produces higher attack success rates and interpretability benefits relative to few-shot prompting. |
ACING: Actor-Critic for Instruction Learning in Black-Box LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have impressive capabilities across tasks like classification, summarization, and reasoning. |
| Approach: | They propose an actor-critic reinforcement learning framework that formulates instruction optimization as a stateless, continuous-action problem. |
| Outcome: | The proposed framework outperforms human-written prompts in 76% of instruction-induction tasks with gains of 33 points and 10-point improvement over baseline. |
Women, Infamous, and Exotic Beings: A Comparative Study of Honorific Usages in Wikipedia and LLMs for Bengali and Hindi (2025.emnlp-main)
Copied to clipboard
| Challenge: | Honorifics encode nuanced socio-pragmatic cues such as power, age, gender, fame, and cultural distance. |
| Approach: | They propose to study third-person honorific usage across 10,000 Hindi and Bengali Wikipedia articles . honorifics are more prevalent in Bengali than in Hindi, while non-honorifics dominate . |
| Outcome: | The authors show that large language models internalize similar socio-pragmatic norms . their analysis shows that honorifics are more prevalent in Bengali than in Hindi . |
Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise (2025.emnlp-main)
Copied to clipboard
Hanyin Wang, Chufan Gao, Qiping Xu, Bolun Liu, Guleid Hussein, Hariprasad Reddy Korsapati, Mohamad El Labban, Kingsley Iheasirim, Mohamed Hassan, Gokhan Anil, Brian Bartlett, Jimeng Sun
| Challenge: | Currently, no automated, scalable method exists to evaluate the quality of LLM-generated clinical notes, leaving manual evaluation the gold standard. |
| Approach: | They propose a framework for training PRMs to deliver step-level reward signals for LLM-generated clinical notes. |
| Outcome: | The proposed framework outperforms reasoning and non-reasoning models on key evaluations and selects physician-preferred clinical notes with 56.2% accuracy. |
GCML: Gradient Coherence Guided Meta-Learning for Cross-Domain Emerging Topic Rumor Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing domain adaptation rumor detection methods ignore the data generalization differences and rely on a large amount of unlabeled target domain samples to achieve domain adaptation. |
| Approach: | They propose a Gradient Coherence guided Meta-Learning approach for emerging topics rumor detection that selectively learns more "generalizable" tasks that are more beneficial in adapting to the target domain. |
| Outcome: | The proposed method outperforms baselines on real-world datasets and significantly outperformed traditional methods on the in-domain condition. |
Can LLMs Generate and Solve Linguistic Olympiad Puzzles? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for solving linguistic puzzles are outdated and require expert linguists to ensure validity. |
| Approach: | They propose a novel task that uses Large Language Models to generate and solve linguistic puzzles for high school students. |
| Outcome: | The proposed model outperforms humans on most puzzle types except for those centered on writing systems and for the understudied languages. |
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Considerable efforts have been and are still being put into increasing the context length of Large Language Models (LLMs) |
| Approach: | They propose an approach that divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. |
| Outcome: | The proposed approach outperforms 8 state-of-the-art methods in effectiveness and efficiency for document summarization and question answering, and achieves the best performance on LongBench v2 among models of comparable size. |
DivScore: Zero-Shot Detection of LLM-Generated Text in Specialized Domains (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot detectors fail when applied to specialized content due to domain shift . DivScore outperforms state-of-the-art detectors in specialized domains . |
| Approach: | They propose a zero-shot detection framework that uses normalized entropy-based scoring and domain knowledge distillation to identify LLM-generated text in specialized domains. |
| Outcome: | The proposed framework outperforms state-of-the-art detectors on medical and legal datasets with 14.4% higher AUROC and 64.0% higher recall. |
Multi-Document Event Extraction Using Large and Small Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multi-document event extraction have limited attention . despite its practical significance, this task has inherent challenges . |
| Approach: | They propose a collaborative framework that integrates large language models for multi-step reasoning and fine-tuned small language models to handle key subtasks. |
| Outcome: | The proposed framework outperforms existing methods and provides new insights into collaborative reasoning to tackle the complexities of multi-document event extraction. |
MA-GTS: A Multi-Agent Framework for Solving Complex Graph Problems in Real-World Applications (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for solving complex problems are expensive and inefficient when handling large-scale, high-complexity problems. |
| Approach: | They propose a multi-agent framework that decomposes complex problems through agent collaboration by mapping implicitly expressed graph data into clear, structured graph representations and dynamically selecting the most suitable algorithm based on problem constraints and graph structure scale. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on multiple benchmarks with robust performance on both closed- and open-source models. |
Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders (2025.emnlp-main)
Copied to clipboard
Weiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo, Chen Xu, Xiaofeng Zhao, Long Meng, Yunfei Lu, Min Zhang, Hao Yang, Tong Xiao, JingBo Zhu
| Challenge: | Existing work on integrating audio encoders with large language models (LLMs) has focused on semantic understanding tasks, but different tasks may require distinct features that emphasize either semantic or acoustic aspects. |
| Approach: | They propose to use a prompt-aware mixture to enhance the Speech LLM that uses multiple audio encoders to extract different features based on the prompt. |
| Outcome: | The proposed approach outperforms all single-encoder Speech LLMs on ASR, speaker number verification, and AC tasks. |
CIKT: A Collaborative and Iterative Knowledge Tracing Framework with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge Tracing (KT) aims to model a student’s learning state over time and predict their future performance. |
| Approach: | They propose a framework that harnesses Large Language Models to enhance both prediction accuracy and explainability by a synergistic optimization loop. |
| Outcome: | The proposed framework improves both prediction accuracy and explainability by using a synergistic optimization loop. |
Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing mitigation strategies for Text-to-Speech systems require excessive training resources or inference latency. |
| Approach: | They propose a GFlOwNet-guided distribution AlignmenT framework that mitigates hallucinations without relying on massive resources or inference latency. |
| Outcome: | The proposed framework reduces over 50% character error rates and lowers uncertainty by up to 58% on challenging test cases. |
MolErr2Fix: Benchmarking LLM Trustworthiness in Chemistry via Modular Error Detection, Localization, Explanation, and Correction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown growing potential in molecular sciences, but they often produce chemically inaccurate descriptions and struggle to recognize or justify potential errors. |
| Approach: | They propose a benchmark to assess LLMs on error detection and correction in molecular descriptions. |
| Outcome: | The proposed benchmark targets LLMs on error detection and correction in molecular descriptions. |
Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using multilingual models, we find that treating languages in isolation obscures the true patterns of memorization. |
| Approach: | They propose a graph-based correlation metric that incorporates language similarity to analyze cross-lingual memorization. |
| Outcome: | The proposed model incorporates language similarity to analyze cross-lingual memorization in 95 languages. |
Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to embed knowledge into large language models have some limitations . static nature of training data and lack of knowledge in domains create knowledge gaps . |
| Approach: | They propose a method that iteratively cycles between sampling generations and optimizing the model through calculated rewards. |
| Outcome: | The proposed method outperforms baseline approaches on medical, legal, astronomy, and current events datasets. |
LLM-Driven Completeness and Consistency Evaluation for Cultural Heritage Data Augmentation in Cross-Modal Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Cross-modal retrieval is essential for interpreting cultural heritage data, but its effectiveness is limited by incomplete or inconsistent textual descriptions. |
| Approach: | They propose a data augmentation framework that enhances cross-modal retrieval performance by improving the completeness and consistency of LLM-generated descriptions. |
| Outcome: | The proposed framework improves cross-modal retrieval performance by improving completeness and consistency of LLM-generated descriptions. |
Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions (2025.emnlp-main)
Copied to clipboard
| Challenge: | We introduce and study artificial impressions–patterns in LLMs’ internal representations of prompts that resemble human impressions and stereotypes based on language. |
| Approach: | They introduce and study artificial impressions–patterns in LLMs’ internal representations of prompts that resemble human impressions and stereotypes based on language. |
| Outcome: | The proposed models predict impressions and model behavior based on the two-dimensional Stereotype Content Model (SCM). |
Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization (2025.emnlp-main)
Copied to clipboard
Jiulong Wu, Zhengliang Shi, Shuaiqiang Wang, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao, Min Zhang
| Challenge: | Existing preference alignment methods focus on aligning model responses with human preferences while neglecting image-text modality alignment. |
| Approach: | They propose Entity-centric Multimodal Preference Optimization to improve modality alignment . they use open-source instruction datasets to automatically construct high-quality preference data . |
| Outcome: | The proposed approach reduces hallucination rates by 80.4% on Object HalBench and 52.6% on MM HalBech. |
3DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selection (2025.emnlp-main)
Copied to clipboard
Hongxin Ding, Yue Fang, Runchuan Zhu, Xinke Jiang, Jinyang Zhang, Yongxin Xu, Weibin Liao, Xu Chu, Junfeng Zhao, Yasha Wang
| Challenge: | Effective domain adaptation typically involves supervised fine-tuning on carefully selected instruction-tuned data. |
| Approach: | They propose a model-centric data selection framework that aligns data selection with the model’s knowledge distribution to improve model performance. |
| Outcome: | The proposed framework outperforms existing methods by up to 2.97% accuracy in the healthcare domain. |
InfiniBench: A Benchmark for Large Multi-Modal Models in Long-Form Movies and TV Shows (2025.emnlp-main)
Copied to clipboard
Kirolos Ataallah, Eslam Mohamed Bakr, Mahmoud Ahmed, Chenhui Gou, Khushbu Pahwa, Jian Ding, Mohamed Elhoseiny
| Challenge: | Existing benchmarks fail to test the full range of cognitive skills needed to process long-form videos . |
| Approach: | They propose a benchmark to evaluate models' ability to process long-form videos rigorously. |
| Outcome: | The benchmark measures the cognitive skills of models in understanding long-form videos . it offers the largest set of question-answer pairs for long video comprehension . |
Intrinsic Test of Unlearning Using Parametric Knowledge Traces (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for “unlearning” information captured in large language models rely on behavioral tests without monitoring residual knowledge in model parameters. |
| Approach: | They propose a general evaluation methodology that uses vocabulary projections to inspect concepts encoded in model parameters. |
| Outcome: | The proposed method detects changes in parametric traces of unlearned concepts and localizes them in two open-source LLMs. |
Speculative Streaming: Efficient and Scalable Speculative Decoding with Multi-Stream Attention (2025.emnlp-main)
Copied to clipboard
Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Antonie Lin, Mohammad Rastegari, Mahyar Najibi
| Challenge: | Speculative decoding is a prominent technique for accelerating LLM inference by leveraging an auxiliary draft model, but its effectiveness is limited by the autoregressive nature of draft generation. |
| Approach: | They propose a method that integrates speculative draft generation directly within the target model using multi-stream attention. |
| Outcome: | The proposed method improves acceptance but also latency and speculation latency, limiting overall speedup. |
Evaluating Cognitive-Behavioral Fixation via Multimodal User Viewing Patterns on Social Media (2025.emnlp-main)
Copied to clipboard
| Challenge: | Digital media platforms often contribute to cognitive-behavioral fixation, a phenomenon in which users exhibit sustained and repetitive engagement with narrow content domains. |
| Approach: | They propose a multimodal topic extraction module and a cognitive-behavioral fixation quantification module that collaboratively enable adaptive, hierarchical, and interpretable assessment of user behavior. |
| Outcome: | The proposed framework lays the groundwork for scalable computational analysis of cognitive fixation. |
Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have highlighted the significant performance variation that can arise from minor changes in prompt design. |
| Approach: | They propose to tokenize the space following the colon to facilitate automated answer extraction via next-token probabilities. |
| Outcome: | The proposed tokenization improves model calibration and improves confidence estimates. |
VocalNet: Speech LLMs with Multi-Token Prediction for Faster and High-Quality Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show VocalNet outperforms existing open-source speech LLMs despite limited training data. |
| Approach: | They propose a scalable and model-agnostic training framework and a novel multi-token prediction paradigm for speech generation. |
| Outcome: | The proposed model outperforms open-source speech LLMs while outperforming existing open-sourced models. |
Path Drift in Large Reasoning Models: How First-Person Commitments Override Safety (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on prompt injection and jailbreak attacks primarily target the surface structure of input prompts. |
| Approach: | They propose a three-stage approach to mitigate the risk of Long-CoT reasoning drift . they propose 'path-level defense' strategy that incorporates role attribution correction and metacognitive reflection . |
| Outcome: | The proposed framework reduces refusal rates and ethical evaporation, while ethical escalation and layered disclaimers progressively steer models toward unsafe completions. |
CBP-Tuning: Efficient Local Customization for Black-box Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Customized black-box prompt tuning is a new approach to customize large language models . however, as models grow, the resources required for training and deployment become increasingly expensive . |
| Approach: | They propose a framework that facilitates efficient local customization while preserving bidirectional privacy. |
| Outcome: | The proposed framework facilitates efficient local customization while preserving bidirectional privacy. |
Beyond the Score: Uncertainty-Calibrated LLMs for Automated Essay Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) systems attain near–human agreement on some public benchmarks, but real-world adoption is limited. |
| Approach: | They propose a distribution-free wrapper that equips any classifier with set-valued outputs enjoying formal coverage guarantees. |
| Outcome: | The proposed model achieves coverage targets while keeping prediction sets compact. |
Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts (2025.emnlp-main)
Copied to clipboard
Georgios Chochlakis, Peter Wu, Tikka Arjun Singh Bedi, Marcus Ma, Kristina Lerman, Shrikanth Narayanan
| Challenge: | Existing approaches to model complex subjective tasks in natural language are limited by significant variation in annotations. |
| Approach: | They propose a simple in-context learning binary filtering baseline that estimates the reasonableness of a document-label pair. |
| Outcome: | The proposed approach can be integrated into annotation pipelines to enhance signal-to-noise ratios. |
Do It Yourself (DIY): Modifying Images for Poems in a Zero-Shot Setting Using Weighted Prompt Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a novel method to enhance imagery in poetic language is proposed . weighted prompt manipulation is a new approach to enhance poetry images . current diffusion models struggle to interpret metaphorical language, symbolism, and nuanced themes. |
| Approach: | They propose a weighted prompt manipulation technique that modifies attention weights and text embeddings within diffusion models to enhance or suppress specific words' influence in the final generated image. |
| Outcome: | The proposed technique enhances or suppresses the influence of specific words in the final generated image, leading to semantically richer and more contextually accurate visualizations. |
Looking Beyond Text: Reducing Language Bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) have been criticized for their language bias. |
| Approach: | They propose to use a dual-attention mechanism to construct separate attention for visual and text inputs to enhance integration of visual inputs across models. |
| Outcome: | Experiments show that the proposed model debiases LVLMs from their language bias, enhancing visual comprehension and reducing hallucinations without additional resources. |
Who Holds the Pen? Caricature and Perspective in LLM Retellings of History (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are increasingly used to simulate human perspectives, authors say . authors: asymmetries in tone, stance, and emphasis can quietly, yet systematically, distort how history is told and remembered. |
| Approach: | They analyze LLM-generated responses across 197 historically significant events . they find that LLMs reliably distinguish persona-based responses from neutral baselines . |
| Outcome: | The findings show that LLMs distinguish persona-based responses from neutral baselines and that directly affected personas exhibit higher exaggeration. |
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs (2025.emnlp-main)
Copied to clipboard
Minxuan Lv, Zhenpeng Su, Leiyu Pan, Yizhe Xiong, Zijia Lin, Hui Chen, Wei Zhou, Jungong Han, Guiguang Ding, Wenwu Ou, Di Zhang, Kun Gai, Songlin Hu
| Challenge: | Existing sparsification methods like pruning can lose model knowledge through parameter removal. |
| Approach: | They propose a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. |
| Outcome: | The proposed approach achieves superior performance across language modeling and downstream tasks under equivalent computational constraints. |
Search Wisely: Mitigating Sub-optimal Agentic Searches By Reducing Uncertainty (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have propelled their use in informationintensive tasks such as question answering and knowledge synthesis. |
| Approach: | They propose a reinforcement learning-based training method that incorporates confidence thresholds to reward high-certainty search decisions. |
| Outcome: | The proposed method outperforms baseline models on seven QA benchmarks and demonstrates that it is more efficient than existing methods. |
Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | prevailing view in language acquisition research has long held that child-directed language is more effective than adultdirected language (ADL) |
| Approach: | They propose a frequency-controlled testing methodology to enable balanced comparisons across training corpora. |
| Outcome: | The proposed method outperforms models trained on English Child-Directed Language (CDL) but it does not yield stronger generalizations for acquiring syntax. |
Benchmarking Debiasing Methods for LLM-based Parameter Estimates (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are expensive yet powerful ways to annotate text, and can be inconsistent when compared with experts. |
| Approach: | They propose to combine LLM annotations with a limited number of expensive expert annotations to produce valid estimates. |
| Outcome: | The proposed methods produce consistent estimates under theoretical assumptions, but they are not comparable across finite datasets. |
(Almost) Free Modality Stitching of Foundation Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-modal foundation models often use modality-specific (uni-modal) models as sub-components, which are stitched together via a connector module. |
| Approach: | They propose a framework that allows for optimal uni-modal model selection and connector training by leveraging hypernetworks. |
| Outcome: | The proposed framework reduces the cost of searching for the best performing uni-modal model pair by 10 while matching the ranking and trained connector performance across diverse multi-modal benchmarks. |
VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | supervised fine-tuning (SFT) data is critical for large multimodal models . current methods suffer from factual errors and hallucinations due to inadequate visual perception . |
| Approach: | They propose a pipeline that integrates vision priors and state-of-the-art LMMs with statistical methods to enhance SFT data quality. |
| Outcome: | The proposed pipeline outperforms existing models in text-rich reasoning tasks while enhancing reasoning efficiency. |
Rescorla-Wagner Steering of LLMs for Undesired Behaviors over Disproportionate Inappropriate Context (2025.emnlp-main)
Copied to clipboard
Rushi Wang, Jiateng Liu, Cheng Qian, Yifan Shen, Yanzhou Pan, Zhaozhuo Xu, Ahmed Abbasi, Heng Ji, Denghui Zhang
| Challenge: | Incorporating external context can enhance the response quality of Large Language Models (LLMs). however, real-world contexts often mix relevant information with disproportionate inappropriate content. |
| Approach: | They propose a Poisoned Context Testbed to pair queries with real-world contexts . they propose 'rw-Steering' to internalize inappropriate signals . |
| Outcome: | The proposed model improves response quality by 39.8% and reverses undesirable behavior curve. |
Exploring Artificial Image Generation for Stance Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to stance detection focus on textual content, which may not capture the implicit stance conveyed by the author. |
| Approach: | They propose a novel approach that transforms original texts into artificially generated images and uses the visual representation to enhance stance detection. |
| Outcome: | The proposed model is able to detect author's stance from a set of artificially generated images and then leverages both the original textual content and the generated image to identify the author' stance. |
Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines how users interact with LGBTQ+ news content . a corpus of 1,419,047 comments on 3,161 YouTube news videos is used to analyze the content - both positive and negative - of cable news outlets. |
| Approach: | They analyze how users interact with LGBTQ+ news content via a corpus of 1,419,047 comments on 3,161 YouTube news videos of major US cable news outlets. |
| Outcome: | The proposed classifier detects positive (hope speech), negative, neutral, and irrelevant content. |
Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a high prompt sensitivity has been widely accepted as a core limitation of large language models . a recent study suggests that prompt senescence may be an artifact of evaluation processes . |
| Approach: | They examine whether prompt sensitivity is an inherent weakness or an artifact of evaluation . they find that heuristic evaluation methods overlook semantically correct responses . large language models have achieved remarkable success across a wide range of tasks . |
| Outcome: | The proposed model is more robust to prompt templates than previously thought . the authors show that prompt sensitivity may be an artifact of evaluation rather than a flaw . |
Topic Coverage-based Demonstration Retrieval for In-Context Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior methods to retrieve demonstrations based on embedding similarity or generation probability, resulting in irrelevant or redundant examples. |
| Approach: | They propose a topic coverage-based retrieval framework that selects demonstrations to comprehensively cover topic-level knowledge relevant to both the test input and the model. |
| Outcome: | The proposed framework covers all the necessary knowledge for the test input and the model. |
On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) are increasingly used as conversational agents because of their pragmatic reasoning abilities. |
| Approach: | They propose an evaluation framework derived from *Wavelength*, a popular communication game where a speaker and a listener communicate about a broad range of concepts in a granular manner. |
| Outcome: | The proposed evaluation framework outperforms direct and Chain-of-Thought (CoT) prompting on language comprehension and language production tasks. |
MuseScorer: Idea Originality Scoring At Scale (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a large language model, idea-buckets are automatically retrieved and a small number of participants are able to score the idea without human annotation. |
| Approach: | They propose a large-scale, psychometrically validated system for frequency-based originality scoring that integrates a Large Language Model with externally orchestrated retrieval. |
| Outcome: | The proposed system matches human annotations in idea clustering structure and participant-level scoring while showing strong convergent and external validity. |
SAFENUDGE: Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to jailbreak attacks, or adversarial prompts eliciting high-risk behavior. |
| Approach: | They propose a safeguard that combines Controlled Text Generation and "nudging" it adds minimal latency to inference and reduces successful jailbreak attempts by up to 37.3% . |
| Outcome: | The proposed safeguard reduces successful jailbreak attempts by between 28.1% and 37.3% by guiding the LLM towards a safe response. |
RaDeR: Reasoning-aware Dense Retrieval Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | RaDeR retrievers outperform strong baselines in reasoning tasks . large language models (LLMs) have impressive reasoning capabilities on a wide range of tasks - however, they face challenges when reasoning is needed for relevance prediction. |
| Approach: | They propose a set of reasoning-based dense retrieval models trained with data derived from mathematical problem solving using large language models. |
| Outcome: | The proposed model outperforms baselines on the BRIGHT and RAR-b benchmarks and achieves comparable or superior performance while using only 2.5% of the training data used by the concurrent work ReasonIR. |
A Culturally-diverse Multilingual Multimodal Video Benchmark & Model (2025.emnlp-main)
Copied to clipboard
Bhuiyan Sanjid Shafique, Ashmal Vayani, Muhammad Maaz, Hanoona Abdul Rasheed, Dinura Dissanayake, Mohammed Irfan Kurpath, Yahya Hmaiti, Go Inoue, Jean Lahoud, Md. Safirur Rashid, Shadid Intisar Quasem, Maheen Fatima, Franco Vidal, Mykola Maslych, Ketan Pravin More, Sanoojan Baliah, Hasindri Watawana, Yuhao Li, Fabian Farestam, Leon Schaller, Roman Tymtsiv, Simon Weber, Hisham Cholakkal, Ivan Laptev, Shin’ichi Satoh, Michael Felsberg, Mubarak Shah, Salman Khan, Fahad Shahbaz Khan
| Challenge: | Large multimodal models have gained attention for their effectiveness to understand and generate descriptions of visual content. |
| Approach: | They propose a multilingual Video LMM benchmark to evaluate video LMMs across 14 languages . they also introduce a machine translated multilingual video training set . |
| Outcome: | The proposed video LMM benchmark is designed to evaluate video Lmms across 14 languages including Arabic, Bengali, Chinese, English, French, German, Hindi, Japanese, Russian, Sinhala, Spanish, Swedish, Tamil, and Urdu. |
DRES: Fake news detection by dynamic representation and ensemble selection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text-based fake news detection are limited due to context sensitivity and generalization issues. |
| Approach: | They propose a method that leverages instance hardness measures to estimate the classification difficulty for each news article across multiple textual feature representations. |
| Outcome: | The proposed method significantly improves over state-of-the-art methods, confirming the effectiveness of representation selection based on instance hardness and dynamic ensemble selection. |
A Graph-Theoretical Framework for Analyzing the Behavior of Causal Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have popularized causal language models . but their internal behavior remains poorly understood due to the high cost and reliance on large-scale benchmarks . |
| Approach: | They propose a graph-theoretical framework for analyzing causal language models . they construct graphs from model outputs and use metrics to capture linguistic features . |
| Outcome: | The proposed framework provides a macroscopic view of the overall behavior of a language model. |
Membership and Memorization in LLM Knowledge Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Knowledge Distillation (KD) aim to mitigate the high computational demands of Large Language Models (LLMs). |
| Approach: | They characterize and investigate membership privacy risks inherent in six LLM KD techniques . they use instruction-tuning settings that span seven NLP tasks and three teacher model families and various size student models to examine the extent of privacy risks. |
| Outcome: | The proposed methods carry membership and memorization privacy risks from the teacher to students, but differ across different techniques. |
Balanced Multi-Factor In-Context Learning for Multilingual Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches address key factors that influence multilingual ICL, but they do not integrate them into the model. |
| Approach: | They propose a method that quantifies and optimally balances three factors for improved example selection. |
| Outcome: | Experiments on mCSQA and TYDI show that the proposed method outperforms existing methods. |
Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive‐k (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing adaptive methods struggle with aggregation QA where optimal external context is unknown and variable. |
| Approach: | They propose a single-pass method that selects a query-specific number of passages . Adaptivek retrieval matches or outperforms fixedk baselines while using 10x fewer tokens compared to full-context input . |
| Outcome: | Adaptivek retrieval matches or outperforms fixedk baselines on factoid and aggregation QA benchmarks . it uses 10x fewer tokens than full-context input and still retrieves 70% of relevant passages compared to previous methods . |
Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark (2025.emnlp-main)
Copied to clipboard
Chihiro Taguchi, Seng Mai, Keita Kurabe, Yusuke Sakai, Georgina Agyei, Soudabeh Eslami, David Chiang
| Challenge: | Multilingual machine translation (MT) benchmarks are widely used to evaluate the capabilities of modern MT systems. |
| Approach: | They propose to use a multilingual machine translation benchmark to assess the capabilities of modern machine translation systems. |
| Outcome: | The FLORES+ benchmark claims to maintain a translation quality score of over 90% . however, the data in four languages falls short of the 90% quality standard . |
Think Globally, Group Locally: Evaluating LLMs Using Multi-Lingual Word Grouping Games (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit biases in reasoning abilities due to linguistic modality, even with similar content. |
| Approach: | They propose a task inspired by the New York Times Connections: GlobalGroup that evaluates large language models in an abstract reasoning task across several languages. |
| Outcome: | The proposed task evaluates models across multiple languages in English, Spanish, Chinese, Hindi, and Arabic. |
Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal large language models exhibit a pronounced form of visual sycophantic behavior when they process image inputs. |
| Approach: | They propose a technique that allows multimodal large language models to engage in reflective reasoning and determine whether a user’s instruction is misleading or corrective. |
| Outcome: | The proposed model resists misleading instructions but is stubborn even if it is wrong. |
MR. Judge: Multimodal Reasoner as a Judge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective reward modeling is especially valuable in reinforcement learning (RLHF) . |
| Approach: | They propose a paradigm for empowering general-purpose MLLMs judges with strong reasoning capabilities by using multiple-choice problem models instead of directly assigning scores. |
| Outcome: | The proposed model surpasses GPT-4o on VL-RewardBench and improves performance on MM-Vet by up to 7.7%. |
MobiZO: Enabling Efficient LLM Fine-Tuning at the Edge via Inference Engines (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are currently pre-trained and fine-tuned on large cloud servers . fine-timing on resource-constrained edge devices presents significant memory and computational demands . |
| Approach: | They propose a resource-efficient fine-tuning framework for LLMs specifically designed for edge devices. |
| Outcome: | Experiments show that MobiZO achieves substantial runtime speedups and memory savings while improving fine-tuning accuracy. |
Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs (2025.emnlp-main)
Copied to clipboard
Wafa Al Ghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan Pravin More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer
| Challenge: | a benchmark is designed to assess the comprehension of Arabic poetry by large language models in 12 historical eras. |
| Approach: | They propose a benchmark to assess the comprehension of Arabic poetry by large language models in 12 historical eras. |
| Outcome: | The benchmark assesses the comprehension of Arabic poetry by large language models in 12 historical eras. |
CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mathematical reasoning remains a significant challenge for large language models (LLMs), despite advances in prompting techniques such as Chain-of-Thought (CoT). |
| Approach: | They propose a framework that enhances reasoning through two stages: Symbolic Conversion and Reasoning Execution. |
| Outcome: | The proposed framework outperforms traditional CoT on six out of seven benchmarks across four LLMs. |
s1: Simple test-time scaling (2025.emnlp-main)
Copied to clipboard
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, Tatsunori Hashimoto
| Challenge: | OpenAI’s o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. |
| Approach: | They curate a small dataset s1K with 1,000 reasoning questions based on three criteria we validate through ablations: difficulty, diversity, and quality. |
| Outcome: | The proposed model exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). |
Learning Subjective Label Distributions via Sociocultural Descriptors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional approaches aggregate annotator judgements (labels), ignoring minority perspectives, and overlooking the influence of sociocultural context behind such annotations. |
| Approach: | They propose a framework where subjectivity in binary labels is modeled as an empirical distribution accounting for the variation in annotators through human values extracted from sociocultural descriptors using a language model. |
| Outcome: | The proposed model yields well-calibrated toxicity distribution predictions across binary toxicity labels, which are further used for majority label prediction across cultural subgroups. |
COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior exemplar selection methods focus on maximizing predictive accuracy, neglecting model calibration. |
| Approach: | They propose to use a Bayesian optimization algorithm to optimize for predictive accuracy and calibration. |
| Outcome: | The proposed algorithm beats or matches baselines on multiple tasks from un-saturated MMLU-pro benchmarks while requiring minimal number of API calls. |
ML-Promise: A Multilingual Dataset for Corporate Promise Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Promises shape perceptions and drive decisions, but verification of their fulfillment is difficult due to complexity and volume of commitments . authors propose a new approach to verifying promises in environmental, social, and governance reports . complexity of promises, complexity of evidence, difficulty in verifying their fulfillment a pressing need for new approaches . |
| Approach: | They propose a multilingual dataset that includes English, French, Chinese, Japanese, and Korean . they propose ML-Promise to facilitate in-depth verification of corporate promises . |
| Outcome: | The proposed approach includes promise identification, evidence assessment, and evaluation of timing for verification in multiple languages. |
Reading Between the Prompts: How Stereotypes Shape LLM’s Implicit Personalization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that such inferences can lead to lower quality responses for users assumed to be from minority groups. |
| Approach: | They analyze LLMs' latent user representations through both model internals and generated answers to targeted user questions. |
| Outcome: | The proposed models infer demographic attributes based on stereotypical signals, which persists even when the user explicitly identifies with a different demographic group. |
Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a low-resource natural language generation task requires a large number of examples to generate outputs and outputs. |
| Approach: | They propose a teacher-student pipeline that synthesizes accurate input–output pairs without human labels or parallel data. |
| Outcome: | The proposed pipeline synthesizes accurate input–output pairs without human labels or parallel data. |
Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate strong reasoning capabilities for many tasks, often by explicitly decomposing the task via Chain-of-Thought (CoT) reasoning. |
| Approach: | They propose to use a multi-step prompt to decompose translation and train models to incorporate intermediate steps. |
| Outcome: | The proposed model achieves state-of-the-art on WMT24 data and shows that prompting LLMs to “translate again” and self-refine yields even better results than human-like step-by-step prompting. |
How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite advances in Large Vision Language Models, a gap remains in their interpretability and performance. |
| Approach: | They identify the Optical Character Recognition Head (OCR Head) heads that are more efficient at recognizing text from images. |
| Outcome: | The Optical Character Recognition Head (OCR Head) is identified as the most efficient head for recognizing text from images. |
Explainability and Interpretability of Multilingual Large Language Models: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature on multilingual large language models lacks transparency in their internal processes. |
| Approach: | They propose to use multilingual large language models to examine their explainability and interpretability methods. |
| Outcome: | The present study examines the explainability and interpretability of multilingual large language models. |
Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train classifiers that predict norm violations are often opacity-prone . a new approach to identify and extract these implicit criteria from historical moderation data is proposed . |
| Approach: | They propose to extract implicit criteria from historical moderation data using an interpretable architecture. |
| Outcome: | The proposed model replicates neural moderation models while providing transparent insights into decision-making processes. |
AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-Image (T2I) models have been successful in generating images from textual descriptions, but they struggle to capture nuanced and implicit attributes inherent in action depiction. |
| Approach: | They propose a benchmark to evaluate the performance of T2I models in generating images from action-centric prompts. |
| Outcome: | The proposed model achieves an increase of 72% on AcT2I. |
Assessing French Readability for Adults with Low Literacy: A Global and Local Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | illiterate individuals are persons aged 15 years and above who cannot read and write with understanding a short simple statement on their everyday life. |
| Approach: | They propose a novel approach to assess french text readability for adults with low literacy skills using a global and segment-level difficulty scale. |
| Outcome: | The proposed approach addresses both global (full-text) and local (segment-level) difficulty scales. |
LILaC: Late Interacting in Layered Component Graph for Open-domain Multimodal Multihop Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal document retrieval frameworks focus on textual, tabular, and visual elements, but there is a shift toward open-domain multimodal retrieval. |
| Approach: | They propose a multimodal retrieval framework that uses a component graph and a late-interaction-based subgraph retrieval method to capture semantic relationships between components. |
| Outcome: | The proposed framework achieves state-of-the-art retrieval performance on all five benchmarks . it is based on a layered component graph representing multimodal information at two layers . |
DiCoRe: Enhancing Zero-shot Event Detection via Divergent-Convergent LLM Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Understanding the complex event ontology, extracting domain-specific triggers from the passage, and structuring them appropriately overloads and limits the utility of Large Language Models (LLMs). |
| Approach: | They propose a divergent-convergent reasoning framework that decouples the task of ED using Dreamer and Grounder. |
| Outcome: | The proposed framework outperforms baselines on six datasets across five domains and nine LLMs, achieving 4–7% average gains over the best baseline. |
SNaRe: Domain-aware Data Generation for Low-Resource Event Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for ED struggle with label noise and domain drift when applied to specialized domains. |
| Approach: | They propose a domain-aware synthetic data generation framework composed of three components: Scout, Narrator, and Refiner. |
| Outcome: | The proposed framework outperforms baseline approaches on three diverse domain ED datasets and achieves average F1 gains of 3-7% in the zero-shot/few-shot settings and 4-20% improvement for multilingual generation. |
Table-R1: Inference-Time Scaling for Table Reasoning Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | In this study, we explore inference-time scaling on table reasoning tasks. |
| Approach: | They propose a large-scale dataset of reasoning traces and a reinforcement learning with verifiable rewards approach to enable inference-time scaling on table reasoning tasks. |
| Outcome: | The proposed model matches or exceeds GPT-4.1 and DeepSeek-R1 models on diverse table reasoning tasks. |
LimRank: Less is More for Reasoning-Intensive Information Reranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to rerank information require large-scale fine-tuning, which is computationally expensive. |
| Approach: | They propose an open-source pipeline for generating diverse, challenging, and realistic reranking examples. |
| Outcome: | The proposed model performs competitively on two benchmarks, while being trained on less than 5% of the data typically used in prior work. |
PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solving (2025.emnlp-main)
Copied to clipboard
Mihir Parmar, Xin Liu, Palash Goyal, Yanfei Chen, Long Le, Swaroop Mishra, Hossein Mobahi, Jindong Gu, Zifeng Wang, Hootan Nakhost, Chitta Baral, Chen-Yu Lee, Tomas Pfister, Hamid Palangi
| Challenge: | Existing methods for natural planning lack constraint-guided iterative verification and adaptive selection . a recent study found that LLMs are not good at such planning. |
| Approach: | They propose a model-agnostic and easily scalable agent framework with three key components: constraint, verification, and selection agents. |
| Outcome: | The proposed framework improves inference-time algorithms on NATURAL PLAN and OlympiadBench benchmarks. |
An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated impressive capabilities across complex reasoning and generation tasks. |
| Approach: | They evaluate a broad spectrum of collaboration strategies for repository-level code generation where the weak model handles simpler tasks at lower cost and the most challenging tasks are delegated to the strong model. |
| Outcome: | The proposed model achieves equivalent performance to the strong model while reducing the cost by 40%. |
What are Foundation Models Cooking in the Post-Soviet World? (2025.emnlp-main)
Copied to clipboard
| Challenge: | During the Soviet era, these identities were pressured through forced assimilation under the Russian language and culture. |
| Approach: | They construct a multi-modal dataset encompassing 1147 and 823 dishes in the Russian and Ukrainian languages, centered around the Post-Soviet region. |
| Outcome: | The results show that leading models struggle to correctly identify the origins of dishes from Post-Soviet nations in both text-only and multi-modal Question Answering (QA) the weak correlation between this task and QA suggests that QA alone may be insufficient as an evaluation of cultural understanding. |
LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoning (2025.emnlp-main)
Copied to clipboard
Tianshi Zheng, Cheng Jiayang, Chunyang Li, Haochen Shi, Zihao Wang, Jiaxin Bai, Yangqiu Song, Ginny Wong, Simon See
| Challenge: | Modern large language models (LLMs) employ diverse logical inference mechanisms for reasoning. |
| Approach: | They analyze the comparative dynamics of inductive (System 1) versus abductive/deductive (system 2) inference in large language models by using a controlled analogical reasoning environment and a MCQ/free-text task format. |
| Outcome: | The proposed methods can significantly scale LLM reasoning. |
EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Han Liu, Ruoyao Wen, Srijith Nair, Jia Liu, Wenjing Lou, Chongjie Zhang, William Yeoh, Yevgeniy Vorobeychik, Ning Zhang
| Challenge: | Recurrent exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. |
| Approach: | They propose a federated fine-tuning framework that uses a round-robin segment sharing scheme to reduce network bandwidth and adaptive sparsification methods tailored to LoRA’s training dynamics. |
| Outcome: | The proposed framework reduces communication overhead without compromising performance on question-answering and value-alignment tasks. |
Memorization ≠ Understanding: Do Large Language Models Have the Ability of Scenario Cognition? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance across NLP tasks. |
| Approach: | They propose a framework to assess LLMs’ scenario cognition . they examine the ability to link semantic scenario elements with their arguments in context . |
| Outcome: | The proposed framework assesses large language models’ scenario cognition . it shows that current models rely on superficial memorization, failing to achieve robust semantic scenario cognition even in simple cases. |
Priority on High-Quality: Selecting Instruction Data via Consistency Verification of Noise Injection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for instruction selection rely on external models or rules, overlooking the intrinsic association between pre-trained model and instruction data. |
| Approach: | They propose a method that utilizes noise injection to identify the quality of instruction data without relying on external models. |
| Outcome: | The proposed method outperforms the model trained on the entire dataset and established baselines. |
Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used for temporal prediction tasks . however, their reliance on pretraining data can lead to contamination concerns . |
| Approach: | They investigate the capability of prompting to simulate an earlier knowledge cutoff in large language models. |
| Outcome: | The proposed model fails to induce forgetting when the forgotten content is not directly asked but causally related to the query. |
DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reliability of large language models often lack self-correction or use costly post-hoc verification. |
| Approach: | They propose a decoding framework that enhances generation reliability through real-time hallucination detection and efficient error correction. |
| Outcome: | Extensive experiments across five benchmarks show the proposed framework improves truthfulness and factual accuracy. |
Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. |
| Approach: | They propose a benchmark to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. |
| Outcome: | The proposed benchmarks show that they can perform step-by-step execution, instruction adherence, numerical computation, and long-range consistency in handling intermediate results. |
Generative Annotation for ASR Named Entity Correction (2025.emnlp-main)
Copied to clipboard
Yuanchang Luo, Daimeng Wei, Shaojun Li, Hengchao Shang, Jiaxin Guo, Zongyao Li, Zhanglin Wu, Xiaoyu Chen, Zhiqiang Rao, Jinlong Yang, Hao Yang
| Challenge: | Existing named entity correction models fail to transcribe domain-speciffcnamed entities when theforms of the wrongly-transcribed words and the ground-truth entity are signiffcantly different. |
| Approach: | They propose a method that utilizes speech sound features to retrieve candidate entities . it uses speech sound feature to annotate entityerrors in ASR transcripts . |
| Outcome: | The proposed method can bring signiffcant improvement to entity accuracy. |
SOLAR: Towards Characterizing Subjectivity of Individuals through Modeling Value Conflicts and Trade-offs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies suggest that Large Language Models can account for individual-level subjectivity, yet exploring whether LLMs can generate perspectives and reasoning that align well with a specific persona or demographic information has not been adequately studied. |
| Approach: | They propose a framework that observes value conflicts and trade-offs in user-generated texts to better represent subjective ground of individuals. |
| Outcome: | The proposed framework improves inference performance for users with limited data and in controversial situations. |
LogicTree: Structured Proof Exploration for Coherent and Rigorous Logical Reasoning with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have remarkable multi-step reasoning capabilities, but they still face challenges in complex logical reasoning. |
| Approach: | They propose an algorithm-guided search framework that automates structured proof exploration and ensures logical coherence. |
| Outcome: | The proposed framework outperforms o3-mini and chain-of-thought with average gains of 23.6% and 12.5% on five datasets. |
Unmasking Fake Careers: Detecting Machine-Generated Career Trajectories via Multi-layer Heterogeneous Graphs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate convincing career trajectories in fake resumes . a novel heterogeneous, hierarchical multi-layer graph framework is proposed to model career entities and their relations in a unified global graph built from genuine resumes. |
| Approach: | They propose a novel heterogeneous, hierarchical multi-layer graph framework that models career entities and their relations in a unified global graph built from genuine resumes. |
| Outcome: | The proposed framework outperforms state-of-the-art models by 5.8-85.0% relative to baselines. |
GAP: a Global Adaptive Pruning Method for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing structured pruning methods employ uniform compression rates across network layers, neglecting the varying importance of different network depths. |
| Approach: | They propose a pruning framework that minimizes global capability loss by layer-adaptive pruning rates. |
| Outcome: | The proposed approach achieves comparable performance with state-of-the-art methods at high pruning rates and shows significant advantages at low pruning rates. |
Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive neural language models (LMs) generate a probability distribution over tokens at each time step given a prompt. |
| Approach: | They propose to find a prompt that induces LMs to output a distribution as close as possible to the target, using either soft or hard gradient-based prompt tuning. |
| Outcome: | The proposed model is able to generate a distribution as close as possible to a target given a prompt, and it can be used to approximate distributions with low or high entropy. |
LGA: LLM-GNN Aggregation for Temporal Evolution Attribute Graph Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current methods focus on 1-hop neighborhood aggregation, lacking capability to capture complex structural interactions. |
| Approach: | They propose a framework that integrates structural information into attribute embeddings through an attribute embedded loss. |
| Outcome: | The proposed framework shows significant improvements over existing methods on real-world datasets. |
EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focusing on single-task environments with limited constraints lack the complexity required to fully reflect the evolution of large language models (LLMs). |
| Approach: | They propose to use a Segment Policy Optimization algorithm to enhance the LLM's ability to accurately fulfill multi-task workflows. |
| Outcome: | The proposed benchmarks show that existing benchmarks lack the complexity required to fully reflect the evolution of large language models. |
Tool Preferences in Agentic LLMs are Unreliable (2025.emnlp-main)
Copied to clipboard
Kazem Faghih, Wenxiao Wang, Yize Cheng, Siddhant Bharti, Gaurang Sriramanan, Sriram Balasubramanian, Parsa Hosseini, Soheil Feizi
| Challenge: | Large language models (LLMs) can now access a wide range of external tools thanks to the Model Context Protocol (MCP). |
| Approach: | They expose a vulnerability in prevalent tool/function-calling protocols by editing tool descriptions to find out which tools are used by LLMs. |
| Outcome: | The proposed changes in the tool descriptions can increase the usage of tools from LLMs when competing with alternatives. |
Enhancing Large Language Model for Knowledge Graph Completion via Structure-Aware Alignment-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge graph completion methods ignore inconsistent representation spaces between natural language and graph structures, leading to duplicate works and time-consuming processes. |
| Approach: | They propose a framework that enhances LLMs for KGC via structure-aware alignment-tuning to align graph embeddings with the natural language space through multi-task contrastive learning. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two KGC tasks across four benchmark datasets. |
MultiDocFusion : Hierarchical and Multimodal Chunking Pipeline for Enhanced RAG on Long Industrial Documents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text chunking methods neglect complex and long industrial document structures, causing information loss and reduced answer quality. |
| Approach: | They propose a multimodal chunking pipeline that detects document regions and extracts text from them via OCR. |
| Outcome: | Extensive tests show that MultiDocFusion improves retrieval precision by 8–15% and ANLS QA scores by 2–3% compared to baselines. |
Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Hallucination is a significant barrier to the effective application of Large Language Models (LLMs). |
| Approach: | They propose an Attention-Guided SElf-Reflection approach for hallucination detection in Large Language Models. |
| Outcome: | The proposed method significantly outperforms existing methods in zero-shot hallucination detection on four widely-used LLMs across three different halluciation benchmarks. |
‘Rich Dad, Poor Lad’: How do Large Language Models Contextualize Socioeconomic Factors in College Admission ? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models are increasingly involved in high-stakes domains, yet how they reason about socially sensitive decisions remains underexplored. |
| Approach: | They propose a dual-process audit framework to probe LLMs’ reasoning behaviors in sensitive applications using a synthetic dataset of 30,000 applicant profiles grounded in real-world correlations. |
| Outcome: | The proposed framework exploits a synthetic dataset of 30,000 applicant profiles grounded in real-world correlations to probe LLMs' reasoning behaviors in sensitive applications. |
Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often refuse to answer legitimate queries, causing models to treat many reasonable prompts as potentially risky. |
| Approach: | They propose a framework that automatically generates and selects overrefusal prompts near the safety boundary. |
| Outcome: | The proposed framework identifies and curates boundary-aligned prompts, enabling more effective and targeted mitigation of overrefusal. |
MMAG: Multimodal Learning for Mucus Anomaly Grading in Nasal Endoscopy via Semantic Attribute Prompting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Accurate grading of rhinitis severity relies heavily on the characterization of key secretions, notably clear nasal discharge (CND) and purulent nasal secretion (PUS). |
| Approach: | They propose a framework that integrates structured prompts with rank-aware vision-language modeling for joint detection and grading. |
| Outcome: | The proposed model improves AUC and F1 scores on CND and PUS datasets by 6.31% and 4.79%. |
The Emperor’s New Reasoning: Format Imitation Overshadows Genuine Mathematical Understanding in SFT (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have yielded impressive gains on mathematical reasoning benchmarks via supervised fine-tuning (SFT). |
| Approach: | They investigate the mechanisms behind SFT improvements in small-scale large language models by examining four key questions: (1) Are performance gains primarily due to format alignment rather than reasoning? (2) Can high-quality supervision encourage genuine reasoning? (4) Are format alignment gains consistent across model sizes and architectures? |
| Outcome: | The proposed models outperform the proprietary models on OlympiadBench and Omni-Math, but lack the brittleness of the models under perturbations to test their reasoning abilities. |
Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve mathematical reasoning require extensive datasets for training or depend on few-shot methods that compromise computational accuracy. |
| Approach: | They propose a training-free adaptation framework that efficiently equips general-purpose pre-trained language models with enhanced mathematical reasoning capabilities. |
| Outcome: | The proposed framework outperforms Qwen2.5-72B-Math-Instruct on MMLU-STEM with a score of 90.9%, compared to 87.3%. |
Flexibly Utilize Memory for Long-Term Conversation via a Fragment-then-Compose Framework (2025.emnlp-main)
Copied to clipboard
Cai Ke, Yiming Du, Bin Liang, Yifan Xiang, Lin Gui, Zhongyang Li, Baojun Wang, Yue Yu, Hui Wang, Kam-Fai Wong, Ruifeng Xu
| Challenge: | Large language models extract useful information from conversation history to enhance the response in long-term conversations. |
| Approach: | They propose a Fragment-then-Compose framework to optimize memory utilization for long-term open-domain conversation. |
| Outcome: | The proposed framework can be used to extract useful information from conversation history . it can be adapted to different situations and improve response generation . |
STRICT: Stress-Test of Rendering Image Containing Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite the advances in diffusion models, the generation of coherent text remains a major bottleneck. |
| Approach: | They propose a benchmark to test the ability of diffusion models to render coherent text in images. |
| Outcome: | The proposed model fails to generate coherent and legible text in images despite its iterative nature . the model fails in both the maximum length of readable text and correctness and legibility of the generated text . |
A Sequential Multi-Stage Approach for Code Vulnerability Detection via Confidence- and Collaboration-based Decision Making (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong capabilities across diverse domains, but their application to code vulnerability detection raises significant concerns regarding efficiency, scalability and cost. |
| Approach: | They propose a sequential multi-stage approach via confidence- and collaboration-based decision making via a three-stage sequential classification framework with a single agent, retrieval-augmented generation with external examples, and multi-agent reasoning enhanced with RAG. |
| Outcome: | The proposed approach improves code vulnerability detection performance on a benchmark dataset and a low-resource language. |
Leveraging Large Models to Evaluate Novel Content: A Case Study on Advertisement Creativity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating creativity is challenging, even for humans, because of its subjectivity and complex cognitive processes. |
| Approach: | They propose a set of tasks to break down visual advertisement creativity into atypicality and originality with fine-grained annotations by humans. |
| Outcome: | The proposed tasks demonstrate the promise and challenges of using VLMs for automated creativity assessment. |
BIRD: Bronze Inscription Restoration and Dating (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing applications of artificial intelligence to bronze inscriptions focus almost exclusively on computer vision . |
| Approach: | They propose a fully encoded dataset that integrates domain- and task-adaptive pretraining with a Glyph Net to model bronze inscriptions. |
| Outcome: | The proposed framework improves restoration, while glyph-biased sampling yields gains in dating. |
DCP: Dual-Cue Pruning for Efficient Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning methods for large vision language models use visual tokens to prune . existing methods fail to balance efficiency and semantic alignment due to large number of visual token. |
| Approach: | They propose a cross-modal pruning framework that considers textual semantics and visual self-attention to combine them to achieve efficient inference acceleration. |
| Outcome: | The proposed pruning framework can retain only 25% of the visual tokens, with a minimal performance degradation of only 0.063% on LLaVA-1.5-13B. |
Improving Context Fidelity via Native Retrieval-Augmented Reasoning (2025.emnlp-main)
Copied to clipboard
Suyuchen Wang, Jinlin Wang, Xinyu Wang, Shiqi Li, Xiangru Tang, Sirui Hong, Xiao-Wen Chang, Chenglin Wu, Bang Liu
| Challenge: | Existing approaches to fidelity to contexts rely on expensive supervised fine-tuning to generate evidence post-answer or train models to perform web searches without improving utilization of the given context. |
| Approach: | They propose a native retrieval-augmented reasoning framework that integrates in-context evidence with the model’s own retrieval capabilities. |
| Outcome: | The proposed approach outperforms supervised fine-tuning, retrieval-augmented generation methods, and external retrieval solutions on multiple real-world and counterfactual QA benchmarks. |
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance (2025.emnlp-main)
Copied to clipboard
Shehzeen Samarah Hussain, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Subhankar Ghosh, Roy Fejgin, Mikyas T. Desta, Rafael Valle, Jason Li
| Challenge: | Autoregressive speech token generation models suffer from hallucinations and undesired vocalizations that do not conform to conditioning inputs. |
| Approach: | They propose an encoder-decoder transformer model that improves contextual adherence of speech token generation LLMs through preference alignment and classifier-free guidance. |
| Outcome: | The proposed model outperforms previous LLM-based models on intelligibility, speaker similarity and naturalness. |
Mixing Inference-time Experts for Enhancing LLM Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving reasoning quality in large language models are limited to using a single expert. |
| Approach: | They propose a framework that finetunes and merges expert logits from one LLM . they use commonsense and entailment reasoning experts to improve chain-of-thought reasoning . |
| Outcome: | The proposed framework outperforms baselines on three question-answering datasets. |
Reinforced Query Reasoners for Reasoning-intensive Retrieval Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retrieval of information excel at textual and semantic matching but struggle in reasoning-intensive retrieval tasks. |
| Approach: | They propose a family of small-scale language models for query reasoning and rewriting in reasoning-intensive retrieval. |
| Outcome: | The proposed model outperforms existing models on a BRIGHT benchmark with BM25 retrievers. |
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Rapid advances in Large Language Models have spurred demand for processing extended context sequences . however, performance degradation due to sequence lengths out-of-distribution and excessively long inference times are limiting LLMs in long-context scenarios. |
| Approach: | They propose a training-free method for efficient and accurate long-context inference . they selectively involves a few critical KV cache tokens in attention calculation . |
| Outcome: | The proposed method speeds up attention computation and accelerates inference time while reducing selection overhead. |
MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Models (2025.emnlp-main)
Copied to clipboard
Siyu Yan, Long Zeng, Xuecheng Wu, Chengcheng Han, Kongcheng Zhang, Chong Peng, Xuezhi Cao, Xunliang Cai, Chenjuan Guo
| Challenge: | Existing defenses target single-turn attacks, but real-world usage involves multi-turn dialogues, exposing models to attacks that exploit conversational context to bypass safety measures. |
| Approach: | They propose a framework that tackles multi-turn jailbreaks from both attack and defense angles. |
| Outcome: | Experiments on large language models show that MUSE effectively mitigates multi-turn jailbreaks. |
EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated strong machine translation capabilities for English-centric language pairs but underperform in direct non-English (x2x) translation. |
| Approach: | They propose a synthetic data generation framework that leverages models’ established English-to-x (en2x) capabilities by extending English parallel corpora into omnidirectional datasets and developing an English-referenced quality evaluation proxy. |
| Outcome: | The proposed framework achieves significant improvement across 72 x2x directions while generalizing to enhance en2x performance. |
“I’ve Decided to Leak”: Probing Internals Behind Prompt Leakage Intents (2025.emnlp-main)
Copied to clipboard
Jianshuo Dong, Yutong Zhang, Liu Yan, Zhenyu Zhong, Tao Wei, Ke Xu, Minlie Huang, Chao Zhang, Han Qiu
| Challenge: | Large language models (LLMs) exhibit prompt leakage vulnerabilities, raising intellectual property and confidentiality concerns. |
| Approach: | They use probing techniques to capture LLMs’ intent-related internal representations and show that they internalize prompt leakage intents in their hidden states before generating tokens. |
| Outcome: | The proposed probes achieve 90%+ AUROC across all tested models, even when applied to new system prompts and attacks. |
Nullspace Disentanglement for Red Teaming Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work mainly leverages crowd workers to construct test cases. |
| Approach: | They propose a black-box approach that exploits the unique properties of the nullspace to disentangle and regulate the crucial success information within test cases. |
| Outcome: | The proposed approach outperforms baseline methods regarding the attack success rate and excels in aspects of diversity and fluency. |
Supervised Attention Mechanism for Low-quality Multimodal Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current studies address missing and noisy modalities separately in multimodal data . missing modality is often caused by unavailable data collection equipment or sensor failures . |
| Approach: | They propose a framework for multimodal affective computing that addresses missing and noisy modalities to enhance model robustness in low-quality data scenarios. |
| Outcome: | The proposed model outperforms state-of-the-art baselines on multiple datasets under the settings of complete modalities, missing modalités, and noisy modality. |
Reinforcement Learning for Large Language Models via Group Preference Reward Shaping (2025.emnlp-main)
Copied to clipboard
Huaisheng Zhu, Siyuan Xu, Hangfan Zhang, Teng Xiao, Zhimeng Guo, Shijie Zhou, Shuyue Hu, Vasant G. Honavar
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) are expensive and sensitive to reward model quality. |
| Approach: | They propose a method that leverages preference-based comparisons rather than precise numerical rewards. |
| Outcome: | Experiments show that GPRS outperforms critic-model-free RL algorithms on RLHF and reasoning tasks. |
zFLoRA: Zero-Latency Fused Low-Rank Adapters (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed with task-specific adapters catering to multiple downstream applications. |
| Approach: | They propose a low-latency fused low-rank adapter that introduces zero latency overhead on top of the base model. |
| Outcome: | The proposed adapter reduces the inference time of the model by 2.5x . the proposed adapters are tested on 18 different tasks on different platforms . |
PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving (2025.emnlp-main)
Copied to clipboard
Mihir Parmar, Palash Goyal, Xin Liu, Yiwen Song, Mingyang Ling, Chitta Baral, Hamid Palangi, Tomas Pfister
| Challenge: | Recent studies have shown that decomposing complex problems into simple subtasks has significantly boosted the performance of large language models (LLMs). |
| Approach: | They propose a unified post-training framework that distills synthetic task decompositions and fine-tunes smaller LLMs via supervised and reinforcement-learning objectives to improve complex reasoning. |
| Outcome: | The proposed framework outperforms strong baselines on GSM8k and MATH benchmarks and shows that it can improve generalization capabilities on out-of-domain datasets. |
Semantic Inversion, Identical Replies: Revisiting Negation Blindness in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Negation is a common occurrence in the real world and is essential for logical reasoning as it helps understand the opposite or absence of a statement. |
| Approach: | They propose a verification framework that includes task design and measurement methods to verify this phenomenon negation blindness on the query. |
| Outcome: | The proposed framework can be used to verify the model fails to capture semantic contradictions in negated queries despite its accurate understanding of knowledge about positive queries. |
AMACE: Automatic Multi-Agent Chart Evolution for Iteratively Tailored Chart Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that chart generation requires manual input of code and intent . despite the benefits of large language models, chart generation still requires user input of many parameters . |
| Approach: | They propose a loop-based framework for automatically evolving charts in a multi-agent environment using large language models. |
| Outcome: | The proposed framework improves performance by 29.97% compared to first generation while reducing generation time by 86.9% compared with manual prompt-based methods. |
ActionStudio: A Lightweight Framework for Data and Training of Large Action Models (2025.emnlp-main)
Copied to clipboard
Jianguo Zhang, Thai Quoc Hoang, Ming Zhu, Zuxin Liu, Shiyu Wang, Tulika Manoj Awalgaonkar, Akshara Prabhakar, Haolin Chen, Weiran Yao, Zhiwei Liu, Juntao Tan, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | Existing infrastructure for efficient agentic data processing and model training remains underdeveloped. |
| Approach: | They propose a lightweight and extensible data and training framework for large action models . they propose to unify diverse agent trajectories using Unified Format 2.0 . |
| Outcome: | The proposed framework shows 9 higher throughput than existing frameworks and performs well across public and realistic agent benchmarks. |
Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing surveys focus on interpretation or safety, but safety and understanding are core motivations for interpretation research. |
| Approach: | They propose a framework that connects interpretation methods, enhancements they inform, and tools that operationalize them. |
| Outcome: | The proposed framework summarizes nearly 70 studies at their intersections and concludes with open challenges and future directions. |
Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) generate contextually relevant responses by jointly interpreting visual and textual inputs. |
| Approach: | They propose a method to classify whether an input token is visually grounded by reinterpreting question prompts or replacing the detected absent tokens during generation. |
| Outcome: | The proposed method mitigates the models’ tendency to falsely presume the visual presence of text input and its generality across various LVLMs. |
Improving Task Diversity in Label Efficient Supervised Finetuning of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across domains . but, for challenging tasks, finetuning often requires substantial human annotations - a process that is time-consuming, labor-intensive, and expensive . |
| Approach: | They propose a method that leverages task-diversity as a principle for effective data selection. |
| Outcome: | The proposed method achieves better accuracy than training on the complete dataset (4% increase in MMLU score). |
Look Beyond Feeling: Unveiling Latent Needs from Implicit Expressions for Proactive Emotional Support (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are gaining popularity as scalable tools for mental health support . however, nearly half of individuals do not receive timely support due to limited selfawareness or reluctance to seek help. |
| Approach: | They propose a proactive emotional support framework that leverages principles of active listening to uncover implicit user needs. |
| Outcome: | The proposed model elicits implicit emotional needs and delivers empathetic support compared to baselines . |
s3: You Don’t Need That Much Data to Train a Search Agent via RL (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to optimize retrieval using search-only metrics ignore downstream utility and fine-tune entire LLM to jointly reason and retrieve limit retrieval utility and compatibility with frozen or proprietary models. |
| Approach: | They propose a lightweight, model-agnostic framework that decouples the searcher from the generator and trains the search user using a Gain Beyond RAG reward. |
| Outcome: | The proposed framework outperforms baselines trained on over 70 more data with 2.4k training samples. |
FuseChat: Knowledge Fusion of Chat Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are costly and require significant computational resources and time. |
| Approach: | They propose a fuse-and-merge framework for the knowledge fusion of chat LLMs . they conduct pairwise knowledge fusing on source chat LRMs to create multiple target LLM . |
| Outcome: | The proposed framework is superior to baselines of various sizes. |
Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to optimize attention for long sequences have been limited by their computational cost. |
| Approach: | They propose a framework that infuses partial differential equations into the Transformer’s attention mechanism to better handle long sequences. |
| Outcome: | The proposed framework achieves consistent performance gains over standard and long-sequence Transformer variants across a range of tasks. |
Forget What You Know about LLMs Evaluations - LLMs are Like a Chameleon (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel on public benchmarks, but high scores may mask overreliance on dataset-specific surface cues rather than true language understanding. |
| Approach: | They propose a meta-evaluation framework that systematically rephrases benchmark inputs to detect overfitting. |
| Outcome: | The proposed framework detects performance degradation indicative of superficial pattern reliance on dataset-specific cues and distortion levels. |
Memorization or Reasoning? Exploring the Idiom Understanding of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions. |
| Approach: | They propose to use a large-scale dataset of idioms in six languages to evaluate LLMs' idiomatic processing ability. |
| Outcome: | The proposed model integrates contextual cues and reasoning to improve idiom understanding in LLMs, suggesting that their performance is influenced by memorization and reasoning. |
RD-MCSA: A Multi-Class Sentiment Analysis Approach Integrating In-Context Classification Rationales and Demonstrations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-class sentiment analysis (MCSA) are difficult due to subtle semantic differences between adjacent sentiment levels and the scarcity of high-quality annotated data. |
| Approach: | They propose a framework to integrate classification rationales with adaptively selected demonstrations to enhance MCSA performance under limited supervision. |
| Outcome: | The proposed framework outperforms baseline and standard ICL methods on five benchmark datasets. |
Puzzled by Puzzles: When Vision-Language Models Can’t Take a Hint (2025.emnlp-main)
Copied to clipboard
| Challenge: | rebus puzzles encode language through imagery, spatial arrangement, and symbolic substitution. |
| Approach: | They construct a benchmark of rebus puzzles in english language to test their ability to interpret and solve them. |
| Outcome: | The proposed model performs well on a set of english-language rebus puzzles. |
CREPE: Rapid Chest X-ray Report Evaluation by Predicting Multi-category Error Counts (2025.emnlp-main)
Copied to clipboard
| Challenge: | CREPE is a metric for rapid, interpretable, and clinically grounded automated chest X-ray report generation. |
| Approach: | They propose to use a domain-specific BERT model fine-tuned with a multi-head regression architecture to predict multi-category error counts across six clinically meaningful categories. |
| Outcome: | CREPE outperforms traditional and recent metrics on a large-scale synthetic dataset of 32,000 annotated report pairs. |
TIDES: Technical Information Discovery and Extraction System (2025.emnlp-main)
Copied to clipboard
| Challenge: | Traditional pre-trained LLMs struggle with domain-specific terminology, while fine-tuned LLM requires substantial computational resources. |
| Approach: | They propose a training-free approach that combines TF-IDF with prompt-based LLMs to address technical questions. |
| Outcome: | The proposed system improves the accuracy and efficiency of QA systems in technical domains without LLM retraining. |
Learning to Ask: When LLM Agents Meet Unclear Instruction (2025.emnlp-main)
Copied to clipboard
Wenxuan Wang, Shi Juluan, Zixuan Ling, Yuk-Kit Chan, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, Michael R. Lyu
| Challenge: | Despite their impressive capabilities, LLMs struggle with complex computations and delivering accurate, timely information. |
| Approach: | They propose a framework that prompts LLM agents to ask questions when they encounter obstacles due to unclear instructions and an automated evaluation tool called ToolEvaluator. |
| Outcome: | The proposed framework outperforms existing frameworks for tool learning in the Noisy ToolBench. |
RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction (2025.emnlp-main)
Copied to clipboard
Yuchi Wang, Yishuo Cai, Shuhuai Ren, Sihan Yang, Linli Yao, Yuanxin Liu, Yuanxing Zhang, Pengfei Wan, Xu Sun
| Challenge: | Existing recaptioning methods suffer from inaccuracies due to missing fine-grained details. |
| Approach: | They propose a framework that refines captions through visual reconstruction using a text-to-image model and a visual reconstruction framework. |
| Outcome: | The proposed framework outperforms baselines on CapsBench and CompreCap by 10%. |
StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has demonstrated unprecedented capabilities in sophisticated linguistic comprehension and generative tasks. |
| Approach: | They propose a framework for search LLMs that trains with step-wise proximal policy optimization method to improve QA performance. |
| Outcome: | The proposed framework outperforms global-reward benchmarks on multi-hop QA with a stepwise proximal policy optimization method and richer and more detailed intermediate search rewards and token-level process supervision. |
Dynamic Model-Bank Test-Time Adaptation for Automatic Speech Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing ASR TTA methods struggle with instability under continual and long-term distribution shifts. |
| Approach: | They propose a continuous adaptive model-bank framework that adapts to domain shifts in ASR test-time scenarios. |
| Outcome: | Experiments on diverse, continuously shifting ASR benchmarks show that DMSUTA outperforms existing continual TTA baselines. |
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. |
| Approach: | They propose a pruning-based approach to balance CF and downstream task performance by integrating the ratio of the task vector to pre-trained model parameters into the pruning criteria. |
| Outcome: | The proposed pruning-based approach limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks. |
Does Localization Inform Unlearning? A Rigorous Examination of Local Parameter Attribution for Knowledge Unlearning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies emphasize localized unlearning, restricting parameter updates to specific regions to remove unrelated general knowledge. |
| Approach: | They revisit existing localized unlearning approaches and conduct experiments to evaluate their effectiveness. |
| Outcome: | The proposed method can remove unrelated knowledge without retraining . the proposed method is not robust enough to evaluate the trade-off between the competing goals of unlearning. |
ArgCMV: An Argument Summarization Benchmark for the LLM-era (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for key point extraction are limited by the popular ArgKP21 dataset . a novel dataset for long-context online discussions is proposed . |
| Approach: | They propose to use a long-context argument key point extraction dataset to test this method. |
| Outcome: | The proposed dataset exhibits higher complexity, co-referencing arguments, higher presence of subjective discourse units, and a larger range of topics over the existing dataset. |
VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown significant promise in embodied decision-making tasks within virtual open-world environments, but lack domain-specific knowledge. |
| Approach: | They propose a cost-effective agent framework that integrates cross-modal domain knowledge and finetunes a dedicated object detection model for visual analysis. |
| Outcome: | The proposed framework reduces the requirement for domain-specific training data from millions of samples to a few hundred. |
GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Efficient Key-Value (KV) cache management is essential for processing long text sequences in large language models (LLMs). |
| Approach: | They propose a graph-based framework that redefines token selection for KV cache compression. |
| Outcome: | The proposed framework can be used in existing KV cache eviction methods such as SnapKV and PyramidKV in a plug-and-play manner. |
Joint Modeling of Entities and Discourse Relations for Coherence Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on coherence modeling focuses on entity features or discourse relation features, with little attention given to combining the two. |
| Approach: | They propose two methods for jointly modeling entities and discourse relations for coherence assessment. |
| Outcome: | The proposed methods significantly improve the performance of coherence models on three benchmark datasets. |
Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Context faithfulness is essential for reliable reasoning in context-dependent scenarios. |
| Approach: | They propose a method that identifies and fine-tunes context-faithful experts . they propose 'context-faither fine- tuning' which selectively fine- tunes them . |
| Outcome: | The proposed method identifies experts with specialization in context utilization and improves context grounding. |
HMoE: Heterogeneous Mixture of Experts for Language Modeling (2025.emnlp-main)
Copied to clipboard
An Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, Weidong Han, Zhanhui Kang, Di Wang, Naoaki Okazaki, Cheng-zhong Xu
| Challenge: | Mixture of Experts (MoE) models use homogeneous experts with diverse capacities, resulting in a lack of expert specialization and parameter utilization. |
| Approach: | They propose a framework where experts differ in size and possess diverse capacities . they propose HMoE to encourage frequent activation of smaller experts . |
| Outcome: | The proposed framework outperforms homogeneous homogenous MoE models on evaluation benchmarks and achieves lower loss rate with fewer activated parameters. |
The Ranking Blind Spot: Decision Hijacking in LLM-based Text Ranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated strong performance in information retrieval tasks like passage ranking. |
| Approach: | They propose two attacks that aim to force the LLM ranker to prefer a specific passage and rank it at the top. |
| Outcome: | The proposed attacks aim to force the LLM ranker to prefer a specific passage and rank it at the top. |
Uniform Information Density and Syntactic Reduction: Revisiting *that*-Mentioning in English Complement Clauses (2025.emnlp-main)
Copied to clipboard
| Challenge: | Uniform Information Density (UID) hypothesis suggests that speakers exploit this variability to maintain a consistent rate of information transmission during language production. |
| Approach: | They propose that speakers exploit this variability to maintain a consistent rate of information transmission during language production. |
| Outcome: | The proposed hypothesis replicates the established relationship between information density and *that*-mentioning . |
GRIT: Guided Relational Integration for Efficient Multi-Table Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for encoding table structures are limited to single-table settings . end users are increasingly turning to LLMs with natural language queries . |
| Approach: | They propose a method that converts relational schemas into LLM-friendly textual representations. |
| Outcome: | The proposed method improves table-column retrieval performance across multiple tables while reducing memory and computational overhead. |
RPDR: A Round-trip Prediction-Based Data Augmentation Framework for Long-Tail Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Long-tail question answering presents significant challenges for large language models due to limited ability to acquire and accurately recall less common knowledge. |
| Approach: | They propose a data augmentation framework that selects high-quality easy-to-learn training data to enhance dense retrieval models. |
| Outcome: | The proposed framework improves on two long-tail retrieval benchmarks, PopQA and EntityQuestion, and shows that it outperforms existing retrievers on extremely long-tailed questions. |
Discrepancy Detection at the Data Level: Toward Consistent Multilingual Question Answering (2025.emnlp-main)
Copied to clipboard
Lorena Calvo-Bartolomé, Valérie Aldana, Karla Cantarero, Alonso Madroñal de Mesa, Jerónimo Arenas-García, Jordan Lee Boyd-Graber
| Challenge: | Multilingual question answering systems must ensure factual consistency across languages while also accounting for cultural variation in subjective responses. |
| Approach: | They propose a user-in-the-loop fact-checking pipeline to detect factual and cultural discrepancies in multilingual QA knowledge bases. |
| Outcome: | The proposed tool detects factual and cultural discrepancies in bilingual question answering systems. |
Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific LLMs (2025.emnlp-main)
Copied to clipboard
Yizhou Ying, Geng Zhang, Cui Danxin, Chengyu Du, Guanglei Yue, Sihang Jiang, Jiaqing Liang, Yifei Fu, Hailin Hu, Yanghua Xiao
| Challenge: | Existing data selection strategies for continual pre-training of large language models often rely on scarce labeled data or computationally expensive LLMs. |
| Approach: | They propose an annotation-independent data selection framework for CPT that evaluates grammatical complexity using lexical diversity and syntactic complexity. |
| Outcome: | The proposed framework outperforms baselines on a financial dataset and surpasses full-data training by 1.7% using only 20% of the data. |
Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts to develop lightweight and practical sentiment analysis models are limited by manual instruction and large-scale user texts. |
| Approach: | They propose a framework for sentiment analysis that uses attribute-based instruction construction and difficulty-based data filtering to distill knowledge. |
| Outcome: | The proposed framework outperforms baseline methods in data efficiency and performance. |
One Planner To Guide Them All ! Learning Adaptive Conversational Planners for Goal-oriented Dialogues (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for goal-oriented dialogues involve training separate models for specific combinations of objectives, leading to computational and scalability issues. |
| Approach: | They propose a new dialogue policy method that can adapt to varying objective preferences at inference time without retraining. |
| Outcome: | The proposed method can adapt to varying objective preferences at inference time without retraining. |
Unsupervised Hallucination Detection by Inspecting Reasoning Processes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Unsupervised hallucination detection aims to identify hallucines generated by large language models without relying on labeled data. |
| Approach: | They propose an unsupervised method to detect hallucinated content by large language models . they use internal representations intrinsic to factual correctness to prompt the model to verify the truthfulness of a given statement . |
| Outcome: | The proposed framework outperforms existing unsupervised methods and is fully unsupervised and low cost. |
Multimodal Neural Machine Translation: A Survey of the State of the Art (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal neural machine translation (MNMT) is a task that aims to translate text into the target language using neural networks. |
| Approach: | They propose to integrate other modalities with textual data to enhance translation performance. |
| Outcome: | The proposed task aims to integrate visual modality with textual data to improve translation quality. |
Lemmatization of Polish Multi-word Expressions (2025.emnlp-main)
Copied to clipboard
| Challenge: | PolEval task is complicated by linguistic irregularities and historical factors. |
| Approach: | They propose to use fine-tuned plT5 and mT5 models to improve lemmatization of multi-word expressions and proper names in Polish instead of rule-based methods. |
| Outcome: | The proposed model achieved 86.23% AccCS, 89.43% accCI and 88.79% on a gold-standard dataset derived from Wikipedia . |
Targeted Distillation for Sentiment Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies demonstrate that large language models exhibit remarkable capabilities and achieve state-of-the-art performance in diverse sentiment analysis tasks. |
| Approach: | They propose a distillation framework that decouples knowledge from alignment and introduces a sentiment analysis benchmark that covers a diverse set of tasks. |
| Outcome: | The proposed framework improves models' generalization to unseen tasks and their generalization is strong against existing small-scale models. |
DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. |
| Approach: | They propose an end-to-end generative approach for jailbreak rewriting inspired by diffusion models that uses a sequence-tosequence (seq2sequ) diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. |
| Outcome: | Experiments on Advbench and Harmbench show that the proposed method outperforms autoregressive jailbreak models across evaluation metrics including ASR, fluency, diversity and diversity. |
Rank-Awareness and Angular Constraints: A New Perspective on Learning Sentence Embeddings from NLI Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | High-quality sentence embeddings are critical for advancing a wide range of Natural Language Processing tasks. |
| Approach: | They propose a framework that leverages the full NLI dataset augmented with pre-computed continuous similarity scores (S) they employ a Rank Margin objective that enforces rank consistency against S using an explicit margin and a Gated Angular objective that conditionally refines embedding geometry based on NLI label (L) and S score agreement. |
| Outcome: | The proposed framework outperforms baseline models on STS and the MTEB benchmarks. |
LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for understanding intents from multimodal signals exhibit limitations in their modality-level reliance, constraining relational reasoning over fine-grained semantics for complex intent understanding. |
| Approach: | They propose a method that harnesses the expansive knowledge of large language models to establish semantic foundations that boost smaller models’ relational reasoning performance. |
| Outcome: | The proposed method outperforms state-of-the-art methods on multimodal intent and dialogue act recognition tasks and shows consistent performance gains across diverse semantic understanding scenarios. |
Seeing Culture: A Benchmark for Visual Reasoning and Grounding (2025.emnlp-main)
Copied to clipboard
Burak Satar, Zhixin Ma, Patrick Amadeus Irawan, Wilfried Ariel Mulyawan, Jing Jiang, Ee-Peng Lim, Chong-Wah Ngo
| Challenge: | Multimodal vision-language models (VLMs) have made significant progress in cultural understanding tasks . but these datasets often fall short of providing cultural reasoning while underrepresenting many cultures. |
| Approach: | They propose a Seeing Culture Benchmark that requires VLMs to reason on culturally rich images in two stages. |
| Outcome: | The proposed approach requires VLMs to reason on culturally rich images in two stages . the Seeing Culture Benchmark identifies cultural reasoning shortcomings in multimodal models . |
GRADA: Graph-based Reranking against Adversarial Documents Attack (2025.emnlp-main)
Copied to clipboard
Jingjie Zheng, Aryo Pradipta Gema, Giwon Hong, Xuanli He, Pasquale Minervini, Youcheng Sun, Qiongkai Xu
| Challenge: | Retrieval Augmented Generation (RAG) frameworks are susceptible to adversarial attacks that manipulate the retrieval process by introducing documents that are adversarially similar to the query. |
| Approach: | They propose a framework that integrates external retrieval modules into RAG frameworks to improve the factual accuracy of large language models. |
| Outcome: | The proposed framework reduces adversarial attacks by 80% while maintaining minimal loss in accuracy. |
Orchestrating Audio: Multi-Agent Framework for Long-Video Audio Synthesis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for video-to-audio dubbing for long-form content are fragmented and lack dedicated datasets. |
| Approach: | They propose a multi-agent framework that offers a coordinated, multi-component approach to long-video audio generation. |
| Outcome: | The proposed method outperforms state-of-the-art V2A models in audio quality. |
MADAWSD: Multi-Agent Debate Framework for Adversarial Word Sense Disambiguation (2025.emnlp-main)
Copied to clipboard
Kaiyuan Zhang, Qian Liu, Luyang Zhang, Chaoqun Zheng, Shuaimin Li, Bing Xu, Muyun Yang, Xinxiao Qiao, Wenpeng Lu
| Challenge: | Word sense disambiguation (WSD) is a fundamental yet challenging task in natural language processing. |
| Approach: | a novel multi-agent Debate framework for adversarial word Sense disambiguation is proposed . the framework simulates a real-world debate environment where multiple agents engage in discussions about ambiguous words in the context of adversarials. |
| Outcome: | The proposed framework integrates with existing LLMs and improves models in Chinese language . it shows that it can be used to improve models in the Chinese language and improve performance . |
Interpretable Text Embeddings and Text Similarity Explanation: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text embeddings are a fundamental component in many NLP tasks, but their interpretation and explanation remain challenging. |
| Approach: | They propose a framework for interpretable text embeddings and text similarity explanation . they characterize the main ideas, approaches, and trade-offs and discuss lessons learned . |
| Outcome: | The proposed methods are compared with existing models and compare them with existing ones. |
Dyve: Thinking Fast and Slow for Dynamic Process Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing process verification methods struggle with reliably assessing incomplete reasoning traces and are limited by the cost of high-quality human annotations or the inherent noise in automatically generated labels. |
| Approach: | They propose a dynamic process verifier that integrates fast and slow thinking to enhance reasoning error detection in large language models. |
| Outcome: | The proposed system outperforms existing process-based verifiers and maintains computational efficiency while maintaining high performance. |
PERSEVAL: A Framework for Perspectivist Classification Evaluation (2025.emnlp-main)
Copied to clipboard
Soda Marem Lo, Silvia Casola, Erhan Sezerer, Valerio Basile, Franco Sansonetti, Antonio Uva, Davide Bernardi
| Challenge: | Perspectivist evaluation practices in NLP remain fragmented and inconsistent . |
| Approach: | They propose a framework that evaluates perspectivist models at the individual annotator level and treats annotators and users as distinct entities, consistent with real-world scenarios. |
| Outcome: | The proposed framework evaluates annotators and users as distinct entities consistent with real-world scenarios. |
Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have greatly improved natural language understanding and generation. |
| Approach: | They train a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks. |
| Outcome: | The results show that training–task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies. |
IndiGEC: Multilingual Grammar Error Correction for Low-Resource Indian Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | despite growing interest in GEC, most research has focused on English due to the lack of benchmark datasets for low-resource lan-guages. |
| Approach: | They propose a new approach to generate high-quality synthetic data for GEC using monolingual corpora. |
| Outcome: | The proposed framework outperforms other monolingual methods in English, Hindi, Bengali, Marathi, and Tamil. |
Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized product recommenders, but their susceptibility to adversarial manipulations is difficult to detect. |
| Approach: | They propose to use large language models to investigate cognitive biases as adversarial strategies in product research using LLMs. |
| Outcome: | The proposed approach is the first to tap into human psychological principles, making such manipulations hard to detect. |
T2R-BENCH: A Benchmark for Real World Table-to-Report Task (2025.emnlp-main)
Copied to clipboard
Jie Zhang, Changzai Pan, Sishi Xiong, Kaiwen Wei, Yu Zhao, Xiangyu Li, Jiaxin Peng, Xiaoyan Gu, Jian Yang, Wenhan Chang, Zhenhe Wu, Jiang Zhong, Shuangyong Song, Xuelong Li
| Challenge: | Existing table benchmarks lack the capacity to adequately assess the practical application of table reasoning in industrial applications. |
| Approach: | They propose a bilingual table-to-report task and a table-based benchmark to assess the quality of table reasoning. |
| Outcome: | The proposed task is based on a bilingual benchmark with 457 industrial tables and evaluation criteria to measure the quality of report generation. |
TCP: a Benchmark for Temporal Constraint-Based Planning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks evaluate temporal reasoning and planning in isolation and under limited forms of complexity. |
| Approach: | They propose a temporal constraint-based planning benchmark that assesses temporal reasoning and planning capabilities in large language models. |
| Outcome: | The proposed model fails to perform well under limited constraints and lacks temporal grounding. |
The Role of Outgoing Connection Heterogeneity in Feedforward Layers of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a new fine-tuning loss, we show that inner neurons with diverse outgoing connections are more critical to model performance than those with uniform connections. |
| Approach: | They propose a new loss that reduces the outgoing connection entropy in feedforward layers and elucidates the role of outgoing connections in large language models. |
| Outcome: | The proposed method is significantly more effective than removing neurons randomly or based on their magnitude. |
Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents (2025.emnlp-main)
Copied to clipboard
Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha
| Challenge: | Flowcharts are a critical tool for visualizing decision-making processes, but their non-linear structure and complex visual-textual relationships make it difficult to interpret them using LLMs. |
| Approach: | They propose a task of Fine-grained Flowchart Attribution to trace components grounding a flowchart referring LLM response. |
| Outcome: | The proposed agent mitigates visual hallucinations in LLM answers over baselines by 10–14% on a FlowExplainBench dataset. |
Collaborative Rational Speech Act: Pragmatic Reasoning for Multi-Turn Dialog (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing extensions of Rational Speech Act face challenges in scaling to multi-turn, collaborative scenarios. |
| Approach: | They propose a Rational Speech Act extension that optimizes a gain function adapted from rate-distortion theory to model multi-turn dialog by optimizing a model gain . they demonstrate the effectiveness of CRSA on referential games and template-based doctor–patient dialogs in the medical domain. |
| Outcome: | The proposed model yields more consistent, interpretable, and collaborative behavior than baselines, paving the way for more pragmatic and socially aware language agents. |
Understanding Subword Compositionality of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) take sequences of subwords as input, requiring them to compose subword representations into meaningful word-level representations. |
| Approach: | They propose to probe how large language models compose subword information . they find structural similarity, semantic decomposability, and form retention are key aspects . |
| Outcome: | The proposed models can be classified into three distinct groups, the authors show . they show that they can achieve great performance when probing layer by layer their sensitivity to semantic decompositionality . |
Internal Chain-of-Thought: Empirical Evidence for Layer‐wise Subtask Scheduling in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | et al., 2024) show large language models exhibit internal chain-of-thought, meaning they decompose and execute composite tasks layer-by-layer. |
| Approach: | They propose a method to decode hidden states using LogitLens . they also propose 'chain of thought' to decompose and execute composite tasks . |
| Outcome: | The proposed method decodes hidden states and shows consistent execution pattern . it opens avenues for fine-grained, instruction-level activation steering. |
From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Iterative evaluation of large language models during training can be time- and compute-intensive. |
| Approach: | They reformulate generative tasks into computationally cheaper NLU alternatives and test their performance correlation between them. |
| Outcome: | The proposed alternatives reduce evaluation time by 35x compared to NLU benchmarks. |
Debiasing Multilingual LLMs in Cross-lingual Latent Space (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have evaluated their cross-lingual transferability by directly applying these methods to LLM representations, revealing their limited effectiveness across languages. |
| Approach: | They propose to perform debiasing in a joint latent space rather than directly on LLM representations by using an autoencoder trained on parallel TED talk scripts. |
| Outcome: | The proposed method improves both the overall debiasing performance and cross-lingual transferability of the proposed techniques across four languages. |
Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern document retrieval embedding methods typically encode passages (chunks) from documents independently, often overlooking contextual information from the rest of the document. |
| Approach: | They propose a benchmark to evaluate retrieval models' ability to leverage document-wide context. |
| Outcome: | The proposed method significantly improves retrieval quality on ConTEB without sacrificing base model performance. |
MS-RAG: Simple and Effective Multi-Semantic Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models suffer from poor indexing and inference speed . graph-based RAGs heavily rely on LLM for retrieval thus inference slow . |
| Approach: | They propose retrieval-augmented generation (RAG) which integrates knowledge with dense vectors to build a multi-semantic RAG. |
| Outcome: | The proposed method achieves state-of-the-art performance with faster inference speed compared to existing methods . |
Transitive self-consistency evaluation of NLI models without gold labels (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that NLI models make inconsistent model decisions during inference . previous studies tested only variations of two transitive consistency rules out of four . |
| Approach: | They propose to generate adversarial examples using antonym replacements to test transitive consistency of NLI models. |
| Outcome: | The proposed approach can test all four rules automatically by generating adversarial examples via antonym replacements. |
MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries (2025.emnlp-main)
Copied to clipboard
| Challenge: | Information Retrieval (IR) research on mixed-language queries remains sparse and outdated. |
| Approach: | They propose a test set for mixed-language queries that is realistic and preferred by bilingual speakers. |
| Outcome: | The proposed benchmarks show that multilingual IR models perform moderately on MiLQ and inconsistently across native, English, and mixed-language queries. |
Enhancing Chinese Offensive Language Detection with Homophonic Perturbation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Detecting offensive language in Chinese is challenging due to homophonic substitutions used to evade detection. |
| Approach: | They propose to use HED-COLD to build a large-scale homophonic dataset for Chinese offensive language detection and a homophone-aware pretraining strategy to learn phonetics and orthography. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the COLD test set and the toxicity benchmark ToxiCloakCN. |
Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current benchmarks for evaluating Large Language Models do not capture the rich variety of communication patterns exhibited by humans. |
| Approach: | They propose a low-cost method to emulate diverse writing styles by rewriting evaluation prompts using persona-based LLM prompting. |
| Outcome: | The proposed method improves the external validity of the benchmarks for Large Language Models (LLMs) based on persona-based prompting. |
Computational Analysis of Character Development in Holocaust Testimonies (2025.emnlp-main)
Copied to clipboard
| Challenge: | This work examines character development along the narrative timeline by analyzing changes in the protagonist’s views and behavior and the interplay between them. |
| Approach: | They propose to analyze character development along the narrative timeline using a transcript of Holocaust survivor testimonies as a test case. |
| Outcome: | The proposed approach characterizes changes in the protagonist’s views and behavior and the interplay between them. |
TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptation (2025.emnlp-main)
Copied to clipboard
Daiye Miao, Yufang Liu, Jie Wang, Changzhi Sun, Yunke Zhang, Demei Yan, Shaokang Dong, Qi Zhang, Yuanbin Wu
| Challenge: | Existing studies have shown that LoRA introduces substantial parameter redundancy, which not only increases the number of trainable parameters but also hinders the effectiveness of fine-tuning. |
| Approach: | They propose a method that leverages importance information from the pretrained model’s weights to mitigate LoRA redundancy. |
| Outcome: | The proposed method significantly reduces the number of trainable parameters required for task adaptation while providing a task-aligned perspective for LoRA redundancy reduction. |
Dual-Path Counterfactual Integration for Multimodal Aspect-Based Sentiment Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal aspect-based sentiment classification rely on superficial correlations and spurious cues. |
| Approach: | They propose a Dual-Path Counterfactual Integration framework that explicitly models counterfactual reasoning in multimodal contexts. |
| Outcome: | The proposed framework improves model robustness by explicitly modeling counterfactual reasoning in multimodal contexts. |
Job Unfair: An Investigation of Gender and Occupational Bias in Free-Form Text Completions by LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has identified that LLMs are used in domains where they support or replace human decision-making . a systematic review of LLM outputs shows that many facets of social bias remain unaccounted for . |
| Approach: | They propose to disentangle gender and occupational biases in Italian and English as expressed by LLMs. |
| Outcome: | The proposed method captures gender and occupational biases in Italian and English . it also shows that models struggle with gender-neutral expressions, especially beyond English - the authors conclude . |
C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent developments in spoken dialogue models have created a gap in understanding their effectiveness in comprehending and emulating human conversations. |
| Approach: | They present a benchmark dataset which comprises 1,079 instances in English and Chinese to examine their effectiveness in emulating human conversations. |
| Outcome: | The proposed model outperforms existing models in English and Chinese by using an LLM-based evaluation method that closely aligns with human judgment. |
Understanding LLMs’ Cross-Lingual Context Retrieval: How Good It Is And Where It Comes From (2025.emnlp-main)
Copied to clipboard
| Challenge: | Cross-lingual context retrieval is a fundamental aspect of cross-lingual alignment, but the performance and mechanism of it for large language models (LLMs) remains unclear. |
| Approach: | They evaluate cross-lingual context retrieval of over 40 large language models . they use cross-linguistic machine reading comprehension as a representative scenario . |
| Outcome: | The results show that open LLMs show strong cross-lingual context retrieval ability . the results also show that their oracle performances improve after training . |
Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for measuring gender stereotypical bias in language models are inconsistencies . lack of explicit standards in data gathering can have detrimental effects on results . |
| Approach: | They propose that currently available benchmarks capture only partial facets of gender stereotypes . they apply a framework from social psychology to balance data across components of gender stereotypes based on stereotypical benchmarks. |
| Outcome: | The proposed framework improves correlation between different benchmarks by using simple balancing techniques. |
Linguistic and Embedding-Based Profiling of Texts Generated by Humans and Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on using LLMs to classify text as either human-written or machine-generated . |
| Approach: | They characterize human-written and machine-generated texts using a set of linguistic features across different linguistic levels such as morphology, syntax, and semantics. |
| Outcome: | The proposed model reveals that human-written texts exhibit simpler syntactic structures and more diverse semantic content. |
An Interdisciplinary Approach to Human-Centered Machine Translation (2025.emnlp-main)
Copied to clipboard
Marine Carpuat, Omri Asscher, Kalika Bali, Luisa Bentivogli, Fred Blain, Lynne Bowker, Monojit Choudhury, Hal Daumé Iii, Kevin Duh, Ge Gao, Alvin C Grissom II, Marzena Karpinska, Elaine C Khoong, William D. Lewis, Andre Martins, Mary Nurminen, Douglas W. Oard, Maja Popovic, Michel Simard, François Yvon
| Challenge: | Despite progress in MT, a gap persists between how the technology is developed and how it is used in real-world contexts. |
| Approach: | They propose a human-centered approach to machine translation (MT) they argue that MT should be evaluated with diverse goals and contexts of use . |
| Outcome: | The proposed approach emphasizes alignment of evaluation and design with diverse communicative goals and contexts of use. |
Exploring the Hidden Capacity of LLMs for One-Step Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Approach: | They show that large language models can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Outcome: | The proposed model can generate hundreds of accurate tokens in one token-parallel forward pass, when provided with only two learned embeddings. |
Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for token-level KV optimization and grouping of tokens are inefficient and strain compute and storage resources. |
| Approach: | They propose a mixture-of-expert approach that dynamically optimizes token-wise computation and memory allocation by a token-based expert-choice routing mechanism guided by learned importance scores. |
| Outcome: | The proposed approach retains all tokens while adaptively routing them to specialized experts with varying KV group sizes, balancing granularity and efficiency. |
PathwiseRAG: Multi-Dimensional Exploration and Integration Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing retrieval-augmented generation systems employ rigid retrieval strategies . static retrieval produces knowledge blind spots, missing connections between quantum algorithms and encryption vulnerabilities . |
| Approach: | PathwiseRAG addresses these challenges through intent-aware strategy selection . it constructs a directed acyclic graph of interconnected sub-problems and explores multiple reasoning trajectories . |
| Outcome: | The proposed framework achieves higher accuracy and better reliability than current systems. |
“Mm, Wat?” Detecting Other-initiated Repair Requests in Dialogue (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current conversational agents (CAs) do not recognize repair initiation, leading to breakdowns or disengagement. |
| Approach: | They propose a multimodal model to automatically detect repair initiation in Dutch dialogues by integrating linguistic and prosodic features grounded in Conversation Analysis. |
| Outcome: | The proposed model integrates linguistic and prosodic features grounded in Conversation Analysis to detect repair initiation in Dutch dialogues. |
R-BPE: Improving BPE-Tokenizers with Token Reuse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained language models prioritize high-resource languages in their vocabularies, leaving others with poor coverage. |
| Approach: | They propose a framework that reuses existing tokenizers and creates ID-based maps to resolve the new tokens of the chosen language. |
| Outcome: | The proposed framework reduces subword fertility by 24.4% on Arabic models and preserves performance on EnglishMMLU. |
Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) neglect the embedding layer. |
| Approach: | They propose a PEFT approach that modifies input embeddings without altering hidden layers. |
| Outcome: | Experiments show that TinyTE modifies embeddings without altering hidden layers . the proposed approach achieves competitive performance while requiring 0.0001% of parameters . |
Adversarial Attacks Against Automated Fact-Checking: A Survey (2025.emnlp-main)
Copied to clipboard
Fanzhen Liu, Sharif Abuadbba, Kristen Moore, Surya Nepal, Cecile Paris, Jia Wu, Jian Yang, Quan Z. Sheng
| Challenge: | Existing fact-checking systems are vulnerable to adversarial attacks that manipulate or generate claims, evidence, or claim-evidence pairs. |
| Approach: | They examine the impact of adversarial attacks on existing AFC systems and examine their impact on existing ones. |
| Outcome: | The findings highlight the need for resilient fact-checking frameworks in limiting misinformation spread and supporting public trust. |
WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild? (2025.emnlp-main)
Copied to clipboard
An-Lan Wang, Jingqun Tang, Lei Liao, Hao Feng, Qi Liu, Xiang Fei, Jinghui Lu, Han Wang, Hao Liu, Yuliang Liu, Xiang Bai, Can Huang
| Challenge: | Existing benchmarks for document understanding in the wild are based on scanned or digital documents . however, these benchmarks fail to capture the challenges posed by documents in the real world . |
| Approach: | They propose a new benchmark that incorporates a diverse set of manually captured document images reflecting real-world conditions. |
| Outcome: | The proposed model is based on a set of manually captured document images reflecting real-world conditions and is compared with digital or scanned documents. |
DCR: Quantifying Data Contamination in LLMs Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) memorize evaluation data during training, inflating performance metrics and undermining genuine generalization assessment. |
| Approach: | They propose a framework to detect and quantify benchmark data contamination (BDC) by synthesizing contamination scores via a fuzzy inference system. |
| Outcome: | The proposed framework detects and quantifies BDC risk across semantic, informational, data, and label levels. |
Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency (2025.emnlp-main)
Copied to clipboard
Svetlana Maslenkova, Clement Christophe, Marco AF Pimentel, Tathagata Raha, Muhammad Umar Salman, Ahmed Al Mahrooqi, Avani Gupta, Shadab Khan, Ronnie Rajan, Praveenkumar Kanithi
| Challenge: | Current dataset curation and bias assessment practices lack transparency . current approaches lack a thorough understanding of how data characteristics influence model behavior . |
| Approach: | They propose a comprehensive bias evaluation framework that integrates general benchmarks with a healthcare-specific methodology to probe for biases in a sensitive healthcare context. |
| Outcome: | The proposed approach to bias evaluation leverages established benchmarks and a healthcare-specific methodology. |
Surprise Calibration for Better In-Context Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for in-context learning apply fixed class priors across all inputs . existing methods rely on retraining and retrain models . |
| Approach: | They propose a Bayesian-based method to capture the temporal dynamics of class priors . they identify "surprise" as an informative signal for class prior shift . |
| Outcome: | The proposed method outperforms existing methods on a range of benchmark tasks. |
SPARK: Simulating the Co-evolution of Stance and Topic Dynamics in Online Discourse with LLM-based Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new framework for topic evolution and stance dynamics is needed to understand online discourse . topic evolution is central to understanding fragmentation of debates, spread of misinformation . |
| Approach: | They propose a stance and topic evolution reasoning framework for co-evolution of topics and stances through natural language interactions. |
| Outcome: | The proposed framework captures key empirical patterns across five real-world domains. |
Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite excelling at many natural language processing tasks, large language models fail to grasp the layered semantics of Drivelological text. |
| Approach: | They construct a benchmark dataset of over 1,200+ carefully curated and diverse examples across English, Mandarin, Spanish, French, Japanese, and Korean to examine their Drivelological characteristics. |
| Outcome: | The proposed models lack conceptual understanding and lack conceptual and semantic accuracy. |
Can Large Language Models be Effective Online Opinion Miners? (2025.emnlp-main)
Copied to clipboard
| Challenge: | OOMB is a novel benchmark designed to assess the ability of large language models (LLMs) to extract and analyze opinions from diverse and complex online environments. |
| Approach: | They propose an online opinion mining benchmark to assess the ability of large language models to extract and analyze opinions from diverse online environments. |
| Outcome: | The proposed benchmark assesses the ability of large language models to mine opinions effectively from diverse and complex online environments. |
Can Large Language Models Translate Unseen Languages in Underrepresented Scripts? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance in machine translation, but struggle with unseen low-resource languages. |
| Approach: | They propose a benchmark to evaluate translation for Mongolian and Yi using linguistic resources. |
| Outcome: | The proposed model can translate Mongolian (in traditional script) and Yi with the help of linguistic resources, but is limited in its ability to handle these languages effectively. |
InterIDEAS: Philosophical Intertextuality via LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new dataset aims to bridge philosophy, literary studies, and natural language processing (NLP) by integrating theories of intertextuality with bibliometric techniques. |
| Approach: | They propose a dataset that bridges philosophy, literary studies, and natural language processing (NLP) it combines theories of intertextuality from literary studies with bibliometric techniques and recent LLMs . |
| Outcome: | a new dataset bridges philosophy, literary studies, and natural language processing (NLP) to analyze intertextuality . the proposed method helps scholars understand the intellectual, social, and historical relations embedded in texts . it also contributes to the development of language models, authors say . |
KCS: Diversify Multi-hop Question Generation with Knowledge Composition Sampling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multi-hop question answering focus on generating simple questions and neglecting the integration of essential knowledge, such as relevant sentences within documents. |
| Approach: | They propose a framework to expand the diversity of generated multi-hop questions by sampling varied knowledge compositions within a given context. |
| Outcome: | The proposed framework improves the overall accuracy of knowledge composition selection by 3.9% on hotpotQA and 2WikiMultihopQA datasets. |
Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has found that large vision–language models are vulnerable to visual biases that inflate scores without altering semantic content. |
| Approach: | They propose a novel meta-evaluation benchmark that exhibits diverse score distributions. |
| Outcome: | The proposed model exhibits vulnerability across all domains, and combines multiple biases amplifies their effects, and pairwise evaluations are similarly susceptible. |
Disentangled Information Bottleneck for Adversarial Text Defense (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have proven that these deep models are super vulnerable to adversarial examples, which are slightly modified inputs. |
| Approach: | They propose a novel text defense method that separates the robust and non-robust features with a disentangled two-line framework rather than the one-line compression network in IB. |
| Outcome: | The proposed method outperforms six baselines on four datasets with accuracy improvements ranging from 3.8% to 20.7%. |
How do Language Models Reshape Entity Alignment? A Survey of LM-Driven EA Methods: Advances, Benchmarks, and Future (2025.emnlp-main)
Copied to clipboard
| Challenge: | Entity alignment (EA) is critical for knowledge graph (KG) integration. |
| Approach: | They propose a taxonomy that categorizes methods in three stages: data preparation, feature embedding, and alignment. |
| Outcome: | The proposed taxonomy categorizes methods in three key stages: data preparation, feature embedding, and alignment. |
Enhancing LLM-Based Social Bot via an Adversarial Learning Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Social media platforms provide an ideal testbed for large language models that exhibit human-like behavior. |
| Approach: | They propose an LLM-based social **Bot that enhances human-like generative capabilities through an adversarial learning framework. |
| Outcome: | The proposed framework generates human-like content aligned with diverse user profiles . it exhibits strong social responsiveness, more accurately modeling opinion dynamics . |
GER-LLM: Efficient and Effective Geospatial Entity Resolution with Large Language Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for integrating spatial data from diverse sources are limited by their reliance on large amounts of training data and their inability to incorporate commonsense knowledge. |
| Approach: | They propose a framework that integrates large language models into the GER pipeline. |
| Outcome: | The proposed framework improves on real-world geospatial datasets and shows that it is more efficient than state-of-the-art methods. |
CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in code large language models have produced repository-level code completion methods that automatically predict the unfinished code based on the broader information from the repository. |
| Approach: | They propose a framework to identify relevant knowledge for retrieval-augmented repository-level code completion. |
| Outcome: | The proposed framework significantly outperforms state-of-the-art methods on ReccEval and CCEval. |
Searching for the Most Human-like Emergent Language (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on emergent communication systems to generate languages with high statistical similarity to human languages has not been done. |
| Approach: | They propose to optimize a signalling game-based emergent communication environment to generate state-of-the-art emergentic languages with a high degree of similarity to human language. |
| Outcome: | The proposed language generates state-of-the-art on XferBench benchmark, demonstrating its similarity to human language and entropy-minimization properties. |
Does Context Matter? A Prosodic Comparison of English and Spanish in Monolingual and Multilingual Discourse Settings (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large number of studies on prosody in languages have focused on monolingual discourse contexts . a recent study focused on the prosodic features of monolingual speech in multilingual contexts. |
| Approach: | They compare prosody of monolingual English and Spanish in monolingual and multilingual settings . they find that monolingual speech produced in a monolingual context is prosodically different from that produced in multilingual context . |
| Outcome: | The proposed study is the first to incorporate multilingual discourse contexts into the study of native-level monolingual prosody. |
ZERA: Zero-init Instruction Evolving Refinement Agent – From Zero Instructions to Structured Prompts via Principle-based Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve large language model performance focus on user prompts and require large sample sizes and long iteration cycles. |
| Approach: | They propose a framework that jointly optimizes both system and user prompts . they evaluate ZERA across five LLMs and nine diverse datasets spanning reasoning, summarization, and code generation tasks. |
| Outcome: | The proposed framework improves prompt construction over baselines and is available on github . it scores prompts using eight generalizable criteria and revises prompts based on structured critiques. |
Toward Machine Interpreting: Lessons from Human Interpreting Studies (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current speech translation systems are static and do not adapt to real-world situations in ways human interpreters do. |
| Approach: | They propose to model human interpreting using a new language model to improve usability . they argue that there is great potential to adopt many human interpreted principles . |
| Outcome: | The proposed models can be used to improve human interpreting and improve translation performance. |
FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing game benchmarks lack diversity and evaluate GUI agents on completing entire storylines. |
| Approach: | They propose a benchmark of 34 Flash-based adventure games to test full story arc completion and tackle observation-behavior gap. |
| Outcome: | The proposed benchmarks show GUI agents struggle with full story arc completion while others improve on observation-behavior gaps. |
FLARE: Faithful Logic-Aided Reasoning and Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern Question Answering (QA) and Reasoning approaches with Large Language Models (LLMs) use Chain-of-Thought (CoT) prompting but struggle with ambiguous tasks. |
| Approach: | They propose a method that uses large language models to plan solutions and formalize queries without external solvers to generate outputs faithful to their intermediate reasoning chains. |
| Outcome: | The proposed method achieves SOTA results on 7 out of 9 diverse reasoning benchmarks and 3 out of 3 logic inference benchmarks while enabling measurement of reasoning faithfulness. |
Discourse-Driven Code-Switching: Analyzing the Role of Content and Communicative Function in Spanish-English Bilingual Speech (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that a range of speaker and listener attributes affect or correlate with the prevalence of code-switching during conversation. |
| Approach: | They analyze the names of entities and dialogue acts present in a Spanish-English spontaneous speech corpus and build a predictive model of CSW. |
| Outcome: | The proposed model is the first to take a discourse-sensitive approach to understanding pragmatic and referential cues of bilingual speech. |
Can Large Language Models Translate Spoken-Only Languages through International Phonetic Transcription? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on spoken-only languages has focused on low-resource languages . spoken- only languages are among the most vulnerable to extinction . |
| Approach: | They propose a unified language understanding framework that learns to translate spoken-only languages via in-context learning. |
| Outcome: | The proposed framework can translate spoken-only languages into high-resource languages using phonetic transcription and automatic dictionary construction and knowledge retrieval. |
ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Scientific fact-checking has largely focused on textual and tabular sources, neglecting scientific charts. |
| Approach: | They propose a benchmark for scientific fact-checking grounded in scientific charts . climateViz comprises 49,862 claims paired with 2,896 visualizations . results show current models struggle to perform fact- checking when statistical reasoning is required . |
| Outcome: | The climateviz benchmark is the first large-scale benchmark for scientific fact-checking . it includes 49,862 claims paired with 2,896 visualizations labeled as support, refute, or not enough . |
Bridging the Gap Between Molecule and Textual Descriptions via Substructure-aware Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models lack the ability to learn fine-grained alignments between molecules and their descriptions. |
| Approach: | They propose a molecule–text learning framework based on substructure-aware alignments that augments original molecule-description pairs with additional alignment signals derived from molecular substructures and chemical phrases. |
| Outcome: | The proposed framework outperforms state-of-the-art models on a wide range of molecular benchmarks. |
SLlama: Parameter-Efficient Language Model Architecture for Enhanced Linguistic Competence Under Strict Data Constraints (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) have shown remarkable performance across a wide array of tasks. |
| Approach: | They propose an architecture that preserves parameter efficiency of tied models without sacrificing representational benefits of untied embeddings. |
| Outcome: | The proposed architecture achieves a 31.72% improvement in linguistic knowledge acquisition over the baseline model. |
What You See is What You Ask: Evaluating Audio Descriptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies evaluate audio descriptions (ADs) using trimmed clips, but writing them is subjective. |
| Approach: | They propose a QA benchmark that evaluates audio descriptions at the level of short, coherent video segments. |
| Outcome: | The proposed evaluation paradigm addresses two themes central to ADs . it compares two humannarrated AD tracks and shows that current methods lag behind human-authored ADs. |
TAPS: Tool-Augmented Personalisation via Structured Tagging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to personalise tool use overlook the role of personalisation in guiding tool use. |
| Approach: | They propose a tool-augmented large language model that integrates user preferences into goal-oriented dialogue agents by leveraging a structured tagging tool and an uncertainty-based tool detector. |
| Outcome: | The proposed solution significantly improves the ability of LLMs to incorporate user preferences, achieving the new state-of-the-art for open source models on the NLSI task. |
Investigating How Pre-training Data Leakage Affects Models’ Reproduction and Detection Capabilities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies do not examine how leaked instances in training datasets influence LLMs’ output and detection capabilities. |
| Approach: | They conduct an experimental survey to examine the relationship between data leakage in training datasets and its effects on the generation and detection by Large Language Models (LLMs). |
| Outcome: | The results show that enhancing leakage detection through few-shot learning can help mitigate the impact of the leakage rate in the training data on detection performance. |
Walk and Read Less: Improving the Efficiency of Vision-and-Language Navigation via Tuning-Free Multimodal Token Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large models achieve strong performance on Vision-and-Language Navigation tasks, but are costly to run in resource-limited environments. |
| Approach: | They propose a method to prune large models to minimize information loss . they use navigation-specific traits to filter the model into foreground and background . |
| Outcome: | The proposed method outperforms previous work on standard VLN benchmarks while saving 50% FLOPS. |
Connecting the Knowledge Dots: Retrieval-augmented Knowledge Connection for Commonsense Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that large language models exhibit a limited understanding of commonsense reasoning due to the necessity of implicit knowledge that is rarely expressed in text. |
| Approach: | They propose a retrieval-augmented knowledge connection framework that transforms indirectly relevant documents into a direct explanation to answer a given question. |
| Outcome: | The proposed framework outperforms state-of-the-art (SOTA) benchmarks and achieves +2.0% and +4.6% average accuracy on in-domain (ID) and out-of domain (OOD) benchmark. |
Agent-as-Judge for Factual Summarization of Long Narratives (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated near-human performance in summarization tasks based on traditional metrics such as ROUGE and BERTScore . however, these metrics do not adequately capture critical aspects of summarizing quality, such as factual accuracy, especially for long narratives. |
| Approach: | They propose a framework that evaluates and refines factuality in narrative summarization by leveraging a Character Knowledge Graph extracted from input narrative. |
| Outcome: | The proposed framework evaluates factuality and provides actionable guidance for refinement. |
DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent measures of factual precision use a decompose-then-verify framework . decontextualization is the process of augmenting subclaims with necessary context . |
| Approach: | They evaluate different decomposition, decontextualization and verification strategies . they introduce a deconstructualization aware verification method that validates subclaims in context . |
| Outcome: | The proposed method decomposes claims and independently verifyes them . it introduces a decontextualization aware verification method that validates subclaims in context . |
RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models that address ambiguity are limited in their ability to address it . ambiguities are an inherent feature of human language, according to research . |
| Approach: | They propose a dataset targeting referential ambiguity in image-based question answering . they find that current language models lack robust strategies to deal with ambiguities . |
| Outcome: | The proposed dataset shows that state-of-the-art models fail to address ambiguity . ambiguities are an inherent feature of human language, according to research . |
Resource-Rational Noisy-Channel Language Processing: Testing the Effect of Algorithmic Constraints on Inferences (2025.emnlp-main)
Copied to clipboard
| Challenge: | a fundamental question in psycholinguistics is how comprehenders form interpretations of utterances that they hear or see. |
| Approach: | They propose to use a language model as a prior and an error model to encode likelihoods to perform incremental and approximate probabilistic inferences over intended sentences and production errors. |
| Outcome: | The proposed model captures previously established patterns in human sentence processing, and trade-off between human-like noisy-channel inferences and computational resources falls out of the model. |
In Benchmarks We Trust ... Or Not? (2025.emnlp-main)
Copied to clipboard
Ine Gevers, Victor De Marez, Jens Van Nooten, Jens Lemmens, Andriy Kosar, Ehsan Lotfi, Nikolay Banar, Pieter Fivez, Luna De Bruyne, Walter Daelemans
| Challenge: | Existing benchmarks for Large Language Models (LLMs) are inadequate and lack a clear solution. |
| Approach: | They propose checklists to cover all aspects of benchmarking issues, both for benchmark creation and usage. |
| Outcome: | The proposed checklists cover all aspects of benchmarking issues, both for benchmark creation and usage. |
Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to RPAs focus on static role profiles, overlooking dynamic perceptual abilities inherent to humans. |
| Approach: | They propose a framework that combines adaptive temporal sampling with dynamic and static role profiles. |
| Outcome: | The proposed framework combines adaptive temporal sampling with dynamic and static role profiles. |
Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that multilingual models encode languagespecific information and language-agnostic features, but the nature and interaction of these representations is not fully understood. |
| Approach: | They propose a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning). |
| Outcome: | The proposed tasks show that language discrimination declines over training and strengthens over time and stabilizes in deeper layers. |
Rethinking Text-based Protein Understanding: Retrieval or LLM? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on integrating protein-related knowledge into large language models through continued pretraining and multi-modal alignment. |
| Approach: | They propose a retrieval-enhanced method which significantly outperforms fine-tuned LLMs for protein-to-text generation and shows accuracy and efficiency in training-free scenarios. |
| Outcome: | The proposed method significantly outperforms fine-tuned LLMs for protein-to-text generation and shows accuracy and efficiency in training-free scenarios. |
Grounded Semantic Role Labelling from Synthetic Multimodal Data for Situated Robot Commands (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing symbolic parsers lack flexibility to operate in complex, dynamic environments. |
| Approach: | They propose a framework that combines frame semantics with perceptual grounding to enable robots to interpret commands via multimodal logical forms. |
| Outcome: | The proposed framework produces over 11,000 image-command pairs and lowers the cost of manual parsers. |
Easy as PIE? Identifying Multi-Word Expressions with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multiword expressions (MWEs) are a semantically non-compositional subclass of multiword expression . authors show that prompt-based LLMs can perform competitively with supervised models . |
| Approach: | They propose a prompt-based approach to identify idiomatic expressions in running text . they find prompt-driven LLMs can perform competitively with supervised models . |
| Outcome: | The proposed approach can perform well with supervised models on annotated data. |
Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has identified retrieval heads as a subset of attention heads responsible for retrieving salient information in long-context language models. |
| Approach: | They introduce a retrieval head that uses attention scores to enhance retrieval from long context . they use QRRetriever to select the most relevant parts with the highest retrieval scores . |
| Outcome: | The proposed retrieval heads outperform other retrieval-based retrieval retrievers on BEIR benchmarks. |
Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have shown promise in hateful meme detection, but they face limitations like sub-optimal performance and limited out-of-domain generalization capabilities. |
| Approach: | They propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. |
| Outcome: | The proposed framework outperforms larger agentic systems in detecting hateful memes under adversarial attacks while maintaining the general vision-language capabilities of LMMs. |
Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in multimodal reasoning overlook the audio modality. |
| Approach: | They propose a large-scale audio language model for deep reasoning that leverages a multitask audio dataset. |
| Outcome: | The proposed model performs well across key benchmarks including MMAU-mini, AIR-Bench chat/foundation, and MELD. |
From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that map variable acoustic inputs into appropriate articulatory movements without explicit instruction are inadequate for infants. |
| Approach: | They propose a model that maps acoustic inputs into articulatory movements without explicit instruction for infants. |
| Outcome: | The proposed model outperforms MFCC features in both single- and multi-speaker settings and provides optimal representations for articulatory learning. |
REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs face ranking uncertainty, unstable top-k recovery, and high token cost due to token-intensive prompting. |
| Approach: | They propose a re-ranking framework that captures uncertainty and refines LLM-derived relevance through recursive Bayesian updates. |
| Outcome: | The proposed framework outperforms state-of-the-art re-rankers while reducing token usage and latency. |
PLLuM-Align: Polish Preference Dataset for Large Language Model Alignment (2025.emnlp-main)
Copied to clipboard
Karolina Seweryn, Anna Kołos, Agnieszka Karlińska, Katarzyna Lorenc, Katarzyna Dziewulska, Maciej Chrabaszcz, Aleksandra Krasnodebska, Paula Betscher, Zofia Cieślińska, Katarzyna Kowol, Julia Moska, Dawid Motyka, Paweł Walkowiak, Bartosz Żuk, Arkadiusz Janz
| Challenge: | Large language models generate preferred responses while avoiding harmful or inappropriate outputs, despite their ability to generate cross-language transferability. |
| Approach: | They introduce the first Polish preference dataset PLLuM-Align, created entirely through human annotation to reflect Polish language and cultural nuances. |
| Outcome: | The proposed dataset lays the groundwork for more aligned Polish LLMs and contributes to the broader goal of multilingual alignment in underrepresented languages. |
Graph-R1: Incentivizing the Zero-Shot Graph Learning Capability in LLMs via Explicit Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Reasoning Models (LLMs) provide a zero-shot alternative via explicit, long chain-of-thought reasoning. |
| Approach: | They propose a GNN-free approach that reformulates graph tasks as textual reasoning problems solved by LRMs. |
| Outcome: | The proposed approach outperforms state-of-the-art baselines in zero-shot settings, producing interpretable and effective predictions. |
Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches for detecting and mitigating embedded stereotypes rely on carefully annotated datasets like StereoSet and CrowS-Pairs, which are only in English and reflect stereotypes from a few English-speaking countries. Existing datasets, especially translation-based ones, often overlook such cultural distinctions. |
| Approach: | They propose a cost-efficient human-LLM collaborative annotation framework to construct a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries. |
| Outcome: | The proposed framework can identify nuanced, region-specific biases across Spanish-supporting LLMs and is adaptable to other languages and regions. |
Can Large Language Models Be Good Language Teachers? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved remarkable success across diverse domains, but their potential as effective language teachers remains inadequately assessed. |
| Approach: | They propose a framework to evaluate Chinese language teachers' pedagogical competence against international standards. |
| Outcome: | The proposed framework evaluates 13 latest multilingual and Chinese LLMs against international standards for Chinese language teachers. |
Empowering Math Problem Generation and Reasoning for Large Language Model via Synthetic Data based Continual Learning Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing learning frameworks for large language models (LLMs) for math problem generation are limited and lack quality data. |
| Approach: | They propose a synthetic data based continual learning framework to improve LLMs ability for MPG and math reasoning. |
| Outcome: | The proposed framework improves performance on large language models and math reasoning using supervised fine-tuning, data synthesis and direct preference optimization. |
Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) pre-trained on massive text data in many languages are preferred solution for various Natural Language processing tasks. |
| Approach: | They compare tokenization parity and information parity as representational biases in pre-trained models . they find TP is better predictor of performance on tasks reliant on syntactic and morphological cues . |
| Outcome: | The proposed model improves on dialect classification, topic classification, and extractive question answering tasks. |
Evaluating the Evaluators: Are readability metrics good measures of readability? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Plain language summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. |
| Approach: | They conduct a thorough survey of literature on plain language summarization (PLS) and find that traditional readability metrics are not compared to human judgments. |
| Outcome: | The proposed language models better capture deeper measures of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. |
Text Takes Over: A Study of Modality Bias in Multimodal Intent Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines the effectiveness of large language models and non-LLMs in multimodal intent detection . large-scale multimodal data integrations include text, audio, and visual inputs . |
| Approach: | They propose a framework to debias multimodal intent detection datasets by using human evaluation. |
| Outcome: | The proposed framework debiases the datasets and shows that mistral-7B outperforms most competitive models by approximately 9% on MIntRec-1 and 4% on MIndRec2.0. |
What’s in a prompt? Language models encode literary style in prompt embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models encode textual information using high-dimensional latent spaces . many studies have investigated how conceptual content of words translates into geometrical relationships between their vector representations . |
| Approach: | They use literary pieces to show that intangible, rather than factual, aspects of the prompt are contained in deep representations. |
| Outcome: | The results show that word-to-vec(tor) embeddings are more complex than other models. |
Identifying and Answering Questions with False Assumptions: An Interpretable Approach (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate misleading answers because of hallucinations . despite their capabilities, LLMs suffer from hallucinisms, which leads to unfaithful answers . |
| Approach: | They propose a method to identify and answer questions with false assumptions . they first investigate whether the problem reduces to fact verification . then, they leverage external evidence to mitigate hallucinations . |
| Outcome: | The proposed approach reduces the problem to fact verification and provides interpretable answers by pinpointing the false assumptions. |
VisFinEval: A Scenario-Driven Chinese Multimodal Benchmark for Holistic Financial Understanding (2025.emnlp-main)
Copied to clipboard
Zhaowei Liu, Xin Guo, Haotian Xia, Lingfeng Zeng, Fangqi Lou, Jinyi Niu, Mengping Li, Qi Qi, Jiahuan Li, Wei Zhang, Yinglong Wang, Weige Cai, Weining Shen, Liwen Zhang
| Challenge: | Existing benchmarks focus on text comprehension, but MLLMs lack the ability to integrate visual data over financial visuals. |
| Approach: | They evaluate 21 state-of-the-art multimodal large language models in a zero-shot setting . they use an annotated question–answer pair from eight common financial image modalities . |
| Outcome: | The new benchmark outperforms existing models but trailed financial experts by 14 percentage points. |
Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research in vision-language models has centered around the possibility of equipping them with implicit long-form chain-of-thought reasoning via distillation and reinforcement learning. |
| Approach: | They propose a Monte Carlo Tree Search-inspired algorithm that injects subquestion–subanswer pairs into the model’s output stream to elicit hidden knowledge and induce long reasoning traces. |
| Outcome: | The proposed method yields a 2% improvement on MMMU-PRO, including a significant 9% gain in Liberal Arts. |
LLMs Don’t Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations (2025.emnlp-main)
Copied to clipboard
Harry Mayne, Ryan Othniel Kearns, Yushi Yang, Andrew M. Bean, Eoin D. Delaney, Chris Russell, Adam Mahdi
| Challenge: | Existing studies on language models' ability to explain their decisions in natural language have focused on self-generated counterfactual explanations (SCEs). |
| Approach: | They evaluate whether LLMs can generate valid counterfactuals and minimal ones . authors suggest that SCEs are, at best, an ineffective explainability tool . |
| Outcome: | The proposed language models can explain their decisions in natural language, the study finds . the models can produce valid counterfactual explanations, but make small edits that fail to change predictions. |
Grounding Multilingual Multimodal LLMs With Cultural Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new data-centric approach could address cultural gaps in multimodal large language models . despite being trained on billions of image-text pairs, today's models are biased towards English and Western data. |
| Approach: | They propose a data-centric approach that directly grounds MLLMs in cultural knowledge. |
| Outcome: | The proposed approach outperforms open-source models on cultural-focused benchmarks without degrading results on mainstream vision–language tasks. |
Following Length Constraints in Instructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing instruction following models fail to follow length constraints in their evaluations. |
| Approach: | They propose to train models that can be controlled at inference time with instructions containing desired length constraints. |
| Outcome: | The proposed models outperform standard instruction following models in length instructed evaluations. |
Memory-QA: Answering Recall Questions Based on Multimodal Memories (2025.emnlp-main)
Copied to clipboard
Hongda Jiang, Xinyuan Zhang, Siddhant Garg, Rishab Arora, Shiun-Zu Kuo, Jiayang Xu, Aaron Colak, Xin Luna Dong
| Challenge: | Memory-QA is a real-world task that involves answering recall questions about visual content from previously stored multimodal memories. |
| Approach: | They propose a memory-QA task that involves answering recall questions about visual content from previously stored multimodal memories. |
| Outcome: | The proposed solution improves memory recording, compression, storage, and search accuracy over state-of-the-art solutions. |
NEXUS: Network Exploration for eXploiting Unsafe Sequences in Multi-Turn LLM Jailbreaks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to large language models suffer from limited exploration of adversarial space . multi-turn jailbreaks that distribute malicious intent across benign exchanges are vulnerable . NEXUS aims to exploit the adversarials of LLMs for maximum effectiveness in jailbreak scenarios . |
| Approach: | They propose a framework for constructing, refining, and executing optimized multi-turn attacks . NEXUS builds a semantic network of thought that captures a comprehensive representation of the adversarial space . |
| Outcome: | NEXUS can achieve higher attack success rate than state-of-the-art approaches . it builds a semantic network of thought that captures a comprehensive representation of the adversarial space . |
Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enabled strong reasoning capabilities through Chain-of-Thought (CoT) prompting. |
| Approach: | They propose a framework that integrates cognitively inspired reasoning paradigms with linguistic constraints to reduce token usage while preserving reasoning accuracy. |
| Outcome: | The proposed framework reduces token usage while preserving reasoning accuracy across 18 reasoning datasets across multiple domains, languages, and modalities. |
From Language to Cognition: How LLMs Outgrow the Human Language Network (2025.emnlp-main)
Copied to clipboard
Badr AlKhamissi, Greta Tuckute, Yingtian Tang, Taha Osama A Binhuraib, Antoine Bosselut, Martin Schrimpf
| Challenge: | Large language models exhibit remarkable similarity to neural activity in the human language network, but their properties remain unclear. |
| Approach: | They benchmark 34 training checkpoints spanning 300B tokens across 8 different model sizes . they find that brain alignment tracks the development of formal linguistic competence more closely than functional linguistic competency. |
| Outcome: | The results show that large language models exhibit similarity to human language networks . they show that the correlation between next-word prediction and brain alignment fades once models surpass human language proficiency. |
Logos as a Well-Tempered Pre-train for Sign Language Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing data on sign languages is limited, resulting in ambiguity in dataset labeling . similar signs can have different semantic meanings, which leads to ambiguous labeling. |
| Approach: | They propose to use a Russian sign language dataset as a universal encoder for other languages . they propose to explicitly annotate visually similar signs to improve model quality . |
| Outcome: | The proposed model outperforms current state-of-the-art models and gets competitive results for the AUTSL dataset. |
Hallucination Detection in LLMs Using Spectral Features of Attention Maps (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance across tasks but remain prone to hallucinations. |
| Approach: | They propose a method that uses attention maps to detect hallucinations . they propose to use top-k eigenvalues of the attention maps as input to probes . |
| Outcome: | The proposed method achieves state-of-the-art hallucination detection performance among attention-based methods. |
Composable Cross-prompt Essay Scoring by Merging Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to cross-prompt automated essay scoring use all available sources . however, using multiple sources for continual adaptation raises privacy concerns . |
| Approach: | They propose a source-free adaptation approach that selectively merges the parameters of individual models without further access to the source datasets. |
| Outcome: | The proposed method outperforms joint-training methods on all sources while maintaining computational efficiency. |
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts (2025.emnlp-main)
Copied to clipboard
Yuho Lee, Jiaqi Deng, Nicole Hee-Yeon Kim, Hyangsuk Min, Taewon Yun, Minjeong Ban, Kim Yul, Hwanjun Song
| Challenge: | HAMLET is a framework for evaluating the long-context comprehension of large language models. |
| Approach: | They propose a framework for evaluating the long-context comprehension of large language models . HAMLET structures key information of source texts into a three-level hierarchy . |
| Outcome: | HAMLET achieves 90% agreement with expert judgments while reducing evaluation cost by up to 25. |
Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templates (2025.emnlp-main)
Copied to clipboard
Hy Dang, Tianyi Liu, Zhuofeng Wu, Jingfeng Yang, Haoming Jiang, Tao Yang, Pei Chen, Zhengyang Wang, Helen Wang, Huasheng Li, Bing Yin, Meng Jiang
| Challenge: | Large language models (LLMs) have strong reasoning and tool-use capabilities, yet fail in real-world tool-interactions due to incorrect parameterization, poor tool selection, or misinterpretation of user intent. |
| Approach: | They propose a curriculum-inspired framework that leverages structured reasoning templates to guide LLMs through more deliberate step-by-step instructions for generating function calls. |
| Outcome: | The proposed framework reduces tool-use errors and improves interpretability and transparency of tool-using agents. |
Evaluation and Facilitation of Online Discussions in the LLM Era: A Survey (2025.emnlp-main)
Copied to clipboard
Katerina Korre, Dimitris Tsirmpas, Nikos Gkoumas, Emma Cabalé, Danai Myrtzani, Theodoros Evgeniou, Ion Androutsopoulos, John Pavlopoulos
| Challenge: | Recent advances in LLMs enable artificial facilitation agents to not only moderate content, but also actively improve the quality of interactions. |
| Approach: | They propose a taxonomy on discussion quality evaluation and a new taxonomies for intervention and facilitation strategies. |
| Outcome: | The proposed methods synthesize ideas from Natural Language Processing (NLP) and Social Sciences to provide a taxonomy on discussion quality evaluation, and a roadmap of good practices and future research directions. |
Temporal Scaling Law for Large Language Models (2025.emnlp-main)
Copied to clipboard
Yizhe Xiong, Xiansheng Chen, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Zhenpeng Su, Wei Huang, Jianwei Niu, Jungong Han, Guiguang Ding
| Challenge: | Existing studies have found that the test loss of LLMs scales as power-laws with model size, computational budget, and dataset size. |
| Approach: | They propose a concept of Temporal Scaling Law to study test loss of LLMs . they break down test loss into fine-grained token positions and develop a dynamic hyperbolic-law . |
| Outcome: | The proposed model predicts the test loss of LLMs as the training steps scale up. |
Reframe Your Life Story: Interactive Narrative Therapist and Innovative Moment Assessment with Large Language Models (2025.emnlp-main)
Copied to clipboard
Yi Feng, Jiaqi Wang, Wenxuan Zhang, Zhuang Chen, Shen Yutong, Xiyao Xiao, Minlie Huang, Liping Jing, Jian Yu
| Challenge: | Existing approaches to mental health support lack realism and capture therapeutic progression over time. |
| Approach: | They propose a framework that simulates expert narrative therapists by planning therapeutic stages, guiding reflection levels, and generating contextually appropriate responses through retrieval-augmentation. |
| Outcome: | The proposed framework outperforms standard methods in quality and depth on 260 simulated clients and 230 human participants. |
From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual and cross-cultural WAT reveal how culture modulates perceptual and interactive patterns. |
| Approach: | They propose to embed cultural-specific semantic associations directly within large language models (LLMs) to address cultural preference. |
| Outcome: | The proposed model significantly improves cross-cultural alignment, capturing diverse semantic associations. |
Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data (2025.emnlp-main)
Copied to clipboard
Shenglai Zeng, Jiankun Zhang, Pengfei He, Jie Ren, Tianqi Zheng, Hanqing Lu, Han Xu, Hui Liu, Yue Xing, Jiliang Tang
| Challenge: | Existing literature suggests that RAG systems may face privacy issues when the retrieval process involves private data. |
| Approach: | They propose a two-stage synthetic data generation paradigm that uses attributes to preserve contextual information from the original data. |
| Outcome: | The proposed approach preserves key contextual information from the original data while reducing privacy risks. |
AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender (2025.emnlp-main)
Copied to clipboard
Weixiang Zhao, Jiahe Guo, Yulin Hu, Yang Deng, An Zhang, Xingyu Sui, Xinyang Han, Yanyan Zhao, Bing Qin, Tat-Seng Chua, Ting Liu
| Challenge: | Activation steering offers training-free defense but relies on fixed steering coefficients, resulting in suboptimal protection and increased false rejections of benign inputs. |
| Approach: | They propose an adaptive activation steering method that dynamically adjusts model behavior based on input characteristics. |
| Outcome: | The proposed method outperforms baseline methods across multiple jailbreak attacks with minimal impact on utility. |
Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable performance on question-answering tasks due to their superior capabilities in natural language understanding and generation. |
| Approach: | They propose a structured taxonomy that categorizes the methodology of synthesizing LLMs and knowledge graphs for QA according to the categories of QA and the KG’s role when integrating with LLM. |
| Outcome: | The proposed taxonomy categorizes the methods according to the categories of QA and the KG’s role when integrating with LLMs. |
TFDP: Token-Efficient Disparity Audits for Autoregressive LLMs via Single-Token Masked Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for auditing autoregressive Large Language Models for disparities are limited and expensive. |
| Approach: | They propose a method to detect disparities in autoregressive Large Language Models by token querying . they propose 'token-focused disparity probing' to measure disparities between sentence pairs . |
| Outcome: | The proposed method detects disparities with 42 times fewer output tokens than previous methods. |
Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on cultural understanding with vision-language models primarily emphasize geographic diversity, often overlooking the critical temporal dimensions. |
| Approach: | They propose a multimodal vision-language model that examines temporal features and cultural image transcreation. |
| Outcome: | The novel model performs better than non-experts on visual cutural understanding but falls short to human experts on cultural image transcreation task. |
MERMAID: Multi-perspective Self-reflective Agents with Generative Augmentation for Emotion Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models struggle to handle ambiguous emotional expressions and implicit affective cues, which are crucial for affective understanding but largely overlooked. |
| Approach: | They propose a multi-agent framework that integrates a self-reflection module, an emotion-guided visual augmentation module, and a cross-modal verification module to enhance emotion recognition. |
| Outcome: | Extensive experiments show that MERMAID outperforms existing methods and achieves absolute accuracy gains of 8.70%–27.90% across diverse benchmarks. |
Personality Vector: Modulating Personality of Large Language Models by Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to induce personality in large language models (LLMs) fail to capture the continuous nature of human traits. |
| Approach: | They propose a method for personality modulation in large language models by model merging by subtracting weights of pre-trained models from those of fine-tuned models. |
| Outcome: | The proposed method allows LLMs to exhibit desired personality traits without additional training. |
Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current writing agents rely on predefined workflows and rigid thinking patterns to generate outlines before writing . authors propose a framework for long-form writing agents built on heterogeneous recursive planning . |
| Approach: | They propose a general agent framework that achieves human-like adaptive writing . they propose recursive task decomposition and dynamic integration of task types . |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on both fiction and technical report generation. |
Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal large language models are increasingly deployed in open-ended, real-world environments where inputs are messy, underspecified, and not always trustworthy. |
| Approach: | They evaluate multimodal large language models in real-world environments where inputs are messy, underspecified, and not always trustworthy. |
| Outcome: | The proposed models fail to detect hidden issues even when they possess the necessary perceptual and reasoning skills. |
PrimeX: A Dataset of Worldview, Opinion, and Explanation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that an individual's worldview -or beliefs about the overall character of the world -can explain persistent behavioral patterns and correlates with personality, well-being, political, religious, and demographic variables. |
| Approach: | They develop a dataset of public opinion survey data from 858 US residents with written explanations from the respondents for why they hold specific opinions and the Primal World Belief survey for assessing respondent worldview. |
| Outcome: | The proposed model can be used to better represent an individual's belief system and improve opinion prediction. |
LASER: An LLM-based ASR Scoring and Evaluation Rubric (2025.emnlp-main)
Copied to clipboard
| Challenge: | Standard ASR evaluation metrics like word error rate penalize morphological and syntactic nuances that do not significantly alter sentence semantics. |
| Approach: | They propose an LLM-based scoring rubric LASER that leverages state-of-the-art LLMs’ in-context learning abilities to learn from prompts with detailed examples. |
| Outcome: | The proposed scoring rubric combines state-of-the-art learning capabilities with a high correlation score with human annotations. |
Improving Zero-shot Sentence Decontextualisation with Content Selection and Planning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extracting sentences from documents leave some implicit discourse information in the sentence unresolved due to their lack of context. |
| Approach: | They propose a content selection framework for zero-shot decontextualisation which determines what content should be mentioned and in what order for a sentence to be understood out of context. |
| Outcome: | The proposed framework outperforms existing methods in rewriting sentences that lack context while maintaining original meaning. |
Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) can process text, images, and audio, but they introduce privacy vulnerabilities. |
| Approach: | They propose a compositional structured prompt attack to exploit MRAG privacy vulnerabilities . they show that LMMs can generate outputs resembling retrieved content . |
| Outcome: | The proposed approach generates outputs resembling retrieved content and exposes sensitive information. |
Code Execution as Grounded Supervision for LLM Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating high-quality CoT data rely on costly human annotations and error-prone CoT. |
| Approach: | They propose a method that extracts verifiable, step-by-step reasoning traces from code execution and transforms them into a natural language CoT reasoning. |
| Outcome: | The proposed method produces highly accurate reasoning data and reduces overall token length during inference by reducing meaningless repetition and overthinking. |
Subjective Behaviors and Preferences in LLM: Language of Browsing (2025.emnlp-main)
Copied to clipboard
Sai Sundaresan, Harshita Chopra, Atanu R. Sinha, Koustava Goswami, Nagasai Saketh Naidu, Raghav Karan, N Anushka
| Challenge: | Large Language Models (LLMs) fuel expectations that a single trained model can effectively align with preferences of myriad users for a given task within a domain. |
| Approach: | They introduce clusterwise LM training, HeTLM, appropriate for subjective behaviors . authors say small LM outperforms large pretrained LMs; heterogeneous cluster specific set of parameters outperformed single LM . |
| Outcome: | The proposed model outperforms large pretrained or finetuned models in the domain of subjective behavior and preferences. |
Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts (2025.emnlp-main)
Copied to clipboard
Michal Golovanevsky, William Rudman, Michael A. Lepori, Amir Bar, Ritambhara Singh, Carsten Eickhoff
| Challenge: | Multimodal Large Language Models perform well on visual question answering tasks, but it remains unclear whether their reasoning relies more on memorized world knowledge or on visual information present in the input image. |
| Approach: | They propose a dataset of visual-realistic counterfactuals that put world knowledge priors into conflict with visual input. |
| Outcome: | The proposed dataset puts world knowledge priors into conflict with visual input . it shows that model predictions shift toward visual evidence in mid-to-late layers . |
Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models (2025.emnlp-main)
Copied to clipboard
Benyamin Jamialahmadi, Parsa Kavehzadeh, Mehdi Rezagholizadeh, Parsa Farinneya, Hossein Rajabzadeh, Aref Jafari, Boxing Chen, Marzieh S. Tahaei
| Challenge: | Existing methods for dynamic inference are limited by hardware inefficiencies or performance degradation. |
| Approach: | They propose a framework for depth-based dynamic inference that freezes the pre-trained model and inserts additional transformer layers at selected exit points. |
| Outcome: | The proposed framework outperforms state-of-the-art methods such as Flextron and Layerskip on multiple models at various scales, as well as other leading compression techniques across a variety of benchmarks. |
Social Genome: Grounded Social Reasoning Abilities of Multimodal Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Social reasoning is a core competency of social intelligence and requires specialized neural and cognitive systems to be able to interpret multimodal interactions. |
| Approach: | They propose to use social reasoning traces to generate fine-grained explanations using external knowledge. |
| Outcome: | The proposed model is based on 272 videos of human interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. |
Profiler: Black-box AI-generated Text Origin Detection via Context-aware Inference Pattern Analysis (2025.emnlp-main)
Copied to clipboard
Hanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang, Guangyu Shen, Kaiyuan Zhang, Shengwei An, Guanhong Tao, Xiangyu Zhang
| Challenge: | Existing methods to identify the origin of AI-generated texts fail to identify origin due to the high similarity of different LLMs. |
| Approach: | They propose a black-box AI-generated text origin detection method which accurately predicts the origin of an input text by extracting distinct context inference patterns. |
| Outcome: | The proposed method outperforms 10 state-of-the-art baselines and achieves a 25% increase in AUC score on average across natural language and code datasets. |
Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Speech Large Language Models (SpeechLLMs) have emerged as dominant speech processing approaches. |
| Approach: | They compare self-supervised learning-based discrete and continuous features . they compare performance across six spoken language understanding-related tasks . |
| Outcome: | The proposed models outperform discrete tokens and continuous features in six spoken language understanding-related tasks. |
RAG-Zeval: Enhancing RAG Responses Evaluator through End-to-End Reasoning and Ranking-Based Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation frameworks rely on direct prompting of resource-intensive models with complex multi-stage prompts, introducing significant computational cost and underutilizing models’ reasoning capabilities. |
| Approach: | They propose a framework that trains evaluators with reinforcement learning to generate comprehensive and sound assessments with detailed explanation in one-pass. |
| Outcome: | The proposed framework outperforms baseline evaluation frameworks that rely on LLMs with 10-100 more parameters and achieves the strongest correlation with human judgments. |
Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models are trained on text data downsampled from massive text corpora like Common Crawl. |
| Approach: | They propose an efficient and scalable system that can make petabyte-level text corpora searchable by using the FM-index data structure. |
| Outcome: | The proposed system indexes 83TB of Internet text in 99 days with a single 128-core CPU node (or 19 hours if using 137 such nodes). |
Mahānāma: A Unique Testbed for Literary Entity Discovery and Linking (2025.emnlp-main)
Copied to clipboard
| Challenge: | High lexical variation, ambiguous references, and long-range dependencies make entity resolution in literary texts particularly challenging. |
| Approach: | They present a large-scale dataset for end-to-end Entity Discovery and Linking (EDL) in Sanskrit. |
| Outcome: | The proposed dataset is aligned with an English knowledge base to support cross-lingual linking. |
Adaptively profiling models with task elicitation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language model evaluations fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. |
| Approach: | They propose a method that automatically builds new evaluations to profile model behavior. |
| Outcome: | The proposed method finds that language models fail in hundreds of tasks . it also finds that o3-mini is prone to hallucination when fabrications are repeated . |
Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax. |
| Approach: | They propose to use causal interpretability methods to characterize abstract mechanisms that LMs learn to use by transferring a wh-filler-gap structure into a gap-less c++ class. |
| Outcome: | The proposed methods can characterize the abstract mechanisms that LMs learn to use, and challenge claims that they can be learned only with strong innate priors. |
TactfulToM: Do LLMs have the Theory of Mind ability to understand White Lies? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies explore Large Language Models’ (LLMs) performance on Theory of Mind (ToM) reasoning tasks, but research on ToM abilities that require more nuanced social context is limited, such as white lies. |
| Approach: | They propose a novel English benchmark to evaluate Large Language Models’ ability to understand white lies within real-life conversations and reason about prosocial motivations behind them. |
| Outcome: | The proposed model outperforms state-of-the-art models on ToM reasoning tasks and reveals significant gaps between humans and LLMs. |
Don’t Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to calculate agreement are acceq and acc eq, but there are many ways to compute agreement based on how scores are grouped together. |
| Approach: | They propose a segment-level meta-evaluation metric that utilizes pairwise differences rather than raw scores to refine Global Pearson to intra-segment comparisons. |
| Outcome: | The proposed metric correctly ranks sentinel evaluation metrics and better aligns with human error weightings than acceq. |
SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, estimating item difficulties requires real students to respond to items . a cold-start approach cannot be applied to previously unseen items either . |
| Approach: | They propose a method for aligning simulated students with instructed ability to predict difficulty of open-ended items. |
| Outcome: | The proposed method outperforms existing methods on two real-world student responses. |
HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin America (2025.emnlp-main)
Copied to clipboard
Guido Ivetta, Marcos J Gomez, Sofía Martinelli, Pietro Palombini, M Emilia Echeveste, Nair Carolina Mazzeo, Beatriz Busaniche, Luciana Benotti
| Challenge: | a dataset of 46,499 sentences created in a professional development course captures intersectional biases across multiple demographic axes and school subjects. |
| Approach: | They present a large-scale dataset of 46,499 sentences created in a professional development course . they show that the dataset contains more stereotypes unrecognized by current LLMs . |
| Outcome: | The proposed dataset captures intersectional biases across multiple demographic axes and school subjects. |
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation (2025.emnlp-main)
Copied to clipboard
Rabiul Awal, Mahsa Massoud, Aarash Feizi, Zichao Li, Suyuchen Wang, Christopher Pal, Aishwarya Agrawal, David Vazquez, Siva Reddy, Juan A. Rodriguez, Perouz Taslakian, Spandana Gella, Sai Rajeswar
| Challenge: | Existing benchmarks focus on specific aspects of web tasks but lack comprehensive coverage. |
| Approach: | They propose a multilingual benchmark that evaluates three core web tasks: (1) website visual question answering, (2) code editing involving HTML/CSS/JavaScript, and (3) mockup-to-code generation. |
| Outcome: | The proposed model performs well on basic information extraction, but struggles with reasoning and grounding, editing code to preserve functionality, and generating design-to-code that maintains hierarchy and supports multilingual content. |
Analyzing values about gendered language reform in LLMs’ revisions (2025.emnlp-main)
Copied to clipboard
| Challenge: | In the past years, LLMs have been used in everyday tasks, especially the formulation and revision of text. |
| Approach: | They examine LLMs' revision of gendered role nouns and their justifications using a prompt set-up to examine their alignment with feminist and trans-inclusive language reforms for English. |
| Outcome: | The proposed revision choices are based on the literature and empirical evidence. |
ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used to solve the entity recognition task. |
| Approach: | They propose a framework to select the most informative and representative samples for LLM in-context learning. |
| Outcome: | The proposed framework outperforms baselines on three specialized domain datasets. |
HyperKGR: Knowledge Graph Reasoning in Hyperbolic Space with Graph Neural Network Encoding Symbolic Path (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for linking knowledge graphs are incomplete and rely on Euclidean embeddings . a hyperbolic GNN framework embeds recursive learning trees in hyperbolical space . |
| Approach: | They propose a hyperbolic GNN framework that embeds recursive learning trees in hyperbolical space and generates query-specific embeddings. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on multiple benchmark datasets. |
LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with hallucinations, handling domain-specific data effectively, and integrating experimental workflows. |
| Approach: | They propose a hierarchical multi-agent framework to emulate the materials science research workflow by combining a new uncertainty and confidence estimate to evaluate the self-consistency of responses from LLaMP and baseline methods. |
| Outcome: | The proposed framework performs better than existing methods in material property retrieval, crystal structure editing, and annealing molecular dynamics simulations. |
ReSeeding Latent States for Sequential Language Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing grounding approaches depend on auxiliary modules at inference or implicitly align encoder-only models that lack generative capacity. |
| Approach: | They propose a method that produces latent embeddings aligned with the true state of the environment and refeeds these embeddables into the model before generating its output. |
| Outcome: | The proposed method outperforms commercial LLMs on three new reasoning benchmarks. |
DPED: Multi-Layer Noise Distillation for Privacy-Preserving Text Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train text embedding models under differential privacy constraints are difficult due to high dimensionality of language data and the presence of rare, identifying linguistic features. |
| Approach: | They propose a framework that leverages teacher-student distillation with noise injection to learn high-quality embeddings while providing differential privacy guarantees. |
| Outcome: | The proposed framework outperforms standard differentially private training methods on benchmark datasets and provides higher privacy-utility trade-offs. |
Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | ambiguities in natural language can lead to outputs that seem correct but fail to reflect the speaker’s intent. |
| Approach: | They propose to identify and then resolve ambiguities in natural language and propose metrics to quantify them. |
| Outcome: | The proposed metrics better correlate with human annotations than uncertainty baselines. |
Morpheme Induction for Emergent Language (2025.emnlp-main)
Copied to clipboard
| Challenge: | CSAR is a greedy algorithm that weights morphemes based on mutual information between forms and meanings, then removes it from the corpus and repeats the process to induce more morphs. |
| Approach: | They propose an algorithm that weights morphemes based on mutual information between forms and meanings, selects highest-weighted pair, removes it from corpus, and repeats process to induce further morphs. |
| Outcome: | The proposed algorithm makes reasonable predictions in adjacent domains. |
Stepwise Informativeness Search for Improving LLM Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have improved multistep reasoning but they lose focus over the middle of long contexts. |
| Approach: | They propose a tree search framework that proactively identifies underutilized steps and minimizing redundant information between steps. |
| Outcome: | The proposed framework generates more accurate and concise rationales with reduced errors and redundancy. |
Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. |
| Approach: | They propose to automate analysis of NLP research by extracting key elements and linking them through interpretable rules and contextual reasoning. |
| Outcome: | The proposed system improves on two domains of fact-checking and hate speech detection. |
FairGen: Controlling Sensitive Attributes for Fair Generations in Diffusion Models via Adaptive Latent Guidance (2025.emnlp-main)
Copied to clipboard
Mintong Kang, Vinayshekhar Bannihatti Kumar, Shamik Roy, Abhishek Kumar, Sopan Khosla, Balakrishnan Murali Narayanaswamy, Rashmi Gangadharaiah
| Challenge: | Text-to-image diffusion models often exhibit generation biases toward specific demographic groups, raising ethical concerns and limiting their adoption. |
| Approach: | They propose an adaptive latent guidance mechanism which controls the generation distribution during inference by dynamically adjusting the diffusion process to enforce specific attributes. |
| Outcome: | The proposed model outperforms existing models on HBE and Stable Bias datasets and achieves substantial bias reduction. |
Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D (2025.emnlp-main)
Copied to clipboard
Artemis Panagopoulou, Le Xue, Honglu Zhou, Silvio Savarese, Ran Xu, Caiming Xiong, Chris Callison-Burch, Mark Yatskar, Juan Carlos Niebles
| Challenge: | a recent study shows that multimodal models can reason across multiple modalities . a limited number of models are able to reason across a variety of inputs . |
| Approach: | They propose a dataset for contrastive cross-modal reasoning across four modalities . they use human annotations and a mixture-of-models round-trip-consistency filter . |
| Outcome: | a new model evaluates models on multiple modalities to determine which one best answers a natural language prompt . the model must select the one that best satisfies the query and then fine-tune it . state-of-the-art models still achieve only 56% accuracy overall and 42% in four-modal settings . |
Proactive Hearing Assistants that Isolate Egocentric Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing hearing assistants are "reactive" in that users manually prompt them to pick specific sound sources via spatial filtering or phone-based interfaces. |
| Approach: | They propose a dual-model architecture that uses the wearer's self-speech as an anchor to infer conversational partners and suppress others. |
| Outcome: | The proposed system can identify and separate conversation partners in multi-conversation settings without explicit user commands or prompts. |
fLSA: Learning Semantic Structures in Document Collections Using Foundation Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to solve new tasks by inducing high-level strategies from example solutions to similar problems and adapting these strategies to solve unseen problems. |
| Approach: | They propose a foundation-model-based Latent Semantic Analysis method that iteratively clusters and tags document segments based on document-level contexts. |
| Outcome: | The proposed method is more informative in reconstructing the original texts than existing tagging methods. |
SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) introduce a new paradigm of explicitly reasoning before answering, but they pose great safety risks against harmful queries and adversarial attacks. |
| Approach: | They propose a safety aha moment that activates safety reasoning and leads to a safe response. |
| Outcome: | The proposed model can generalize to unseen jailbreak prompts while maintaining general abilities. |
HypER: Literature-grounded Hypothesis Generation and Distillation with Provenance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches focus on retrieval augmentation and focus on the quality of the output . Existing methods focus on generating a highly specific declarative statement ignoring the underlying reasoning process behind ideation. |
| Approach: | They propose a large language model that generates evidence-based hypotheses using literature-guided reasoning and a multi-task setting. |
| Outcome: | The proposed model outperforms the base model and generates evidence-grounded hypotheses with high feasibility and impact as judged by human experts. |
Empowering GraphRAG with Knowledge Filtering and Integration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models suffer from knowledge gaps and hallucinations, resulting in incorrect or poor reasoning. |
| Approach: | They propose Graph retrieval-augmented generation (GraphRAG) which integrates structured knowledge from external graphs to enhance model's reasoning. |
| Outcome: | Experiments on knowledge graph QA tasks show that GraphRAG significantly improves reasoning performance across multiple backbone models. |
Interpretable Mnemonic Generation for Kanji Learning via Expectation-Maximization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for mnemonic generation in Japanese are limited in their interpretability due to script differences. |
| Approach: | They propose a method that models the mnemonic construction process as driven by common rules. |
| Outcome: | The proposed method performs well in the cold-start setting for new learners while providing insight into the mechanisms behind effective mnemonic creation. |
Refining Attention for Explainable and Noise-Robust Fact-Checking with Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional transformer-based models falter due to noise sensitivity and lack explainability . ATTUN is a transformer architecture designed to enhance model transparency and resilience to noise. |
| Approach: | They propose a transformer architecture that enhances model transparency and resilience to noise . ATTUN is a module that directly modifies attention weights . they validated their approach using fact-checking datasets based on their results . |
| Outcome: | The proposed model improves predictions and identify relevant sections of input data. |
Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities but their misuse for harmful purposes remains a concern. |
| Approach: | They propose a jailbreaking technique that exploits weaknesses in LLMs' architecture . they propose abductive framing and symbolic encoding to bypass safeguards . |
| Outcome: | The proposed technique achieves over 95% attack success rate on GPT-series models and 70% across all targets. |
Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25 (2025.emnlp-main)
Copied to clipboard
| Challenge: | Interpretability in information retrieval (IR) models is coarse-grained and poorly understood . a cross-encoder model extracts traditional relevance signals, such as term frequency and inverse document frequency . |
| Approach: | They analyze how a common IR model extracts traditional relevance signals . this is similar to the probabilistic ranking function BM25 . |
| Outcome: | The proposed model extracts traditional relevance signals in early-to-middle layers, similar to BM25 . the model then combine these concepts in later layers, laying the groundwork for future interventions . |
Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement learning is emerging as a primary driver for improving language model reasoning capabilities. |
| Approach: | They propose a method for explicitly up-weighting rare but correct solutions to overcome rank bias in group relative policy optimization (GRPO) . |
| Outcome: | The proposed method mitigates rank bias and improves pass@N across a large range of N in both synthetic and real theorem proving settings. |
PhoniTale: Phonologically Grounded Mnemonic Generation for Typologically Distant Language Pairs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vocabulary acquisition is a challenge for second-language learners when learning typologically distant languages such as English and Korean, where phonological and structural mismatches complicate vocabulary learning. |
| Approach: | They propose a cross-lingual mnemonic generation system that performs IPA-based phonological adaptation and syllable-aware alignment to retrieve L1 keyword sequence and uses LLMs to generate verbal cues. |
| Outcome: | The proposed system outperforms human-written and automated mnemonics in a short-term recall test with human participants and achieves quality comparable to human-writing mnms. |
Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries (2025.emnlp-main)
Copied to clipboard
| Challenge: | a typical human-assistant conversation is lengthy and shows significant diversity in topics, intents, and requirements across turns. |
| Approach: | They propose a framework that leverages pertinent linguistic concepts of dialog-acts and maxims to improve the accuracy of LLM-judges on preference data with complex, multi-turn conversational context. |
| Outcome: | The proposed framework improves on 4 challenging datasets showing that humans frequently change their intents from one turn of the conversation to the next. |
Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are typically trained to reflect a relatively uniform set of values, which limits their applicability to tasks that require understanding of nuanced human perspectives. |
| Approach: | They propose to use Chain-of-Thought reasoning techniques to build steerable pluralistic models by fine-tuning on human-authored CoT and synthetic explanations. |
| Outcome: | The proposed methods outperform others and demonstrate strong sample efficiency. |
CMedCalc-Bench: A Fine-Grained Benchmark for Chinese Medical Calculations in LLM (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing medical NLP benchmarks focus on qualitative reasoning and textual comprehension, but lack of fine-grained evaluation of intermediate reasoning. |
| Approach: | They propose a Chinese medical calculation benchmark that disentangles clinical entity extraction from numerical computation. |
| Outcome: | The proposed framework disentangles clinical entity extraction from numerical computation, enabling systematic diagnosis of model deficiencies. |
Evaluating Robustness of Large Audio Language Models to Audio Injection: An Empirical Study (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Audio-Language Models (LALMs) are increasingly being deployed in real-world applications, yet their robustness against malicious audio injection remains underexplored. |
| Approach: | They quantitatively assess their vulnerabilities and resilience using metrics: the Defense Success Rate, Context Robustness Score, and Judgment Robustic Index. |
| Outcome: | The proposed models demonstrate significant performance disparities across four attack scenarios. |
How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparison (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks primarily assess static knowledge, while intelligence also entails the ability to rapidly learn from experience. |
| Approach: | They propose to use semantic games to evaluate test-time learning . they recruit eight human participants to complete the same task . |
| Outcome: | The proposed framework compares model performance under limited and cumulative experience settings and contains four forms of experience representation. |
Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Making (2025.emnlp-main)
Copied to clipboard
Yejin Son, Minseo Kim, Sungwoong Kim, Seungju Han, Jian Kim, Dongju Jang, Youngjae Yu, Chan Young Park
| Challenge: | Existing safety evaluations rely on coarse success rates and domain-specific setups, making it difficult to diagnose why and where these models fail. |
| Approach: | They propose a framework for systematically evaluating the physical safety of LLMs in embodied decision making. |
| Outcome: | The proposed framework assesses the physical safety of LLMs in embodied decision making. |
SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are an integral enabler of enterprise applications such as summarization, retrieval augmented generation, and agentic workflows. |
| Approach: | They propose a model transformation and distillation procedure that prefills later layers’ KV cache using an earlier layer’s output, allowing prompt tokens to skip those later layers. |
| Outcome: | The proposed procedure can reduce prefill computation by 25-50% across several LLM families while incurring minimum quality degradation. |
Co-Eval: Augmenting LLM-based Evaluation with Machine Metrics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs suffer from biases and misalignment due to limited functional understanding and knowledge gaps. |
| Approach: | They introduce a framework that leverages a criteria planner model and optimized machine metrics to enhance the scalability and fairness of LLM-based evaluation. |
| Outcome: | The proposed framework reduces biases and improves alignment with human preferences, with gains of up to 0.324 in Spearman correlation. |
Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes end-to-end models but suffer from limitations . prior work focused on generating captions from long video streams . |
| Approach: | They propose a saliency-aware framework that localizes events and generates captions for each event. |
| Outcome: | The proposed framework achieves state-of-the-art results on YouCook2 and ViTT. |
Semantic Networks Extracted from Students’ Think-Aloud Data are Correlated with Students’ Learning Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Largescale open online courses (MOOCs) are available to hundreds of millions of learners, but efficiently evaluating these students' performance remains a crucial task for educators. |
| Approach: | They propose to use textbook-based information as a semantic network to extract concepts and relations from students' verbal data. |
| Outcome: | The proposed models extract concepts and relations from students’ verbal data and show that denser and more interconnected networks were associated with more elaborated knowledge acquisition. |
Less is More: The Effectiveness of Compact Typological Language Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Linguistic feature datasets such as URIEL+ have high dimensionality and sparsity, especially for low-resource languages. |
| Approach: | They propose a pipeline to optimize the URIEL+ typological feature space by feature selection and imputation. |
| Outcome: | The proposed pipeline produces compact yet interpretable typological representations on linguistic distance alignment and downstream tasks. |
Sparse Activation Editing for Reliable Instruction Following in Narratives (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks fail to capture the challenges of instruction following in complex narrative contexts. |
| Approach: | They propose a training-free framework that identifies and edits instruction-relevant neurons using only natural language instructions without requiring labelled data. |
| Outcome: | The proposed framework improves instruction following by identifying and editing instruction-relevant neurons using only natural language instructions, without requiring labelled data. |
Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Encoder transformer models compress information from all tokens into a single [CLS] token to represent global context. |
| Approach: | They propose a 1-D convolution module that augments token representations with multi-scale local features to improve performance. |
| Outcome: | Experiments on five diverse tasks show that the proposed framework outperforms baseline models by 1% to 14% while maintaining efficiency. |
Causal Tree Extraction from Medical Case Reports: A Novel Task for Experts-like Text Comprehension (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extract causal relationships from medical case reports are insufficient for capturing causal relationships of an entire case. |
| Approach: | They propose a task that generates a causal tree with the primary disease as the root and extracts causal relationships from a medical case report. |
| Outcome: | The proposed method outperforms the baseline method by 20.2 points in the human evaluation and introduces evaluation metrics that reflect clinician preferences. |
OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature (2025.emnlp-main)
Copied to clipboard
Alisha Srivastava, Emir Kaan Korukluoglu, Minh Nhat Le, Duyen Tran, Chau Minh Pham, Marzena Karpinska, Mohit Iyyer
| Challenge: | Large language models (LLMs) are known to memorize and recall English text from their pretraining data, but the extent to which this ability generalizes to non-English languages or transfers across languages remains unclear. |
| Approach: | They propose a dataset of 31.5K aligned excerpts from 20 books in ten languages, including English originals, official translations and new translations in six low-resource languages. |
| Outcome: | The proposed model can recall English content in translations, but perturbations reduce performance, causing the model to fail. |
Enhanced Noun-Noun Compound Interpretation through Textual Enrichment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent benchmarks frame Noun-Noun Compound Interpretation as a multiple-choice question . but, it still faces key limitations: vague relation descriptions as options and inability to handle polysemous compounds. |
| Approach: | They propose a textual enrichment framework that parses relations into eventoriented descriptions . the framework explicitly surfaces the hidden event connecting head and modifier . |
| Outcome: | The proposed framework yields consistently higher accuracy across three LLM families. |
ICL CIPHERS: Quantifying ”Learning” in In-Context Learning via Substitution Ciphers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that In-Context Learning operates in dual modes . however, disentangling these modes remains a challenging goal . |
| Approach: | They propose a class of task reformulations based on substitution ciphers borrowed from classic cryptography. |
| Outcome: | The proposed model can solve tasks with a BIJECTIVE mapping, but it requires 'deciphering' the latent cipher. |
Corrupted but Not Broken: Understanding and Mitigating the Negative Impacts of Corrupted Data in Visual Instruction Tuning (2025.emnlp-main)
Copied to clipboard
Yunhao Gou, Hansi Yang, Zhili Liu, Kai Chen, Yihan Zeng, Lanqing Hong, Zhenguo Li, Qun Liu, Bo Han, James Kwok, Yu Zhang
| Challenge: | Visual Instruction Tuning (VIT) aims to enhance Multimodal Large Language Models (MLLMs), but its effectiveness is often compromised by corrupted datasets with issues such as hallucinated content and poor OCR quality. |
| Approach: | They propose a corruption-robust training paradigm that surpasses existing strategies for mitigating the effects of corrupted data. |
| Outcome: | The proposed training paradigm surpasses existing strategies for mitigating the effects of corrupted data. |
Memory OS of AI Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) face a shortage of long-term memory capabilities and limited personalization due to fixed context windows. |
| Approach: | They propose a Memory Operating System to achieve efficient memory management for AI agents . MemoryOS enables hierarchical memory integration and dynamic updating . |
| Outcome: | The proposed architecture enables hierarchical memory integration and dynamic updating. |
Rule Discovery for Natural Language Inference Data Generation Using Out-of-Distribution Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing training rules for natural language inference do not cover the diversity of natural language. |
| Approach: | They propose a framework that combines out-of-distribution detection and clustering to identify new premise–hypothesis pairs in a dataset that are not covered by existing rules. |
| Outcome: | The proposed framework achieves +0.85%p accuracy on 2k and +0.15%p on 550k samples. |
Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models lack spatial reasoning capability, despite their ability to comprehend spatial arrangements and model structural relations. |
| Approach: | They propose a benchmark to evaluate vision-language models' spatial perception, structural understanding, and reasoning capabilities by minimizing reliance on domain-specific knowledge. |
| Outcome: | The proposed benchmark is based on 1,100 carefully curated real-world images with high spatial complexity. |
Definition Generation for Word Meaning Modeling: Monolingual, Multilingual, and Cross-Lingual Perspectives (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in text generation have opened up new opportunities for word meaning modeling. |
| Approach: | They extend definition generation task beyond English to a suite of 22 languages . they use Llama-based models to evaluate models in monolingual, multilingual, cross-lingual settings . |
| Outcome: | The proposed model outperforms pretrained models in monolingual, multilingual, and cross-lingual settings. |
Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive language models excel in text-to-audio generation, but lag behind diffusion models by a non-trivial margin. |
| Approach: | They propose a framework that integrates multiple isolated transformers with causal conditioning and anti-causal alignment via reinforcement learning. |
| Outcome: | The proposed framework outperforms existing LM-based and diffusion-based systems in audio synthesis. |
HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) face memory challenges due to the high cost of backpropagation. |
| Approach: | They propose a zeroth-order (ZO) optimization that matches memory usage to inference . they propose scalable and memory-efficient zeroth order (ZE) optimizer that integrates annealed A-GNB gradients with diagonal Hessian estimation and layer-wise clipping as a second-order pre-conditioner. |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods with an average speedup of 20 over MeZO on RoBERTa-large and OPT-1.3B. |
Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models struggle when faced with unseen domains or languages. |
| Approach: | They propose a framework that leverages the broad knowledge of an MLLM to generate cross-modal pre-questions (preQs) before retrieval. |
| Outcome: | Experiments show that PREMIR outperforms existing retrievers on out-of-distribution benchmarks, including closed-domain and multilingual settings, outperforming strong baselines across all metrics. |
From Parameters to Performance: A Data-Driven Study on LLM Structure and Development (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have revolutionized a wide range of domains, driving significant advancements in both technology and real-world applications. |
| Approach: | They present a large-scale dataset encompassing diverse open-source LLM structures and their performance across multiple benchmarks. |
| Outcome: | The proposed model validates the relationship between structural configurations and performance across multiple benchmarks and further corroborates the findings using mechanistic interpretability techniques. |
Logical Reasoning with Outcome Reward Models for Test-Time Scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), but it is under-explored in deductive reasoning. |
| Approach: | They propose to use Chain-of-Thought to generate data using single and multiple samples to train ORMs. |
| Outcome: | The proposed model expands the type of errors covered in the training dataset, covering previously unexplored error types. |
Speculating LLMs’ Chinese Training Data Pollution from Their Tokens (2025.emnlp-main)
Copied to clipboard
Qingjie Zhang, Di Wang, Haoting Qian, Liu Yan, Tianwei Zhang, Ke Xu, Qi Li, Minlie Huang, Hewu Li, Han Qiu
| Challenge: | Experiments on GPT and other 23 LLMs indicate that tokens widely exist while GPT’s vocabulary behaves the worst: more than 23% long Chinese tokens (i.e., a token with more than two Chinese characters) are either porn or online gambling. |
| Approach: | They propose to locate Polluted Chinese (PoC) tokens in LLMs and build a PoC token detector to label them in vocabularies by considering each token’s semantics and related contents from the search engines. |
| Outcome: | The proposed method predicts that the ratio of “*” related webpages in GPT-4o's training data is around 0.5%. |
NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current large language models struggle to answer questions that span tens of thousands of tokens. |
| Approach: | They evaluate 1–4 hop QA over 64k–128k-token excerpts from 83 novels . they find consistent accuracy drops with increased hops and context length . |
| Outcome: | The novelhopqa benchmark evaluates 1–4 hop QA over 64k–128k-token excerpts from 83 public-domain novels. |
Weights-Rotated Preference Optimization for Large Language Models (2025.emnlp-main)
Copied to clipboard
Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang
| Challenge: | Existing methods to align large language models with high reward hacking are limited by the complexity of the parameter space and the complexity. |
| Approach: | They propose a weights-rotated preference optimization algorithm that constrains the output layer logits with the KL divergence inherited from DPO and fine-tunes the intermediate hidden states. |
| Outcome: | The proposed algorithm achieves a 3.27-point improvement on AlpacaEval 2 and surpasses the best baseline by 6.2 to 7.5 points on MT-Bench with merely 0.015% of the trainable parameters. |
The Stepwise Deception: Simulating the Evolution from True News to Fake News with LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies assume fake news is inherently existing rather than exploring its gradual formation. |
| Approach: | They propose a Large Language Model-based simulation approach explicitly focusing on fake news evolution from real news. |
| Outcome: | The proposed framework captures fake news evolution patterns and accurately reproduces known fake news, aligning closely with human evaluations. |
How to inject knowledge efficiently? Knowledge Infusion Scaling Law for Pre-training Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that strategically infusing domain knowledge during pretraining can substantially improve downstream performance. |
| Approach: | They propose a knowledge infusion scaling law that predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. |
| Outcome: | The proposed model predicts the optimal amount of domain knowledge to inject into large LLMs by analyzing their smaller counterparts. |
SMEC:Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models generate high-dimensional embeddings that capture rich semantic and syntactic information. |
| Approach: | They propose a training framework to reduce dimensionality and complexity of large language models. |
| Outcome: | Experiments on image, text, and multimodal datasets show that the proposed training framework reduces dimensionality while maintaining performance. |
Reverse Prompt Engineering: A Zero-Shot, Genetic Algorithm Approach to Language Model Inversion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on a large number of outputs for training and inference, and they can produce garbled text. |
| Approach: | They propose a training-free framework that reconstructs prompts using only a limited number of text outputs from a language model. |
| Outcome: | The proposed framework generates high-quality prompt recovery and more semantically and functionally aligned with the originals than current state-of-the-art methods. |
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | DiMo-GUI is a training-free framework for GUI grounding that splits input into textual elements and iconic elements, allowing the model to reason over each modality independently using general-purpose vision-language models. |
| Approach: | They propose a training-free framework for GUI grounding that leverages two core strategies: dynamic visual grounding and modality-aware optimization. |
| Outcome: | The proposed framework splits the input into textual elements and iconic elements, allowing the model to reason over each modality independently using general-purpose vision-language models. |
SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) lack large-scale, systematically constructed benchmarks for evaluating their alignment with real-world social attitudes. |
| Approach: | They propose a benchmark to assess LLMs' alignment with real-world social attitudes . they find LLM models achieve only 30–40% accuracy when simulating individuals . |
| Outcome: | The proposed benchmark shows that LLMs achieve only 30% accuracy when simulating individuals in complex survey scenarios. |
Financial Risk Relation Identification through Dual-view Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Identifying inter-firm risk relations is crucial for applications like portfolio management and investment strategy. |
| Approach: | They propose a method for extracting inter-firm risk relations using Form 10-K filings . their method captures implicit and abstract risk connections through unsupervised fine-tuning . |
| Outcome: | The proposed method outperforms baselines across multiple evaluation settings. |
CopySpec: Accelerating LLMs with Speculative Copy-and-Paste (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) suffer from slower inference as context size grows, but CopySpec leverages larger contexts to accelerate inference. |
| Approach: | They propose a technique that speculates that the same tokens will follow repeated sequences in the model’s chat history or context and enables seamless copying without compromising output quality. |
| Outcome: | The proposed technique can generate responses that closely resemble previous outputs or responses that can be verbatim extracted from context without compromising output quality and without requiring additional GPU memory. |
GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that many layers are functionally redundant in large language models (LLMs), enabling model compression by removing these layers to reduce inference cost. |
| Approach: | They propose a framework that removes redundant layers to reduce inference cost by preserving sensitivity-aware singular values. |
| Outcome: | The proposed framework outperforms existing methods in 90% of the original model under a 20% compression ratio. |
GraphAgent: Agentic Graph Language Assistant (2025.emnlp-main)
Copied to clipboard
| Challenge: | Real-world data combines structured and unstructured formats, capturing explicit relationships and implicit semantic interdependencies. |
| Approach: | They propose GraphAgent, an automated agent pipeline addressing both explicit and implicit graph-enhanced semantic dependencies for predictive and generative tasks. |
| Outcome: | Extensive experiments on diverse datasets validate GraphAgent’s effectiveness in graph-related predictive and text generative tasks. |
DDO: Dual-Decision Optimization for LLM-Based Medical Consultation via Multi-Agent Collaboration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs fail to capture the dual nature of medical consultation (MC) this mismatch often results in ineffective symptom inquiry and unreliable disease diagnosis. |
| Approach: | They propose a novel LLM-based framework that performs Dual-Decision Optimization by decoupling the two sub-tasks and optimizing them with distinct objectives through a collaborative multi-agent workflow. |
| Outcome: | The proposed framework outperforms existing LLM-based approaches on three real-world MC datasets and achieves competitive performance with state-of-the-art generation-based methods. |
FedMABench: Benchmarking Mobile GUI Agents on Decentralized Heterogeneous User Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mobile GUI agents have attracted tremendous research participation recently. traditional approaches to mobile agent training rely on centralized data collection. |
| Approach: | They propose a benchmark for federated training and evaluation of mobile GUI agents . they find that federation algorithms consistently outperform local training . |
| Outcome: | The first benchmark for federated training and evaluation of mobile GUI agents is released . it features 6 datasets with 30+ subsets, 8 federation algorithms, 10+ base models, and over 800 apps across 5 categories . |
VLA-Mark: A cross modal watermark for large vision-language alignment models (2025.emnlp-main)
Copied to clipboard
Shuliang Liu, Zheng Qi, Jesse Jiaxi Xu, Yibo Yan, Junyan Zhang, He Geng, Aiwei Liu, Peijie Jiang, Jia Liu, Yik-Cheung Tam, Xuming Hu
| Challenge: | Existing text watermarking methods disrupt visual-textual alignment, leaving semantic-critical concepts vulnerable. |
| Approach: | They propose a vision-aligned framework that embeds detectable watermarks into outputs . they combine localized patch affinity, global semantic coherence, contextual attention patterns . |
| Outcome: | The proposed framework shows lower PPL and higher BLEU than conventional methods with near-perfect detection (98.8% AUC). |
Sentence Smith: Controllable Edits for Evaluating Text Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Controllable and transparent text generation has been a long-standing goal in NLP . but previous approaches were hindered by parsing and generation insufficiencies . |
| Approach: | They propose a framework for English that has three steps: 1. Parsing a sentence into a semantic graph. 2. Applying human-designed semantic manipulation rules. 3. Generating text from the manipulated graph. |
| Outcome: | The proposed framework for English is based on a neural network and parsers. |
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning (2025.emnlp-main)
Copied to clipboard
Yu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang, Chenghao Xiao, Long Li, Deli Zhao, Wenbing Huang, Tingyang Xu, Qifeng Bai, Yu Rong
| Challenge: | Existing medical reasoning datasets are limited in scale and typically rely on incomplete data. |
| Approach: | They propose to use ReasonMed to train medical reasoning models using a multi-agent generation, verification, and refinement pipeline. |
| Outcome: | The largest medical reasoning dataset to date surpasses the prior best sub-10B models by 4.17% and even exceeds LLaMA3.1-70B on PubMedQA by 4.60%. |
Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing sparse retrieval methods suffer from a lack of interpretability . we propose a new interpretability framework that decomposes dense embeddings into distinct, interpretable latent concepts. |
| Approach: | They propose a new interpretability framework that leveragesSparse Autoencoders to decompose uninterpretable dense embeddings fromDPR models into distinct, interpretable latent concepts. |
| Outcome: | The proposed interpretability framework achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches. |
UICOMPASS: UI Map Guided Mobile Task Automation via Adaptive Action Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mobile task automation is an emerging technology that leverages AI to automatically execute routine tasks by users’ commands on mobile devices like Android. |
| Approach: | They propose a UI Map-guided LLM-based approach to automate mobile tasks using static analysis and LLMs. |
| Outcome: | The proposed approach achieves a 15.87% higher task execution success rate than SOTA approaches even when only APK is available. |
Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large reasoning models (LRMs) are being adopted more widely as personal agents thanks to their enhanced planning skills enabled by reasoning traces (RTs). |
| Approach: | They propose to increase the budget of models with increased reasoning steps to amplify such leakage by enlarging their internal thinking to the model's internal thinking. |
| Outcome: | The proposed model can reason more verbosely and leak more in their own thinking, while improving utility but enlarges the privacy attack surface. |
Model Unlearning via Sparse Autoencoder Subspace Guided Projections (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing unlearning strategies lack interpretability or fail to provide robust defense against adversarial prompts. |
| Approach: | They propose a framework that leverages SAE features to drive targeted updates in the model’s parameter space. |
| Outcome: | The proposed framework reduces harmful knowledge accuracy by 3.22% compared to baselines and improves adversarial robustness under jailbreak prompts. |
ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to conversational Query Reformulation (CQR) suffer from high dependency on external supervision from annotations or large language models and insufficient alignment between the rewriter and downstream retrievers. |
| Approach: | They propose a framework that transforms context-dependent queries into self-contained forms suitable for off-the-shelf retrievers. |
| Outcome: | The proposed framework outperforms existing methods on topiOCQA and QReCC datasets while using smaller 3B parameter models without external supervision. |
How to Make Large Language Models Generate 100% Valid Molecules? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can learn to perform a wide range of tasks, but generating valid molecules using representations like SMILES is challenging in few-shot settings. |
| Approach: | They propose a language framework that converts invalid SMILES to SELFIES and LLMs as post-hoc correctors to ensure that the molecules generated by LLM are 100% valid. |
| Outcome: | The proposed model performs worse with SELFIES than with SMILES and improves on other metrics. |
Exploring Quality and Diversity in Synthetic Data Generation for Argument Mining (2025.emnlp-main)
Copied to clipboard
| Challenge: | Argument Mining (AM) is hindered by the scarcity of structure-annotated datasets, which are expensive to create manually. |
| Approach: | They propose to use quality-oriented synthesis and diversity-oriented approach to generate argumentative texts with diverse topics and argument structures. |
| Outcome: | The proposed approach significantly improves existing models in full-data and low-resource settings. |
Dynamic Jointly Batch Selection for Data Efficient Machine Translation Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Data quality and effective selection are key to improving machine translation performance . study focuses on fine-tuning models using a batch selection strategy . |
| Approach: | They propose a data selection methodology for fine-tuning machine translation systems that leverages the synergy between a learner model and a pre-trained reference model to enhance overall training effectiveness. |
| Outcome: | The proposed method improves training efficiency by up to fivefold compared to baseline methods. |
3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark (2025.emnlp-main)
Copied to clipboard
Ivan Sviridov, Amina Miftakhova, Tereshchenko Artemiy Vladimirovich, Galina Zubkova, Pavel Blinov, Andrey Savchenko
| Challenge: | Large Vision-Language Models (LVLMs) are being explored in medicine but their ability to conduct complex real-world telemedicine consultations remains underexplored. |
| Approach: | They propose to use large vision-language models to conduct telemedicine consultations using a framework that simulates patient variability and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. |
| Outcome: | The proposed framework compares diagnostic strategies for open and closed-source LVLMs and shows that multimodal dialogue improves F1 score by 6.5% over non-dialogue settings. |
OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Open Large Language Models (OLLMs) are increasingly leveraged in generative AI applications, posing new challenges for detecting their outputs. |
| Approach: | They propose a benchmark to train and evaluate machine-generated text detectors on Turing Test and Authorship Attribution problems. |
| Outcome: | The proposed detector outperforms existing detectors in varying degrees of difficulty and relevance across tasks. |
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios (2025.emnlp-main)
Copied to clipboard
| Challenge: | a number of tools are used to perform complex tasks, but the tool utilization process can cause errors. |
| Approach: | They propose a critique evaluation benchmark for tool learning that analyzes function-calling errors on tool evaluation benchmarks. |
| Outcome: | The proposed critique evaluation benchmark holds diverse tool-use errors with varying complexities, which better reflects real-world scenarios. |
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. |
| Approach: | They propose a probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
| Outcome: | The proposed probing technique decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. |
Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data. |
| Approach: | They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages. |
| Outcome: | The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks. |
Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferral (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing quality estimation metrics are used to design effective deferral rules for machine translation. |
| Approach: | They propose a simple yet effective approach for machine translation using existing quality estimation metrics as deferral rules. |
| Outcome: | The proposed approach outperforms existing models in large translation tasks while reducing computational costs. |
iVISPAR — An Interactive Visual-Spatial Reasoning Benchmark for VLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) struggle with spatial reasoning and visual alignment, despite their performance on 2D tasks. |
| Approach: | They propose a multimodal benchmark to evaluate VLMs' spatial reasoning capabilities based on the sliding tile puzzle . |
| Outcome: | The proposed model performs better on 2D tasks compared to 3D or text-based settings, but struggles with complex spatial configurations and consistently falls short of human performance. |
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. |
| Approach: | They propose to use an ensemble of large language models to flag mislabeled examples by using an LLM-as-a-judge approach to detect label errors in existing datasets. |
| Outcome: | The proposed method improves label accuracy and consistency in large language models. |
Detecting Legal Citations in United Kingdom Court Judgments (2025.emnlp-main)
Copied to clipboard
| Challenge: | citation detection in court judgments is challenging because of the complexity of legal language . citation analysis is critical for many legal applications, but the complexity is not always easy to solve. |
| Approach: | They compare three different models for citation detection in court judgments using the Cambridge Law Corpus . they compare rulebased regular expressions, transformer-based encoders and large language models . |
| Outcome: | The proposed model outperforms the existing models in the citation analysis and analysis of 190 court judgments. |
Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks have exposed patterns and may not truly assess generalization ability of Large Language Models (LLMs). |
| Approach: | They propose a “Generalization Stress Test” to assess Large Language Models’ generalization ability under slight and controlled perturbations, including option length, problem types, and irrelevant noun replacements. |
| Outcome: | The proposed test shows that LLMs exhibit severe accuracy drops and unexpected biases when faced with minor but content-preserving modifications. |
Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that retrieval-augmented language models can perform better than larger models, but the optimal degree of overlap remains unclear. |
| Approach: | They investigate how varying levels of query–context overlap affect model performance . they specifically generate synthetic context through paraphrasing queries . |
| Outcome: | The proposed model can improve test-time perplexity and training time by 40% without compromising performance. |
Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on persona prompting has shown mixed results on its effectiveness . prior work did not consider when and why personas should affect performance . |
| Approach: | They analyze literature on persona prompting and distill three desiderata for their effectiveness . they propose mitigation strategies to improve robustness but find they only work for the largest, most capable models . |
| Outcome: | The authors find that expert personas usually lead to positive or non-significant performance changes . they propose mitigation strategies to improve robustness but only for the largest models . |
HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods that produce a fixed trade-off between storage size and performance are often ineffective due to the growing size of large language models. |
| Approach: | They propose a model merging technique that capitalizes on similarities between low-rank adapters to reduce storage costs and improve performance. |
| Outcome: | The proposed method significantly reduces storage size (48% reduction) while outperforms existing merging techniques in terms of performance (0.2-1.8% drop). |
Parrot: A Training Pipeline Enhances Both Program CoT and Natural Language CoT for Reasoning (2025.emnlp-main)
Copied to clipboard
Senjie Jin, Lu Chen, Zhiheng Xi, Yuhui Wang, Sirui Song, Yuhao Zhou, Xinbo Zhang, Peng Sun, Hong Lu, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing work focuses on enabling models to generate natural language chain-of-thought rationales or leverage executable and verifiable code, such as Python. |
| Approach: | They propose a novel training pipeline that integrates sequential P-CoT and N-Co T generation and a subtask hybrid training strategy to facilitate natural language transferability. |
| Outcome: | The proposed training pipeline improves both N-CoT and P-Co T performance over the RL baseline. |
Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Language Models (VLMs) have significant parameter size and autoregressive (AR) decoding nature impose considerable computational demands on VLA models. |
| Approach: | They propose a framework to relax acceptance utilizing the relative distances represented by the action tokens of the VLA model. |
| Outcome: | Empirical results show that the proposed framework improves the speed of the prediction task by 44%. |
Leveraging Text-to-Text Transformers as Classifier Chain for Few-Shot Multi-Label Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilabel text classification (MLTC) is an essential task in NLP applications. |
| Approach: | They propose a distillation-based T5 generalist model for zero-shot MLTC and few-shot fine-tuning. |
| Outcome: | The proposed model outperforms baselines of similar size on three few-shot tasks. |
M-Wanda: Improving One-Shot Pruning for Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual LLM performance is often dependent on model size, resulting in performance loss. |
| Approach: | They propose a pruning method that models cross-lingual variation by incorporating language-aware activation statistics into its pruning criterion and dynamically adjusts layerwise sparsity based on cross-linguistic importance. |
| Outcome: | The proposed method improves performance at minimal additional costs while maintaining multilinguality. |
Beyond Hate Speech: NLP’s Challenges and Opportunities in Uncovering Dehumanizing Language (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing hate speech datasets rarely contain enough instances of dehumanizing content, and current models struggle to distinguish such language from more benign forms of hate or offense. |
| Approach: | They evaluate four state-of-the-art large language models for dehumanization detection. |
| Outcome: | The proposed models perform only moderately under an optimized configuration, while others over-predict dehumanization for some identities, while under-identifying it for others. |
Conflict-Aware Soft Prompting for Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that REtrieval-augmented generation (RAG) fails to resolve the conflict between incorrect external context and correct parametric knowledge. |
| Approach: | They propose a conflict-aware REtrieval-augmented generation system that encodes external context into compact memory embeddings and captures a guidance signal that directs reasoning toward the more reliable knowledge source. |
| Outcome: | Extensive experiments show that CARE effectively mitigates context-memory conflicts, leading to an average performance gain of 5.0% on QA and fact-checking benchmarks. |
R-CHAR: A Metacognition-Driven Framework for Role-Playing in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing role-playing structures lack cognitive consistency in complex scenarios . Existing models excel in math and coding tasks but lack coherent reasoning . |
| Approach: | They propose a metacognition-driven framework that enhances role-playing performance . experimental results show performance improvements across varying scenario complexities . |
| Outcome: | The proposed framework outperforms existing models in social intelligence tasks and shows strength in long-context comprehension and group-level social interactions. |
Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Semantic similarity between two sentences depends on the aspects considered between those sentences. |
| Approach: | They propose a Conditional Semantic Textual Similarity task which measures the similarity between two sentences under a specified condition. |
| Outcome: | The proposed method improves Spearman correlation by 5.4% by training a supervised model on the re-annotated dataset. |
When Words Smile: Generating Diverse Emotional Facial Expressions from Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that generate only coarse facial expressions ignore the rich and dynamic nature of face-to-face communication. |
| Approach: | They propose an end-to-end text-to expression model that explicitly focuses on emotional dynamics. |
| Outcome: | The proposed model outperforms baselines on 15,000 text–3D expression pairs on a large-scale dataset. |
Improving Online Job Advertisement Analysis via Compositional Entity Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on IE in OJAs has focused on skills extraction, but other information is extracted using job tasks, job titles, and work tools. |
| Approach: | They propose a compositional entity modeling framework for requirement extraction from online job advertisements (OJAs) they annotate a manually annotated dataset of 500 German job ads that captures roles, tools, experience levels, attitudes, and their functional context. |
| Outcome: | The proposed framework can extract requirements from a manually annotated dataset of 500 German job ads. |
Correlation-Aware Example Selection for In-Context Learning with Nonsymmetric Determinantal Point Processes (2025.emnlp-main)
Copied to clipboard
Qiunan Du, Zhiliang Tian, Zhen Huang, Kailun Bian, Tianlun Liu, Zhaoning Zhang, Xinwang Liu, Feng Liu, Dongsheng Li
| Challenge: | Existing studies on in-context learning (ICL) focus on the selection of individual examples and ignore correlations among examples. |
| Approach: | They propose a method to capture positive and negative correlations using the determinantal point process . they optimize the method via kernel decomposition-based MLE to fit a constructed pseudo-labeled dataset . |
| Outcome: | The proposed method outperforms baselines in ICL example selection. |
Leveraging Cognitive Complexity of Texts for Contextualization in Dense Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to estimate semantic similarity of queries and documents rely on token-level information derived from query/document interactions. |
| Approach: | They propose a new DRM that leverages query/document interactions based on full embedding representations generated by a Transformer-based model. |
| Outcome: | The proposed model outperforms fine-tuning techniques on lightweight bi-encoders and traditional late-interaction models. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
CAVE : Detecting and Explaining Commonsense Anomalies in Visual Environments (2025.emnlp-main)
Copied to clipboard
Rishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, Antoine Bosselut
| Challenge: | a new benchmark for computer vision fails to capture richness and unpredictability of real-world anomalies . state-of-the-art VLMs struggle with visual anomaly perception and commonsense reasoning . elucidating the nature of anomalies is a fundamental human trait . |
| Approach: | They propose a benchmark for visual anomalies that includes annotations for visual grounding and categorizing anomalies based on their visual manifestations, their complexity, severity, and commonness. |
| Outcome: | The proposed benchmark improves on existing vision models by incorporating visual annotations. |
Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing cross-lingual transfer are limited by parallel resources and lack linguistic and domain coverage. |
| Approach: | They propose a cross-lingual in-context pre-training approach that leverages semantically related bilingual Wikipedia documents to enhance cross-linguistic transfer. |
| Outcome: | The proposed approach improves multilingual performance on three models across six target languages. |
SemVink: Advancing VLMs’ Semantic Understanding of Optical Illusions via Visual Global Thinking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vision-language models excel in semantic tasks but fail at detecting hidden content . current architectures prioritize abstract reasoning over low-level visual operations . |
| Approach: | They propose a benchmark to test vision-language models that can detect hidden content . they propose HC-Bench to scale images to low resolutions to unlock 99% accuracy . |
| Outcome: | HC-Bench shows that leading VLMs achieve near-zero accuracy even with explicit prompting . et al.: current models prioritize abstract reasoning over low-level visual operations . they urge a shift toward hybrid models bridging gap between computational vision and human cognition . |
Order Doesn’t Matter, But Reasoning Does: Training LLMs with Order-Centric Augmentation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Logical reasoning is essential for large language models (LLMs) to ensure accurate and coherent inferences. |
| Approach: | They propose an order-centric data augmentation framework based on commutativity in logical reasoning that randomly shuffles independent premises to introduce condition order augmentation. |
| Outcome: | The proposed framework improves LLMs’ reasoning performance and adaptability to diverse logical structures. |
Type-Less yet Type-Aware Inductive Link Prediction with Pretrained Language Models (2025.emnlp-main)
Copied to clipboard
Alessandro De Bellis, Salvatore Bufi, Giovanni Servedio, Vito Walter Anelli, Tommaso Di Noia, Eugenio Di Sciascio
| Challenge: | Existing methods to predict links between entities are limited in dynamic environments where new entities are incrementally introduced. |
| Approach: | They propose a Type-less yet type-awaRe approach for subgraph-based inductive link prediction that leverages pre-trained language models for semantic enrichment. |
| Outcome: | The proposed approach outperforms state-of-the-art models in scenarios with scarce type annotations and sparse graph connectivity. |
Extracting Linguistic Information from Large Language Models: Syntactic Relations and Derivational Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models, we study their morphosyntactic competence and generalization capabilities. |
| Approach: | They propose to use morphosyntactic tasks to study their linguistic knowledge and generalization capabilities to extract different types of morphological structure for typologically diverse languages. |
| Outcome: | The proposed models outperform GPT-4o and LLaMA 3.3-70B in all diagnostic tasks, but show little evidence of abstract morphological rule learning. |
Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have shifted the post-training paradigm from instruction tuning and human preference alignment to reinforcement learning (RL) based on rule-based evaluations of answer correctness, these models often receive rewards for speculative answers without generating coherent reasoning chains. |
| Approach: | They propose a confidence-based reward model tailored for enhancing STEM reasoning capabilities. |
| Outcome: | The proposed model outperforms state-of-the-art open-source reward models across diverse STEM benchmarks. |
TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has focused on the (un)intended leakage of sensitive information through LLM outputs. |
| Approach: | They propose a threat model that embeds context information into natural-looking outputs via linguistic steganography without requiring explicit control over inference inputs. |
| Outcome: | The proposed model transmits 32-bit secrets with 87% accuracy on held-out prompts and can reach over 97% accuracy using majority voting across three generations. |
Frequency & Compositionality in Emergent Communication (2025.emnlp-main)
Copied to clipboard
| Challenge: | Natural languages exhibit a universal tendency to resist regular patterns, developing idiosyncratic forms. |
| Approach: | They investigate the relationship between frequency and compositionality in emergent languages . they use a referential game setting to manipulate input frequency through Zipfian distributions . |
| Outcome: | The proposed method shows that the frequency distributions of the most frequent words resist regular patterns, resulting in less compositional structure. |
Summarizing Speech: A Comprehensive Survey (2025.emnlp-main)
Copied to clipboard
Fabian Retkowski, Maike Züfle, Andreas Sudmann, Dinah Pfau, Shinji Watanabe, Jan Niehues, Alexander Waibel
| Challenge: | Podcasts and other audiovisual content are becoming more and more a part of everyday communication and the digital age is changing from text to voice. |
| Approach: | They synthesize the current state of the field and highlight the need for realistic evaluation benchmarks and multilingual datasets. |
| Outcome: | The proposed frameworks are based on evaluation protocols and datasets and highlight the need for realistic benchmarks and multilingual datasets. |
CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to role-playing language models rely on prompt engineering or supervised fine-tuning to emulate character behaviors but neglect the underlying cognitive mechanisms driving these behaviors. |
| Approach: | They propose a novel RPLA adopting a cognize-then-respond reasoning paradigm that leverages dual cognition for more contextually grounded and psychologically coherent responses. |
| Outcome: | The proposed RPLA outperforms baselines and generalizes effectively across diverse role-playing tasks. |
Assay2Mol: Large Language Model-based Drug Design Using BioAssay Context (2025.emnlp-main)
Copied to clipboard
| Challenge: | Unstructured text that describes biological mechanisms of assays is untapped for early-stage drug discovery. |
| Approach: | They propose a large language model-based workflow that can capitalize on existing biochemical screening assays for early-stage drug discovery. |
| Outcome: | Assay2Mol outperforms machine learning approaches that generate candidate compounds for protein structures while promoting more synthesizable molecule generation. |
Frame First, Then Extract: A Frame-Semantic Reasoning Pipeline for Zero-Shot Relation Triplet Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extract triplets for unseen relations rely on costly fine-tuning and lack structured semantic guidance. |
| Approach: | They propose a framework that adopts a "frame first, then extract" paradigm to extract triplets from unstructured text. |
| Outcome: | The proposed framework achieves competitive zero-shot performance on multiple benchmarks and can be used to enhance existing extraction methods. |
MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking, which can elicit harmful or unsafe behaviors. |
| Approach: | They propose a multilingual guardrail with reasoning for prompt classification that integrates culturally and linguistically nuanced variants and supervised fine-tuning. |
| Outcome: | The proposed guardrail outperforms baselines across in-domain and out-of-domain languages by more than 15%. |
TALON: A Multi-Agent Framework for Long-Table Exploration and Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to query-relevant content retrieval fail to retrieve contextually relevant data. |
| Approach: | They propose a multi-agent framework for table question answering over long tables . TALON features a planning agent that iteratively invokes a tool agent to access tabular data . |
| Outcome: | The proposed framework achieves average accuracy improvements of 7.5% and 12.0% across all language models. |
You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using sparse contextually rich examples, we demonstrate a strong association between training data sparsity and model performance. |
| Approach: | They propose two training strategies to leverage contextually rich examples in training data . they demonstrate strong association between sparsity and model performance . |
| Outcome: | The proposed training strategies improve translation accuracy by 6 and 8 percentage points on the ctxPro evaluation. |
Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL (2025.emnlp-main)
Copied to clipboard
Jessica Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin MacMurray van Liemt, Nithum Thain, Hakim Sidahmed, Lucas Dixon
| Challenge: | Parameter-efficient reinforcement learning (PE-RL) is a highly effective training regime to improve large language models’ ability to answer queries on sensitive topics with a Neutral Point of View (NPOV). |
| Approach: | They propose to use parameter-efficient reinforcement learning to train large language models to answer queries with a Neutral Point of View (NPOV) This is compared to the strongest baseline, LoRA finetuning, SFT and RLHF. |
| Outcome: | The proposed training regime improves on NPOV quality and scores higher on features identified by linguists as key to separating good answers from the best answers. |
Randomized Smoothing Meets Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Randomized smoothing (RS) is a well-known technique to ensure the correctness of machine learning models. |
| Approach: | They propose a randomized smoothing technique that can be used to ensure correctness of machine learning models. |
| Outcome: | The proposed method can be used to classify oracle outputs as harmful vs. harmless . the results are validated against recent jailbreak-style adversarial attacks . |
PIIvot: A Lightweight NLP Anonymization Framework for Question-Anchored Tutoring Dialogues (2025.emnlp-main)
Copied to clipboard
| Challenge: | Understanding and improving affective learning strategies continues to be one of computing's primary contributions to education research. |
| Approach: | They propose a framework for PII anonymization that leverages knowledge of the data context to simplify the PI I detection problem. |
| Outcome: | The proposed framework simplifies the detection problem by leveraging knowledge of the data context. |
Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025.emnlp-main)
Copied to clipboard
Yinuo Wang, Baiyang Wang, Robert Mercer, Frank Rudzicz, Sudipta Singha Roy, Pengjie Ren, Zhumin Chen, Xindi Wang
| Challenge: | achieving comprehensive trustworthiness in medical QA poses significant challenges due to complexity of healthcare data, critical nature of clinical scenarios, and multifaceted dimensions of trustworthy AI. |
| Approach: | They examine six key dimensions of trustworthiness in medical QA . they compare how each dimension is evaluated in existing LLM-based systems . |
| Outcome: | The findings show that large language models have improved patient safety and effectiveness . the models exhibit critical trust failures when deployed in clinical settings . |
Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent evidence suggests that language models with human-scale pretraining data may possess a similar generalization ability by generalizing from frequent to rare constructions. |
| Approach: | They construct a synthetic benchmark that targets syntactic and semantic properties of the English Let-Alone construction and compare it with a human-scale transformer language model. |
| Outcome: | The proposed model can generalize from frequent to rare constructions, but human-scale models do not make correct generalizations about Let-Alone’s meaning. |
BOUQuET : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation (2025.emnlp-main)
Copied to clipboard
Pierre Andrews, Mikel Artetxe, Mariano Coria Meglioli, Marta R. Costa-jussà, Joe Chuang, David Dale, Mark Duppenthaler, Nathanial Paul Ekberg, Cynthia Gao, Daniel Edward Licht, Jean Maillard, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Eduardo Sánchez, Ioannis Tsiamas, Arina Turkatenko, Albert Ventayol-Boada, Shireen Yates
| Challenge: | BOUQUET is a multi-way, multicentric and multi-register/domain dataset and benchmark . the dataset is handcrafted in 8 non-English languages . |
| Approach: | They propose to use BOUQuET to collect a multi-way, multicentric and multi-register/domain dataset and benchmark in 8 non-English languages. |
| Outcome: | The proposed dataset is available at https://huggingface.co/datasets/facebook/bouquet. |
HealthCards: Exploring Text-to-Image Generation as Visual Aids for Healthcare Knowledge Democratizing and Education (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-image (T2I) generation has the potential to advance knowledge democratization and education. |
| Approach: | They explore ways to harness T2I models for generating health knowledge flashcards . they curated a high-quality healthcare knowledge flash card dataset . |
| Outcome: | The proposed models can generate health knowledge flashcards with appealing images . the results show that the open-source models can be fine tuned to generate health content . |
When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have shifted focus toward scaling inference-time compute. |
| Approach: | They propose to scale inference-time compute in a multilingual, multi-task setting . they propose to use m-ArenaHard-v2.0 prompts to sample multiple outputs in parallel . |
| Outcome: | The proposed solutions achieve an average +6.8 jump in win-rates for 8B models on m-ArenaHard-v2.0 prompts in non-English languages against proprietary models like Gemini. |
Creativity in LLM-based Multi-Agent Systems: A Survey (2025.emnlp-main)
Copied to clipboard
Yi-Cheng Lin, Kang-Chieh Chen, Zhe-Yan Li, Tzu-Heng Wu, Tzu-Hsuan Wu, Kuan-Yu Chen, Hung-yi Lee, Yun-Nung Chen
| Challenge: | Large language model (LLM)-driven multi-agent systems (MAS) are transforming how humans and AIs collaboratively generate ideas and artifacts. |
| Approach: | They present a taxonomy of agent proactivity and persona design and an overview of generation techniques. |
| Outcome: | The proposed framework and roadmap offers a roadmap for advancing the development, evaluation, and standardization of creative MAS. |
Context and POS in Action: A Comparative Study of Chinese Homonym Disambiguation in Human and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Ambiguity is pervasive in language, yet we resolve it effortlessly and unconsciously . ambiguity is common because languages allow words to take on multiple meanings . |
| Approach: | They build a sentence-pair dataset to examine how context and POS influence homonym resolution in humans and large language models. |
| Outcome: | The proposed dataset compared humans and large language models to determine how POS and context influence homonym resolution in humans. |
Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can be used to attack content filtering algorithms in social media platforms. |
| Approach: | They propose to generate adversarial examples to test the robustness of social media content filtering algorithms. |
| Outcome: | The proposed model outperforms existing models in the case of propaganda, false claims, rumours and hyperpartisan news. |
Leveraging Loanword Constraints for Improving Machine Translation in a Low-Resource Multilingual Context (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study addresses the challenge of adapting loanwords during the translation process in low-resource languages. |
| Approach: | They propose a method that augments source sentences with loanword constraints . they then integrate loanwords as external linguistic knowledge into machine translation systems . |
| Outcome: | The proposed approach improves translation quality and handling loanword adaptation correctly in target languages. |
Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on data-efficient methods, particularly Cross-lingual In-Context Learning (X-ICL) |
| Approach: | They propose a method to improve cross-lingual in-context learning for low-resource languages by using language-specific neurons. |
| Outcome: | The proposed method improves cross-lingual performance on low-resource languages by ensuring full activation of language overlap neurons. |
Scaling Low-Resource MT via Synthetic Data Generation with LLMs (2025.emnlp-main)
Copied to clipboard
Ona de Gibert, Joseph Attieh, Teemu Vahtola, Mikko Aulamo, Zihao Li, Raúl Vázquez, Tiancheng Hu, Jörg Tiedemann
| Challenge: | a recent study has shown that LLM-generated synthetic data can improve low-resource machine translation performance . traditional data augmentation techniques like back-translation preserve the human-written target and synthesize the other . |
| Approach: | They construct a document-level synthetic corpus from English Europarl and extend it via pivoting to 147 additional language pairs. |
| Outcome: | The proposed model can significantly improve low-resource machine translation performance even when noisy. |
Tailoring Table Retrieval from a Field-aware Hybrid Matching Perspective (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirical results show that a hybrid retrieval approach to table retrieval outperforms state-of-the-art benchmarks. |
| Approach: | They propose a table-tailored HYbrid matching rEtriever which addresses table matching needs from a field-aware hybrid perspective. |
| Outcome: | Empirical results show that the proposed rEtriever outperforms state-of-the-art retrieval methods. |
Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on text embeddings focus less on how information is encoded. |
| Approach: | They find that truncating embedding dimensions causes an increase in performance when removed. |
| Outcome: | The proposed method improves performance across 6 state-of-the-art text encoders and 26 downstream tasks. |
Morables: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables (2025.emnlp-main)
Copied to clipboard
Matteo Marcuzzo, Alessandro Zangari, Andrea Albarelli, Jose Camacho-Collados, Mohammad Taher Pilehvar
| Challenge: | Literature-based benchmarks provide a compelling framework for evaluating LLMs' capacity for complex abstract reasoning and inference. |
| Approach: | They propose a novel moral reasoning benchmark built from fables and short stories that uses adversarial variants to stress-test model robustness. |
| Outcome: | The proposed model outperforms models on fables and short stories, but is susceptible to adversarial manipulation and rely on superficial patterns rather than true moral reasoning. |
MessIRve: A Large-Scale Spanish Information Retrieval Dataset (2025.emnlp-main)
Copied to clipboard
Francisco Valentini, Viviana Cotik, Damián Furman, Ivan Bercovich, Edgar Altszyler, Juan Manuel Pérez
| Challenge: | Information retrieval (IR) is the task of finding relevant documents in response to a user query. |
| Approach: | They propose a large-scale Spanish IR dataset with almost 700,000 queries from Google’s autocomplete API and relevant documents sourced from Wikipedia. |
| Outcome: | The proposed dataset covers a wide variety of topics, unlike smaller datasets. |
AFRIDOC-MT: Document-level MT Corpus for African Languages (2025.emnlp-main)
Copied to clipboard
Jesujoba Oluwadara Alabi, Israel Abebe Azime, Miaoran Zhang, Cristina España-Bonet, Rachel Bawden, Dawei Zhu, David Ifeoluwa Adelani, Clement Oyeleke Odoje, Idris Akinade, Iffat Maab, Davis David, Shamsuddeen Hassan Muhammad, Neo Putini, David O. Ademuyiwa, Andrew Caines, Dietrich Klakow
| Challenge: | AFRIDOC-MT is a document-level multi-parallel translation dataset covering five languages . AFRITIC-MT models perform better on sentences than general-purpose LLMs . |
| Approach: | They propose a document-level multi-parallel translation dataset covering English and five African languages. |
| Outcome: | The proposed dataset covers 334 health and 271 information technology news documents . it shows that NLLB-200 achieves the best average performance among standard models . |
Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead (2025.emnlp-main)
Copied to clipboard
| Challenge: | African languages are often left behind in state-of-the-art natural language processing systems and large language models. |
| Approach: | They analyze 884 research papers on NLP for African languages published over past five years . they identify key trends shaping the field and outline promising directions . |
| Outcome: | The findings identify key trends shaping the field and outline promising directions . the authors analyze 884 research papers on NLP for African languages published over the past five years . |
GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them? (2025.emnlp-main)
Copied to clipboard
Yiyang Zhou, Linjie Li, Shi Qiu, Zhengyuan Yang, Yuyang Zhao, Siwei Han, Yangfan He, Kangqi Li, Haonian Ji, Zihao Zhao, Haibo Tong, Lijuan Wang, Huaxiu Yao
| Challenge: | Existing video benchmarks often resemble image-based questions with scans of only a few key frames, without deep temporal reasoning. |
| Approach: | They propose a video benchmark to assess whether large vision-language models can genuinely think with videos rather than perform superficial frame-level analysis. |
| Outcome: | The proposed benchmark consists of 3,269 videos and over 4,342 highly visual-centric questions across 11 categories, including Trajectory Analysis, Temporal Reasoning, and Forensics Detection. |
Social Bias in Multilingual Language Models: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit the same social bias as models processing English texts. |
| Approach: | They examine the literature on bias evaluation and mitigation approaches in multilingual and non-English contexts and identify gaps in the field. |
| Outcome: | The proposed models perform well on multilingual language understanding benchmarks and are consistent with the current literature. |
BYOKG-RAG: Multi-Strategy Graph Retrieval for Knowledge Graph Question Answering (2025.emnlp-main)
Copied to clipboard
Costas Mavromatis, Soji Adeshina, Vassilis N. Ioannidis, Zhen Han, Qi Zhu, Ian Robinson, Bryan Thompson, Huzefa Rangwala, George Karypis
| Challenge: | Existing approaches to knowledge graph question answering (KGQA) rely on Large Language Model (LLM) agents for graph traversal and retrieval. |
| Approach: | They propose a framework that synergizes Large Language Models with specialized graph retrieval tools to enhance KGQA. |
| Outcome: | The proposed framework outperforms the second-best graph retrieval method by 4.5% points while showing better generalization to custom KGs. |
Synth-SBDH: A Synthetic Dataset of Social and Behavioral Determinants of Health for Clinical Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing SBDH datasets lack detailed annotations and are limited in their availability and coverage. |
| Approach: | They propose a synthetic SBDH annotation dataset with detailed SBDH status, temporal information, and rationale across 15 categories. |
| Outcome: | The proposed dataset outperforms models with no Synth-SBDH training on three tasks using real-world clinical datasets from two distinct hospital settings. |
Pun Unintended: LLMs and the Illusion of Humor Understanding (2025.emnlp-main)
Copied to clipboard
Alessandro Zangari, Matteo Marcuzzo, Andrea Albarelli, Mohammad Taher Pilehvar, Jose Camacho-Collados
| Challenge: | Existing models for pun detection lack nuanced grasp typical of human interpretation. |
| Approach: | They analyze existing pun detection benchmarks and human evaluation across recent LLMs to find subtle changes in puns that mislead LLM. |
| Outcome: | The proposed models lack the nuance typical of human interpretation and lack the depth of their analysis to detect puns. |
RACCooN: Versatile Instructional Video Editing with Auto-Generated Narratives (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent video generative models rely on detailed, labor-intensive text prompts for tasks, limiting adaptability for personal/raw videos. |
| Approach: | They propose a video-to-paragraph-to video editing method that supports diverse video editing capabilities, such as removal, addition, and modification, through a unified pipeline. |
| Outcome: | The proposed method supports diverse video editing capabilities, such as removal, addition, and modification, through a unified pipeline. |
Pre-trained Models Perform the Best When Token Distributions Follow Zipf’s Law (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models typically fix a vocabulary size in advance, then use Byte Pair Encoding (BPE) to construct the tokenizer. |
| Approach: | They propose a method for determining the vocabulary size by analyzing token frequency distributions through Zipf’s law and propose to use it to optimize model performance. |
| Outcome: | The proposed method improves model efficiency and effectiveness across NLP, genomics, and chemistry. |
Do RAG Systems Really Suffer From Positional Bias? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval Augmented Generation (RAG) improves the factual accuracy of LLMs on knowledgeintensive tasks by including in the prompt passages retrieved from an external corpus. |
| Approach: | They propose to use a retrieval algorithm to add passages from an external corpus to the LLM prompt to improve the factual accuracy of LLMs. |
| Outcome: | The proposed approach improves the factual accuracy of LLMs on knowledgeintensive tasks by including in the prompt passages retrieved from an external corpus. |
Aspect-Oriented Summarization for Psychiatric Short-Term Readmission Prediction (2025.emnlp-main)
Copied to clipboard
WonJin Yoon, Boyu Ren, Spencer Thomas, Chanhwi Kim, Guergana K Savova, Mei-Hua Hall, Timothy A. Miller
| Challenge: | Recent advances in large language models have enabled the automated processing of lengthy documents even without supervised training on a task-specific dataset. |
| Approach: | They propose a method for processing the summaries of long documents using different aspect-oriented prompts and integrate the information signals from these different prompts for supervised training of transformer models. |
| Outcome: | The proposed method improves on a high-impact task predicting readmissions from a psychiatric discharge using real-world data from four hospitals. |
Adapting Bias Evaluation to Domain Contexts using Generative Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to assess social bias in NLP systems face limitations in scalability and fidelity across domains. |
| Approach: | They propose a domain-adaptive framework that uses prompting with Large Language Models to automatically transform template-based bias datasets into domain-specific variants. |
| Outcome: | The proposed framework improves the accuracy and contextual relevance of bias evaluations in socially relevant datasets. |
Emergent morpho-phonological representations in self-supervised speech models (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that self-supervised speech models do not represent phonological and morphological phenomena in frequent English noun and verb inflections. |
| Approach: | They study how S3Ms represent phonological and morphological phenomena in English . they propose alternative representational strategies that may support human spoken word recognition . |
| Outcome: | a new study shows that S3M models can represent phonological and morphological phenomena in English . the models can be trained to recognize spoken words in naturalistic, noisy environments . |
Multilingual Language Model Pretraining using Machine-translated Data (2025.emnlp-main)
Copied to clipboard
Jiayi Wang, Yao Lu, Maurice Weber, Max Ryabinin, David Ifeoluwa Adelani, Yihong Chen, Raphael Tang, Pontus Stenetorp
| Challenge: | Existing methods for collecting and filtering multilingual web data lead to most languages lagging behind English performance due to the Internet's English-centric nature. |
| Approach: | They propose to translate a high-quality English web corpus into nine languages and pretrain a 1.3B-parameter model on it. |
| Outcome: | The proposed model matches or outperforms multilingual LLMs of similar size across Non-English understanding and reasoning tasks despite being trained on an order of magnitude less data. |
IntentionFrame: A Semi-Structured, Multi-Aspect Framework for Fine-Grained Conversational Intention Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for understanding user intentions in multi-turn dialogues fail to capture conversational complexity. |
| Approach: | They propose a semi-structured framework which organizes conversational intents into four interrelated aspects: situation, emotion, action, and knowledge. |
| Outcome: | The proposed framework retains interpretability and provides a rich context to accurately parse and respond to nuanced user inputs. |
Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite advances in reinforcement learning, data collection and fine-tuning remain costly and hard to scale. |
| Approach: | They propose a video-adaptive test-time scaling strategy that combines RL with a supervised fine-tuning strategy to improve video reasoning capability. |
| Outcome: | The proposed method surpasses existing models by 2.4% in accuracy using only 3.6% training samples. |
Efficient Compositional Multi-tasking for On-device Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Adapter parameters provide a mechanism to modify the behavior of machine learning models and have gained significant popularity in the context of large language models (LLMs). |
| Approach: | They propose a benchmark for text-based compositional multi-tasking where multiple tasks are executed simultaneously. |
| Outcome: | The proposed method is optimized for on-device applications where computational resources are limited. |
Improving Large Language Model Safety with Contrastive Representation Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing defenses against large language models (LLMs) are limited by their ability to generate responses to diverse inputs. |
| Approach: | They propose a model defense framework that finetunes a large-scale model using a triplet-based loss combined with adversarial hard negative mining to encourage separation between benign and harmful representations. |
| Outcome: | The proposed model defense outperforms previous representation engineering-based defenses while improving robustness against input-level and embedding-space attacks. |
Leveraging What’s Overfixed: Post-Correction via LLM Grammatical Error Overcorrection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce overcorrection often result in significantly decreased recall, limiting the usability of correction systems. |
| Approach: | They propose a novel approach that leverages the strengths of large language models to balance recall and precision by triggering overcorrection via LLMs and fine-tuning smaller models to identify and refine erroneous outputs. |
| Outcome: | The proposed approach maximizes recall and precision by leveraging the generative power of LLMs while preserving the reliability of smaller supervised models. |
Scaling Up Temporal Domain Generalization via Temporal Experts Averaging (2025.emnlp-main)
Copied to clipboard
Aoming Liu, Kevin Miller, Venkatesh Saligrama, Kate Saenko, Boqing Gong, Ser-Nam Lim, Bryan A. Plummer
| Challenge: | Temporal Domain Generalization (TDG) aims to generalize across temporal distribution shifts, e.g., lexical change over time. |
| Approach: | They propose a framework that updates the entire model using weight averaging to maximize generalization potential while minimizing computational costs. |
| Outcome: | The proposed framework outperforms previous methods by up to 69% while being up to 60x more efficient. |
LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encoder (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior research on linguistic mechanisms of large language models is limited by coarse granularity, limited analysis scale, and narrow focus. |
| Approach: | They propose a framework for analyzing the linguistic mechanisms of large language models based on Sparse Auto-Encoders. |
| Outcome: | The proposed framework extracts Chinese and English linguistic features across four dimensions . it uncovers intrinsic representations of linguistic knowledge in LLMs and can control outputs . |
The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models fail at simple character-level tasks due to low mutual information, study finds . authors propose a lightweight architectural modification that improves character- level reasoning . |
| Approach: | They propose a lightweight architectural modification that improves character-level reasoning while preserving the inductive advantages of subword models. |
| Outcome: | The proposed model improves character-level reasoning while preserving the advantages of subword models. |
Improving the Quality of Web-mined Parallel Corpora of Low-Resource Languages using Debiasing Heuristics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Parallel Data Curation (PDC) techniques aim to filter out noisy parallel sentences from web-mined corpora. |
| Approach: | They propose to rank parallel sentences using similarity scores on sentence embeddings derived from Pre-trained Multilingual Language Models (multiPLMs) . previous research has shown that the choice of multiPLM significantly impacts the quality of the filtered parallel corpus. |
| Outcome: | The proposed methods reduce disparities between multiPLMs while producing better results. |
Weaver: Interweaving SQL and LLM for Table Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches that combine SQL and LLM rely on rigid workflows . Tables play a critical role across various domains such as finance, healthcare and scientific research . |
| Approach: | Weaver is a modular pipeline that integrates SQL and LLM for table-based question answering. |
| Outcome: | Weaver outperforms state-of-the-art methods on four Table QA datasets. |
ECO Decoding: Entropy-Based Control for Controllability and Fluency in Controllable Dialogue Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Controllable Dialogue Generation (CDG) enables chatbots to generate responses tailored to desired attributes like emotion and dialog-act. |
| Approach: | They propose a method which dynamically adjusts the control strength at each generation step according to the model’s entropy in both the language model and attribute classifier probability distributions. |
| Outcome: | The proposed method outperforms existing methods on DailyDialog and MultiWOZ datasets while maintaining fluency and grammar. |
Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles (2025.emnlp-main)
Copied to clipboard
| Challenge: | Across languages, numeral systems vary widely in how they construct and combine numbers. |
| Approach: | They conduct experiments to examine the linguistic and mathematical aspects of numbers in language. |
| Outcome: | The models can't solve linguistic-mathematical puzzles involving cross-linguistic numeral systems, the authors found . they lack the ability to flexibly infer compositional rules from implicit patterns in human-scale data. |
Unsupervised Concept Vector Extraction for Bias Control in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are known to perpetuate stereotypes and exhibit biases. |
| Approach: | They propose a method that extracts concept representations via probability weighting without labeled data and efficiently selects a steering vector for measuring and manipulating the model’s representation. |
| Outcome: | The proposed method can be used to predict gender bias and generalizes to racial bias. |
Seeing the Same Story Differently: Framing‐Divergent Event Coreference for Computational Framing Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new task aims to capture subtle differences in how news articles frame events . a central challenge is capturing how same real-world event can evolve into sharply divergent narratives . |
| Approach: | They propose a task that identifies pairs of event mentions referring to the same underlying occurrence but differing in framing across documents. |
| Outcome: | The proposed method enables scalable, interpretable analysis of how media frame the same events differently. |
LLMs are Better Than You Think: Label-Guided In-Context Learning for Named Entity Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) tasks are performed using only a few demonstrations. |
| Approach: | They propose a method that leverages training labels through token-level statistics to improve ICL performance. |
| Outcome: | The proposed method outperforms existing methods on five NER datasets and is robust in low-resource settings. |
COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have created significant computational inefficiencies due to their size and complexity. |
| Approach: | They propose to use a linear combination to deactivate non-essential parameters during inference to reduce computational costs. |
| Outcome: | The proposed methods can omit 90% of computations with performance loss as low as 5.5% ideally, while M-COUNTDOWN provides a predictor-free solution with up to 29.4% better performance preservation compared to existing methods. |
SimpleDoc: Multi‐Modal Document Understanding with Dual‐Cue Page Retrieval and Iterative Refinement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Document Visual Question Answering (DocVQA) is a task to answer questions based on documents containing text, tables, and images. |
| Approach: | They propose a lightweight retrieval framework that uses visual language models to embed and retrieve relevant pages as images and generate answers with VLMs that can accept an image as input. |
| Outcome: | The proposed framework outperforms baselines by 3.2% on average on 4 DocVQA datasets with much fewer pages retrieved. |
VLP: Vision-Language Preference Learning for Embodied Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reward engineering are time-consuming and expensive to collect human preference labels. |
| Approach: | They propose a vision-language preference learning framework which learns from human feedback . they define three types of language-conditioned preferences and construct a visual preference dataset . |
| Outcome: | The proposed framework outperforms baselines on embodied manipulation tasks and can be applied to other tasks. |
QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompting methods for multimodal large language models lack fine-grained perception across disparate images . existing methods fail to integrate perception and reasoning, causing problems with general multi-image reasoning tasks. |
| Approach: | They propose a generalized prompting method that integrates perception and reasoning . they evaluate the method on open-source and closed-source MLLMs . |
| Outcome: | The proposed method shows competitive performance across tasks and improves in challenging scenarios. |
EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understanding (2025.emnlp-main)
Copied to clipboard
Ashish Seth, Utkarsh Tyagi, Ramaneswaran Selvakumar, Nishit Anand, Sonal Kumar, Sreyan Ghosh, Ramani Duraiswami, Chirag Agarwal, Dinesh Manocha
| Challenge: | Multimodal Large Language Models excel at visual perception and reasoning in third-person and egocentric videos, but are prone to hallucinations, generating coherent yet inaccurate responses. |
| Approach: | They propose to use a benchmark to evaluate MLLM hallucinations in egocentric videos. |
| Outcome: | EGOILLUSION comprises 1,400 videos paired with 8,000 human-annotated open and closed-ended questions designed to trigger hallucinations in both visual and auditory cues in egocentric videos. |
MULTIVOX: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions (2025.emnlp-main)
Copied to clipboard
Ramaneswaran Selvakumar, Ashish Seth, Nishit Anand, Utkarsh Tyagi, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha
| Challenge: | omni models lack spoken dialogues, which is essential for assessing conversational and auditory capabilities of voice assistants. |
| Approach: | They propose a benchmark to evaluate the ability of voice assistants to integrate paralinguistic speech features into their models. |
| Outcome: | The multivox voice assistant benchmark evaluates the ability of models to integrate spoken and visual cues including paralinguistic speech features for truly multimodal understanding. |
Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has identified MLP modules in early layers as key contributors to factual recall. |
| Approach: | They conduct a comprehensive evaluation of factual recall across several models . they find that Qwen-based models behave differently from previous patterns . |
| Outcome: | The results show that attention modules in the earliest layers contribute more to factual recall than MLP modules. |
Probing Narrative Morals: A New Character-Focused MFT Framework for Use with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to categorize moral foundations in storytelling are limited. |
| Approach: | They propose a character-centric method to quantify moral foundations in storytelling using large language models and a novel Moral Foundations Character Action Questionnaire to validate their approach against human annotations. |
| Outcome: | The proposed method validates against human annotations and then applies to 2,697 folktales from 55 countries. |
Probing and Boosting Large Language Models Capabilities via Attention Heads (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to identifying capabilities rely on external signals with limited structural grounding . emergence of specific capabilities remains poorly understood . |
| Approach: | They propose a lightweight approach that links LLM capabilities to internal components by identifying correspondences at the level of attention heads. |
| Outcome: | The proposed approach improves accuracy on MMLU and BBH by 1 to 1.5 points over gradient-based method and 5 to 6 points over other intermediate-state baselines. |
A Survey of Link Prediction in N-ary Knowledge Graphs (2025.emnlp-main)
Copied to clipboard
Jiyao Wei, Saiping Guan, Da Li, Zhongni Hou, Miao Su, Yucan Guo, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
| Challenge: | N-ary Knowledge Graphs (NKGs) capture n-ary facts containing more than two entities. |
| Approach: | They present the first comprehensive survey of link prediction in NKGs . they provide an overview of the field and analyze their performance and application scenarios . |
| Outcome: | The proposed methods provide an overview of the field and analyze performance and application scenarios. |
Multi-Frequency Contrastive Decoding: Alleviating Hallucinations for Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies attribute object hallucinations to linguistic priors and data biases . MFCD method removes hallucinian distribution in the original output distribution . |
| Approach: | They propose a method that removes the hallucination distribution in the original output distribution . they propose MFCD to mitigate hallucinism in large visual-language models . |
| Outcome: | The proposed method reduces hallucination distributions without training or external tools . the proposed method can be applied to various LVLMs without modifying model architecture or training . |
ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has explored model-driven strategies for prompt optimization, but these methods suffer from high computational overhead or require strong optimization capabilities from the model itself, which limits their broad applicability. |
| Approach: | They propose a framework that optimizes and generates role-playing prompts by limiting the prompt search space to role-player scenarios. |
| Outcome: | The proposed framework matches and surpasses existing prompt optimization methods in terms of performance. |
BrailleLLM: Braille Instruction Tuning with Large Language Models for Braille Domain Tasks (2025.emnlp-main)
Copied to clipboard
Tianyuan Huang, Zepeng Zhu, Hangdi Xing, Zirui Shao, Zhi Yu, Chaoxiong Yang, Jiaxian He, Xiaozhong Liu, Jiajun Bu
| Challenge: | Existing Braille research focuses on isolated tasks while mixed-content Braille tasks face data scarcity and ambiguities. |
| Approach: | They propose a syntax tree-based augmentation method tailored for Braille data. |
| Outcome: | The proposed method improves Braille translation, formula-to-Braille conversion, and mixed-text translation. |
MAviS: A Multimodal Conversational Assistant For Avian Species (2025.emnlp-main)
Copied to clipboard
Yevheniia Kryklyvets, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly, Jinxing Zhou, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Hisham Cholakkal
| Challenge: | Existing multimodal large language models face challenges when it comes to specialized topics like avian species. |
| Approach: | They propose a large-scale multimodal avian species dataset that integrates image, audio, and text modalities for over 1,000 bird species. |
| Outcome: | The proposed model outperforms the baseline MiniCPM-o-2.6 by a large margin. |
Refining Text Generation for Realistic Conversational Recommendation via Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Conversational Recommender Systems (CRSs) deviate from real human interactions by rapidly recommending items in brief sessions. |
| Approach: | They propose to use Large Language Models to generate dialogue summaries from dialogue history and item recommendation information from item description to extract both explicit user statements and implicit preferences inferred from the dialogue context. |
| Outcome: | The proposed method extracts both explicit user statements and implicit preferences inferred from the dialogue context. |
Large Language Models Threaten Language’s Epistemic and Communicative Foundations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are reshaping the norms of human communication, sometimes decouping words from genuine human thought. |
| Approach: | They propose to model humans, LLMs, and texts in a provenance network . they propose to use epistemic doppelgängers to generate texts that are indis- tinguishable from human-authored texts . |
| Outcome: | The proposed models induce semantic drift, erode account-ability, and obfuscate intent and authorship. |
Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to detect the knowledge boundary of Vision Large Language Models (VLLMs) are expensive and require indiscriminate retrieval to address questions that require real-time information or are knowledge-intensive. |
| Approach: | They propose a method that fine-tunes a VLLM on an automatically constructed dataset for boundary identification. |
| Outcome: | The proposed method reduces indiscriminate retrieval while maintaining or improving the performance of a VLLM on an automatically constructed dataset. |
Multi-view-guided Passage Reranking with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models rely on autoregressive generation and sliding window strategies to rank passages, which incur heavy computational overhead as the number of passages increases. |
| Approach: | They propose a non-generative LLM-based reranking method that encodes query-passage information into diverse view embeddings without being influenced by external biases. |
| Outcome: | The proposed model matches the performance of much larger 7B-scale fine-tuned models while achieving a 100x reduction in inference latency. |
Disentangling Subjectivity and Uncertainty for Hate Speech Annotation and Modeling using Gaze (2025.emnlp-main)
Copied to clipboard
Özge Alacam, Sanne Hoeken, Andreas Säuberli, Hannes Gröner, Diego Frassinelli, Sina Zarrieß, Barbara Plank
| Challenge: | Variation is inherent in opinion-based annotation tasks like sentiment or hate speech analysis. |
| Approach: | They propose to use annotators' confidence ratings to disentangle subjective variation from uncertainty without relying on specific features present in the data. |
| Outcome: | The proposed approach shows that human gaze patterns offer valuable indicators of subjective evaluation and uncertainty. |
VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Due to the nature of speech modality, social bias in Spoken Language Models (SLMs) can emerge from two distinct sources: 1) content aspect and 2) acoustic aspect. |
| Approach: | They propose a dataset that measures social bias by presenting ambiguous or disambiguated contexts followed by questions that may elicit stereotypical responses. |
| Outcome: | The proposed dataset converts every BBQ context into controlled voice conditions, enabling per-axis accuracy, bias, and consistency scores comparable to the original text benchmark. |
Explaining Differences Between Model Pairs in Natural Language through Sample Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | a framework that generates faithful natural language explanations of when and how two ML models converge or diverge in their predictions requires access to training data. |
| Approach: | They propose a framework that generates faithful natural language explanations of when and how two ML models converge or diverge in their predictions. |
| Outcome: | The proposed framework generates faithful natural language explanations of when and how two models diverge in their predictions. |
Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) and AI systems have led to a paradigm shift in the design and optimization of complex workflows. |
| Approach: | They propose a systematic review of recent progress in optimizing compound AI systems . they formalize the notion of compound AI system optimization and classify existing methods along several key dimensions . |
| Outcome: | The proposed methods outperform existing methods in the field of compound AI and highlight open research challenges and future directions. |
A Multi-Level Benchmark for Causal Language Understanding in Social Media Discourse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets focus on explicit causality in structured text, providing limited support for detecting implicit causal expressions. |
| Approach: | They propose a dataset of Reddit posts annotated across four causal tasks . they use a binary causal classification, explicit vs. implicit causality, cause–effect span extraction and causal gist generation to bridge causal detection and reasoning over informal discourse. |
| Outcome: | The proposed dataset analyzes 10,120 Reddit posts discussing public health related to the COVID-19 pandemic. |
Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality Missingness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Clinical notes are often missing from clinical notes, resulting in modality missing-not-at-random (MMNAR) . large language models fine-tuned or adapted to clinical tasks have shown promise in medical reasoning, outcome prediction, and decision support. |
| Approach: | They propose a framework that leverages observed data and informative missingness in multimodal clinical records. |
| Outcome: | The proposed framework leverages observed data and informative missingness in multimodal clinical records. |
XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown significant progress in Open-domain question answering (ODQA) but most evaluations focus on English and assume locale-invariant answers across languages. |
| Approach: | They propose a benchmark specifically designed for locale-sensitive multilingual ODQA that uses 3,000 English seed questions expanded to eight languages. |
| Outcome: | The proposed benchmarks are based on 3,000 English seed questions expanded to eight languages and a human-verified annotation distinguishing locale-invariant and locale-sensitive cases. |
Transformer-Based Temporal Information Extraction and Application: A Review (2025.emnlp-main)
Copied to clipboard
| Challenge: | Temporal information extraction (IE) aims to extract structured temporal information from unstructured text, thereby uncovering the implicit timelines within. |
| Approach: | They summarize and analyze the work using Transformers to highlight potential future directions. |
| Outcome: | The proposed method is applied across healthcare, newswire, and intelligence analysis domains. |
How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current studies evaluate LLMs on explicit false statements, overlooking how misinformation manifests subtly as unchallenged premises in real-world interactions. |
| Approach: | They propose to use EchoMist to analyze implicit misinformation from diverse sources . they also investigate two mitigation methods to enhance LLMs’ capability to counter implicit mis information. |
| Outcome: | The proposed model fails to detect false premises and generate counterfactual explanations. |
AmpleHate: Amplifying the Attention for Versatile Implicit Hate Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to detect hate speech rely on contrastive learning to distinguish hate from non-hate sentences. |
| Approach: | They propose a novel approach to detect implicit hate speech by identifying explicit targets . they use a pretrained Named Entity Recognition model to capture explicit target information . |
| Outcome: | The proposed approach outperforms current methods and achieves faster convergence. |
Can Large Language Models Act as Ensembler for Multi-GNNs? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing graph neural networks lack the inherent semantic understanding capability of rich textual attributes, limiting their effectiveness in applications. |
| Approach: | They propose a model that integrates multiple GNNs and LLMs to provide an ensemble for multi-GNNs. |
| Outcome: | The proposed model outperforms existing models in terms of semantic understanding of graph structures and graph structures. |
Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has focused on situational awareness, which refers to a model's ability to recognize its operating phase and constraints, but it has neglected the complementary capacity to identify and adapt to the identity and characteristics of a dialogue partner. |
| Approach: | They formalize interlocutor awareness and evaluate its emergence in contemporary LLMs. |
| Outcome: | The proposed model reliably identify same-family peers and certain prominent model families, such as GPT and Claude. |
From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text (2025.emnlp-main)
Copied to clipboard
Ridwan Mahbub, Mohammed Saidul Islam, Mir Tafseer Nayeem, Md Tahmid Rahman Laskar, Mizanur Rahman, Shafiq Joty, Enamul Hoque
| Challenge: | Existing VLMs produce more positive descriptions for high-income countries compared to middle- or low-income nations, even when country attribution is the only variable changed. |
| Approach: | They propose to automate the process by generating textual summaries of charts using vision-language models to understand how a country’s economic status influences the sentiment of generated summary. |
| Outcome: | The proposed model amplifys geo-economic biases in 6,000 chart-country pairs from six widely used vision-language models to understand how a country’s economic status influences the sentiment of generated summaries. |
Real-time Ad Retrieval via LLM-generative Commercial Intention for Sponsored Search Advertising (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retrieving documents and ads use one-to-few mappings and time-consuming content extraction. |
| Approach: | They propose a framework that leverages LLM-generated commercial intents as an intermediate semantic representation to directly retrieve ads for queries in real-time. |
| Outcome: | The proposed framework has been implemented in a real-world online system, handling daily search volumes in billions. |
Toward Efficient Sparse Autoencoder-Guided Steering for Improved In-Context Learning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) have emerged as a powerful analytical tool in mechanistic interpretability for large language models (LLMs). |
| Approach: | They propose a novel approach that leverages SAEs to enhance the general in-context learning performance of large language models (LLMs). |
| Outcome: | The proposed method yields a 3.5% improvement across diverse text classification tasks and exhibits greater robustness to hyperparameter variations compared to standard steering approaches. |
CLMTracing: Black-box User-level Watermarking for Code Language Model Tracing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Open-source code language models (code LMs) are a growing threat for intellectual property protection. |
| Approach: | They propose a black-box code LM watermarking framework that uses rule-based watermarks and utility-preserving injection method for user-level model tracing. |
| Outcome: | The proposed framework shows that it performs well across multiple state-of-the-art code LMs and is harmless compared to existing baselines. |
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Providing constructive feedback to authors is a core component of peer review . authors lack guidance on how to improve their review, a problem that is often overlooked . |
| Approach: | They use a RevUtil dataset to benchmark fine-tuned models for assessing review comments . they find that machine-generated reviews generally underperform human reviews on these aspects . |
| Outcome: | The proposed model outperforms closed models on four aspects of review comments . the proposed model achieves agreement levels comparable to and exceeding those of human models . |
Evolving Chinese Spelling Correction with Corrector-Verifier Collaboration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent methods address Chinese Spelling Correction (CSC) with either BERT-based models or large language models (LLMs) however, both of them face challenges. |
| Approach: | They propose a model collaboration pipeline to iteratively optimize a BERT-based corrector. |
| Outcome: | The proposed model outperforms existing methods and outperformed human annotation methods. |
M2Edit: Locate and Edit Multi-Granularity Knowledge in Multimodal Large Language Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge editing methods for MLLMs lack multi-granularity knowledge . existing knowledge editing approaches lack multimodality knowledge and generalize to multimodal data. |
| Approach: | They propose a multimodal knowledge editing method which integrates key knowledge layers within MLLMs and collaboratively edits them. |
| Outcome: | The proposed method improves visual generality performance on knowledge data of different granularities. |
Do LLMs Behave as Claimed? Investigating How LLMs Follow Their Own Claims using Counterfactual Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation frameworks rely on curated datasets that, once public, may be accessed by newer LLMs. |
| Approach: | They propose a framework that generates counterfactual questions and answers from existing evaluation datasets and uses them to evaluate LLMs. |
| Outcome: | The proposed evaluation framework reduces the risk of data leakage by allowing the LLMs to respond to counterfactual questions and verify their claims. |
Multilingual vs Crosslingual Retrieval of Fact-Checked Claims: A Tale of Two Approaches (2025.emnlp-main)
Copied to clipboard
| Challenge: | Previous work has mostly tackled the task monolingually, i.e., having both the input and the retrieved claims in the same language. |
| Approach: | They examine strategies to improve multilingual and crosslingual performance by selecting negative examples and re-ranking. |
| Outcome: | The proposed methods improve performance on a multilingual and crosslingual dataset. |
How Much Do LLMs Hallucinate across Languages? On Realistic Multilingual Estimation of LLM Hallucination (2025.emnlp-main)
Copied to clipboard
| Challenge: | despite LLMs becoming increasingly multilingual, most studies on detecting and quantifying LLM hallucination are English-centric . |
| Approach: | They train a multilingual hallucination detection model and conduct a large-scale study across 30 languages and 6 open-source LLM families. |
| Outcome: | The proposed model is based on an English-centric model and annotates gold data for five high-resource languages. |
LiTransProQA: An LLM-based Literary Translation Evaluation Metric with Professional Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for literature prioritize mechanical accuracy over artistic expression . this bias could result in an irreversible decline in translation quality and cultural authenticity . |
| Approach: | They propose a novel, reference-free, LLM-based question-answering framework for literary translation evaluation. |
| Outcome: | a novel, reference-free, LLM-based question-answering framework is developed for literary translation evaluation. |
Improving Handshape Representations for Sign Language Processing: A Graph Neural Network Approach (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for sign language recognition process a signing sequence holistically, leaving handshape information implicit, which limits both recognition accuracy and linguistic analysis. |
| Approach: | They propose a graph neural network that separates temporal dynamics from static handshape configurations in continuous signing sequences. |
| Outcome: | The proposed approach achieves 46% accuracy across 37 handshape classes, compared to 25% for baseline methods. |
Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque (2025.emnlp-main)
Copied to clipboard
Oscar Sainz, Naiara Perez, Julen Etxaniz, Joseba Fernandez de Landa, Itziar Aldabe, Iker García-Ferrero, Aimar Zabala, Ekhi Azurmendi, German Rigau, Eneko Agirre, Mikel Artetxe, Aitor Soroa
| Challenge: | Instructing language models with user intent requires large instruction datasets limited to a limited set of languages. |
| Approach: | They propose to use existing LLMs and synthetically generated instructions to train models with user intent. |
| Outcome: | The proposed model outperforms base non-instructed models on Basque without Basque instructions. |
SOCIAL SCAFFOLDS: A Generalization Framework for Social Understanding Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing tools for social reasoning tasks lack explicit social cues, such as intents and implications. |
| Approach: | They propose a framework for facilitating generalization across social reasoning tasks by generating rationales that make social cues explicit. |
| Outcome: | The proposed framework improves cross-task performance and inferred speaker intent in 44% of cases. |
Beyond A Single AI Cluster: A Survey of Decentralized LLM Training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Decentralized LLM training leverages dispersed resources at varying scales. |
| Approach: | They propose a resource-driven paradigm that leverages dispersed resources across clusters, datacenters and even regions. |
| Outcome: | The proposed model scales are 175 billion to 660 billion parameters, and the exponential growth in computational requirements poses significant challenges. |
Can LLM Agents Maintain a Persona in Discourse? (2025.emnlp-main)
Copied to clipboard
Pranav Bhandari, Nicolas Fay, Michael J Wise, Amitava Datta, Stephanie Meek, Usman Naseem, Mehwish Nasim
| Challenge: | Large language models are often subjected to context-shifting behaviour, resulting in a lack of consistent and interpretable personality-aligned interactions. |
| Approach: | They propose to use two conversation agents to generate a discourse with an assigned personality from the OCEAN framework and then use multiple judge agents to infer original traits. |
| Outcome: | The proposed model is based on two conversation agents with a personality assigned from the OCEAN framework and then multiple judge agents to infer the original traits assigned. |
Iterative Multilingual Spectral Attribute Erasure (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for debiasing are unable to exploit this opportunity because they operate on individual languages. |
| Approach: | They propose to iterate multilingual spectral attribute error (IMSAE) to mitigate joint bias subspaces across multiple languages through iterative SVD-based truncation. |
| Outcome: | The proposed method outperforms monolingual and cross-lingual approaches while maintaining model utility. |
TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-SQL datasets are too complex and noisy for rigorous interpretability analysis. |
| Approach: | They propose text-to-SQL generation as an ideal task to study mechanistic interpretability . they use edge attribution patching and sparse autoencoders to identify minimal circuits . |
| Outcome: | The proposed task combines the formal structure of toy tasks with real-world complexity. |
SCRIBE: Structured Chain Reasoning for Interactive Behaviour Explanations using Tool Calling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models can be used to provide personalized feedback in educational settings, but they face privacy concerns, limited computational resources, and the need for pedagogically valid responses. |
| Approach: | They propose a framework for multi-hop, tool-augmented reasoning to generate valid responses to student questions about feedback reports. |
| Outcome: | The proposed framework can generate valid responses to student questions about feedback reports using domain-specific tools and self-reflective inference pipelines. |
Logit Space Constrained Fine-Tuning for Mitigating Hallucinations in LLM-Based Recommender Systems (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLM-based recommender systems rely on standard fine-tuning methodologies, often ignoring hallucination issues during the fine-uning process. |
| Approach: | They propose a logit space constraint-based fine-tuning framework to mitigate hallucination in LLM-based recommenders by incorporating Kullback–Leibler divergence into the training objective. |
| Outcome: | Experiments on two recommendation models with distinct LLM backbones and four real-world datasets show that LCFT reduces hallucination and enhances recommendation performance. |
PACHAT: Persona-Aware Speech Assistant for Multi-party Dialogue (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive research on spoken dialogue systems has advanced the development of intelligent voice assistants, but integration of role information within speech remains an underexplored area. |
| Approach: | They propose a language-based spoken dialogue system that integrates role information within speech to generate contextually appropriate responses. |
| Outcome: | The proposed architecture achieves speaker-specific responses, character understanding, and the generation of targeted replies in multi-party dialogue scenarios, surpassing existing spoken dialogue systems. |
Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LLMs) have demonstrated impressive performances across diverse domains, but how their safety benefits from enhanced reasoning capabilities against jailbreak queries remains unexplored. |
| Approach: | They propose a safety-aware reasoning paradigm that integrates a pivot token-based safety-based reasoning mechanism into LLMs’ generation process. |
| Outcome: | The proposed model improves the safety of large language models against jailbreak queries while minimizing attacks and maintaining the original performance. |
Graph-Guided Textual Explanation Generation Framework (2025.emnlp-main)
Copied to clipboard
Shuzhou Yuan, Jingyi Sun, Ran Zhang, Michael Färber, Steffen Eger, Pepa Atanasova, Isabelle Augenstein
| Challenge: | Existing work has questioned their faithfulness, as they may not accurately reflect the model’s internal reasoning process regarding its predicted answer. |
| Approach: | They propose a Graph-Guided Textual Explanation Generation framework that generates a graph neural network layer that guides the NLE generation and generates explanations with greater semantic and lexical similarity to human-written ones. |
| Outcome: | The proposed framework improves NLE faithfulness by up to 12.12% compared to baseline methods on encoder-decoder and decoder-only models. |
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored methods to enhance self-correction in large language models, but little attention has been given to understanding the models’ internal mechanisms underlying error detection. |
| Approach: | They propose to use a large language model to analyze arithmetic errors in four smaller-sized LLMs and identify their internal mechanisms. |
| Outcome: | The proposed models heavily rely on consistency headstextemdashattention heads that assess surface-level alignment of numerical values in arithmetic solutions. |
A Causal Lens for Evaluating Faithfulness Metrics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) offer natural language explanations as an alternative to feature attribution methods for model interpretability, but they may not reflect the model’s truereasoning faithfully. |
| Approach: | They propose a testbed framework for evaluating faithfulness metrics for natural language explanations using diagnosticity and model-editing methods. |
| Outcome: | The proposed framework evaluates faithfulness metrics for natural language explanations on four tasks including fact-checking, analogy, object counting, and multi-hop reasoning. |
Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts (2025.emnlp-main)
Copied to clipboard
Yifei Yu, Qian-Wen Zhang, Lingfeng Qiao, Di Yin, Fang Li, Jie Wang, Chen Zeng Xi, Suncong Zheng, Xiaolong Liang, Xing Sun
| Challenge: | Recent models have extended Corresponding Author. context lengths to millions of tokens while maintaining reasoning and comprehension capabilities. |
| Approach: | They propose a benchmark to evaluate the ability of large language models to extract sequential information items from long contexts. |
| Outcome: | The proposed model achieves maximum accuracy of 63.50% on six well-known LLMs. |
FISTAPruner: Layer-wise Post-training Pruning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning methods require inefficient retraining for billion-scale LLMs or rely on heuristicically designed metrics to determine pruning masks, leading to performance degradation. |
| Approach: | They propose a convex optimization model that induces sparsity in large language models by leveraging FISTA. |
| Outcome: | The proposed method can remove 50% of model parameters while retaining 98.6% and 95.6% of the zero-shot performance. |
Do LLMs Encode Frame Semantics? Evidence from Frame Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using the FrameNet lexical resource, we evaluate large language models under prompt-based inference and observe that they can perform frame identification effectively even without explicit supervision. |
| Approach: | They evaluate large language models under prompt-based inference and observe that they encode latent knowledge of frame semantics. |
| Outcome: | The proposed model can generate coherent frame definitions while generalizing well to out-of-domain benchmarks. |
StepER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods overlook the need for different reasoning abilities at different steps, hindering transfer in multi-step retrieval-augmented frameworks. |
| Approach: | They propose a method that uses step-wise supervision to align with evolving information and reasoning demands across stages. |
| Outcome: | The proposed method outperforms previous methods on multi-hop QA benchmarks with an 8B model achieving performance comparable to a 70B teacher model. |
How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is a popular choice of safety fine-tuning algorithms, but prior explanations of its effects only account for dampened toxic neurons in the MLP layers. |
| Approach: | They analysed four language models and found that toxic neurons only account for 2.5% to 24% of DPO’s effects across models. |
| Outcome: | The proposed method outperforms DPO in reducing toxicity while preserving perplexity, without requiring any weight updates. |
It’s All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-resource languages, especially those written in rare scripts, remain unsupported by large language models due to lack of training data. |
| Approach: | They evaluate 20 under-represented languages across three state-of-the-art multilingual LLMs and compare their methods to parameter-efficient fine-tuning. |
| Outcome: | The proposed methods compare with parameter-efficient fine-tuning (PEFT) on low-resource languages. |
Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a critical emerging capability of large language models (LLMs), enabling few-shot learning during inference by including a few demonstrations in the prompt. |
| Approach: | They propose to use positional bias to study ICL's performance for the first time by examining the positional variation in demos, system prompt, and user message in LLM input. |
| Outcome: | The proposed model can predict accuracy and accuracy when demos are placed at different positions in the input prompt and in the user message. |
Multilingual Pretraining for Pixel Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | PIXEL-M4 model pretrains on four visually and linguistically diverse languages . previous work on pixel-based language models focused on monolingual pretraining on English data . |
| Approach: | They propose a pixel-based language model that is pretrained on four visually diverse languages. |
| Outcome: | The proposed model outperforms an English-only counterpart on non-Latin scripts on semantic and syntactic tasks. |
MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for faithful calibration of large language models (LLMs) are insufficient and can harm faithful calibration. |
| Approach: | They propose a new prompt-based calibration approach inspired by human metacognition that measures faithfulness across diverse models and task domains and enables up to 61% improvement in faithfulness. |
| Outcome: | The proposed approach improves faithfulness across diverse models and task domains and achieves an 83% win rate over original generations as judged by humans. |
Machine-generated text detection prevents language model collapse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly prevalent across the web, resulting in a degenerative process whereby LLMs reinforce their own errors and reduce output diversity. |
| Approach: | They propose to use machine-generated text to reduce model collapse by up-sampling likely human content in training data. |
| Outcome: | The proposed approach prevents model collapse and improves performance compared to training on purely human data. |
Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting hate speech data are expensive and time-consuming . labeled data is expensive and difficult to collect, especially for low-resource languages . |
| Approach: | They propose a method that leverages nearest-neighbor retrieval to augment minimal labeled data in target language. |
| Outcome: | The proposed method outperforms existing models on eight languages and is highly data-efficient. |
V-VAE: A Variational Auto Encoding Framework Towards Fine-Grained Control over Human-Like Chat (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing role-play and persona-based chat approaches rely on static role descriptions, coarse-grained signal space, and low-quality synthetic data. |
| Approach: | They propose a Verbal Variational Auto-Encoding framework which dynamically adapts dialogue behaviour based on latent variables across talking style, interaction patterns, and personal attributes. |
| Outcome: | The proposed framework outperforms baselines on HumanChatBench and DialogBench to address the scarcity of high-quality data in the human-like domain. |
Mixture of Languages: Improved Multilingual Encoders Through Language Grouping (2025.emnlp-main)
Copied to clipboard
João Maria Janeiro, Belen Alastruey, Francisco Massa, Maha Elbayad, Benjamin Piwowarski, Patrick Gallinari, Loic Barrault
| Challenge: | Recent work in this field relies on training transformer encoders on a large amount of multilingual data, with all parameters shared across all languages. |
| Approach: | They propose a mixture of languages strategy to pretrain largely multilingual encoders using masked language modeling. |
| Outcome: | The proposed architecture outperforms a dense counterpart, MoE models and public multilingual encoders on downstream tasks while minimizing interference. |
Too Helpful, Too Harmless, Too Honest or Just Right? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods optimize for individual alignment dimensions in isolation, leading to trade-offs and inconsistent behavior. |
| Approach: | They propose a modular alignment framework that integrates a Mixture of Calibrated Experts (MoCaE) within the Transformer architecture. |
| Outcome: | The proposed framework outperforms baselines on three alignment benchmarks, achieving 32.5% win rate, 33.9% safety score, and 28.4% truthfulness. |
Cardiverse: Harnessing LLMs for Novel Card Game Prototyping (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have enhanced the possibilities for game prototyping, but they face significant challenges. |
| Approach: | They propose a graph-based indexing method for generating novel game variations and an LLM-driven system for consistent game code generation validated by gameplay records. |
| Outcome: | The proposed framework accelerates card game prototyping, reduces human labor, and lowers barriers to entry for game developers. |
Assessing effective de-escalation of crisis conversations using transformer-based models and trend statistics (2025.emnlp-main)
Copied to clipboard
| Challenge: | a lack of quantitative approaches to assess emotion in crisis conversations hinders the science of crisis intervention. |
| Approach: | They propose a transformer-based emotional valence scoring model that measures emotion in crisis conversations . they compare numerical emotional vs. verbal valencies to a corpus of hand-scored social media messages . |
| Outcome: | The proposed model outperforms dictionary-based tools and an LLM in matching scores from human annotators. |
Measuring and Mitigating Media Outlet Name Bias in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored the potential political biases of large language models, but limited attention has been devoted to the effects of media outlet names. |
| Approach: | They propose to quantify media outlet name biases in large language models and leverage this metric to develop an automated prompt optimization framework. |
| Outcome: | The proposed framework mitigates media outlet name biases, offering a scalable approach to enhancing the fairness of LLMs in news-related applications. |
The Good, the Bad, and the Debatable: A Survey on the Impacts of Data for In-Context Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is an emergent capability of large language models that allows them to learn new tasks at inference time without updating parameter updates. |
| Approach: | They propose to examine the relationship between data and in-context learning by examining the qualities of demonstrations that are desirable when selecting demonstrations, the "bad" qualities of demonstrators that can negatively impact the model and the "debatable" qualities. |
| Outcome: | The proposed model can learn unseen tasks by seeing a number of examples in the context window without updating parameters. |
Where Confabulation Lives: Latent Feature Discovery in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite advances in large language models, hallucination remains a critical failure mode . despite these advances, LLMs are prone to hallucinic outputs that contain illusory information presented as if it were factual or grounded in reality. |
| Approach: | They propose to isolate and analyze confabulation, a foundational aspect of hallucination, where the model fabricates facts about unknown entities. |
| Outcome: | The proposed method reveals that the model can fabricate facts with minimal disruption, shedding light on the inner representations that drive factual and non-factual output. |
Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has demonstrated that using chain of thought (CoT) on soft-reasoning tasks can yield limited or even negative performance gains. |
| Approach: | They investigate how chain of thought (CoT) is used in soft-reasoning tasks across instruction-tuned, reasoning and reasoning-distilled models. |
| Outcome: | The proposed model can steer predictions without faithfully reflecting reasoning, indicating a disconnect between CoT influence and faithfulness. |
Playpen: An Environment for Exploring Learning From Dialogue Game Feedback (2025.emnlp-main)
Copied to clipboard
Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia
| Challenge: | In this paper, we investigate whether Dialogue Games—goal-directed and rule-governed activities driven predominantly by verbal actions—can also serve as a source of feedback signals for learning. |
| Approach: | They introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning, direct alignment and reinforcement learning with Group Relative Policy Optimization. |
| Outcome: | The proposed model improves performance on unseen instances, but negatively impacts other skills, while interactive learning shows balanced improvements without loss of skills. |
GenLink: Generation-Driven Schema-Linking via Multi-Model Learning for Text-to-SQL (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results on BIRD and Spider benchmarks validate the effectiveness of GenLink. |
| Approach: | They propose a generation-driven schema-linking framework based on multi-model learning . experimental results validate the effectiveness of GenLink . |
| Outcome: | Experimental results show that GenLink improves schema-linking recall rate and cross-domain adaptability. |
TSVer: A Benchmark for Fact Verification Against Time-Series Evidence (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for fact-checking lack structured evidence, provide insufficient justifications for verdicts, or rely on synthetic claims. |
| Approach: | They propose a temporal and numerical reasoning dataset based on time-series evidence that is annotated with time frames and a verdict and justifications reflecting how the evidence is used to reach the verdict. |
| Outcome: | The proposed dataset improves the quality of the annotations and achieves an inter-annotator agreement of = 0.745 on verdicts. |
Cross-MoE: An Efficient Temporal Prediction Framework Integrating Textual Modality (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models ignore dynamic and different relations between time series patterns and textual features, which leads to poor performance in temporal-textual feature fusion. |
| Approach: | They propose a temporal-textual fusion framework that replaces Cross Attention with Cross-Ranker to reduce computational complexity and enhances modality-aware correlation memorization with Mixture-of-Experts (MoE) networks to tolerate the distributional shifts in time series. |
| Outcome: | The proposed framework reduces MSE by 8.78% compared to the current SOTA model and requires only 75% of computational overhead and 12.5% of activated parameters. |
Sparse Autoencoder Features for Classifications and Transferability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) provide potential for uncovering structured, human-interpretable representations in Large Language Models (LLMs). |
| Approach: | They analyze SAEs for interpretable feature extraction from Large Language Models in safety-critical classification tasks. |
| Outcome: | The proposed framework outperforms hidden-state and BoW models while demonstrating cross-lingual toxicity detection and visual classification tasks. |
KGE Calibrator: An Efficient Probability Calibration Method of Knowledge Graph Embedding Models for Trustworthy Link Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for probability calibration of knowledge graph embedding models are ill-suited for KGEs. |
| Approach: | They propose a method to calibrate knowledge graph embedding models for ranking-based link prediction using a Jump Selection Strategy and Multi-Binning Scaling to enhance reliability. |
| Outcome: | Experiments show that the KGEC outperforms existing calibration methods in terms of effectiveness and efficiency. |
LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to automate essay scoring rely on LLMs to generate absolute scores . however, these methods diverge from human evaluations due to model biases and inconsistent scoring . |
| Approach: | They propose a method that formulates AES as a pairwise comparison task using large language models. |
| Outcome: | The proposed method outperforms conventional zero-shot methods in accuracy while maintaining computational efficiency. |
The Arabic Generality Score: Another Dimension of Modeling Arabic Dialectness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work addresses this issue by modeling dialectness as a continuous variable . however, ALDi reduces complex variation to a single dimension . |
| Approach: | They propose a way to model Arabic dialectness as a continuous variable . they propose etymology-aware edit distance and a regression model to model AGS . |
| Outcome: | The proposed approach outperforms baselines on a multi-dialect benchmark. |
Lemmatization as a Classification Task: Results from Arabic across Multiple Genres (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing tools for lemmatization in morphologically rich languages with ambiguous orthography face inconsistent standards and limited genre coverage. |
| Approach: | They propose two new approaches that frame lemmatization as classification into a Lemma-POS-Gloss tagset, leveraging machine translation and semantic clustering. |
| Outcome: | The proposed models perform better than existing models and are more interpretable, the authors show. |
A Comprehensive Framework to Operationalize Social Stereotypes for Responsible AI Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen unprecedented gains in generative AI models' capabilities across modalitieslanguage, image, audio, and video domains across the globe. |
| Approach: | They propose a framework to operationalize stereotypes in generative AI evaluations using social psychological research and NLP data. |
| Outcome: | The proposed framework identifies key components of stereotypes that are crucial in AI evaluation, including the target group, associated attribute, relationship characteristics, perceiving group, and context. |
Correct-Detect: Balancing Performance and Ambiguity Through the Lens of Coreference Resolution in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are intended to reflect human linguistic competencies . but when context is absent or insufficient, ambiguity resolution becomes more tenuous . |
| Approach: | They propose a CORRECT-DETECT trade-off between large language models and ambiguity detection . they show that large language model models can achieve good performance with minimal prompting . |
| Outcome: | The proposed models can achieve good performance with minimal prompting in coreference disambiguation and detection of ambiguity in corefertility tasks, but they cannot do both at the same time. |
GRAID: Synthetic Data Generation with Geometric Constraints and Multi-Agentic Reflection for Harmful Content Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are expensive to run within a large-scale system and not ideal for low-latency use cases. |
| Approach: | They propose a pipeline that leverages Large Language Models (LLMs) for dataset augmentation. |
| Outcome: | The proposed pipeline improves the performance of a harmful text classification dataset using Large Language Models (LLMs). |
LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to optimize large language models rely on manual design or focus on optimizing individual components. |
| Approach: | They propose a LaMDAgent framework that constructs and optimizes end-to-end post-training pipelines by exploring various model improving methods, objects, and their applied orderings based on task-based feedback. |
| Outcome: | The proposed framework achieves a 9.0-point gain in tool-use accuracy without degrading instruction-following, and reduces computational costs. |
Finetuning LLMs for Human Behavior Prediction in Social Science Experiments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can be used to simulate social science experiments . finetuning LLMs directly on individual-level responses from past experiments improves accuracy . |
| Approach: | They propose to fine tune large language models directly on individual responses from past experiments to achieve multiple levels of generalization. |
| Outcome: | The proposed model outperforms GPT-4o in completely unseen studies by 36% . the proposed model reduces demographic parity difference by 10.6% compared to GPT-4) |
How Private are Language Models in Abstractive Summarization? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective protection of private information is essential for knowledge dissemination in sensitive domains such as medical and legal. |
| Approach: | They perform a comprehensive study of privacy risks in LM-based summarization using closed- and four-weight models of different sizes and families. |
| Outcome: | The proposed models show that they leak personally identifiable information in their summaries, compared to human-generated summary summators, which show significantly higher privacy protection levels. |
Expectation Preference Optimization: Reliable Preference Estimation for Improving the Reasoning Capability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pairwise preference optimization is used to improve supervised fine-tuning performance of large language models. |
| Approach: | They propose an algorithm that takes pairs of sample groups instead of single samples for preference learning. |
| Outcome: | The proposed algorithm outperforms baseline methods on reasoning benchmarks. |
Split-Merge: Scalable and Memory-Efficient Merging of Expert LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a zero-shot merging framework for large language models consolidates specialized domain experts into a single model without any further training. |
| Approach: | They propose a zero-shot merging framework that consolidates specialized domain experts into a single model without further training. |
| Outcome: | Experiments on code generation, mathematical reasoning, medical question answering, and instruction-following benchmarks confirm the versatility and effectiveness of the proposed framework. |
Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores (2025.emnlp-main)
Copied to clipboard
| Challenge: | a rapid proliferation of large language models (LLMs) makes it difficult to assess which models are best suited for specific tasks. |
| Approach: | They find that the consistency of an LLM's Elo score is 91% correlated with its own human-produced Elo scores. |
| Outcome: | a new method to evaluate large language models is needed to scale with the increasing number of models released . current best way is to measure model's Elo score by comparing it to other models in contests . a simple proxy for Elo scores can be computed cheaply without human data or prior knowledge . |
Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance (2025.emnlp-main)
Copied to clipboard
Xueqing Peng, Triantafillos Papadopoulos, Efstathia Soufleri, Polydoros Giannouris, Ruoyu Xiang, Yan Wang, Lingfei Qian, Jimin Huang, Qianqian Xie, Sophia Ananiadou
| Challenge: | Greek is the dominant language of the world's merchant navy and is a key language for international trade. |
| Approach: | They propose to develop a Greek financial evaluation benchmark and a financial LLM fine-tuned on Greek-specific financial data to bridge this gap. |
| Outcome: | The proposed benchmarks surpass GPT-4 by 8.33%, GPT- 4o by 26.83%, and Deepseek-V3 by 67.74%. |
TaxoAlign: Scholarly Taxonomy Generation Using Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for taxonomy generation do not compare structure of generated surveys with those written by human experts. |
| Approach: | They propose a method that bridges the gap between human-generated and automatically-created taxonomies. |
| Outcome: | The proposed method surpasses baselines on CS-TaxoBench on nearly all metrics. |
DiNaM: Disinformation Narrative Mining with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Disinformation is a powerful force in digital media, posing serious threats such as physical harm and the erosion of democracy. |
| Approach: | They propose to use a multi-step approach to uncover disinformation narratives by using Large Language Models to detect false information and then using clustering techniques to identify underlying disinformation stories. |
| Outcome: | The proposed algorithm outperforms general-purpose narrative mining methods by 16.4–24.7%. |
VeriLocc: End-to-End Cross-Architecture Register Allocation via LLM (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern GPUs evolve rapidly, yet production compilers still rely on hand-crafted register allocation heuristics that require substantial re-tuning for each hardware generation. |
| Approach: | They propose a framework that combines large language models with compiler techniques to enable generalizable and verifiable register allocation across GPU architectures. |
| Outcome: | The framework outperforms expert-tuned libraries on matrix multiplication and multi-head attention and a verifier-guided regeneration loop to ensure correctness. |
MemeIntel: Explainable Detection of Propagandistic and Hateful Memes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for label detection and explanation generation have been limited in understanding complex issues . identifying propaganda and hate in memes is essential for combating misinformation and minimizing harm . |
| Approach: | They propose an explanation-enhanced dataset for propaganda memes in Arabic and hateful memes on English to solve these tasks. |
| Outcome: | The proposed model outperforms the current state-of-the-art in label detection and explanation generation. |
FLUID QA: A Multilingual Benchmark for Figurative Language Usage in Dialogue across English, Chinese, and Korean (2025.emnlp-main)
Copied to clipboard
| Challenge: | Figurative language is a core component of everyday communication . existing benchmarks focus on sentence-level classification or inference tasks . |
| Approach: | They propose a multilingual benchmark that evaluates figurative usage in dialogue . they use a sentence-level diagnostic task to embed figurativ choices into multi-turn contexts . |
| Outcome: | The benchmark evaluates large language models' ability to use figurative expressions coherently in conversation. |
Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in domains requiring moral understanding, yet their reasoning often remains shallow and misaligned with human reasoning. |
| Approach: | They propose a value-grounded framework for evaluating and distilling structured moral reasoning in large language models. |
| Outcome: | The proposed framework evaluates 12 open-source models across four moral datasets. |
VerIF: Verification Engineering for Reinforcement Learning in Instruction Following (2025.emnlp-main)
Copied to clipboard
| Challenge: | Best practices for RL in instruction following remain underexplored. |
| Approach: | They propose a verification method that combines rule-based code verification with LLM-based verification from a large reasoning model. |
| Outcome: | The proposed method achieves state-of-the-art performance among models of comparable size and generalizes well to unseen constraints. |
UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work lacks direct and fair evaluation of Large Language Models’ ability to express uncertainty effectively in long-form generation. |
| Approach: | They propose a benchmark to evaluate uncertainty expression in both long- and short-form question answering (QA) they propose prompt-based and training-based methods to improve models’ performance. |
| Outcome: | The proposed method mitigates this issue but a misalignment persists in uncertainty expression between long- and short-form generation. |
Enhancing Study-Level Inference from Clinical Trial Papers via Reinforcement Learning-Based Numeric Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has framed this task as a textual inference task by retrieving relevant content fragments and inferring conclusions from them. |
| Approach: | They propose to extract structured numerical evidence and apply domain knowledge informed logic to derive outcome-specific conclusions. |
| Outcome: | The proposed approach outperforms general-purpose LLMs of over 400B parameters and achieves a 21% improvement in F1 score over retrieval-based systems. |
Context-aware Biases for Length Extrapolation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Relative Positional Encoding (RPE) lack the capacity to adapt to different input contexts. |
| Approach: | They propose an additive RPE method that learns token-specific, context-aware biases for each attention head in transformers by dynamically adjusting positional biase based on the input sequence. |
| Outcome: | The proposed method significantly improves the extrapolation performance of existing RPE methods on the fineWeb-Edu-10B and WikiText-103 datasets. |
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists (2025.emnlp-main)
Copied to clipboard
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun
| Challenge: | AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. |
| Approach: | They propose an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. |
| Outcome: | The proposed pipeline synthesizes accurate tasks and tasks from a dataset of 5,404 tasks covering four scientific disciplines and 756 Python packages. |
Finding your MUSE: Mining Unexpected Solutions Engine (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches for ideation fail to capture the core functional essence of ideas, enabling a more principled exploration of potential solutions. |
| Approach: | They propose a method for constructing functional concept graphs that support abstraction, problem reframing, and analogical inspiration. |
| Outcome: | The proposed method yields large-scale, high-quality representations with explicit abstraction relations, overcoming limitations of prior work. |
Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Quantization enables efficient deployment of large language models in resource-constrained environments . but impact on truthfulness remains largely unexplored . |
| Approach: | They propose a framework to assess the truthfulness of quantized large language models . they find quantized models retain internally truthful representations but produce false outputs . |
| Outcome: | The framework assesses the truthfulness of quantized models across three dimensions . it finds that quantized model models retain internally truthful representations but are more susceptible to false outputs . |
Leveraging Knowledge Graph-Enhanced LLMs for Context-Aware Medical Consultation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have significantly influenced the field of online medical consultations, but critical challenges remain, such as the generation of hallucinated information and the integration of up-to-date medical knowledge. |
| Approach: | They propose a framework that combines retrieval-augmented generation with a structured medical knowledge graph. |
| Outcome: | The proposed framework outperforms baselines on two medical consultation datasets and shows significant improvements in hallucination reduction and clinical usefulness. |
Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for event extraction are limited in their ability to recall nuanced or rare events. |
| Approach: | They propose a hybrid approach that leverages a self-mixture of agents and a discriminative sequence tagger to resolve ambiguities and enhance overall event prediction quality. |
| Outcome: | The proposed approach outperforms existing state-of-the-art methods across three benchmark datasets. |
Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on calibration and uncertainty quantification focuses on individual models, overlooking the potential of model diversity. |
| Approach: | They propose a method that uses Jensen-Shannon Divergence to identify and aggregate well-calibrated subsets of large language models (LLMs) to improve calibration. |
| Outcome: | The proposed method improves accuracy on binary prediction tasks compared to single-model and naive ensemble baselines. |
Exploring morphology-aware tokenization: A case study on Spanish language modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that subword tokenization improves performance of neural language models. |
| Approach: | They propose a linguistically grounded approach to train a tokenizer on morphologically segmented data. |
| Outcome: | The proposed tokenizer improves on a Spanish language model with morphological information. |
Studying Rhetorically Ambiguous Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets do not contain many rhetorical questions that can be rhetorical or informational depending on context. |
| Approach: | They propose a dataset explicitly constructed to support the study of rhetorical ambiguity . they evaluate the performance of state-of-the-art language models on the dataset . |
| Outcome: | The proposed dataset shows that state-of-the-art language models struggle to recognize rhetorical questions. |
Estimating LLM Consistency: A User Baseline vs Surrogate Metrics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are prone to hallucinations and sensitive to prompt perturbations, resulting in inconsistent or unreliable generated text. |
| Approach: | They propose a logit-based ensemble method to measure LLM consistency and propose to use it to evaluate human ratings of LLM reliability. |
| Outcome: | The proposed method matches the best-performing existing metric in estimating human ratings of LLM consistency. |
Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety evaluations rely on artificial images to evaluate vision-language models . a recent study found that memes are more effective at bypassing safety measures than synthetic or typographic images. |
| Approach: | They propose a benchmark pairing meme images with harmful and benign instructions . they assess multiple VLMs across single and multi-turn interactions . |
| Outcome: | The proposed benchmark pairs real meme images with harmful and benign instructions. |
Improving Rule-based Reasoning in LLMs using Neurosymbolic Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) face challenges in reliably solving reasoning tasks, especially when solving tasks that require strict rule following. |
| Approach: | They propose a method that encodes hidden states into neurosymbolic vectors and decodes them into a neurosample vector space to enable problem-solving within a neural space. |
| Outcome: | The proposed method shows an average of 88.6% lower cross-entropy loss and 15.4 times more problems correctly solved on a suite of mathematical reasoning tasks compared to chain-of-thought prompting and supervised fine-tuning (LoRA). |
Can LLMs Extract Frame-Semantic Arguments? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Frame-semantic parsing is a critical task in natural language understanding . however, the ability of large language models to extract frame-sensical arguments remains unexplored . |
| Approach: | They propose a framework to extract frame-semantic arguments from large language models . they use JSON representations to enhance performance, but smaller models can achieve competitive results . |
| Outcome: | The proposed model achieves state-of-the-art on ambiguous targets while limiting generalization to out-of domain data. |
Accelerated Test-Time Scaling with Model-Free Speculative Sampling (2025.emnlp-main)
Copied to clipboard
Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, Sravan Babu Bodapati
| Challenge: | Language models have demonstrated remarkable capabilities in reasoning tasks through test-time scaling techniques like best-of-N sampling and tree search. |
| Approach: | They propose a model-free speculative decoding approach that exploits redundancy in reasoning trajectories to achieve significant acceleration without compromising accuracy. |
| Outcome: | The proposed approach reduces inference latency by 60-65% while maintaining accuracy. |
Enhancing RLHF with Human Gaze Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning language models with human values and preferences. |
| Approach: | They propose to use gaze-aware reward models and gaze-based distribution of sparse rewards to enhance RLHF. |
| Outcome: | The proposed models achieve faster convergence while maintaining or slightly improving performance, reducing computational requirements during policy training. |
Mapping semantic networks to Dutch word embeddings as a diagnostic tool for cognitive decline (2025.emnlp-main)
Copied to clipboard
| Challenge: | Semantic networks are abstract representations of the semantic memory system and can be used to estimate networks . |
| Approach: | They used Dutch verbal fluency data to explore the relationship between semantic networks and cognitive health. |
| Outcome: | The proposed measures predict cognitive health scores on the Mini-Mental State Examination (MMSE) while the traditional number-of-words measure was not significant, the results suggest that semantic network metrics may provide a more sensitive measure of cognitive health than traditional scoring. |
CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models have shown impressive ability in various language tasks, especially with their emergent in-context learning capability. |
| Approach: | They propose a causal reasoning benchmark for multi-modal in-context learning from large vision-language models that incorporates visual inputs. |
| Outcome: | The proposed model outperforms existing models on three visual causal reasoning tasks and demonstrates their strengths and weaknesses. |
Implicit Behavioral Alignment of Language Agents in High-Stakes Crowd Simulations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language-driven generative agents have enabled large-scale social simulations with transformative uses, from interpersonal training to aiding global policy-making. |
| Approach: | They propose a framework for persona-environment Behavioral Alignment that iteratively refines agent personas and aligns them with real-world expert benchmarks. |
| Outcome: | The proposed framework greatly enhances behavioral realism and reliability in high-stakes social simulations. |
Are Language Models Consequentialist or Deontological Moral Reasoners? (2025.emnlp-main)
Copied to clipboard
Keenan Samway, Max Kleiman-Weiner, David Guzman Piedrahita, Rada Mihalcea, Bernhard Schölkopf, Zhijing Jin
| Challenge: | Recent studies have focused on the moral judgments in large language models rather than their underlying moral reasoning process. |
| Approach: | They propose a taxonomy of moral rationales to classify reasoning traces according to consequentialism and deontology . they use trolley problems to analyze moral reasoning tracing in large language models . |
| Outcome: | The proposed taxonomy of moral rationales sheds light on consequentialism and deontology . it systematically classifies reasoning traces according to two main ethical theories . |
PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing natural language generation (NLG) metrics fail to capture domain-specific nuances . patent claims require precise assessment of structural elements such as antecedent consistency and claim dependency. |
| Approach: | They propose a multi-dimensional evaluation framework specifically designed for patent claims . PatentScore integrates hierarchical decomposition of claim elements, validation patterns and scoring across structural, semantic, and legal dimensions. |
| Outcome: | The proposed evaluation framework outperforms existing evaluation frameworks on patent claims . patentScore achieved highest correlation with expert annotations on 400 patent claims dataset . |
All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) perform well on a multitude of computational tasks, yet their inner workings remain unclear. |
| Approach: | They propose two techniques to inhibit input-specific token computations in initial layers . they propose a transformer that allows for any token to immediately access all preceding tokens . |
| Outcome: | The proposed algorithms can perform on a variety of mental math tasks with high accuracy and transfer across models. |
A Position Paper on the Automatic Generation of Machine Learning Leaderboards (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automated leaderboard generation is a tool for comparing prior work with a tabular overview of experimental results. |
| Approach: | They propose an automatic leaderboard generation framework to standardise how the task is defined. |
| Outcome: | The proposed framework standardises how the ALG task is defined and proposes new directions . the proposed framework includes recommendations for datasets and metrics that promote fair evaluation . |
SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate that SimMark surpasses previous sentence-level watermarking techniques in robustness, sampling efficiency, and applicability across diverse domains. |
| Approach: | They propose a sentence-level watermarking algorithm that makes LLM outputs traceable without requiring access to model internals. |
| Outcome: | The proposed algorithm surpasses previous sentence-level watermarking techniques in robustness, sampling efficiency, and applicability across diverse domains while maintaining the text quality and fluency. |
SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Document Retrieval (VDR) relies on text-to-image retrieval using specialized bi-encoders . et al., 2022, 2024, 2021, 2023, 2026, 2030, 2040, 2050, 2060) document retrieval bridges human or artificial agents to the most relevant information, authors say . |
| Approach: | They propose a zero-shot visual document retrieval method that uses bi-encoders . they propose 63.4% nDCG@5 for visual document capture and a reusable semantic proxy . |
| Outcome: | The proposed method surpasses the strongest specialised multi-vector visual document encoder on the ViDoRe-v2 benchmark and scales similarly on MIRACL-VISION with broader multilingual coverage. |
Meta-Semantics Augmented Few-Shot Relational Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for few-shot relational learning on knowledge graphs focus on leveraging specific relational information, but rich semantics inherent in KGs have been overlooked. |
| Approach: | They propose a meta-learning framework that integrates meta-semantics with relational information for few-shot relational learning. |
| Outcome: | Extensive experiments on two real-world KG benchmarks validate the effectiveness of PromptMeta in adapting to new relations with limited supervision. |
ProLongVid: A Simple but Strong Baseline for Long-context Video Instruction Tuning (2025.emnlp-main)
Copied to clipboard
Rui Wang, Bohao Li, Xiyang Dai, Jianwei Yang, Yi-Ling Chen, Zhen Xing, Yifan Yang, Dongdong Chen, Xipeng Qiu, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Existing approaches to adapt image-focused models for video understanding have not been successful in analyzing long video sequences. |
| Approach: | They propose a video instruction dataset that outperforms existing video instruction data for fine-tuning MLLMs by incrementally increasing input context length. |
| Outcome: | The proposed model outperforms existing models on video benchmarks and outperformed proprietary models on VideoMME even with a compact 7B model. |
ModelCitizens: Representing Community Voices in Online Safety (2025.emnlp-main)
Copied to clipboard
Ashima Suvarna, Christina A Chance, Karolina Naranjo, Hamid Palangi, Sophie Hao, Thomas Hartvigsen, Saadia Gabriel
| Challenge: | Existing toxic language detection models are trained on annotations that collapse diverse perspectives into a single ground truth. |
| Approach: | They propose to augment social media posts with conversational scenarios to reflect the impact of conversational context on toxicity. |
| Outcome: | The proposed model outperforms existing models on social media with conversational scenarios. |
UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets (2025.emnlp-main)
Copied to clipboard
Pengyu Wang, Shaojun Zhou, Chenkun Tan, Xinghao Wang, Wei Huang, Zhen Ye, Zhaowei Li, Botian Jiang, Dong Zhang, Xipeng Qiu
| Challenge: | Existing datasets address understanding and generation in isolation, limiting the performance of unified vision large language models. |
| Approach: | They propose a dataset that facilitates mutual enhancement between multimodal understanding and generation. |
| Outcome: | The proposed framework integrates diverse visual and textual inputs and outputs, enabling comprehensive cross-modal reasoning and precise text-to-image alignment. |
The Pursuit of Empathy: Evaluating Small Language Models for PTSD Dialogue Support (2025.emnlp-main)
Copied to clipboard
Suhas Bn, Yash Mahajan, Dominik O. Mattioli, Andrew M. Sherrill, Rosa I. Arriaga, Christopher Wiese, Saeed Abdullah
| Challenge: | Claude Sonnet 3.5 consistently outperforms all models, but smaller models often approach human-rated empathy levels. |
| Approach: | They introduce a dataset comprising 10,000 two-turn conversations across 500 diverse, clinically-grounded PTSD personas. |
| Outcome: | The proposed model outperforms all models but has a "knowledge transfer ceiling" older adults prefer validation responses while graduate-educated users prefer emotionally layered responses . |
Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) have shown impressive capabilities in document understanding due to different types of annotation noise in training. |
| Approach: | They propose a method to reduce C&P knowledge conflicts across all tested MLLMs . they propose to use annotation noise to train models to understand document content . |
| Outcome: | The proposed method reduces C&P knowledge conflicts across all tested MLLMs and enhances their performance in both cognitive and perceptual tasks. |
AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trials are expensive and time-consuming, and accurate trial prediction is key to advancing medical treatments. |
| Approach: | They propose a framework that combines reasoning capabilities of large language models with the explainability of classical machine learning to generate, evaluate, and refine tabular features without human input. |
| Outcome: | The proposed framework performs better than SOTA methods on clinical trial prediction tasks within a limited number of iterations. |
MMDocIR: Benchmarking Multimodal Retrieval for Long Documents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal document retrieval are lacking for evaluating performance of systems. |
| Approach: | They propose a benchmark that evaluates page-level and layout-level retrieval tasks . they use a rich dataset featuring 1,685 questions annotated by experts . |
| Outcome: | The proposed benchmark outperforms existing benchmarks in page-level and layout-level retrieval tasks. |
Program of Thoughts for Financial Reasoning: Leveraging Dynamic In-Context Examples and Generative Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Numerical reasoning remains a challenging area for large language models (LLMs). |
| Approach: | They propose a two-step framework to enhance LLM's capabilities in financial numerical reasoning by using a generative retriever and context-aware program of thought prompting. |
| Outcome: | The proposed model surpasses previous benchmarks with execution accuracy improvements of 5.98% and 4.05%, respectively. |
Waste-Bench: A Comprehensive Benchmark for Evaluating VLLMs in Cluttered Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have paved the way for VisionLarge Language Model (VLLM) capabilities have not been thoroughly explored in cluttered datasets where there is complex environment having deformedshaped objects. |
| Approach: | They propose a dataset specifically designed for waste classification in real-world scenarios, characterized by complex environments and deformed shaped objects. |
| Outcome: | The proposed dataset provides valuable insights into the performance of VLLMs under challenging conditions. |
Demystifying Domain-adaptive Post-training for Financial LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Domain-adaptive post-training of large language models (LLMs) has emerged as promising approach for specialized domains such as medicine and finance. |
| Approach: | They propose a system to identify optimal adaptation criteria and training strategies for LLMs for the finance domain. |
| Outcome: | The proposed model achieves state-of-the-art performance across a wide range of financial tasks. |
HICode: Hierarchical Inductive Coding with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-grained corpus analysis are limited to small subsets and require manual labeling and statistical tools like topic modeling. |
| Approach: | They propose a pipeline that inductively generates labels from analysis data and then hierarchically clusters them to surface emergent themes. |
| Outcome: | The proposed pipeline validates the approach across three datasets and shows it is robust through automated and human evaluations. |
Cacheback: Speculative Decoding With Nothing But Cache (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that large language models are unable to model locality in language. |
| Approach: | They propose a training-free and model-agnostic speculative decoding method that exploits locality in language to accelerate Large Language Models. |
| Outcome: | The proposed method achieves state-of-the-art performance among comparable methods . it leverages only LRU cache tables of token n-grams to generate draft sequences . |
MA-DPR: Manifold-aware Distance Metrics for Dense Passage Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirical evidence suggests that manifold-aware distance allows DPR to leverage context from related neighboring passages. |
| Approach: | They propose a "manifold-aware" distance metric that measures query–passage distance . they propose to model the intrinsic manifold structure of passages using a nearest-neighbor graph . |
| Outcome: | Empirical evidence suggests MA-DPR outperforms Euclidean and cosine distances on OOD . it can be applied to a wide range of dense embedding and retrieval tasks . |
LLM-Guided Co-Training for Text Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirical results show that it achieves state-of-the-art performance on 4 out of 5 benchmark datasets and ranks first among 14 compared methods according to the Friedman test. |
| Approach: | They propose a weighted co-training approach that is guided by Large Language Models (LLMs) they use LLM labels on unlabeled data as target labels and co-train two encoder-only based networks that train each other over multiple iterations. |
| Outcome: | The proposed approach outperforms conventional methods on 4 out of 5 benchmark datasets and ranks first among 14 compared methods according to the Friedman test. |
LeanK: Learnable K Cache Channel Pruning for Efficient Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing efforts to optimize the key-value (KV) cache include: (1) Eviction, which discards cache of less important tokens; (2) Selection, which retains the full KV cache but selectively reads relevant entries. |
| Approach: | They propose a learning-based method that prunes unimportant key (K) cache channels by leveraging static channel sparsity. |
| Outcome: | Experiments show that LeanK reduces GPU memory and accelerates decoding without sacrificing accuracy. |
DELOC: Document Element Localizer (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to edit PDFs using natural language are ineffective at grounding the edit location effectively. |
| Approach: | They propose a system to ground PDF edit request spatially using a model to predict the edit location in the PDF. |
| Outcome: | The proposed system outperforms existing Multimodal Large Language Models and specialized models on DocEdit. |
NL2Lean: Translating Natural Language into Lean 4 through Multi-Aspect Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
Yue Fang, Shaohan Huang, Xin Yu, Haizhen Huang, Zihan Zhang, Weiwei Deng, Furu Wei, Feng Sun, Qi Zhang, Zhi Jin
| Challenge: | Existing formal proof assistants rely on instruction tuning and lack fine-grained structural and semantic alignment. |
| Approach: | They propose a reinforcement learning framework that enables LLMs to translate natural language into formal language such as Lean 4 . they use a model with basic translation ability to refine the model's reinforcement learning . |
| Outcome: | The proposed method outperforms baseline models on NL-to-Lean 4 tasks. |
A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications (2025.emnlp-main)
Copied to clipboard
| Challenge: | Misgendering is the act of referring to someone by using words that do not match their chosen identity. |
| Approach: | They propose to use a participatory-design approach to assess and mitigate misgendering across 42 languages and dialects using a human-in-the-loop approach. |
| Outcome: | The proposed guardrails reduce misgendering rates across all languages and dialects without loss of quality and without loss in quality. |
X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-video retrieval systems use embedding models for feature extraction and compute cosine similarities for ranking. |
| Approach: | They propose an explainable retrieval framework upon LLM CoT reasoning to replace embedding models for feature extraction and ranking. |
| Outcome: | The proposed retrieval framework improves retrieval performance and produces detailed rationales. |
Token-level Proximal Policy Optimization for Query Generation (2025.emnlp-main)
Copied to clipboard
Yichen Ouyang, Lu Wang, Fangkai Yang, Pu Zhao, Chenghua Huang, Jianfeng Liu, Bochen Pang, Yaming Yang, Yuefeng Zhan, Hao Sun, Qingwei Lin, Saravan Rajmohan, Weiwei Deng, Dongmei Zhang, Feng Sun
| Challenge: | Large Language Models (LLMs) have improved search engines and recommendation systems through their text understanding capabilities. |
| Approach: | They propose a token-level proximal policy optimization approach to empower LLMs to perform better in query generation through fine-tuning. |
| Outcome: | The proposed approach outperforms existing LLMs on an open-source and industrial dataset. |
Prior Prompt Engineering for Reinforcement Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on algorithms, reward shaping, and data curation, but prior prompt engineering is understudied. |
| Approach: | They investigate prior prompt engineering (pPE) in reinforcement fine-tuning . they translate five representative iPE strategies into corresponding pPE approaches . |
| Outcome: | The proposed approaches outperform iPE-prompted models on in-domain and out-of-domain benchmarks. |
Beyond WER: Probing Whisper’s Sub‐token Decoder Across Diverse Language Resource Levels (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large multilingual automatic speech recognition models achieve remarkable performance, but the internal mechanisms of the end-to-end pipeline remain underexplored. |
| Approach: | They propose to analyze Whisper's multilingual decoder to uncover systematic decoding disparities masked by aggregate error rates. |
| Outcome: | The proposed model performs better on higher resource languages, but lower resource languages fare worse on these metrics. |
ThinkTuning: Instilling Cognitive Reflections without Distillation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning. |
| Approach: | They propose a GRPO-based interactive training approach that augments the rollouts of a student model with the guidance of . a teacher poses a problem, lets the student try an answer, then gives corrective feedback–enough to point the mind in the right direction and then show the correct solution. |
| Outcome: | The proposed method shows 3.69% improvement over zero-shot baselines and 2.08% and 3.99% improvement over the vanilla-GRPO baselines. |
Droid: A Resource Suite for AI-Generated Code Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing detectors fail to generalise to diverse coding domains and programming languages outside of their narrow training data. |
| Approach: | They propose to use DroidCollection to train machine-generated code detectors that can be trained on a multi-task objective. |
| Outcome: | The proposed detectors fail to generalise to diverse coding domains and programming languages outside of their narrow training data. |
LoRACoE: Improving Large Language Model via Composition-based LoRA Expert (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that the Mixture of Experts architecture improves performance of large language models. |
| Approach: | They propose a method to build static experts using LoRA parameters . they propose to use rank-level parameters to build experts based on rank-based parameters based in LoRA module. |
| Outcome: | The proposed method improves task performance across a broader range of tasks. |
Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing solutions to the problem of semantically-preserving variations of prompts are expensive and require trial-and-error prompt engineering. |
| Approach: | They propose a dual-loop adversarial framework that optimizes a trainable perturbation as "latent continuous paraphrase" they demonstrate a 0.5%-4% improvement on worst-case win-rate on the RobustAlpaca benchmark . |
| Outcome: | Extensive experiments show that the proposed framework improves on the RobustAlpaca benchmark with a 0.5%-4% improvement on the worst-case win-rate. |
Pluralistic Alignment for Healthcare: A Role-Driven Framework (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to align large language models fail to reflect diversity in sensitive domains like healthcare, where personal, cultural, and situational factors shape pluralism. |
| Approach: | They propose a lightweight, generalizable, pluralistic alignment approach to model diverse perspectives and values in open and closed models. |
| Outcome: | The proposed approach advances the pluralistic alignment for all three modes across seven varying-sized open and closed models. |
Flexible-length Text Infilling for Discrete Diffusion Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing discrete diffusion models lack flexibility for text infilling without ground-truth positional data. |
| Approach: | They propose a discrete diffusion model that jointly denoises token values and token positions using a novel sample-level Optimal Transport coupling. |
| Outcome: | The proposed method outperforms existing methods on infilling benchmarks such as One-Billion-Word and Yelp. |
Beyond the Leaderboard: Understanding Performance Disparities in Large Language Models via Model Diffing (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that benchmarking fails to explain why models outperform others . open-weight large language models have transformed the AI landscape . |
| Approach: | They use model diffing to analyze capability differences between Gemma-2-9b-it and SimPO-enhanced variants. |
| Outcome: | The proposed model diffing approach can provide fine-grained insights beyond leaderboard metrics . it can also help to identify model performance gaps, the authors say . |
Explicit Learning and the LLM in Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a growing number of researchers are examining whether large language models can learn to translate a "new" language using grammar books. |
| Approach: | They examine an LLM's ability to learn new languages using grammar books . authors suggest alternative fine-tuning strategies to improve explicit learning . |
| Outcome: | The proposed model can learn low-resource languages described in grammar books but lacking extensive corpora. |
Towards Language-Agnostic STIPA: Universal Phonetic Transcription to Support Language Documentation at Scale (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing ASR systems focus on orthographic output for high-resource languages, but STIPA can be used as a language-agnostic interface for documenting under-resourced and unwritten languages. |
| Approach: | They propose to use the International Phonetic Alphabet (STIPA) to generate phonetic transcriptions using a language-agnostic interface. |
| Outcome: | The proposed model reduces phonetic error rates even in low-resource settings and can be used for documenting under-resourced and unwritten languages. |
Beyond Pairwise: Global Zero-shot Temporal Graph Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Temporal relation extraction (TRE) is a fundamental task in natural language processing (NLP) that involves identifying the temporal relationships between events in a document. |
| Approach: | They propose a method that generates a document’s complete temporal graph in a single step, followed by temporal constraint optimization to refine predictions and enforce temporal consistency across relations. |
| Outcome: | The proposed method outperforms existing zero-shot approaches and offers a competitive alternative to supervised TRE models. |
“Feels Feminine to Me”: Understanding Perceived Gendered Style through Human Annotations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using gender identity-based framing, language–gender associations are often grounded in the author’s gender identity, inferred from their language use. |
| Approach: | They propose to operationalize the language–gender association as a perceived gender expression of language, focusing on how expression is externally interpreted by humans, independent of the author’s gender identity. |
| Outcome: | The first dataset of itskind identifies 5,100 human annotations of perceived gendered style—human-written texts rated on a five-point scale from very feminine to very masculine. |
RALS: Resources and Baselines for Romanian Automatic Lexical Simplification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text simplification is the process of transforming texts into variants that are simpler to understand by larger audiences or easier to process by existing NLP systems. |
| Approach: | They propose a method for ordering simplification suggestions using a pairwise ranking approximation method, arranging candidates from simple to complex based on a separate set of human judgments. |
| Outcome: | The proposed system is the first to combine lexical simplification and complexity prediction in Romanian with human lexicals. |
How Do Social Bots Participate in Misinformation Spread? A Comprehensive Dataset and Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Social media platforms provide an ideal environment to spread misinformation, where social bots can accelerate the spread. |
| Approach: | They construct a large-scale dataset that includes annotations for misinformation and social bots on the Sina Weibo platform. |
| Outcome: | The proposed dataset contains 65,749 social bots and 345,886 genuine accounts, annotated using a weakly supervised annotator. |
Are Stereotypes Leading LLMs’ Zero-Shot Stance Detection ? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models inherit stereotypes from their pretraining data, leading to biased behavior toward certain social groups in many tasks. |
| Approach: | They propose to annotate posts in pre-existing stance detection datasets with dialect or vernacular of a specific group and text complexity/readability to investigate whether these attributes influence the model’s stance detect decisions. |
| Outcome: | The proposed model exhibits significant stereotypes when performing stance detection tasks in a zero-shot setting. |
Multi-Modal Framing Analysis of News (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automated frame analysis of political communication has been limited by the use of predefined frames and the visual contexts in which they appear. |
| Approach: | They propose a method for doing multi-modal, multi-label framing analysis at scale using large (vision-) language models. |
| Outcome: | The proposed method provides a more complete picture for understanding media bias. |
TempParaphraser: “Heating Up” Text to Evade AI-Text Detection through Paraphrasing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing detectors that perform well on benchmark datasets have weaknesses that can be exploited to manipulate AI-text. |
| Approach: | They propose a framework that simulates high-temperature sampling effects through multiple normal-temperaturing generations, effectively evading detection. |
| Outcome: | The proposed framework reduces detector accuracy by an average of 82.5% while preserving high text quality. |
ComicScene154: A Scene Dataset for Comic Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Comics offer compelling yet under-explored domain for computational narrative analysis . authors highlight potential of comics for narrative-driven, multimodal data analysis based on novel comics . |
| Approach: | They propose a dataset of scene-level narrative arcs derived from comic books . they highlight their potential to inform broader research on multimodal storytelling . |
| Outcome: | The dataset provides an initial benchmark that future studies can build upon. |
MedLinkDE – MedDRA Entity Linking for German with Guided Chain of Thought Reasoning (2025.emnlp-main)
Copied to clipboard
Roman Christof, Farnaz Zeidi, Manuela Messelhäußer, Dirk Mentzer, Renate Koenig, Liam Childs, Alexander Mehler
| Challenge: | In pharmacovigilance, effective automation of medical data structuring is critical. |
| Approach: | They propose a German real-world MedDRA dataset that uses embedding models to retrieve medical terms and guided chain-of-thought re-ranking using LLMs. |
| Outcome: | The proposed approach shows that it is feasible to integrate German data with German MedDRA datasets under the constraints of data scarcity. |
HookMoE: A learnable performance compensation strategy of Mixture-of-Experts for LLM inference acceleration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mixture of Experts (MoE) models have been a promising paradigm for scaling model capacity through top-k routing mechanisms. |
| Approach: | They propose a plug-and-play single-layer compensation framework that strategically inserts a lightweight trainable Hook module immediately preceding selected transformer blocks. |
| Outcome: | The proposed framework reduces the number of activated experts by more than 50% and achieves a 1.42 inference speed-up during the prefill stage. |
Cross-Document Cross-Lingual NLI via RST-Enhanced Graph Fusion and Interpretability Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite the development of many subdirections, Cross-Document Cross-Lingual NLI remains largely unexplored. |
| Approach: | They propose a novel paradigm that extends traditional NLI capabilities to multi-document, multilingual scenarios by integrating RST-enhanced graph fusion with interpretability-aware prediction. |
| Outcome: | The proposed method improves on existing models and document-level NLI to multi-document, multilingual scenarios. |
3R: Enhancing Sentence Representation Learning via Redundant Representation Reduction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve sentence representations lack fine-grained guidance on reducing redundant information. |
| Approach: | They propose a method that dynamically identifies redundant information from a dimensional perspective and trains the SRL model to redistribute semantics on different dimensions. |
| Outcome: | The proposed method improves sentence representations on seven semantic text similarity benchmarks. |
When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision and language models have demonstrated remarkable performance in visual question answering tasks. |
| Approach: | They introduce a framework to optimize L-VLMs by leveraging unlabeled images . they conduct extensive experiments on four diverse VQA benchmarks . |
| Outcome: | The proposed framework improves L-VLMs on four visual question answering benchmarks. |
ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom (2025.emnlp-main)
Copied to clipboard
Jingqi Zhou, Sheng Wang, Jingwei Dong, Kai Liu, Lei Li, Jiahui Gao, Jiyue Jiang, Lingpeng Kong, Chuan Wu
| Challenge: | Large vision-language models often prioritize language knowledge over image information on visual reasoning tasks, incurring performance degradation. |
| Approach: | They propose a visual reasoning framework that decouples vision-reasoning capabilities and multi-run proactive perception. |
| Outcome: | The proposed framework outperforms existing models on benchmarks for open-source and closed-source models with 13.2% performance gain. |
Extractive Fact Decomposition for Interpretable Natural Language Inference in one Forward Pass (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in Natural Language Inference (NLI) uses atomic fact decomposition to enhance interpretability and robustness. |
| Approach: | They propose an encoder-only architecture that performs extractive atomic fact decomposition and interpretable inference without generative models. |
| Outcome: | The proposed architecture achieves competitive accuracy and improves robustness out of distribution and in adversarial settings over models based on extractive rationale supervision. |
Structure-Conditional Minimum Bayes Risk Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Minimum Bayes Risk (MBR) decoding has been used in machine translation for many years. |
| Approach: | They propose three adaptations to the minimum bayes risk utility function to make it more sensitive to structural variability in the outcome space. |
| Outcome: | The proposed adaptations significantly improve generation quality by up to 13.7 percentage points in win rate. |
Label Set Optimization via Activation Distribution Kurtosis for Zero-Shot Classification with Generative Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) performance is highly sensitive to prompt design, yet the impact of class label options (e.g. lexicon or order) in zero-shot classification remains underexplored. |
| Approach: | They propose a post-hoc method for selecting optimal label sets in zero-shot ICL with large language models. |
| Outcome: | The proposed method consistently achieves performance gains of 0.54 to 0.76 compared to the conventional method. |
The Transfer Neurons Hypothesis: An Underlying Mechanism for Language Latent Space Transitions in Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest a processing framework for multilingual inputs in decoder-based LLMs. |
| Approach: | They propose a framework for multilingual inputs in decoder-based LLMs that enables transfer of representations between latent spaces and shared semantic latent space. |
| Outcome: | The proposed framework is validated by a new study on transfer neurons in multilingual LLMs. |
VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions (2025.emnlp-main)
Copied to clipboard
| Challenge: | VEHME is a vision language model for assessing handwritten math answers . traditional methods of assessing student work are limited by time constraints, class sizes and cognitive load . |
| Approach: | They propose a Vision-Language Model for Evaluating Handwritten Mathematics Expressions to assess handwritten math responses with high accuracy and interpretable reasoning traces. |
| Outcome: | VEHME achieves state-of-the-art performance among open-source models and approaches accuracy of proprietary systems. |
All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for confidence estimation are primarily designed for factual QA tasks and fail to generalize to reasoning tasks. |
| Approach: | They propose a set of training-free, graph-based confidence estimation methods tailored to reasoning tasks that exploit graph properties such as centrality, path convergence, and path weighting. |
| Outcome: | The proposed methods improve confidence estimation and performance on two downstream tasks. |
SEMMA: A Semantic Aware Knowledge Graph Foundation Model (2025.emnlp-main)
Copied to clipboard
Arvindh Arun, Sumit Kumar, Mojtaba Nayyeri, Bo Xiong, Ponnurangam Kumaraguru, Antonio Vergari, Steffen Staab
| Challenge: | Existing Knowledge Graph Foundation Models (KGFMs) rely on graph structure, overlooking the rich semantic signals encoded in textual attributes. |
| Approach: | They propose a dual-module KGFM that integrates transferable textual semantics alongside structure to generate relation identifiers. |
| Outcome: | The proposed model outperforms ULTRA and ULtra in fully inductive link prediction in more challenging generalization settings. |
Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in generating visualizations from natural language, but lack of comprehensive benchmarks limits their capabilities. |
| Approach: | They propose a framework that jointly refines the textual answer and visualization code to improve GPT-4o's pass rate from 26% to 42% over direct approach. |
| Outcome: | The proposed framework increases GPT-4o’s pass rate from 26% to 42% over the direct approach and improves chart quality. |
Predicting Prosodic Boundaries for Children’s Texts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a dataset of 54 leveled English stories annotated for potential pauses, we find that nearly 30% of pause occur at non-punctuation locations of the text. |
| Approach: | They propose to use a text-based model to predict pause locations in children's reading material using a curated dataset of 54 leveled English stories annotated for potential pauses, or prosodic boundaries, by 21 fluent speakers. |
| Outcome: | The proposed model can model both allowed and “forbidden” pauses . it uses a curated dataset of 54 leveled English stories annotated for potential pause locations by 21 fluent speakers . |
Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown strong performance in many reasoning benchmarks, but lack robust planning or symbolic abstractions. |
| Approach: | They propose to synthesize high-quality symbolic reasoning trajectories with stepwise pseudo-labels at scale via Monte Carlo estimation. |
| Outcome: | The proposed method can be trained on high-quality symbolic reasoning trajectories with stepwise pseudo-labels at scale using Monte Carlo estimation. |
Can Large Language Models Outperform Non-Experts in Poetry Evaluation? A Comparative Study Using the Consensual Assessment Technique (2025.emnlp-main)
Copied to clipboard
| Challenge: | Consensual Assessment Technique (CAT) for large language models is used to evaluate creativity, but is costly and time-consuming with non-experts. |
| Approach: | They adapt the Consensual Assessment Technique (CAT) for Large Language Models to a 90-poem dataset with a ground truth based on publication venue. |
| Outcome: | The proposed method outperforms the best human non-expert evaluations by significantly outperforming the best language models. |
Beyond Human Labels: A Multi-Linguistic Auto-Generated Benchmark for Evaluating Large Language Models on Resume Parsing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Efficient resume parsing is critical for global hiring, yet the lack of dedicated benchmarks for evaluating large language models (LLMs) on multilingual, structure-rich resumes hinders progress. |
| Approach: | They propose to use a human-in-the-loop pipeline to generate 2,500 synthetic resumes spanning 50 templates, 30 career fields, and 5 languages to evaluate large language models. |
| Outcome: | The proposed benchmarks show that the models perform poorly on multilingual resumes and lack of standardized templates. |
Orthogonal Finetuning Made Scalable (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent shift in foundation models has slowed the adoption of finetuning methods . however, its high runtime and memory demands limit its scalability . |
| Approach: | They propose an input-centric reformulation that uses matrix-vector multiplications instead of cubic multiplication . they extend OFTv2 to support finetuning quantized foundation models and show it outperforms QLoRA . |
| Outcome: | The proposed model outperforms the popular QLoRA in training stability, efficiency, and memory usage. |
AIR: Complex Instruction Generation via Automatic Iterative Refinement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating complex instructions are resource-intensive and lack diversity. |
| Approach: | They propose a framework to generate complex instructions with constraints using a document-generated initial instruction and an iterative refinement framework to incorporate LLM-as-judge guidance. |
| Outcome: | The proposed framework significantly outperforms existing methods for generating complex instructions, and outperformed existing methods. |
SQUiD: Synthesizing Relational Databases from Unstructured Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Relational databases are central to modern data management, but most data exists in unstructured forms like text documents. |
| Approach: | They propose a framework that decomposes the task into four stages, each with specialized techniques. |
| Outcome: | The proposed framework outperforms baselines across diverse datasets. |
RAG+: Enhancing Retrieval-Augmented Generation with Application-Aware Reasoning (2025.emnlp-main)
Copied to clipboard
Yu Wang, Shiwan Zhao, Zhihu Wang, Ming Fan, Xicheng Zhang, Yubo Zhang, Zhengfan Wang, Heyuan Huang, Ting Liu
| Challenge: | Existing RAG paradigms often overlook the cognitive step of applying knowledge, leaving a gap between retrieved facts and task-specific reasoning. |
| Approach: | They introduce a module extension that integrates application-aware reasoning into the RAG pipeline. |
| Outcome: | Experiments show that RAG+ outperforms standard RAG variants and achieves gains of 3–5% in complex scenarios. |
Rapid Word Learning Through Meta In-Context Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for few-shot word learning are underexplored. |
| Approach: | They propose a method that trains language models to generate new examples of a word’s usage given a few in-context examples, using a special placeholder token to represent the new word. |
| Outcome: | The proposed method trains language models to generate new examples of a word’s usage given a few in-context examples, using a special placeholder token to represent the new word. |
EuroGEST: Investigating gender stereotypes in multilingual language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models encode social biases, but most benchmarks for gender bias remain English-centric. |
| Approach: | They propose a dataset to measure gender-stereotypical reasoning in large language models across English and 29 European languages. |
| Outcome: | The proposed method is highly accurate across languages and strong in translations and gender labels. |
How Persuasive Is Your Context? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Empirically, through aseries of experiments, we show that TPS captures a more nuanced notion of persuasiveness than previously proposed metrics. |
| Approach: | They introduce a targeted persuasion score to quantify how persuasive a given context is to an LM. |
| Outcome: | Empirically, the proposed model captures a more nuanced notion of persuasiveness than previously proposed metrics. |
The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025.emnlp-main)
Copied to clipboard
| Challenge: | Embedding-based similarity metrics can be influenced by content dimensions and spurious attributes like the text’s source or language. |
| Approach: | They propose a debiasing algorithm that removes observed confounders from encoder representations and removes them from the encoder. |
| Outcome: | The proposed method improves on out-of-distribution benchmarks and on benchmarks, but performance is not affected. |
Measuring scalar constructs in social science with LLMs (2025.emnlp-main)
Copied to clipboard
Hauke Licht, Rupak Sarkar, Patrick Y. Wu, Pranav Goel, Niklas Stoehr, Elliott Ash, Alexander Miserlis Hoyle
| Challenge: | Valid scalar measurement of skalar constructs is a fundamental task in text analysis. |
| Approach: | They evaluate four approaches to measuring scalar constructs using large language models . pairwise comparisons produced better measurements than prompting LLMs, they say . validation of skalar measurement enables wide range of substantive applications in social science research . |
| Outcome: | The proposed methods improve on pairwise comparisons and finetuning . the proposed methods can be used in social science research . |
Text Detoxification: Data Efficiency, Semantic Preservation and Model Generalization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detoxification of text often rely on manually annotated data . xiangli: "detoxification of texts is a powerful way to remove toxic content" |
| Approach: | They propose a reinforcement learning framework that optimizes detoxification and semantic preservation without annotating large amounts of data. |
| Outcome: | The proposed method overcomes major limitations and surpasses humanannotated references across multiple benchmarks. |
Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss (2025.emnlp-main)
Copied to clipboard
Kiana Aghakasiri, Noopur Zambare, JoAnn Thai, Carrie Ye, Mayur Mehta, J Ross Mitchell, Mohamed Abdalla
| Challenge: | De-identification is an application of NLP where automated algorithms remove identifying information of patients and providers. |
| Approach: | They propose to use generative large language models to de-identify patients and providers . they propose to validate existing metrics to quantify extent of inappropriate removal . |
| Outcome: | The proposed method is based on a survey of LLM-based de-identification research . it shows that the models perform poorly in identifying clinically relevant changes . |
Reasoning under Uncertainty: Efficient LLM Inference via Unsupervised Confidence Dilution and Convergent Adaptive Sampling (2025.emnlp-main)
Copied to clipboard
Zhenning Shi, Yijia Zhu, Yi Xie, Junhan Shi, Guorui Xie, Haotian Zhang, Yong Jiang, Congcong Miao, Qing Li
| Challenge: | Large language models suffer from overconfidence and computational inefficiency due to fixed computation budgets and miscalibrated confidence estimates. |
| Approach: | They propose a framework for computationally efficient, trustworthy reasoning under uncertainty using Diversity-Aware Self-Signal Dilution and Convergent Adaptive Weighted Sampling techniques. |
| Outcome: | The proposed framework reduces inference cost by 70% while maintaining accuracy levels while reducing inference costs. |
Africa Health Check: Probing Cultural Bias in Medical LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in global healthcare . yet their outputs reflect Western-centric training data and omit indigenous medical systems . |
| Approach: | They evaluate cultural bias in instruction-tuned medical LLMs using a curated dataset of African traditional herbal medicine. |
| Outcome: | The findings show that cultural biases remain embedded in model training . the findings highlight the need for culturally informed evaluation strategies . |
Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Terms (2025.emnlp-main)
Copied to clipboard
| Challenge: | ailsntua researchers examine whether machine translation systems exhibit gender biases that reinforce societal stereotypes. |
| Approach: | They propose a probability-based metric to evaluate gender bias by analyzing aggregated model responses. |
| Outcome: | The proposed metric evaluates whether translations in Greek and French align with or diverge from societal stereotypes. |
REVIVING YOUR MNEME: Predicting The Side Effects of LLM Unlearning and Fine-Tuning via Sparse Model Diffing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods assess performance after LLMs are fine-tuned or unlearned to adapt to new tasks or eliminate undesirable behaviors. |
| Approach: | They propose a framework for identifying unintended side effects using sparse model diffing. |
| Outcome: | The proposed framework can detect unintended side effects without fine-tuning data . it achieves 95% accuracy in predicting side effects, aligning with known benchmarks . |
ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Theory of Mind (ToM) benchmarks focus on text-only or dyadic interactions, but to address this gap, we propose ToM-SSI: a new benchmark specifically designed to test ToM capabilities in environments rich with social interactions and spatial dynamics. |
| Approach: | They propose to use the Sally-Anne test to test ToM capabilities in environments rich in social interactions and spatial dynamics. |
| Outcome: | The proposed model captures a wider range of social cognition than existing models and demonstrates that existing models are still limited in these new tasks. |
Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used in the creation of online content, creating feedback loops as future generations of models will be trained on this synthetic data. |
| Approach: | They propose to use large language models to create feedback loops as future models are trained on this data. |
| Outcome: | The proposed model collapse effects are found to be detrimental to the results of recursive training on human datasets. |
Detecting LLM Hallucination Through Layer-wise Information Deficiency: Analysis of Ambiguous Prompts and Unanswerable Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate confident yet inaccurate responses, introducing significant risks for deployment in safety-critical domains. |
| Approach: | They propose a method to detect model hallucination by systematic analysis of information flow across model layers. |
| Outcome: | The proposed approach improves model reliability by immediately integrating with universal LLMs without additional training or architectural modifications. |
Extending Automatic Machine Translation Evaluation to Book-Length Documents (2025.emnlp-main)
Copied to clipboard
Kuang-Da Wang, Shuoyang Ding, Chao-Han Huck Yang, Ping-Chun Hsieh, Wen-Chih Peng, Vitaly Lavrukhin, Boris Ginsburg
| Challenge: | Large Language Models (LLMs) have superior translation performance and long-context capabilities, but evaluation methodologies remain constrained to sentence-level assessment due to dataset limitations and token number restrictions in metrics. |
| Approach: | They propose an evaluation scheme that extends existing automatic metrics to long-document translation by treating documents as continuous text and applying sentence segmentation and alignment methods. |
| Outcome: | The proposed evaluation scheme outperforms existing long-form document evaluation schemes while accounting for under-/over-translations and varied sentence boundaries. |
MedFact: A Large-scale Chinese Dataset for Evidence-based Medical Fact-checking of LLM Responses (2025.emnlp-main)
Copied to clipboard
Tong Chen, Zimu Wang, Yiyi Miao, Haoran Luo, Sun Yuanfei, Wei Wang, Zhengyong Jiang, Procheta Sen, Jionglong Su
| Challenge: | Existing medical fact-checking datasets focus on human-generated content, leaving the verification of content generated by large language models (LLMs) relatively unexplored. |
| Approach: | They propose to use Chinese medical fact-checking datasets to verify LLM-generated medical content by combining in-context learning and fine-tuning. |
| Outcome: | The first evidence-based Chinese medical fact-checking dataset of LLM-generated medical content consists of 1,321 questions and 7,409 claims . |
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video-language models excel at understanding video content but struggle with spatial relationships, temporal ordering, and cross-frame continuity. |
| Approach: | They propose a framework that trains video-LLMs to distinguish accurate representations from carefully crafted adversarial examples. |
| Outcome: | Experiments show that VideoPASTA improves performance without human annotation or captioning . the framework can be used on various state-of-the-art video-LLMs with no human annotation . |
Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions (2025.emnlp-main)
Copied to clipboard
Seyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, Manas Gaur
| Challenge: | Exact label definitions are considered as clues to disambiguate unclear labels, helping models perform their tasks more effectively. |
| Approach: | They conducted controlled experiments on multiple explanation benchmark datasets and label definition conditions using expert-curated, LLM-generated, perturbed, and swapped definitions. |
| Outcome: | The results suggest that models often default to internal representations, particularly in general tasks, while domain-specific tasks benefit more from explicit definitions. |
Group-Aware Reinforcement Learning for Output Diversity in Large Language Models (2025.emnlp-main)
Copied to clipboard
Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, Gerard Medioni
| Challenge: | Large Language Models suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist. |
| Approach: | They propose a group-aware policy optimization extension of GRPO that computes rewards over the group as a whole. |
| Outcome: | The proposed model improves on standard LLM benchmarks without compromising accuracy. |
Model-Based Ranking of Source Languages for Zero-Shot Cross-Lingual Transfer (2025.emnlp-main)
Copied to clipboard
| Challenge: | NN-Rank is an algorithm for ranking source languages for cross-lingual transfer . it leverages hidden representations from multilingual models and unlabeled target-language data . |
| Approach: | They propose an algorithm for ranking source languages for cross-lingual transfer which leverages hidden representations from multilingual models and unlabeled target-language data. |
| Outcome: | The proposed algorithm outperforms state-of-the-art models on in-domain data and shows that it can achieve 92.8% of the NDCG achieved using all available target data. |
PruneCD: Contrasting Pruned Self Model to Improve Decoding Factuality (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to decode early exit logits in large language models are lacking in factuality and accuracy. |
| Approach: | They propose a contrastive decoding method that constructs the amateur model via layer pruning rather than early exit. |
| Outcome: | The proposed method improves factuality with minimal inference overhead and is robust and practical. |
Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialogues (2025.emnlp-main)
Copied to clipboard
Jinfeng Zhou, Yuxuan Chen, Jianing Yin, Yongkang Huang, Yihan Shi, Xikun Zhang, Libiao Peng, Rongsheng Zhang, Tangjie Lv, Zhipeng Hu, Hongning Wang, Minlie Huang
| Challenge: | Existing approaches to cognitive restructuring (CR) are limited by entrenched cognitive distortions, emotional resistance, and individual differences. |
| Approach: | They propose a framework that structures CR as theory-grounded multi-stage multi-turn dialogue and a multi-channel loop mechanism to account for diverse individual distortions. |
| Outcome: | The proposed framework integrates supportive strategies for emotional management and a multi-channel loop mechanism to account for diverse individual distortions. |
AccessEval: Benchmarking Disability Bias in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models exhibit disparities in how they handle real life queries. |
| Approach: | They propose a large-scale benchmark to evaluate large language models across six real-world domains and nine disability types. |
| Outcome: | The proposed model outputs show higher factual error, more negative tone, and increased stereotyping with social perception compared to neutral queries. |
The Impact of Language Mixing on Bilingual LLM Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show multilingual speakers intentionally switch languages during reasoning . enforcing monolingual decoding reduces accuracy by 5.6 percentage points . |
| Approach: | They find that multilingual speakers intentionally switch languages during reasoning . enforcing monolingual decoding reduces accuracy by 5.6 percentage points . authors suggest that language mixing is not merely a byproduct of multilingual training . |
| Outcome: | The proposed model can be used to predict whether a language switch would benefit or harm reasoning. |
VISaGE: Understanding Visual Generics and Exceptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | atypical evaluation instances disrupt incontext instance understanding and in-weight conceptual knowledge. |
| Approach: | They propose to use a dataset to analyze atypical visual and textual images to test their models. |
| Outcome: | The proposed model is based on a dataset consisting of typical and exceptional images. |
Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing long-context language models (LMs) can handle tens of thousands of tokens in a single context window. |
| Approach: | They compare two recent multi-stage pipelines, ReadAgent and RAPTOR, against three baselines. |
| Outcome: | The proposed pipelines outperform more complex methods on multiple long-context QA benchmarks. |
Discursive Circuits: How Do Language Models Understand Discourse Relations? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Discourse structure is essential for ensuring language models behave safely and ethically. |
| Approach: | They propose a task where a model completes a discourse given a specified relation . they propose CUDR task that enables activation patching to make circuit discovery feasible . |
| Outcome: | The proposed model recovers discourse understanding in the English PDTB-based CuDR task. |
Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning (2025.emnlp-main)
Copied to clipboard
Chan Young Park, Jillian Fisher, Marius Memmel, Dipika Khullar, Seoho Yun, Abhishek Gupta, Yejin Choi
| Challenge: | Large language models (LLMs) are human-centric, but omit low-level, spatially grounded details needed for robotic execution. |
| Approach: | They propose a lightweight framework for vision-language procedural planning that enables iteratively critique, revise and verify their own plans without external supervision or teacher models. |
| Outcome: | a new framework outperforms weaker models 100X the size in vision-language procedural planning . the framework generates higher-quality, execution-ready plans that can be used at inference and fine-tuning . |
ThinkSLM: Towards Reasoning in Small Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reasoning has long been viewed as an emergent property of large language models (LLMs), appearing at or above a certain scale (100B parameters). |
| Approach: | They propose a benchmark to evaluate the reasoning abilities of small language models (SLMs) using quantization, pruning, and distillation. |
| Outcome: | The proposed benchmark systematically evaluates reasoning abilities of small language models trained from scratch or derived from LLMs using quantization, pruning, and distillation. |
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Excessive refinement can cause over-correction and reduce performance, authors say . they say MAgICoRe is a framework for multi-agent iteration for coarse-to-fine refinement . |
| Approach: | They propose a framework for multi-agent iteration for coarse-to-fine refinement that reduces excessive refinement by categorizing problems as easy or hard. |
| Outcome: | The proposed framework beats Self-Consistency by 3.4%, Best-of-k by 3.2%, and Self-Refine by 4.0% on Llama-3-8B and GPT- 3.5. |
Batched Self-Consistency Improves LLM Relevance Assessment and Ranking (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has focused on a one-by-one pointwise (PW) scoring strategy where each LLM call judges one passage at a time against the query. |
| Approach: | They propose to use batched PW methods to evaluate multiple passages per LLM call to improve efficiency and judgment quality by enabling content from multiple passage to be seen jointly. |
| Outcome: | The proposed methods improve efficiency and judgment quality by enabling content from multiple passages to be seen jointly. |
SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts (2025.emnlp-main)
Copied to clipboard
| Challenge: | SemCSE is an unsupervised method for learning semantic embeddings of scientific texts . |
| Approach: | They propose an unsupervised method for learning semantic embeddings of scientific texts that leverages LLM-generated scientific summaries to train a model that positions semantically related summary closer together in the embeddable space. |
| Outcome: | The proposed method achieves state-of-the-art performance on the SciRepEval benchmark for scientific text embeddings, highlighting the benefits of a semantically focused training approach. |
Controlled Generation for Private Synthetic Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text anonymization is essential for developing and deploying AI in high stakes domains . tools for redacting directly identifying content are unlikely to guarantee 100% recall . |
| Approach: | They propose a method for privacy-preserving synthetic text generation that leverages HIPS theory and de-identification principles. |
| Outcome: | The proposed method achieves a strong balance between privacy protection and utility on legal and clinical datasets. |
Towards AI-Assisted Psychotherapy: Emotion-Guided Generative Interventions (2025.emnlp-main)
Copied to clipboard
Kilichbek Haydarov, Youssef Mohamed, Emilio Goldenhersch, Paul OCallaghan, Li-jia Li, Mohamed Elhoseiny
| Challenge: | Large language models (LLMs) lack rich non-verbal emotional cues essential to real-world therapy. |
| Approach: | They propose a multimodal dataset of 1,441 publicly sourced therapy session videos containing both dialogue and non-verbal signals such as facial expressions and vocal tone. |
| Outcome: | The proposed model improves the quality of generated interventions and evaluators misalign with expert assessments in this domain, highlighting the need for human-centered evaluation. |
From Shortcuts to Balance: Attribution Analysis of Speech-Text Feature Utilization in Distinguishing Original from Machine-Translated Texts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work demonstrated strong performance in distinguishing machine-translated text from human-authored or human-transcribed content using pretrained language models. |
| Approach: | They find that bimodal integration reduces reliance on NEs while moderating overemphasis attribution patterns in speech features. |
| Outcome: | The proposed models show that they are more balanced while relying less on NEs. |
DEBATE, TRAIN, EVOLVE: Self‐Evolution of Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have improved significantly in reasoning through extensive training on massive datasets. |
| Approach: | They propose a ground truth-free training framework that uses multi-agent debate traces to evolve a single language model. |
| Outcome: | The proposed framework achieves 8.92% accuracy gain on the GSM-PLUS dataset. |
From Chat Logs to Collective Insights: Aggregative Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to analyzing large-scale conversation logs treat interactions as independent, missing critical insights. |
| Approach: | They propose a task that requires models to reason explicitly over thousands of user-chatbot interactions to answer aggregational queries. |
| Outcome: | The proposed task requires models to reason over thousands of user-chatbot interactions to answer aggregational queries such as identifying emerging concerns among demographics. |
A Text-Based Recommender System that Leverages Explicit Affective State Preferences (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that leverage user preferences that are implicit in user-item rating histories can be slow to track changes in user preferences and imprecise for users with diverse preferences. |
| Approach: | They propose a novel recommendation task that leverages a wide range of affective states sought explicitly by the user to identify items that induce those affective state. |
| Outcome: | The proposed model can leverage a wide range of affective states sought explicitly by the user to identify items likely to induce those affective state. |
CARE: Multilingual Human Preference Learning for Cultural Awareness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language Models are tuned with human preferences to produce helpful responses, but the impact of preference tuning on the ability to handle culturally diverse queries remains understudied. |
| Approach: | They introduce a multilingual resource that contains culturally specific questions and 31.7k responses with human judgments. |
| Outcome: | The proposed model outperforms models with stronger initial cultural performance . the proposed model has gaps in the literature on culturally relevant data . |
Multilingual Dialogue Generation and Localization with Dialogue Act Scripting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to training or evaluating non-English dialogue datasets often introduce artifacts that reduce their naturalness and cultural appropriateness. |
| Approach: | They propose a structured framework for encoding, localizing, and generating multilingual dialogues from abstract intent representations. |
| Outcome: | The proposed framework outperforms translation models in Italian, German, and Chinese on cultural relevance, coherence, and situational appropriateness. |
SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to estimate uncertainty use predictive confidence, structural characteristics of representation space, or stochastic variation in model outputs. |
| Approach: | They propose a new uncertainty estimation framework based on sparse dictionary learning by identifying dictionary atoms associated with misclassified samples. |
| Outcome: | The proposed framework outperforms or matches existing methods on several NLU benchmarks and sentiment analysis benchmarks. |
Planning-Aware Code Infilling via Horizon-Length Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to fill-in-the-middle (FIM) often fail to generate content that aligns well with the surrounding context. |
| Approach: | They propose a training objective that teaches models to predict the number of remaining middle tokens at each step. |
| Outcome: | The proposed training objective improves FIM performance by up to 24% on diverse benchmarks across file-level and repository-level. |
SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala (2025.emnlp-main)
Copied to clipboard
Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Large Language Models (LLMs) have been evaluated mostly on global or anglocentric subjects, often neglecting low-resource languages and culturally specific content. |
| Approach: | They evaluate 26 Large Language Models using a multiple-choice question answering benchmark for Sinhala. |
| Outcome: | The new benchmarks show that Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies, but overall performance remains limited. |
OG-RAG: Ontology-grounded retrieval-augmented generation for large language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing retrieval-augmented models fail to account for structured domain knowledge, leading to suboptimal context generation. |
| Approach: | They propose an Ontology-Grounded Retrieval Augmented Generation method to enhance LLM-generated responses by anchoring retrieval processes in domain-specific ontologies. |
| Outcome: | The proposed method increases recall of accurate facts by 55% and improves response correctness by 40% across four different LLMs. |
Convergence and Divergence of Language Models under Different Random Seeds (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large body of work has examined the training dynamics of language models. |
| Approach: | They investigate the convergence of language models (LMs) trained under different random seeds . they find that larger models reconverge faster in later training stages, while smaller models never actually reconverge. |
| Outcome: | The proposed model size and training checkpoints influence convergence of language models under different seeds. |
Analyzing and Modeling LLM Response Lengths with Extreme Value Theory: Anchoring Effects and Hybrid Distributions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches treat length as an incidental output property rather than a statistically regular phenomenon worthy of rigorous modeling. |
| Approach: | They propose a statistical framework for modeling and controlling large language model response lengths using extreme value theory and cross-validation on Qwen and DeepSeek architectures. |
| Outcome: | The proposed model improves tail fit and generalizability while maintaining generalizzability. |
Language Models Identify Ambiguities and Exploit Loopholes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that exploit loopholes identify and reason about ambiguity and conflicting goals, presenting a potential safety risk. |
| Approach: | They propose to study the responses of large language models to loopholes by examining ambiguity and pragmatics in LLMs. |
| Outcome: | The proposed models can identify ambiguities and exploit loopholes to satisfy their given goals as opposed to the goals of the user. |
Benchmarking LLMs for Translating Classical Chinese Poetry: Evaluating Adequacy, Fluency, and Elegance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models fall short of translating culturally significant content . existing models fall behind in achieving such translations, authors say . |
| Approach: | They propose a suitable benchmark for translating classical Chinese poetry into English . they propose RAT, a retrieval-augmented machine translation method that enhances the translation process . |
| Outcome: | The proposed method improves translation quality in terms of adequate, fluent, and elegant translations. |
AraEval: An Arabic Multi-Task Evaluation Suite for Large Language Models (2025.emnlp-main)
Copied to clipboard
Alhanoof Althnian, Norah A. Alzahrani, Shaykhah Z. Alsubaie, Eman Albilali, Ahmed Abdelali, Nouf M. Alotaibi, M Saiful Bari, Yazeed Alnumay, Abdulhamed Alothaimen, Maryam Saif, Shahad D. Alzaidi, Faisal Abdulrahman Mirza, Yousef Almushayqih, Mohammed Al Saleem, Ghadah Alabduljabbar, Abdulmohsen Al-Thubaity, Areeb Alowisheq, Nora Al-Twairesh
| Challenge: | AraEval is a suite of evaluation tasks designed to assess the advanced knowledge, reasoning, truthfulness, and instruction following capabilities of large language models. |
| Approach: | They propose to use AraEval to assess the advanced knowledge, reasoning, truthfulness, and instruction following capabilities of large language models in the Arabic context. |
| Outcome: | The evaluation suite covers a broad spectrum of domains, including science, history, religion, and literature. |
QUIDS: Query Intent Description for Exploratory Search via Dual Space Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using QUIDS, we generate user-facing query intent descriptions that surface what the search engine likely inferred the query to mean based on post-retrieval evidence. |
| Approach: | They propose a method that leverages dual-space contrastive learning to isolate intent-relevant information while suppressing irrelevant content. |
| Outcome: | The proposed method outperforms state-of-the-art methods across ROUGE, BERTScore, and human/LLM evaluations. |
A Systematic Survey of Automatic Prompt Optimization Techniques (2025.emnlp-main)
Copied to clipboard
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, Haozhu Wang, Han Ding, Yuzhe Lu, Zhichao Xu, Yun Zhou, Balasubramaniam Srinivasan, Qiaojing Yan, Yueyan Chen, Haibo Ding, Panpan Xu, Lin Lee Cheong
| Challenge: | Recent advances in prompt engineering have created impediments for end users to adopt . however, prompt engineering remains an impedance due to rapid advances in models, tasks, and associated best practices. |
| Approach: | They propose to define APO as a 5-part unifying framework and categorize all relevant works based on their salient features. |
| Outcome: | The proposed framework aims to improve the performance of large language models on various tasks. |
Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown the power of chain-of-thought reasoning in improving complex decision-making tasks. |
| Approach: | They propose a pipeline that generates chain-of-thought (CoT) explanations from CoTs with improved accuracy. |
| Outcome: | The proposed pipeline outperforms a direct generation method and baselines on three datasets. |
MemInsight: Autonomous Memory Augmentation for LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM) agents have evolved to intelligently process information, make decisions, and interact with users or tools. |
| Approach: | They propose an autonomous memory augmentation approach to enhance semantic data representation and retrieval mechanisms by leveraging historical interactions. |
| Outcome: | The proposed approach outperforms a baseline RAG by 34% in recall for LoCoMo retrieval on three task scenarios and boosts persuasiveness of recommendations by 14%. |
Breaking the Noise Barrier: LLM-Guided Semantic Filtering and Enhancement for Multi-Modal Entity Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-modal entity alignment (MMEA) aims to identify equivalent entities between two multimodal knowledge graphs. |
| Approach: | They propose a novel LLMguided MMEA framework that prioritizes noise reduction before fusion. |
| Outcome: | The proposed framework prioritizes noise reduction before fusion and improves semantics on the noisy FB YG dataset. |
ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval systems rely on surface-level cues such as keyword overlap and semantic similarity to evaluate retrieval beyond these shallow signals. |
| Approach: | They propose a benchmark that shifts the reasoning challenge to query-side processing techniques that can help resolve complexity. |
| Outcome: | The proposed benchmarks show that document-side reasoning remains a challenge. |
No Need for Explanations: LLMs can implicitly learn from mistakes in-context (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature assumes that correct answers to large language models must be accompanied by comprehensive rationales to be helpful. |
| Approach: | They propose to show incorrect answers to Large Language Models (LLMs) as a popular strategy to improve their performance in reasoning-intensive tasks. |
| Outcome: | The proposed approach outperforms chain-of-thought prompting in math reasoning tasks. |
MoVa: Towards Generalizable Classification of Human Morals and Values (2025.emnlp-main)
Copied to clipboard
Ziyu Chen, Junfei Sun, Chenxi Li, Tuan Dung Nguyen, Jing Yao, Xiaoyuan Yi, Xing Xie, Chenhao Tan, Lexing Xie
| Challenge: | Identifying human morals and values embedded in language is essential to empirical studies of communication. |
| Approach: | They propose a framework for generalizable classification of human morals and values . they recommend a classification strategy that scores all related concepts simultaneously . |
| Outcome: | The proposed method outperforms fine-tuned models across domains and frameworks. |
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work of GUI action grounding fine-tunes data from pre-trained MLLMs, but data is limited to specific GUI environments. |
| Approach: | They propose to use a GUI-based agent to collect environment-specific data and fine-tune GUI grounding models with the collected data. |
| Outcome: | The proposed model can be extended to other GUI environments to improve performance. |
Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models (LLMs) fail to detect character knowledge errors, leading to low-quality automatic corpus construction. |
| Approach: | They propose to use a large language model to detect known knowledge errors and an agent-based reasoning method to improve error detection. |
| Outcome: | The proposed method improves the ability of LLMs to detect errors in known knowledge errors and unknown knowledge errors while playing roles. |
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations. |
| Approach: | They propose a large-scale multi-turn multimodal dialogue dataset that uses rules and GPT assistance to generate a multi-turned multimodal dialog dataset. |
| Outcome: | The proposed dataset is a strong benchmark for multi-turn multimodal dialogue learning . it features complex dialogues with contextual dependencies that force models to track, ground, and recall information across multiple turns and disparate visual regions. |
Graph-Based Multi-Trait Essay Scoring (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on Automated Essay Scoring (AES) models essay as word sequence, but new approach uses graph-attention network approach to model essay traits. |
| Approach: | They propose a graph-attention network approach to automate essay scoring that models interactions among essay traits as a graphical graph. |
| Outcome: | The proposed approach outperforms competing approaches on the ASAP++ dataset . it allows for multiple-task scoring, allowing for more detailed feedback on essays . |
Benchmarking LLMs on Semantic Overlap Summarization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are the most capable text generation models in a variety of tasks and fields. |
| Approach: | They benchmark Large Language Models (LLMs) on SOS and introduce PrivacyPolicyPairs (3P) a dataset of 135 high-quality privacy policy documents is used to evaluate the model. |
| Outcome: | The proposed dataset complements existing resources and broadens domain coverage. |
N-CORE: N-View Consistency Regularization for Disentangled Representation Learning in Nonverbal Vocalizations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Nonverbal vocalizations are an essential component of human communication, conveying rich information without linguistic content. |
| Approach: | They propose a backbone-agnostic framework to disentangle emotion and speaker information from nonverbal vocalizations by leveraging N views of audio samples to learn invariance to specific transformations. |
| Outcome: | The proposed framework achieves competitive performance compared to state-of-the-art methods on the VIVAE, ReCANVo, and ReCANVO-Balanced datasets. |
Probability Distribution Collapse: A Critical Bottleneck to Compact Unsupervised Neural Grammar Induction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models face expressiveness bottlenecks, resulting in unnecessarily large yet underperforming grammars. |
| Approach: | They propose a method to reduce the expressiveness bottleneck of unsupervised neural grammar induction by leveraging neural parameterization to estimate prob-ability distributions. |
| Outcome: | The proposed approach significantly improves parsing performance while enabling the use of significantly more compact grammars across a wide range of languages. |
Spatial Layouts in News Homepages Capture Human Preferences (2025.emnlp-main)
Copied to clipboard
| Challenge: | Homepage layouts are daily and manually curated by expert human news editors, and serve as a proxy for information prioritization. |
| Approach: | They propose a pairwise preference model to capture ranked preferences between news items . they use a dataset of 3,000 news homepages captured twice daily over a five-year period . |
| Outcome: | The proposed model infers editorial preference with over 0.7 F1 score (based on human trials) it also ranks local city council policies passed over ten years in San Francisco, evaluating their "newsworthiness". |
KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Understanding and reasoning over text within visual contexts poses a significant challenge for Vision-Language Models. |
| Approach: | They propose a benchmark for Korean Reading and rEasoning in Text-rich VQA Attuned to diverse visual contexts to address this challenge. |
| Outcome: | The proposed benchmark is tailored for Korean reading and rEasoning in text-rich VQA attuned to diverse visual contexts. |
ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in LLMs have significantly enhanced their reasoning capabilities, enabling LLM-based agents to perform complex multi-step decision making beyond static problem solving. |
| Approach: | They propose a novel reasoning backbone that shifts reasoning from merely planning next actions to continuously reflecting on the agent’s state relative to its goal. |
| Outcome: | The proposed model outperforms ReAct by 27.7% on average, achieving a 93.3% success rate in ALFWorld. |
CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward (2025.emnlp-main)
Copied to clipboard
Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F. Wong, Songyang Zhang, Kai Chen
| Challenge: | Existing approaches lack robustness to handle complex edge cases and generalizability across different domains. |
| Approach: | They develop an accurate and lightweight verifier model for evaluation and outcome reward that matches unstructured outputs against standard answers. |
| Outcome: | The proposed model can process multiple answer types including multi-subproblems, formulas, and sequence answers while identifying abnormal/invalid responses. |
A Knowledge-driven Adaptive Collaboration of LLMs for Enhancing Medical Decision-making (2025.emnlp-main)
Copied to clipboard
| Challenge: | Medical decision-making often involves integrating knowledge from multiple clinical specialties. static, pre-assigned roles hinder adaptability and dynamic knowledge integration. |
| Approach: | They propose a Knowledge-driven Adaptive Multi-Agent Collaboration framework that emulates large language models to emulate expert teamwork. |
| Outcome: | The proposed framework outperforms single-agent and advanced multi-agend methods on two real-world medical scenarios. |
Castle: Causal Cascade Updates in Relational Databases with Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches focus primarily on SELECT queries, neglecting the challenges of SQL update operations and their ripple effects. |
| Approach: | They introduce the first framework for schema-only cascade update generation using large language models. |
| Outcome: | The framework can produce accurate SQL updates without revealing table content . it can also determine which columns must be updated and how updates propagate through the schema, causing cascading updates . |
Idiosyncratic Versus Normative Modeling of Atypical Speech Recognition: Dysarthric Case Studies (2025.emnlp-main)
Copied to clipboard
| Challenge: | Past studies have focused on fully personalized (or idiosyncratic) models for atypical speech . past studies focused on idiotic models, but current approaches focus on generalizing and handling idiomatic patterns . |
| Approach: | They compare four models that generalize and handle idiosyncrasy to find atypical speech . they find the dysarthric-idios-ync model performs better than the idioconic approach . |
| Outcome: | The proposed model generalizes and handles idiosyncrasy better than the idiocy model . the model requires less personalized data and reduces word error rate from 71% to 32% . |
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls (2025.emnlp-main)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, Xin Wang, Luis A. Lastras, Pavan Kapanipathi
| Challenge: | Existing benchmarks and datasets for tool calling have lagged behind . nested sequencing is a common problem in LLMs, but it is not enough to evaluate them. |
| Approach: | They propose a benchmark to evaluate LLMs on nested sequences of API calls, i.e. sequences where the output of one API call is passed as input to a subsequent call. |
| Outcome: | The proposed model achieves a full sequence match accuracy of 28% and a win-rate of 60% on nested sequences of API calls. |
Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has explored unimodal biases in visual question answering, but the problem of selection bias in Multiple-Choice Question Answering (MCQA) remains underexplored. |
| Approach: | They propose a method that mitigates bias without retraining and is compatible with frozen LVLMs. |
| Outcome: | The proposed method mitigates bias without retraining and is compatible with frozen LVLMs. |
Can Large Language Models Unlock Novel Scientific Research Ideas? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and ChatGPT have marked a turning point in the integration of Artificial Intelligence (AI) into people’s everyday lives. |
| Approach: | They conduct a human evaluation of the novelty, relevancy, and feasibility of the generated future research ideas. |
| Outcome: | The proposed models generate more diverse ideas than GPT-4, GPT-3.5, and Gemini 1.0. |
Word Salad Chopper: Reasoning Models Waste A Ton Of Decoding Budget On Useless Repetitions, Self-Knowingly (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) are often bottlenecked by the high cost of output tokens. |
| Approach: | They propose a lightweight, turnkey component for Large Reasoning Models that is minimally invasive to its reasoning trajectory. |
| Outcome: | The proposed component is lightweight and low overhead, and lacks semantic value. |
DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for cultural awareness and alignment are lacking . Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives. |
| Approach: | They propose a new CSI dataset for Indian culture that measures cultural competence . they use a CSI created by LLM as Judge and human evaluations from diverse regions . |
| Outcome: | The proposed model shows that it is capable of generating culturally relevant adaptations across multiple cultural facets. |
SYNC: A Synthetic Long-Context Understanding Benchmark for Controlled Comparisons of Model Capabilities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing synthetic tasks target narrow skill sets, limiting their ability to comprehensively assess model capabilities. |
| Approach: | They propose a new evaluation suite of synthetic tasks spanning domains including graph understanding and translation that test a wide range of capabilities. |
| Outcome: | The evaluation suite of synthetic tasks spanning domains including graph understanding and translation shows that the tasks perform significantly better on more challenging tasks. |
OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets are not consistently formatted and use a variety of chunk encodings (IOB, BIO, etc.), often without documentation. |
| Approach: | They present OpenNER 1.0, a standardized collection of openly-available named entity recognition (NER) datasets. |
| Outcome: | The proposed datasets correct annotation format issues and provide a structure that enables research in multilingual and multi-ontology NER. |
Mondrian: A Framework for Logical Abstract (Re)Structuring (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using the ABT structure, academic abstracts are structured to provide clear and concise prose, but a lack of clarity and logical coherence is a challenge for authors struggling with English proficiency or academic writing conventions. |
| Approach: | They propose a framework that identifies the key components of an abstract and reorients itself to properly reflect the ABT logical progression. |
| Outcome: | The proposed framework improves comprehensibility of academic writing, particularly for non-native English speakers, and is based on a human evaluation and automated metrics. |
Case-Based Decision-Theoretic Decoding with Quality Memories (2025.emnlp-main)
Copied to clipboard
| Challenge: | Minimum Bayes risk (MBR) decoding is a decision rule of text generation . however, it depends on sample texts drawn from the text generation model . |
| Approach: | They propose a case-based decision-theoretic method to estimate the expected utility using examples of domain data. |
| Outcome: | The proposed method outperforms MAP decoding in translation tasks and image captioning tasks on MSCOCO and nocaps datasets. |
PRIME: Large Language Model Personalization with Cognitive Dual-Memory and Personalized Thought Process (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language model (LLM) personalization aims to align outputs with individuals’ unique preferences and opinions. |
| Approach: | They integrate a cognitive dual-memory model into LLM personalization by mirroring episodic memory to historical user engagements and semantic memory to long-term, evolving user beliefs. |
| Outcome: | The proposed framework integrates the well-established cognitive dual-memory model into LLM personalization, using episodic and semanticmemories. |
Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not evaluated whether probing accuracy predicts syntactic outcomes. |
| Approach: | They evaluate 32 open-weight transformer models and find that probing fails to predict outcomes of targeted syntax evaluations across English linguistic phenomena. |
| Outcome: | The proposed model does not predict syntactic outcomes on English linguistic phenomena. |
Image Difference Captioning via Adversarial Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing supervised approaches to image difference captioning overfit to dataset-specific language patterns and fail to capture accurate preferences. |
| Approach: | They propose an adversarial direct preference optimization framework that aligns captioning policy with pairwise difference preferences via Direct Preference Optimization. |
| Outcome: | The proposed approach outperforms baselines on benchmark IDC datasets in generating fine-grained and accurate difference descriptions. |
seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | **seqBench** allows systematic variation of several key complexity dimensions. |
| Approach: | They introduce a parametrized benchmark for probing sequential reasoning limits in Large Language Models through precise, multi-dimensional control over several key complexity dimensions. |
| Outcome: | The framework allows systematic variation of logical depth, backtracking requirements and noise ratio on state-of-the-art LLMs. |
NormGenesis: Multicultural Dialogue Generation via Exemplar-Guided Social Norm Modeling and Violation Recovery (2025.emnlp-main)
Copied to clipboard
| Challenge: | Social norms govern culturally appropriate behavior in communication, enabling dialogue systems to produce coherent and socially acceptable outputs. |
| Approach: | They propose a framework for generating and annotating socially grounded dialogues in Chinese, English, and Korean. |
| Outcome: | The proposed framework outperforms existing frameworks in refinement quality, dialogue naturalness, and generalization performance. |
SATBench: Benchmarking LLMs’ Logical Reasoning via Automated Puzzle Generation from SAT Formulas (2025.emnlp-main)
Copied to clipboard
Anjiang Wei, Yuheng Wu, Yingjia Wan, Tarun Suresh, Huanmi Tan, Zhanke Zhou, Sanmi Koyejo, Ke Wang, Alex Aiken
| Challenge: | SATBench is a benchmark for evaluating the logical reasoning capabilities of large language models (LLMs) through logical puzzles derived from Boolean satisfiability (SAT) problems. |
| Approach: | They propose a benchmark to evaluate logical reasoning capabilities of large language models (LLMs) using logical puzzles derived from Boolean satisfiability problems. |
| Outcome: | The proposed model achieves 65.0% accuracy on hard UNSAT problems, close to the baseline of 50%. |
Data Descriptions from Large Language Models with Influence Estimation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing explainable AI approaches focus on interpreting how models make predictions. |
| Approach: | They propose a pipeline that generates textual descriptions using large language models . they propose 'cross-modal transfer classification' task to examine effectiveness of textual description . |
| Outcome: | The proposed method improves classification accuracy compared to baselines and sheds light on how the model prioritizes and utilizes information for decision-making. |
EquiBench: Benchmarking Large Language Models’ Reasoning about Program Semantics via Equivalence Checking (2025.emnlp-main)
Copied to clipboard
Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago S. F. X. Teixeira, Diyi Yang, Ke Wang, Alex Aiken
| Challenge: | EquiBench is a new benchmark to evaluate large language models' ability to reason about program semantics . Unlike natural language, code is executable. |
| Approach: | They propose a benchmark to evaluate large language models through equivalence checking . EquiBench consists of 2400 program pairs across four languages and six categories . |
| Outcome: | The proposed benchmark consists of 2400 program pairs across four languages and six categories. |
MicroEdit: Neuron-level Knowledge Disentanglement and Localization in Lifelong Model Editing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing large language models are designed for single or limited edits, lacking the capacity to support long-term, multi-round knowledge updates. |
| Approach: | They propose a neuron-level editing method that performs minimal interventions within large language models (LLMs) by leveraging a sparse autoencoder, MicroEdit disentangles knowledge representations and activates only a minimal set of necessary neurons for precise parameter updates. |
| Outcome: | Extensive experiments show that MicroEdit outperforms prior methods and robustly handles lifelong knowledge editing across QA and Hallucination settings on LLaM and Mistral. |
Do Large Language Models Understand Word Senses? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have set new performance standards in a wide range of tasks. |
| Approach: | They evaluate the Word Sense Disambiguation capabilities of instruction-tuned LLMs and their ability to understand word senses in three generative settings: definition generation, free-form explanation, and example generation. |
| Outcome: | The proposed models can explain the meaning of words in context with 98% accuracy, while demonstrating greater robustness across domains and levels of difficulty. |
Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Diverse language model responses are crucial for creative generation, open-ended tasks, and self-improvement training. |
| Approach: | They propose a length-controlled data selection strategy that improves diversity while maintaining length parity. |
| Outcome: | The proposed method improves diversity while maintaining length parity on LLaMA-3.1-8B and Olmo-2 family. |
Uncovering the Bigger Picture: Comprehensive Event Understanding Via Diverse News Retrieval (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing news retrieval systems prioritize textual relevance, leading to redundancy and limited viewpoint exposure. |
| Approach: | They propose a framework that explicitly models semantic variation at the sentence level. |
| Outcome: | The proposed framework outperforms strong baselines in terms of retrieval diversity without compromising relevance. |
Personalized LLM Decoding via Contrasting Personal Preference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalization of large language models (LLMs) is becoming increasingly important as they are increasingly deployed in real-world applications. |
| Approach: | They propose a decoding-time approach that leverages the user's implicit reward signal by performing parameter-efficient fine-tuning on user-specific data. |
| Outcome: | The proposed approach improves personalization by an average of 10.57% in ROUGE-L without external reward models or additional training procedures. |
The Missing Parts: Augmenting Fact Verification with Half Truth Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing fact verification systems assess whether a claim is true or false . but many real-world claims are half-truths due to omission of critical context . a new framework that detects omitted information can improve existing fact-checking pipelines . |
| Approach: | They propose a framework that detects omission-based misinformation by aligning evidence and inferring implied intent. |
| Outcome: | The proposed framework boosts Half-True classification F1 by up to 16 points . it can be integrated into existing fact-checking pipelines and improves performance across strong baselines. |
Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations (2025.emnlp-main)
Copied to clipboard
Yimin Xiao, Yongle Zhang, Dayeon Ki, Calvin Bao, Marianna J. Martindale, Charlotte Vaughn, Ge Gao, Marine Carpuat
| Challenge: | Using machine translation tools for everyday tasks is becoming more commonplace, but a lack of evaluation strategies and alternatives can cause users to over-rely on it. |
| Approach: | They propose to use MT evaluation techniques to promote MT quality and MT literacy among its users. |
| Outcome: | The findings highlight the need for evaluation and NLP explanation techniques to promote MT quality and MT literacy among its users. |
Personalization up to a Point: Why Personalized Content Moderation Needs Boundaries, and How We Can Enforce Them (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalized content moderation can protect users from harm while facilitating free expression . however, it can also allow highly harmful and even illegal hate speech to spread . |
| Approach: | They propose to enforce legal boundaries on personalized content moderation models to reduce legal violations while maintaining user welfare. |
| Outcome: | The proposed approach reduces legal violations while maintaining user welfare while maintaining a high degree of model performance. |
MPCG: Multi-Round Persona-Conditioned Generation for Modeling the Evolution of Misinformation with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Misinformation evolves as it spreads, shifting in language, framing, and moral emphasis to adapt to new audiences. |
| Approach: | They propose a multi-round, persona-conditioned framework that simulates how claims are iteratively reinterpreted by agents with distinct ideological perspectives. |
| Outcome: | The proposed framework generates persona-specific claims across multiple rounds . it is based on an uncensored large language model and is scalable to multiple tasks . |
LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evidence of human label variation in Natural Language Inference (NLI) however, within-label variation is an additional challenge. |
| Approach: | They propose a linguistically-informed taxonomy for categorizing free-text explanations in English that captures different reasoning strategies behind NLI explanations with a particular focus on within-label variation. |
| Outcome: | The proposed taxonomy can be used to classify explanations in English using a linguistically-informed taxonomies. |
LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Question Answering systems are limited by noisy documents and flawed QA pairs. |
| Approach: | They propose a high-quality subset of NarrativeQA focused on literary works . they identify and correct low-quality QA samples while removing extraneous text . |
| Outcome: | The proposed subset of NarrativeQA is based on literary works. |
FillerSpeech: Towards Human-Like Text-to-Speech Synthesis with Filler Insertion and Filler Style Control (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in speech synthesis have improved audio quality and pronunciation . fillers are an integral part of natural human conversation, but achieving human-like conversational speech remains a challenge. |
| Approach: | They propose a speech synthesis framework that enables natural filler insertion and style control . they propose 'filler-inclusive' speech data that includes fillers with pitch and duration information . |
| Outcome: | The proposed framework enables natural filler insertion and style control . the proposed framework is validated and can be used to predict filler style . |
Multi-LMentry: Can Multilingual LLMs Solve Elementary Tasks Across Languages? (2025.emnlp-main)
Copied to clipboard
Luca Moroni, Javier Aula-Blasco, Simone Conia, Irene Baucells, Naiara Perez, Silvia Paniagua Suárez, Anna Sallés, Malte Ostendorff, Júlia Falcão, Guijin Son, Aitor Gonzalez-Agirre, Roberto Navigli, Marta Villegas
| Challenge: | a recent study focused on complex, high-level tasks, but LMentry is limited to English . a multilingual evaluation of large language models is needed to address this gap, authors say . |
| Approach: | They propose a compact benchmark that enables systematic evaluation of large language models . they propose to use tasks that are trivial for humans but remain surprisingly difficult for LLMs . |
| Outcome: | The proposed benchmark is limited to English, leaving its insights linguistically narrow. |
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries. |
| Approach: | They propose a lookahead q-cache framework that generates low-cost pseudo lookaheaded queries to better approximate the true decoding-stage queries. |
| Outcome: | The proposed framework outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks and can be flexibly combined to yield further improvements. |
PerspectiveMod: A Perspectivist Resource for Deliberative Moderation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Human moderators in online discussions face a heterogeneous range of tasks that go beyond content moderation, or policing. |
| Approach: | They propose a dataset of online comments annotated for the question "Does this comment require moderation?" they aim to improve discussion quality by analyzing annotator perspectives and annotating their views. |
| Outcome: | The proposed model is unique in its intentional variation across the level of moderation experience embedded in the source data, the annotator profiles and the individuality of the annnotator. |
LoCt-Instruct: An Automatic Pipeline for Constructing Datasets of Logical Continuous Instructions (2025.emnlp-main)
Copied to clipboard
Hongyu Sun, Yusuke Sakai, Haruki Sakajo, Shintaro Ozaki, Kazuki Hayashi, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Existing instruction following datasets lack logical coherence across turns, narrow topical breadth and heavy manual effort. |
| Approach: | They propose a pipeline that leverages LLMs’ reasoning capabilities to assemble rich, topic-related single-instruction data into multi-turn dialogues and produce chains that are logically coherent, progressively deepen in content, and span diverse domains without fixed templates or extensive human annotation. |
| Outcome: | The proposed pipeline improves the performance of existing LLMs by integrating multiple topic-related data into multi-turn dialogues without fixed templates or extensive human annotation. |
CodeSSM: Towards State Space Models for Code Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformers have limitations, such as quadratic complexity and high inference costs. |
| Approach: | They propose a state space model that is trained on code corpora to assess its effectiveness. |
| Outcome: | The proposed model reduces memory usage by up to 64% compared to transformers at a context length of 2048. |
EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models produce outputs that are too advanced or vague for younger learners and there are no standardized benchmarks to evaluate their ability to adapt across cognitive and developmental stages. |
| Approach: | They propose to use a benchmark to assess LLMs' ability to adapt to different grade levels and to use it to evaluate their model's performance. |
| Outcome: | The proposed framework assesses the ability of large language models to adapt to grade levels across a range of subjects and grades. |
xCoRe: Cross-context Coreference Resolution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current coreference resolution systems are limited to short-to-medium-sized documents and struggle to scale to very long documents due to architectural limitations and implied memory costs. |
| Approach: | They propose a unified approach to coreference resolution that unifies two challenging settings . they use a pipeline that first identifies mentions, then creates clusters within individual contexts . |
| Outcome: | The proposed model achieves state-of-the-art results on cross-document benchmarks and strong performance on long-document data while retaining top-tier results on traditional datasets. |
Retrieval-Augmented Generation with Estimation of Source Reliability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) is an effective approach to enhance the factual accuracy of large language models (LLMs). |
| Approach: | They propose a multi-source RAG framework that estimates the reliability of sources and prioritizes highly reliable and relevant documents. |
| Outcome: | The proposed framework outperforms baselines in scenarios with heterogeneous source reliability while scaling efficiently as the number of sources increases. |
NitiBench: Benchmarking LLM Frameworks on Thai Legal Question Answering Capabilities (2025.emnlp-main)
Copied to clipboard
Pawitsapak Akarajaradwong, Pirat Pothavorn, Chompakorn Chaksangchaichot, Panuthep Tasawong, Thitiwat Nopparatbundit, Keerakiat Pratai, Sarana Nutanong
| Challenge: | Large language models (LLMs) show promise in legal question answering (QA), yet Thai legal QA systems face challenges due to limited data and complex legal structures. |
| Approach: | They propose a benchmark which uses Thai financial laws and tax rulings to evaluate Thai legal QA systems. |
| Outcome: | The proposed benchmark compared retrieval-augmented generation and long-context LLM approaches across three key dimensions and found that they improve over naive methods. |
From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for error or uncertainty estimation rely on logits, entropy, sampling variance. |
| Approach: | They propose an input-only method for anticipating language model errors using token-level likelihood features inspired by surprisal and the Uniform Information Density hypothesis. |
| Outcome: | The proposed method outperforms baseline models and standard models on linguistic datasets. |
WojoodRelations: Arabic Relation Extraction Corpus and Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on Arabic RE remains limited due to the language’s rich morphology and syntactic complexity, and the lack of large, high-quality datasets. |
| Approach: | They propose to use WojoodRelations to extract relation relationships from Arabic textual data using relation-aware templates and GPT-Joint to perform relation-based retrieval. |
| Outcome: | The proposed method achieves a Cohen’s of 0.92, indicating high reliability, and supervised models achieve 92.89% F1 for RE, while LLMs obtain 72.73% F1 . |
Conflicting Needles in a Haystack: How LLMs behave when faced with contradictory information (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities in retrieving and analyzing complex information, but their reliability in conflicting contexts remains poorly understood. |
| Approach: | They propose an adversarial extension of the Needle-in-a-Haystack framework in which three mutually exclusive “needles” are embedded within long documents. |
| Outcome: | The proposed framework highlights critical limitations in the robustness of current LLMs—including commercial systems—to contradiction. |
Towards Event Extraction with Massive Types: LLM-based Collaborative Annotation and Partitioning Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Event Extraction (EE) is a long-standing target, but lacks an efficient and effective annotation framework to construct the corresponding datasets. |
| Approach: | They propose an LLM-based collaborative annotation framework that refines annotations of triggers from distant supervision and carries out argument annotation. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on the largest EE dataset to date . it achieves the F1 scores of 90% and 85.3% on the human-annotated test set . |
Liaozhai through the Looking-Glass: On Paratextual Explicitation of Culture-Bound Terms in Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to explicitating culturally-embedded meaning have focused on in-text solutions, overlooking paratextual apparatus in the footnotes and endnotes employed by professional translators. |
| Approach: | They formalize Genette's (1987) theory of paratexts and evaluate expert-aligned paratext models . they find that LLM-generated paratext improves audience comprehension . |
| Outcome: | The proposed model improves the comprehension of the Chinese short story Liaozhai by using human evaluations. |
Concept-pedia: a Wide-coverage Semantically-annotated Multimodal Dataset (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current evaluations for Vision-language Models remain heavily anchored to ImageNet . |
| Approach: | They propose a large-scale semantically-annotated multimodal resource that extends the range of visual concepts, including diverse abstract categories. |
| Outcome: | The proposed model expands the range of visual concepts, including diverse abstract categories. |
RAED: Retrieval-Augmented Entity Description Generation for Emerging Entity Linking and Disambiguation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Entity Linking and Entity Disambiguation systems assume static knowledge bases are incomplete and up-to-date, rendering them incapable of handling entities not yet included in the knowledge base. |
| Approach: | They propose a model that retrieves external knowledge to improve factual grounding in entity descriptions. |
| Outcome: | The proposed model outperforms systems that require fixed knowledge sets on Entity Disambiguation and Wikipedia to improve factual grounding in entity descriptions. |
Personalized Language Models via Privacy-Preserving Evolutionary Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for personalization in language models lack explicit mechanisms for privacy preservation. |
| Approach: | They propose a Privacy-Preserving Model Merging via Evolutionary Algorithms to optimize utility while minimizing privacy risks. |
| Outcome: | The proposed approach outperforms baseline models on the LaMP benchmark and achieves 45% improvement in task performance. |
Aligning Text/Speech Representations from Multimodal Models with MEG Brain Activity During Listening (2025.emnlp-main)
Copied to clipboard
Padakanti Srijith, Khushbu Pahwa, Radhika Mamidi, Bapi Raju Surampudi, Manish Gupta, Subba Reddy Oota
| Challenge: | Recent studies have found that speech language models fail to capture brain-relevant semantics beyond low-level features. |
| Approach: | They analyze multimodal models to assess their alignment with MEG brain recordings . they find text embeddings from multimodal and unimodal models significantly outperform unilateral models . |
| Outcome: | a new study shows that text-based models outperform unimodal models in alignment with brain recordings during naturalistic story listening. |
STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Text2SQL systems do not support SQL analysts in their primary work of performing complex analytics on specialized databases. |
| Approach: | They propose to decompose STARQA questions using SQL and Python to perform reasoning on specialized relational databases. |
| Outcome: | The proposed approach decomposes the task through a combination of SQL and Python, and achieves better performance on the more difficult questions. |
Slim-SC: Thought Pruning for Efficient Scaling with Self-Consistency (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that Test-Time Scaling (TTS) can improve reasoning performance without retraining the model. |
| Approach: | They propose a step-wise pruning strategy that identifies and removes redundant chains using inter-chain similarity at the thought level. |
| Outcome: | The proposed method reduces inference latency and KVC usage by up to 45% and 26% with R1-Distill while maintaining or improving accuracy. |
Long Chain-of-Thought Fine-tuning via Understanding-to-Reasoning Transition (2025.emnlp-main)
Copied to clipboard
Chenxin An, Zhihui Xie, Xiaonan Li, Ming Zhong, Shansan Gong, Lei Li, Jun Zhang, Jingjing Xu, Lingpeng Kong
| Challenge: | Existing research on long-context scaling in language models has focused on managing lengthy input prompts instead of producing long outputs. |
| Approach: | They propose a sequence-level curriculum learning framework that shifts a model’s focus from interpreting long chain-of-thoughts to generating them. |
| Outcome: | Experiments on rigorous reasoning benchmarks, including AIME24 and GPQA Diamond, show that the proposed approach surpasses standard fine-tuning by over 10% while maintaining robust performance on understanding tasks. |
Exploring Large Language Models for Detecting Mental Disorders (2025.emnlp-main)
Copied to clipboard
| Challenge: | Detecting mental disorders and patient emotions through text analysis and machine learning is of increasing interest to researchers over the past decade. |
| Approach: | They compare the performance of traditional machine learning methods and encoder-based models on Russian-language datasets to those of large language models. |
| Outcome: | The proposed models outperform traditional methods on small and noisy datasets, but can perform comparable to language models when trained on patients with clinically confirmed depression. |
Efficient Real-time Refinement of Language Model Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate factually incorrect answers, a challenge that remains . Streaming-VR enables on-the-fly verification and correction of tokens as they are generated . |
| Approach: | They propose a method that enables on-the-fly verification and correction of LLM tokens as they are generated. |
| Outcome: | The proposed method improves factual accuracy and improves refinement efficiency compared to prior methods. |
Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Masked diffusion models (MDMs) are promising non-autoregressive alternatives for large language modeling. |
| Approach: | They propose a method that leverages an external reward model to provide a principled global signal during the iterative diffusion process. |
| Outcome: | The proposed method improves non-autoregressive generation orders and performance across evaluation metrics. |
AI Argues Differently: Distinct Argumentative and Linguistic Patterns of LLMs in Persuasive Contexts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Distinguishing LLM-generated text from human-written is a key challenge for safe and ethical NLP, especially in high-stake settings such as persuasive online discourse. |
| Approach: | They propose to use general-purpose linguistic features and domain-specific features related to argument quality to compare human- and LLM-authored arguments. |
| Outcome: | The proposed framework compares arguments by humans and three LLMs using two easily-interpretable feature sets. |
TounsiBench: Benchmarking Large Language Models for Tunisian Arabic (2025.emnlp-main)
Copied to clipboard
| Challenge: | a dataset of Tunisian Arabic instructions and prompts is used to evaluate LLMs' ability to understand and generate responses in Tunisia . we assess the quality, correctness, relevance, and dialectal adherence of LLM responses . |
| Approach: | They propose a benchmark for evaluating the capabilities of large language models in Tunisian Arabic . they use a dataset of Tunisia Arabic instructions and prompts to evaluate their models . |
| Outcome: | The proposed model can judge quality, correctness, relevance, and dialectal adherence . the model can also generate a leaderboard for the Tunisian Arabic language . |
Moral Framing in Politics (MFiP): A new resource and models for moral framing (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on detecting moral values in political communication, trying to identify moral frames used by political actors or parties to convey their messages. |
| Approach: | They propose to code German parliamentary debates to identify moral framing and to detect subtle differences in politicians’ moral framming. |
| Outcome: | The proposed model distinguishes between different types of moral frames and includes narrative roles, together with the moral foundations for each frame. |
ReDepress: A Cognitive Framework for Detecting Depression Relapse from Social Media (2025.emnlp-main)
Copied to clipboard
Aakash Kumar Agarwal, Saprativa Bhattacharjee, Mauli Rastogi, Jemima S. Jacob, Biplab Banerjee, Rashmi Gupta, Pushpak Bhattacharyya
| Challenge: | Almost 50% of depression patients face the risk of going into relapse. |
| Approach: | They propose to validate a social media dataset on depression relapse using cognitive theories of depression. |
| Outcome: | The first clinically validated social media dataset focused on depression relapse comprises 204 Reddit users annotated by mental health professionals. |
iKnow-audio: Integrating Knowledge Graphs with Audio-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastive language-audio pretraining models learn by aligning audio and text in a shared embedding space. |
| Approach: | They propose a framework that integrates knowledge graphs with audio-language models to provide robust semantic grounding. |
| Outcome: | iKnow-audio improves disambiguation of acoustically similar sounds and reduces prompt engineering. |
EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videos (2025.emnlp-main)
Copied to clipboard
| Challenge: | This paper explores using Multimodal Large Language Models (MLLMs) to respond to student questions from online lectures . MLLM is a novel question answering task of real world significance . |
| Approach: | They propose to use Multimodal Large Language Models to automatically respond to student questions from online lectures by using a dataset of 5252 question-answer pairs from 296 computer science videos. |
| Outcome: | The proposed model can fine tune and fine tune questions from 296 computer science videos and show that students' preferences are important to the task. |
The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs (2025.emnlp-main)
Copied to clipboard
Denis Janiak, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, Tomasz Jan Kajdanowicz
| Challenge: | Large language models (LLMs) have revolutionized natural language processing, but their tendency to hallucinate poses serious challenges for reliable deployment. |
| Approach: | They propose to use ROUGE to assess lexical overlap to determine accuracy of hallucination detection methods. |
| Outcome: | The proposed evaluation frameworks can rival complex methods, exposing a fundamental flaw in current evaluation practices. |
Turning Logic Against Itself: Probing Model Defenses Through Contrastive Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety measures detect overt malicious intent but fail to address subtle, reasoning-driven vulnerabilities. |
| Approach: | They propose a two-phase jailbreak technique that exploits contrastive reasoning to bypass safety mechanisms in large language models. |
| Outcome: | The proposed techniques achieve higher attack success rates (44%) than existing methods. |
CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models are not equipped to read diverse languages and scripts found in historical materials. |
| Approach: | They propose to train an open-weight vision-language model for historical text recognition on CHURRO-DS, the largest historical text-recognition dataset to date. |
| Outcome: | The proposed model outperforms existing vision-language models on CHURRO-DS, the largest historical text recognition dataset to date. |
Towards Author-informed NLP: Mind the Social Bias (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of text understanding fail when opinions are conveyed implicitly or sarcastically. |
| Approach: | They propose to model user contexts within a social embedding space that was learned from the Twitter network at large-scale. |
| Outcome: | The proposed model improves generalization of stance prediction and toxicity detection, and also toxicity and incivility detection. |
Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models (2025.emnlp-main)
Copied to clipboard
Sina Semnani, Jirayu Burapacheep, Arpandeep Khatua, Thanawan Atchariyachanvanit, Zheng Wang, Monica Lam
| Challenge: | a new study examines the accuracy of Wikipedia's factual inconsistencies . a corpus-level inconsistent detection system can help editors identify inconsistances . |
| Approach: | They propose a corpus-level inconsistency detection system that combines LLM reasoning with retrieval to detect and contextualize potential contradictions for human review. |
| Outcome: | The proposed system can detect inconsistencies in Wikipedia and human review. |
Leveraging Multilingual Training for Authorship Representation: Enhancing Generalization across Languages and Domains (2025.emnlp-main)
Copied to clipboard
| Challenge: | Authorship representation (AR) models capture an author's distinctive writing style by encoding documents written by the same author as nearby vectors in the embedding space. |
| Approach: | They propose a method that incorporates probabilistic content masking and language-aware batching to improve contrastive learning by reducing cross-lingual interference. |
| Outcome: | The proposed model outperforms monolingual baselines in 21 out of 22 non-English languages and reaches a maximum gain of 15.91% in a single language. |
DrFrattn: Directly Learn Adaptive Policy from Attention for Simultaneous Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn read/write policies from attention mechanism may compromise effectiveness of attention mechanism . |
| Approach: | They propose a method that directly learns adaptive policies from the attention mechanism . experimental results demonstrate that the method achieves an improved balance between translation accuracy and latency. |
| Outcome: | The proposed method achieves improved balance between translation accuracy and latency. |
The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art multimodal language models (MLMs) show promise for supporting SLPs, but their use remains underexplored due to a limited understanding of their performance in high-stakes clinical settings. |
| Approach: | They propose a taxonomy of real-world use cases of multimodal language models in speech-language pathologies to address this gap. |
| Outcome: | The proposed model outperforms 15 state-of-the-art models in speech-language pathologies across five use cases and achieves improvements of over 30% on domain-specific data. |
NormXLogit: The Head-on-Top Never Lies (2025.emnlp-main)
Copied to clipboard
| Challenge: | NormXLogit is a novel approach for assessing the significance of input tokens based on word embeddings . |
| Approach: | They propose a novel method for assessing the significance of input tokens based on the input and output representations associated with each token. |
| Outcome: | The proposed method outperforms gradient-based methods in faithfulness and offers competitive performance compared to leading architecture-specific techniques. |
Doc2Chart: Intent-Driven Zero-Shot Chart Generation from Documents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated strong capabilities in transforming text descriptions or tables to data visualizations . however, it is not straightforward to apply these methods directly for a more real-world use case of visualizing data from long documents . |
| Approach: | They propose an unsupervised method for generating intent-based charts from documents . they propose an attribution-based metric that uses a structured textual representation of charts . |
| Outcome: | The proposed method outperforms baselines in terms of chart data accuracy and chart type over baselines. |
Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have increased the vulnerability of LLMs, but they can cause more severe damage than standalone systems if compromised. |
| Approach: | They propose a new type of attack that induces malfunctions by misleading the agent into executing repetitive or irrelevant actions. |
| Outcome: | The proposed attacks induce failure rates exceeding 80% in multiple scenarios, highlighting the substantial risks associated with this vulnerability. |
FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Spatial reasoning is a fundamental aspect of human intelligence. |
| Approach: | They propose a framework to assess FoR comprehension in large language models (LLMs) by using the Frame of Reference Evaluation in Spatial Reasoning Tasks benchmark. |
| Outcome: | The proposed method improves overall performance across spatial reasoning tasks. |
Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Cross-Lingual Transfer in Sense-Aware Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Cross-lingual transfer allows models to perform tasks in languages unseen during training and is often assumed to benefit from increased multilinguality. |
| Approach: | They challenge this assumption by analyzing polysemy disambiguation and lexical semantic change in 28 languages and using confounding factors to account for perceived advantages. |
| Outcome: | The proposed models and benchmarks are compared across 28 languages and show that multilingual training is neither necessary nor beneficial for effective transfer. |
Translating Domain-Specific Terminology in Typologically-Diverse Languages: A Study in Tax and Financial Education (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing public terminology datasets for MT research are limited in language coverage or domain specificity, making it difficult to assess or improve MT systems in specialized settings. |
| Approach: | They propose a multilingual terminology resource for tax and financial education covering seven typologically diverse languages: English, Spanish, Russian, Vietnamese, Korean, Chinese (traditional and simplified) and Haitian Creole. |
| Outcome: | The proposed terminology resource covers seven typologically diverse languages: English, Spanish, Russian, Vietnamese, Korean, Chinese (traditional and simplified) and Haitian Creole. |
Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work shows that byte-pair encoding (BPE) tokenization uses a learned merge list to iteratively combine subword units into tokens during inference time. |
| Approach: | They propose to use a standard byte-pair encoding algorithm to pair a learned token vocabulary with a detailed merge list to compress text. |
| Outcome: | The proposed algorithms differ from the encoding process during training and show that the targetted deviation from merge lists exhibits significant degradation in language model performance. |
Spectral Scaling Laws in Language Models: emphHow Effectively Do Feed-Forward Networks Use Their Latent Space? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. |
| Approach: | They propose to reframe model width selection as a spectral utilization optimization problem . they quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families . |
| Outcome: | The proposed model maximizes the capacity of feed-forward networks by recasting the problem as a spectral utilization optimization problem. |
TLUE: A Tibetan Language Understanding Evaluation Benchmark (2025.emnlp-main)
Copied to clipboard
Fan Gao, Cheng Huang, Yutong Liu, Nyima Tashi, Xiangxiang Wang, Thupten Tsering, Ban Ma-bao, Renzeng Duojie, Gadeng Luosang, Rinchen Dongrub, Dorje Tashi, Xiao Feng Cd, Yongbin Yu, Hao Wang
| Challenge: | Low-resource languages, like Tibetan, remain underrepresented in large language models' evaluations. |
| Approach: | They propose a Tibetan Language Understanding Evaluation Benchmark to assess LLMs' proficiency in Tibetan . they use a multi-task understanding benchmark and a safety benchmark to evaluate models . |
| Outcome: | The proposed benchmark shows that most large language models perform below the random baseline, especially in Tibetan language processing. |
Retrieving Support to Rank Answers in Open-Domain Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | a novel question answering architecture retrieves content relevant to the combined pair . previous work on automatic claim verification has shown hallucinations . |
| Approach: | They propose a question-answer architecture that prioritizes supporting evidence . it retrieves paragraphs that directly substantiate the correctness of a with respect to q . |
| Outcome: | The proposed approach can be used by large language models to retrieve explanatory paragraphs that ground their reasoning. |
Trojsten Benchmark: Evaluating LLM Problem-Solving in Slovak STEM Competition Problems (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have been used for grading open-ended responses and providing feedback beyond traditional methods. |
| Approach: | They propose a Slovak-language dataset and a rubric-based LLM grading framework . they quantify multistep reasoning performance by difficulty and show consistency under difficult items . |
| Outcome: | The proposed model outperforms existing models on Slovak-language competition problems . the model shows consistent underperformance on harder items and language sensitivity . |
BRSpeech-DF: A Deep Fake Synthetic Speech Dataset for Portuguese Zero-Shot TTS (2025.emnlp-main)
Copied to clipboard
Alexandre Costa Ferro Filho, Rafaello Virgilli, Lucas Alcantara Souza, F S de Oliveira, Marcelo Henrique Lopes Ferreira, Daniel Tunnermann, Gustavo Dos Reis Oliveira, Anderson Da Silva Soares, Arlindo Rodrigues Galvão Filho
| Challenge: | ADD detection is a key area of research for low-resource languages like Portuguese, which lacks high-quality datasets. |
| Approach: | They propose to provide the first publicly available ADD dataset for Portuguese, encompassing both Brazilian and European variants. |
| Outcome: | The proposed dataset contains over 458,000 utterances, including a smaller portion of real speech from 62 speakers and a large collection of synthetic samples generated using multiple zero-shot text-to-speech (TTS) models, each conditioned on the original speaker’s voice. |
A Simple Yet Effective Method for Non-Refusing Context Relevant Fine-grained Safety Steering in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models to meet safety policies are costly and impractical. |
| Approach: | They propose a method to fine-tune large language models to meet evolving safety policies by applying a gradient-free, unsupervised approach. |
| Outcome: | The proposed method provides precise control, avoids blanket refusals, and directs models to generate safe, relevant content. |
Statistical and Neural Methods for Hawaiian Orthography Modernization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Hawaiian orthography employs two distinct spelling systems, both of which are used by communities of speakers today. |
| Approach: | They develop models that convert between the ‘okina letter and kahak diacritic, which represent glottal stops and long vowels, respectively. |
| Outcome: | The proposed models outperform neural seq2seq models and LLMs in a low-resource setting, highlighting the potential for traditional machine learning approaches in . low-cost environments. |
so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite popularity of poetry as both an art form and a generation task for large language models, whitespace has not received sufficient attention from the NLP community. |
| Approach: | They examine how 4k poets have used whitespace in their works . they compare it to 51k LLM-generated poems and 12k unpublished poems posted online . |
| Outcome: | The proposed dataset compares 4k poetry poems with 51k LLM-generated poems and 12k unpublished poems posted in an online community. |
Certified Mitigation of Worst-Case LLM Copyright Infringement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are trained on vast datasets that include copyrighted material or content with usage restrictions. |
| Approach: | They propose a "copyright takedown" method that provides certified copyright take down . they use a combination of inference-time and rewriting techniques to transform potentially infringing segments . |
| Outcome: | The proposed method reduces infringement risk, preserves utility, and accommodates different levels of enforcement stringency with adaptive abstention. |
Quantifying Logical Consistency in Transformers via Query-Key Alignment (2025.emnlp-main)
Copied to clipboard
Eduard Tulchinskii, Laida Kushnareva, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
| Challenge: | Existing solutions for multi-step logical reasoning are unreliable . Existing methods generate intermediate steps but provide no internal check of coherence . |
| Approach: | They propose a method that uses internal Query-Key interactions within transformer attention heads as a proxy for logical consistency. |
| Outcome: | The proposed method reveals latent reasoning structure in large language models and provides a mechanistic alternative to ablation-based analysis. |
SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? (2025.emnlp-main)
Copied to clipboard
Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, Jianfeng Gao
| Challenge: | Large language models (LLMs) are increasingly used in interactive applications, and human evaluation remains the gold standard for assessing their performance in multi-turn conversations. |
| Approach: | They propose to use large language models to simulate users for automatic assistant evaluation. |
| Outcome: | The proposed model outperforms human evaluations on two interactive tasks and achieves Spearman’s of 0.7 on both tasks. |
CourtReasoner: Can LLM Agents Reason Like Judges? (2025.emnlp-main)
Copied to clipboard
Sophia Simeng Han, Yoshiki Takashima, Shannon Zejiang Shen, Chen Liu, Yixin Liu, Roque K. Thuo, Sonia Knowlton, Ruzica Piskac, Scott J Shapiro, Arman Cohan
| Challenge: | Large language models (LLMs) have achieved impressive results across mathematical, logical, and commonsense reasoning tasks. |
| Approach: | They propose a novel expert-annotated judicial reasoning benchmark to measure LLMs' ability to construct goal-oriented legal reasoning. |
| Outcome: | The proposed benchmark measures the LLM agent’s ability to construct goal-oriented legal reasoning. |
Not Your Typical Government Tipline: LLM-Assisted Routing of Environmental Protection Agency Citizen Tips (2025.emnlp-main)
Copied to clipboard
| Challenge: | Regulatory agencies often operate with limited resources and rely on tips to identify violations . but processing these tips at scale presents significant operational challenges . a case study shows advances in large language models can be utilized to support overburdened agencies . |
| Approach: | They analyze thousands of citizen tips from the EPA's "Report a Violation" system to develop an LLM-assisted pipeline for tip routing. |
| Outcome: | The proposed approach filters out 80.5% of irrelevant tips and increases overall routing accuracy from 31.8% to 82.4% compared to the current system. |
Retracing the Past: LLMs Emit Training Data When They Get Lost (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extracting training data from large language models exhibit limited success . existing methods offer limited insight into the fundamental drivers of memorization leakage . |
| Approach: | They propose a framework for extracting memorized data by maximizing model uncertainty . they propose mismatched fine-tuning to weaken alignment and induce confusion . |
| Outcome: | The proposed attacks outperform baselines on unaligned and aligned LLMs . the proposed attacks exploit the model uncertainty of the input snippets induced by the model entropy spike . |
Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study evaluated the extent to which SLMs encode nuanced syntactic and conceptual features . acoustic and phonetic features are shallow, but the extent of nuance is unclear . |
| Approach: | a new study evaluates contextual syntactic and semantic features in transformer-based speech language models . authors compare SLMs to linguistic competence assessments for large language models. |
| Outcome: | a new study compares SLMs with linguistic competence assessments to assess speech recognition and understanding . the results show that SLM models encode grammatical features more robustly than conceptual ones . |
Current Semantic-change Quantification Methods Struggle with Semantic Change Discovery in the Wild (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for lexical semantic-change detection quantify changes in the meaning of words over time. |
| Approach: | They propose to use a top-k setup to evaluate semantic-change discovery despite lacking complete annotations on a battery of semantic-changing detection methods. |
| Outcome: | The proposed setup extends the annotations in the commonly used LiverpoolFC and SemEval-EN benchmarks by 85% and 90%. |
Evaluating Large Language Models for Detecting Antisemitism (2025.emnlp-main)
Copied to clipboard
| Challenge: | Detecting hateful content is a challenging and important problem, but it requires continuous training to adapt to the ever-changing landscape of social media. |
| Approach: | They evaluate eight open-source LLMs’ capability to detect antisemitic content . they use in-context definition as a policy guideline to explore different prompts . |
| Outcome: | The proposed model outperforms the fine-tuned model with Guided-CoT . |
D-RAG: Differentiable Retrieval-Augmented Generation for Knowledge Graph Question Answering (2025.emnlp-main)
Copied to clipboard
Guangze Gao, Zixuan Li, Chunfeng Yuan, Jiawei Li, Wu Jianzhuo, Yuehao Zhang, Xiaolong Jin, Bing Li, Weiming Hu
| Challenge: | Existing approaches to Knowledge Graph Question Answering (KGQA) use Retrieval-Augmented Generation (RAG) but subgraph selection process is non-differentiable, preventing end-to-end training of the retriever and the generator. |
| Approach: | They propose a Differentiable RAG approach that optimizes the retriever and the generator for KGQA. |
| Outcome: | The proposed approach outperforms state-of-the-art approaches on WebQSP and CWQ. |
Towards Robust Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu Hoang Trinh, Quoc V Le, Junehyuk Jung
| Challenge: | IMO-Bench is a suite of advanced reasoning benchmarks that targets the international mathematical Olympiad level. |
| Approach: | They propose IMO-Bench, a suite of advanced reasoning benchmarks that targets the level of the international mathematical Olympiad. |
| Outcome: | IMO-Bench is a suite of advanced reasoning benchmarks that targets the level of the international mathematical Olympiad. |
Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Fine-tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models such as GPT and Llama have shown remarkable ability on diverse natural language tasks, yet their performance on complex table tasks is suboptimal. |
| Approach: | They propose a generator-validator paradigm to iteratively generate-then-validate training data from language models to fine-tune stronger Table-Specialist models that can specialize in a given task, without using manually-labeled data. |
| Outcome: | The proposed model outperforms vanilla language models on diverse table tasks and can match or surpass GPT-4 level quality. |
Introducing Spotlight: A Novel Approach for Generating Captivating Key Information from Documents (2025.emnlp-main)
Copied to clipboard
Ankan Mullick, Sombit Bose, Rounak Saha, Ayan Kumar Bhowmick, Aditya Vempaty, Prasenjit Dey, Ravi Kokku, Pawan Goyal, Niloy Ganguly
| Challenge: | Unlike highlights (fragmented key points) and traditional summaries, spotlights selectively emphasize intriguing content to foster deeper reader engagement with the source material. |
| Approach: | They propose a novel paradigm for information extraction that selectively emphasizes intriguing content to foster deeper reader engagement with the source material. |
| Outcome: | The proposed model improves readability and boosts engagement value of the original document. |
Argument Summarization and its Evaluation in the Era of Large Language Models (2025.emnlp-main)
Copied to clipboard
Moritz Altemeyer, Steffen Eger, Johannes Daxenberger, Yanran Chen, Tim Altendorf, Philipp Cimiano, Benjamin Schiller
| Challenge: | Large Language Models (LLMs) have revolutionized various Natural Language Generation tasks, including Argument Summarization (ArgSum). |
| Approach: | They propose a prompt-based evaluation scheme and validate it through a human benchmark dataset. |
| Outcome: | The proposed evaluation scheme outperforms existing methods and is validated by a human benchmark dataset. |
Computational Analysis of Conversation Dynamics through Participant Responsivity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Growing literature explores toxicity and polarization in discourse, with comparatively little work on characterizing what makes dialogue prosocial and constructive. |
| Approach: | They develop and evaluate methods for quantifying responsivity through semantic similarity of speaker turns and large language models to identify the relation between two speaker turns. |
| Outcome: | The proposed method is based on semantic similarity of speaker turns and large language models to identify the relation between two speaker turns. |
AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Weight-only quantization is a powerful optimization technique for large language models . pushing below 4 bits often leads to substantial accuracy degradation due to increased quantization error. |
| Approach: | They propose a framework that assigns layer-wise quantization bit-widths to optimize model quality and memory usage. |
| Outcome: | The proposed framework can optimize for large language models under memory constraints. |
Beyond Averages: Learning with Annotator Disagreement in STS (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to capture and model disagreement in Semantic Textual Similarity (STS) ignore label dispersion and incentivize models to ignore uncertainty crucial for practical settings. |
| Approach: | They propose to capture and model disagreement in Semantic Textual Similarity (STS) a lightweight truncated Gaussian head and a cross-encoder are used to model disagreement . |
| Outcome: | The proposed approach improves accuracy and calibration of models to human judgments. |
Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models rely on sequential queries . however, existing methods rely heavily on sequential querying . |
| Approach: | They propose a training-free framework that transforms a single LLM into an effective inference-time ensemble. |
| Outcome: | The proposed framework outperforms existing models on reasoning benchmarks, such as MATH, and improves on a DIPPER ensemble of three Qwen2-MATH-1.5B instances. |
Constrained Non-negative Matrix Factorization for Guided Topic Modeling of Minority Topics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Topic models often fail to capture low-prevalence, domain-critical themes such as mental health themes in online comments. |
| Approach: | They propose a topic modeling solution using a constrained NMF to capture minority topics . they incorporate a seed word list characterizing minority content of interest . |
| Outcome: | The proposed model outperforms baselines on synthetic data in terms of topic purity and quality using Jensen-Shannon divergence. |
Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Whether language models have inductive biases favoring typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs). |
| Approach: | They extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) . they also examine the generalization ability of LMs to process unseen longer test sentences . |
| Outcome: | The proposed models better capture features of natural languages and can process unseen longer test sentences. |
Training compute-optimal transformer encoder models (2025.emnlp-main)
Copied to clipboard
| Challenge: | OptiBERT is a family of compute-optimal BERT-style models that matches or surpasses leading baselines while training with dramatically less FLOPS. |
| Approach: | They propose to train OptiBERT models with a Masked Language Model objective . they train a family of compute-optimal BERT-style models that matches or surpasses leading baselines . |
| Outcome: | The proposed model matches or surpasses leading baselines on GLUE and MTEB while training with dramatically less FLOPS. |
Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews (2025.emnlp-main)
Copied to clipboard
Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, Juho Kim
| Challenge: | Large Language Models (LLMs) can automatically draft reviews, but determining whether they are trustworthy requires systematic evaluation. |
| Approach: | They propose an automatic focus-level evaluation pipeline based on two sets of facets . authors evaluated LLM reviews at surface-level or content-level . |
| Outcome: | The proposed framework enables automatic evaluation of paper reviews based on two sets of facets . the framework compared open review paper reviews with human experts on validity, clarity, novelty . |
Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that deep vision-only and language-only models project inputs into a partially aligned representational space. |
| Approach: | They investigate whether a model's representational code is semantically shared . they find that alignment peaks in mid-to-late layers of both model types . |
| Outcome: | a forced-choice "Pick-a-Pic" task shows human preferences for image-caption matches are mirrored in embedding spaces across vision-language model pairs. |
Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models (2025.emnlp-main)
Copied to clipboard
Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing, Gleb Kuzmin, Ivan Lazichny, Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov, Artem Shelmanov
| Challenge: | Uncertainty quantification (UQ) is a promising approach for detecting hallucinations and low-quality outputs of Large Language Models (LLMs). |
| Approach: | They propose to learn conditional dependency between autoregressive LLM generation steps from attention-based features and a two-staged training procedure to incorporate recurrent features. |
| Outcome: | The proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches. |
Chinese Toxic Language Mitigation via Sentiment Polarity Consistent Rewrites (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be effective at rewriting toxic content, but they often default to overly polite rewrites, distorting the emotional tone and communicative intent. |
| Approach: | They evaluate 17 large language models with variant architectures to evaluate their ability to rewrite toxic content while preserving the speaker's original intent. |
| Outcome: | The first Chinese detoxification dataset explicitly designed to preserve sentiment polarity is evaluated across five real-world scenarios. |
A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs (2025.emnlp-main)
Copied to clipboard
Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin
| Challenge: | Uncertainty quantification (UQ) is a framework for assessing the reliability of model outputs. |
| Approach: | They introduce pre-trained UQ heads for LLMs that are highly robust and generalized to languages they were not explicitly trained on. |
| Outcome: | The pre-trained heads significantly improve their ability to capture uncertainty compared to unsupervised methods. |