Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for reducing hallucinations incur a significant increase in latency. |
| Approach: | They propose a task-agnostic attention-guided head suppression strategy that can be seamlessly integrated during inference without incurring significant compute or latency overhead. |
| Outcome: | The proposed approach reduces hallucinations by 2.7x while maintaining F1 and improves throughput by 1.8% compared to existing methods. |
Similar Papers
Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Decode-Only models propagate information from left to right, but the model's attention still focuses on the visual representations, resulting in hallucinations. |
| Approach: | They propose to leverage the core information embedded in semantic representations to enhance the model's visual understanding by leveraging the attention distributions. |
| Outcome: | The proposed method reduces hallucinations by 80% by aligning the attention distribution with the actual information flow. |
Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence (2025.acl-long)
Copied to clipboard
Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, Jinqiao Wang
| Challenge: | Existing methods focus on alignment training or decoding refinements but address symptoms at the generation stage without probing the underlying causes. |
| Approach: | They propose a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads. |
| Outcome: | The proposed method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations while maintaining high efficiency with negligible additional time overhead. |
Whispers that Shake Foundations: Analyzing and Mitigating False Premise Hallucinations in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models generate hallucinated text when confronted with false premise questions . authors propose a method to mitigate false premises hallucinosity . |
| Approach: | They propose a method to constrain false premise attention heads during the model inference process. |
| Outcome: | The proposed method improves performance by constraining false premise attention heads . it yields a notable increase of nearly 20% of model performance . |
Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision–Language Models (VLMs) have demonstrated strong capabilities in tasks that require joint understanding of text and images. |
| Approach: | They propose a strategy that incorporates head-wise attention perturbation via continuous multiplicative noise coupled with a visual-guided loss focused on vision-sensitive text tokens to promote a more balanced attention distribution. |
| Outcome: | The proposed approach outperforms baseline models on three benchmarks and consistently outperformed the baseline model. |
Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings (2026.findings-acl)
Copied to clipboard
Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti, Gauri Jagatap, Jiaxin Yuan, Sarvesh Baskar, Vijay Kamarshi, Andrea Fanelli, Furong Huang
| Challenge: | Hallucinations in Large Vision-Language Models (LVLMs) are a persistent challenge, stemming from inadequate integration of visual information during multimodal reasoning. |
| Approach: | They propose a visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. |
| Outcome: | The proposed method significantly reduces hallucinations and fosters more balanced multimodal reasoning. |
Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention (2025.acl-long)
Copied to clipboard
| Challenge: | Large Vision Language Models (LVLMs) suffer from hallucination where generated textual descriptions fail to align accurately with visual semantics. |
| Approach: | They propose a training-free approach that mitigates hallucination through targeted intervention in the model’s intermediate activations by identifying directional patterns of hallucinism in the activation space using a small calibration set. |
| Outcome: | The proposed approach reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks. |
SHARP: Steering Hallucination in LVLMs via Representation Engineering (2025.emnlp-main)
Copied to clipboard
Junfei Wu, Yue Ding, Guofan Liu, Tianze Xia, Ziyue Huang, Dianbo Sui, Qiang Liu, Shu Wu, Liang Wang, Tieniu Tan
| Challenge: | Large Vision-Language Models (LVLMs) generate responses that are plausible but incorrect or unsupported—commonly referred to as hallucinations. |
| Approach: | They propose a representation-level intervention framework that modulates hallucination-related features during inference by probing their encoded features. |
| Outcome: | The proposed framework reduces hallucinations while maintaining the performance and generalization capabilities of Large Vision-Language Models (LVLMs). |
AHEAD: Attention Head Energy-Aware Dynamics for Hallucination Mitigation in MLLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to hallucination mitigation ignore heterogeneous behaviors of attention heads . hallucinosity is a critical barrier to multimodal large language models' reliability, authors say . |
| Approach: | They propose a framework that quantifies the energetic properties of each attention head during object generation through two potential networks and dynamically adjusts their contributions at inference time. |
| Outcome: | The proposed framework reduces hallucination rates without fine-tuning the base model while maintaining generation quality. |
Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding (2025.naacl-long)
Copied to clipboard
| Challenge: | Multi-modal large language models (MLLMs) generate plausible but incorrect content, resulting in hallucinations . recent advances in MLLM technology have demonstrated their outstanding performance in a variety of visual tasks, such as object detection. |
| Approach: | They propose a plug-and-play method which leverages MLLMs’ internal representations to mitigate hallucinations by analyzing input and output tokens. |
| Outcome: | The proposed method exploits MLLMs’ internal representations to mitigate hallucinations. |
CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention (2025.acl-long)
Copied to clipboard
Zekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin
| Challenge: | Large Vision-Language Models (LVLMs) have impressive multimodal abilities but remain prone to multilingual object hallucination. |
| Approach: | They propose a cross-lingual attention intervention method to mitigate multilingual object hallucination in LVLMs by aligning attention patterns. |
| Outcome: | The proposed method improves 13.56% (up to 30%) on the POPE and 21.75% on the hallucination subsets across languages. |