VADE: Visual Attention Guided Hallucination Detection and Elimination

Challenge:	Vision Language Models (VLMs) are prone to hallucinations, generating outputs that lack grounding in the actual visual data.
Approach:	They propose a sequence modelling approach to learn complex sequential patterns from transformer attention maps.
Outcome:	The proposed approach achieves an average PR-AUC of 80% in hallucination detection on M-HalDetect and an 5% improvement in hallucinosis mitigation on MSCOCO.

Similar Papers

Overview

Challenge

Approach

Outcome

Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation (2026.findings-acl)

Copied to clipboard

Zhenghua Wang, Yixin Wu, Feiran Zhang, Qi Qian, Changze Lv, Xuanjing Huang, Xiaoqing Zheng

Multimodality

Challenge:	Vision–Language Models (VLMs) have demonstrated strong capabilities in tasks that require joint understanding of text and images.
Approach:	They propose a strategy that incorporates head-wise attention perturbation via continuous multiplicative noise coupled with a visual-guided loss focused on vision-sensitive text tokens to promote a more balanced attention distribution.
Outcome:	The proposed approach outperforms baseline models on three benchmarks and consistently outperformed the baseline model.

Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models (2025.findings-acl)

Copied to clipboard

Xinlong Chen, Yuanxing Zhang, Qiang Liu, Junfei Wu, Fuzheng Zhang, Tieniu Tan

Multimodality

Challenge:	Large Vision-Language Models (LVLMs) have impressive capabilities across visual tasks, yet they remain hindered by the persistent challenge of hallucinations.
Approach:	They propose a novel approach that dynamically adapts decoding strategies by evaluating the correctness of the model’s attention on image tokens to distinguish the correct attention.
Outcome:	Extensive experiments show that the proposed approach outperforms existing decoding methods across multiple mainstream benchmarks, effectively mitigating hallucinations in LVLMs.

DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination (2024.emnlp-main)

Copied to clipboard

Xuan Gong, Tianshi Ming, Xinpeng Wang, Zhihua Wei

Multimodality

Challenge:	Despite the success of Large Vision-Language Models, they suffer from hallucination.
Approach:	They propose a training-free strategy that "Dive into" the attention of LVLMs to "Reduce" object hallucination by using classification tokens of ViT.
Outcome:	The proposed method reduces the impact of outlier tokens on LVLMs . the proposed method is based on LLaVA-1.5, LLvaVA-NeXT and InstructBLIP .

Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation (2025.findings-emnlp)

Copied to clipboard

Zhenglin Hua, Jinghan He, Zijun Yao, Tianxu Han, Haiyun Guo, Yuheng Jia, Junfeng Fang

Language Models

Challenge:	Existing approaches to address hallucinations in large vision-language models require substantial computational cost and time.
Approach:	They propose to leverage sparse autoencoders to identify semantic directions closely associated with faithfulness or hallucination, extracting more precise and disentangled hallucinian-related representations.
Outcome:	The proposed method outperforms existing decoding approaches while maintaining transferability across different model architectures with negligible additional time overhead.

Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding (2025.naacl-long)

Copied to clipboard

Xinhao Xu, Hui Chen, Mengyao Lyu, Sicheng Zhao, Yizhe Xiong, Zijia Lin, Jungong Han, Guiguang Ding

Multimodality

Challenge:	Multi-modal large language models (MLLMs) generate plausible but incorrect content, resulting in hallucinations . recent advances in MLLM technology have demonstrated their outstanding performance in a variety of visual tasks, such as object detection.
Approach:	They propose a plug-and-play method which leverages MLLMs’ internal representations to mitigate hallucinations by analyzing input and output tokens.
Outcome:	The proposed method exploits MLLMs’ internal representations to mitigate hallucinations.

Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models (2025.findings-acl)

Copied to clipboard

Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, Changick Kim

Multimodality

Challenge:	Large Vision Language Models suffer from hallucinations, attributing incorrect or misleading features to images.
Approach:	They propose a test-time approach that recalibrates the influence of blind tokens . they identify blind token by analyzing layer-wise attention distributions over image tokens.
Outcome:	The proposed approach reduces hallucinations in large vision language models . it uses a contrastive decoding strategy to balance the influence of blind tokens .

Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence (2025.acl-long)

Copied to clipboard

Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, Jinqiao Wang

Multimodality

Challenge:	Existing methods focus on alignment training or decoding refinements but address symptoms at the generation stage without probing the underlying causes.
Approach:	They propose a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads.
Outcome:	The proposed method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations while maintaining high efficiency with negligible additional time overhead.

Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention (2025.acl-long)

Copied to clipboard

Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, Zhaoxiang Zhang

Multimodality

Challenge:	Large Vision Language Models (LVLMs) suffer from hallucination where generated textual descriptions fail to align accurately with visual semantics.
Approach:	They propose a training-free approach that mitigates hallucination through targeted intervention in the model’s intermediate activations by identifying directional patterns of hallucinism in the activation space using a small calibration set.
Outcome:	The proposed approach reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks.

Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow (2025.findings-emnlp)

Copied to clipboard

Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng

Multimodality

Challenge:	Decode-Only models propagate information from left to right, but the model's attention still focuses on the visual representations, resulting in hallucinations.
Approach:	They propose to leverage the core information embedded in semantic representations to enhance the model's visual understanding by leveraging the attention distributions.
Outcome:	The proposed method reduces hallucinations by 80% by aligning the attention distribution with the actual information flow.

Treble Counterfactual VLMs: A Causal Approach to Hallucination (2025.findings-emnlp)

Copied to clipboard

Li Li, Jiashu Qu, Linxin Song, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao

Multimodality

Challenge:	Existing studies link hallucination to data or representation biases, but their causal origins remain unclear.
Approach:	They propose a causal framework to analyze and mitigate hallucination in vision-language models by using counterfactual analysis to estimate the Natural Direct Effect (NDE) of each modality and their interaction.
Outcome:	The proposed framework significantly reduces hallucination while preserving task performance while retaining reliability.