Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering

Challenge:	Existing approaches to overcome object hallucination are limited . Existing mitigations include costly retraining and a training-free inference framework .
Approach:	They propose a training-free inference framework that simulates a metacognitive self-correction process.
Outcome:	The proposed framework reduces object hallucination rates by 12.67% on MMHal-Bench and improves accuracy by 5.8% on POPE.

Similar Papers

Overview

Challenge

Approach

Outcome

Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence (2025.acl-long)

Copied to clipboard

Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, Jinqiao Wang

Multimodality

Challenge:	Existing methods focus on alignment training or decoding refinements but address symptoms at the generation stage without probing the underlying causes.
Approach:	They propose a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads.
Outcome:	The proposed method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations while maintaining high efficiency with negligible additional time overhead.

Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation (2025.findings-emnlp)

Copied to clipboard

Zhenglin Hua, Jinghan He, Zijun Yao, Tianxu Han, Haiyun Guo, Yuheng Jia, Junfeng Fang

Language Models

Challenge:	Existing approaches to address hallucinations in large vision-language models require substantial computational cost and time.
Approach:	They propose to leverage sparse autoencoders to identify semantic directions closely associated with faithfulness or hallucination, extracting more precise and disentangled hallucinian-related representations.
Outcome:	The proposed method outperforms existing decoding approaches while maintaining transferability across different model architectures with negligible additional time overhead.

Treble Counterfactual VLMs: A Causal Approach to Hallucination (2025.findings-emnlp)

Copied to clipboard

Li Li, Jiashu Qu, Linxin Song, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao

Multimodality

Challenge:	Existing studies link hallucination to data or representation biases, but their causal origins remain unclear.
Approach:	They propose a causal framework to analyze and mitigate hallucination in vision-language models by using counterfactual analysis to estimate the Natural Direct Effect (NDE) of each modality and their interaction.
Outcome:	The proposed framework significantly reduces hallucination while preserving task performance while retaining reliability.

Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings (2026.findings-acl)

Copied to clipboard

Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti, Gauri Jagatap, Jiaxin Yuan, Sarvesh Baskar, Vijay Kamarshi, Andrea Fanelli, Furong Huang

Semantic Text Processing

Challenge:	Hallucinations in Large Vision-Language Models (LVLMs) are a persistent challenge, stemming from inadequate integration of visual information during multimodal reasoning.
Approach:	They propose a visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution.
Outcome:	The proposed method significantly reduces hallucinations and fosters more balanced multimodal reasoning.

Perceptual Hallucination in Vision–Language Models: Definition, Analysis and Verification (2026.findings-acl)

Copied to clipboard

Taewook Hwang, Inbum Heo, Sung Jun Lee, Sangkeun Jung

Multimodality

Challenge:	Recent advances in large language models (LLMs) have dramatically improved text understanding and generation capabilities.
Approach:	They define perceptual hallucination as the phenomenon where VLMs generate information as if perceived, despite absent or damaged visual evidence.
Outcome:	The proposed model reduces hallucination exposure by 36% on average, with reductions of up to 88%.

Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention (2025.acl-long)

Copied to clipboard

Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, Zhaoxiang Zhang

Multimodality

Challenge:	Large Vision Language Models (LVLMs) suffer from hallucination where generated textual descriptions fail to align accurately with visual semantics.
Approach:	They propose a training-free approach that mitigates hallucination through targeted intervention in the model’s intermediate activations by identifying directional patterns of hallucinism in the activation space using a small calibration set.
Outcome:	The proposed approach reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks.

Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decoding (2026.eacl-srw)

Copied to clipboard

Boqi Chen, Xudong Liu, Jianing Qiu

Multimodality

Challenge:	Recent studies improve visual contrastive decoding (VCD) by constructing more informative auxiliary views.
Approach:	They propose to construct an object-aligned auxiliary view that disrupts unsupported tokens and produces a stronger contrast signal.
Outcome:	Empirically, the proposed method shows consistent gains on two popular object hallucination benchmarks across two MLLMs.

CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention (2025.acl-long)

Copied to clipboard

Zekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin

Reasoning

Challenge:	Large Vision-Language Models (LVLMs) have impressive multimodal abilities but remain prone to multilingual object hallucination.
Approach:	They propose a cross-lingual attention intervention method to mitigate multilingual object hallucination in LVLMs by aligning attention patterns.
Outcome:	The proposed method improves 13.56% (up to 30%) on the POPE and 21.75% on the hallucination subsets across languages.

Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning (2025.coling-main)

Copied to clipboard

Xiaoye Qu, Jiashuo Sun, Wei Wei, Daizong Liu, Jianfeng Dong, Yu Cheng

Multimodality

Challenge:	Large Vision-Language Models (LVLMs) have impressive capabilities in multi-modal context comprehension, but they still suffer from hallucination problems due to inconsistent outputs with the image content.
Approach:	They propose a training-free framework MVP to reduce hallucinations in Large Vision-Language Models . they propose multi-view information-seeking strategy to perceive the comprehensive information in the image .
Outcome:	The proposed framework reduces hallucinations in large vision-language models by combining multi-view multi-path reasoning with multi-vision multi-path reasoning.

Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024.findings-acl)

Copied to clipboard

Avshalom Manevich, Reut Tsarfaty

Multimodality

Challenge:	Large Vision-Language Models (LVLMs) often produce object hallucinations due to their reliance on text cues and learned object co-occurrence biases.
Approach:	They propose a language-contrasting decoding algorithm that adjusts LVLM outputs based on LLM confidence levels to mitigate object hallucinations.
Outcome:	The proposed method shows up to %4 improvement in POPE F1 scores and %36 reduction in CHAIR scores on COCO validation set while improving captioning quality scores.