Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination (2026.acl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) are capable of processing visual inputs, but are susceptible to hallucinations. |
| Approach: | They propose a method to localize and localize specific visual tokens, which are defined as **Inert Tokens**, across layers, revealing a rigid semantic collapse. |
| Outcome: | The proposed approach reduces the likelihood of LVLMs being hijacked by visual inputs while maintaining general capabilities. |
Similar Papers
Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision–Language Models (VLMs) have demonstrated strong capabilities in tasks that require joint understanding of text and images. |
| Approach: | They propose a strategy that incorporates head-wise attention perturbation via continuous multiplicative noise coupled with a visual-guided loss focused on vision-sensitive text tokens to promote a more balanced attention distribution. |
| Outcome: | The proposed approach outperforms baseline models on three benchmarks and consistently outperformed the baseline model. |
Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence (2025.acl-long)
Copied to clipboard
Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, Jinqiao Wang
| Challenge: | Existing methods focus on alignment training or decoding refinements but address symptoms at the generation stage without probing the underlying causes. |
| Approach: | They propose a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads. |
| Outcome: | The proposed method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations while maintaining high efficiency with negligible additional time overhead. |
DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite the success of Large Vision-Language Models, they suffer from hallucination. |
| Approach: | They propose a training-free strategy that "D**ive into" the attention of LVLMs to "R**educe" object hallucination by using classification tokens of ViT. |
| Outcome: | The proposed method reduces the impact of outlier tokens on LVLMs . the proposed method is based on LLaVA-1.5, LLvaVA-NeXT and InstructBLIP . |
Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Vision Language Models suffer from hallucinations, attributing incorrect or misleading features to images. |
| Approach: | They propose a test-time approach that recalibrates the influence of blind tokens . they identify blind token by analyzing layer-wise attention distributions over image tokens. |
| Outcome: | The proposed approach reduces hallucinations in large vision language models . it uses a contrastive decoding strategy to balance the influence of blind tokens . |
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing training-free methods are vulnerable to the attention sink phenomenon . Existing methods include contrastive decoding and auxiliary expert models . |
| Approach: | They propose a training-free attention intervention that constructs a PAD map to identify semantically core visual regions and applies per-head Median Absolute Deviation Scaling to adaptively control the intervention strength. |
| Outcome: | The proposed intervention improves visual grounding and reduces hallucinations on multiple LVLMs and benchmarks. |
Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Decode-Only models propagate information from left to right, but the model's attention still focuses on the visual representations, resulting in hallucinations. |
| Approach: | They propose to leverage the core information embedded in semantic representations to enhance the model's visual understanding by leveraging the attention distributions. |
| Outcome: | The proposed method reduces hallucinations by 80% by aligning the attention distribution with the actual information flow. |
TokenPenalty: Alleviating Attention Sinks and Positional Decay in LVLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) often hallucinate due to two relevant phenomena: massive activation phenomenon and positional information decay. |
| Approach: | They propose a token-level intervention strategy that dynamically suppresses irrelevant visual tokens while preserving key contextual signals. |
| Outcome: | Experiments show that TokenTruth significantly improves factual consistency across MLLMs on standard image understanding benchmarks. |
Inject to Heal: Alleviating hallucination in LVLMs via Context Embedding Injection (2026.findings-acl)
Copied to clipboard
| Challenge: | a large vision-language model can generate hallucinations inconsistent with visual input . a lightweight method that embeds the last input token as a grounding signal reduces the likelihood of hallucinosity. |
| Approach: | They propose a training-free mitigation strategy that harnesses the hidden state of the last input token as a grounding signal to maintain visual fidelity throughout decoding and curb hallucinations. |
| Outcome: | The proposed method outperforms state-of-the-art methods on CHAIR, AMBER, and MMHal benchmarks. |
Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention (2025.acl-long)
Copied to clipboard
| Challenge: | Large Vision Language Models (LVLMs) suffer from hallucination where generated textual descriptions fail to align accurately with visual semantics. |
| Approach: | They propose a training-free approach that mitigates hallucination through targeted intervention in the model’s intermediate activations by identifying directional patterns of hallucinism in the activation space using a small calibration set. |
| Outcome: | The proposed approach reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks. |
Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) excel in tasks ranging from image captioning to complex reasoning. |
| Approach: | They propose a contrastive decoding framework that dynamically calibrates each token generation by mining the model’s internal perceptual discrepancies. |
| Outcome: | The proposed framework mitigates hallucination while enhancing general reasoning capabilities. |