Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research in large vision-language models has shown promising results, but the issue of hallucination remains. |
| Approach: | They propose an instruction-based method to reduce hallucinations in large vision-language models . they use disturbance instructions to exacerbate hallucinosity in multimodal fusion modules . |
| Outcome: | The proposed method reduces hallucinations in multimodal fusion modules by reducing alignment uncertainty and subtracting hallucines from the original distribution. |
Similar Papers
Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) often produce object hallucinations due to their reliance on text cues and learned object co-occurrence biases. |
| Approach: | They propose a language-contrasting decoding algorithm that adjusts LVLM outputs based on LLM confidence levels to mitigate object hallucinations. |
| Outcome: | The proposed method shows up to %4 improvement in POPE F1 scores and %36 reduction in CHAIR scores on COCO validation set while improving captioning quality scores. |
Multi-Frequency Contrastive Decoding: Alleviating Hallucinations for Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies attribute object hallucinations to linguistic priors and data biases . MFCD method removes hallucinian distribution in the original output distribution . |
| Approach: | They propose a method that removes the hallucination distribution in the original output distribution . they propose MFCD to mitigate hallucinism in large visual-language models . |
| Outcome: | The proposed method reduces hallucination distributions without training or external tools . the proposed method can be applied to various LVLMs without modifying model architecture or training . |
Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings (2026.findings-acl)
Copied to clipboard
Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti, Gauri Jagatap, Jiaxin Yuan, Sarvesh Baskar, Vijay Kamarshi, Andrea Fanelli, Furong Huang
| Challenge: | Hallucinations in Large Vision-Language Models (LVLMs) are a persistent challenge, stemming from inadequate integration of visual information during multimodal reasoning. |
| Approach: | They propose a visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. |
| Outcome: | The proposed method significantly reduces hallucinations and fosters more balanced multimodal reasoning. |
Alleviating Hallucinations of Large Language Models through Induced Hallucinations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models generate inaccurate or fabricated information, a phenomenon known as hallucinations. |
| Approach: | They propose a simple strategy to induce-then-contrast decode LLMs to enhance their factuality . they first induce hallucinations from the original model and penalize them . |
| Outcome: | The proposed strategy improves factuality of large language models across task formats, model sizes, and model families. |
Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decoding (2026.eacl-srw)
Copied to clipboard
| Challenge: | Recent studies improve visual contrastive decoding (VCD) by constructing more informative auxiliary views. |
| Approach: | They propose to construct an object-aligned auxiliary view that disrupts unsupported tokens and produces a stronger contrast signal. |
| Outcome: | Empirically, the proposed method shows consistent gains on two popular object hallucination benchmarks across two MLLMs. |
Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on coarse-grained hallucination detection and fail to capture hallucinics . vision encoders exhibit unique hallucinian characteristics, but suboptimal of simple feature fusion. |
| Approach: | They propose a visual encoder that employs different training paradigms to instill inductive biases in visual encoded models. |
| Outcome: | The proposed system reduces hallucinations and improves model performance. |
Contrastive Decoding Reduces Hallucinations in Large Multilingual Machine Translation Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Hallucinations occur when the target side sentence is detached from the source side sentence, or in other words, when there is a low contribution of the source sentence to the generation of the target sentence. |
| Approach: | They propose to use Contrastive Decoding to maximise the log-likelihood difference between a model and the same model with reduced contribution from the encoder outputs. |
| Outcome: | The proposed algorithm maximises the log-likelihood difference between a model and the same model with reduced contribution from the encoder outputs. |
Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention (2025.acl-long)
Copied to clipboard
| Challenge: | Large Vision Language Models (LVLMs) suffer from hallucination where generated textual descriptions fail to align accurately with visual semantics. |
| Approach: | They propose a training-free approach that mitigates hallucination through targeted intervention in the model’s intermediate activations by identifying directional patterns of hallucinism in the activation space using a small calibration set. |
| Outcome: | The proposed approach reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks. |
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large vision-language models produce hallucinations that compromise output reliability. |
| Approach: | They propose a dual-stage framework for mitigating hallucinations without performance degradation . they propose semantic-aware component disentanglement and interpretable parameter updates . |
| Outcome: | The proposed model reduces hallucinations by 23.4% while maintaining 97.4% of general generative capability. |
Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024.acl-long)
Copied to clipboard
Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, Bing Qin
| Challenge: | Large Vision-Language Models (LVLMs) suffer from multimodal hallucinations . however, the generated hallucines could influence the models’ subsequent generation . |
| Approach: | They propose a framework to evaluate LVLMs' behaviors when encountering generated hallucinations and a method to revise the output distribution of LVLs with the one derived from the residual visual input. |
| Outcome: | The proposed framework reduces the performance of open-source LVLMs by 31%, indicating that they are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions. |