Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings (2026.findings-acl)
Copied to clipboard
Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti, Gauri Jagatap, Jiaxin Yuan, Sarvesh Baskar, Vijay Kamarshi, Andrea Fanelli, Furong Huang
| Challenge: | Hallucinations in Large Vision-Language Models (LVLMs) are a persistent challenge, stemming from inadequate integration of visual information during multimodal reasoning. |
| Approach: | They propose a visual feature incorporation method that encourages the model to learn visually-informed textual embeddings distinct from those of the base LLM and promotes a more balanced attention distribution. |
| Outcome: | The proposed method significantly reduces hallucinations and fosters more balanced multimodal reasoning. |
Similar Papers
Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024.acl-long)
Copied to clipboard
Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, Bing Qin
| Challenge: | Large Vision-Language Models (LVLMs) suffer from multimodal hallucinations . however, the generated hallucines could influence the models’ subsequent generation . |
| Approach: | They propose a framework to evaluate LVLMs' behaviors when encountering generated hallucinations and a method to revise the output distribution of LVLs with the one derived from the residual visual input. |
| Outcome: | The proposed framework reduces the performance of open-source LVLMs by 31%, indicating that they are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions. |
Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) often produce object hallucinations due to their reliance on text cues and learned object co-occurrence biases. |
| Approach: | They propose a language-contrasting decoding algorithm that adjusts LVLM outputs based on LLM confidence levels to mitigate object hallucinations. |
| Outcome: | The proposed method shows up to %4 improvement in POPE F1 scores and %36 reduction in CHAIR scores on COCO validation set while improving captioning quality scores. |
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research in large vision-language models has shown promising results, but the issue of hallucination remains. |
| Approach: | They propose an instruction-based method to reduce hallucinations in large vision-language models . they use disturbance instructions to exacerbate hallucinosity in multimodal fusion modules . |
| Outcome: | The proposed method reduces hallucinations in multimodal fusion modules by reducing alignment uncertainty and subtracting hallucines from the original distribution. |
Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) generate detailed and coherent responses from visual inputs but are prone to generate hallucinations due to an over-reliance on language priors. |
| Approach: | They propose a method that reduces the text context and controls only the image-related POS tokens to maintain text quality by reducing the text contextualization. |
| Outcome: | The proposed method achieves state-of-the-art performance on object hallucination benchmarks and achieves Pareto optimality among the existing methods. |
Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on coarse-grained hallucination detection and fail to capture hallucinics . vision encoders exhibit unique hallucinian characteristics, but suboptimal of simple feature fusion. |
| Approach: | They propose a visual encoder that employs different training paradigms to instill inductive biases in visual encoded models. |
| Outcome: | The proposed system reduces hallucinations and improves model performance. |
Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence (2025.acl-long)
Copied to clipboard
Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, Jinqiao Wang
| Challenge: | Existing methods focus on alignment training or decoding refinements but address symptoms at the generation stage without probing the underlying causes. |
| Approach: | They propose a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads. |
| Outcome: | The proposed method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations while maintaining high efficiency with negligible additional time overhead. |
Visual Evidence Prompting Mitigates Hallucinations in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | LVLMs have shown impressive progress by integrating visual perception with linguistic understanding to produce contextually grounded outputs. |
| Approach: | They propose a visual evidence prompting method to mitigate hallucinations in large vision-language models by using small visual models to complement them. |
| Outcome: | The proposed method reduces hallucinations by reducing false activation and enhancing correct ones. |
Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) often hallucinate and produce captions that mention concepts that cannot be found in the image. |
| Approach: | They propose to add grounding objectives to captions that explicitly align image regions or objects to text spans to reduce hallucination. |
| Outcome: | The proposed evaluation protocol reduces the amount of hallucination in LVLMs by adding grounding objectives. |
SHARP: Steering Hallucination in LVLMs via Representation Engineering (2025.emnlp-main)
Copied to clipboard
Junfei Wu, Yue Ding, Guofan Liu, Tianze Xia, Ziyue Huang, Dianbo Sui, Qiang Liu, Shu Wu, Liang Wang, Tieniu Tan
| Challenge: | Large Vision-Language Models (LVLMs) generate responses that are plausible but incorrect or unsupported—commonly referred to as hallucinations. |
| Approach: | They propose a representation-level intervention framework that modulates hallucination-related features during inference by probing their encoded features. |
| Outcome: | The proposed framework reduces hallucinations while maintaining the performance and generalization capabilities of Large Vision-Language Models (LVLMs). |
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large vision-language models produce hallucinations that compromise output reliability. |
| Approach: | They propose a dual-stage framework for mitigating hallucinations without performance degradation . they propose semantic-aware component disentanglement and interpretable parameter updates . |
| Outcome: | The proposed model reduces hallucinations by 23.4% while maintaining 97.4% of general generative capability. |