LPOI: Listwise Preference Optimization for Vision Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for aligning large VLMs with human preferences often overfit to textual information or exacerbate hallucinations. |
| Approach: | They propose an object-aware listwise preference optimization for reducing hallucinations in VLMs . they mask a critical object in an image and interpolate the masked region to form more complete images . |
| Outcome: | The proposed method outperforms existing methods in reducing hallucinations and enhancing performance on MMHalBench, AMBER, and Object HalBench. |
Similar Papers
Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization (2025.emnlp-main)
Copied to clipboard
Jiulong Wu, Zhengliang Shi, Shuaiqiang Wang, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao, Min Zhang
| Challenge: | Existing preference alignment methods focus on aligning model responses with human preferences while neglecting image-text modality alignment. |
| Approach: | They propose Entity-centric Multimodal Preference Optimization to improve modality alignment . they use open-source instruction datasets to automatically construct high-quality preference data . |
| Outcome: | The proposed approach reduces hallucination rates by 80.4% on Object HalBench and 52.6% on MM HalBech. |
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing large vision-language models suffer from hallucination due to over-reliance on the Large Language Model (LLM) backbone. |
| Approach: | They propose a method to improve visual context learning by using a large-scale preference learning algorithm to improve hallucination. |
| Outcome: | The proposed method improves on human-annotated hallucination datasets. |
Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Vision Language Models are not free from the issue of Object Hallucination (OH) OH is a phenomenon where LVLMs generate hallucinated objects and descriptions in their outputs. |
| Approach: | They propose a method to suppress OH by referencing images from AI-generated images at the logit level. |
| Outcome: | The proposed method significantly improves existing methods on visual contrast decoding. |
LiPO: Listwise Preference Optimization through Learning-to-Rank (2025.naacl-long)
Copied to clipboard
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J Liu, Xuanhui Wang
| Challenge: | Recent work on language models with curated feedback provides promising alternatives to RLHF . multiple responses can be ranked by reward models or AI feedback, but there is no study on directly fitting upon a list of responses. |
| Approach: | They propose a method that aligns language models with curated human feedback . they propose SLiC and DPO as promising alternatives to traditional RLHF . |
| Outcome: | The proposed method outperforms DPO and SLiC on several preference alignment tasks with curated and real rankwise preference data. |
Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD) (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) often produce object hallucinations due to their reliance on text cues and learned object co-occurrence biases. |
| Approach: | They propose a language-contrasting decoding algorithm that adjusts LVLM outputs based on LLM confidence levels to mitigate object hallucinations. |
| Outcome: | The proposed method shows up to %4 improvement in POPE F1 scores and %36 reduction in CHAIR scores on COCO validation set while improving captioning quality scores. |
MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LVLMs have shown remarkable performance in visual-language understanding for downstream multimodal tasks. |
| Approach: | They propose a method to alleviate hallucinations by masking the “image heads” in LVLMs . |
| Outcome: | The proposed method alleviates the phenomenon of hallucinations and retains the general capabilities of LVLMs. |
Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decoding (2026.eacl-srw)
Copied to clipboard
| Challenge: | Recent studies improve visual contrastive decoding (VCD) by constructing more informative auxiliary views. |
| Approach: | They propose to construct an object-aligned auxiliary view that disrupts unsupported tokens and produces a stronger contrast signal. |
| Outcome: | Empirically, the proposed method shows consistent gains on two popular object hallucination benchmarks across two MLLMs. |
Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for hallucination mitigation are based on external dependency and require external annotations or auxiliary models for preference data collection. |
| Approach: | a new method is proposed to help model-generated hallucinations without external dependencies. |
| Outcome: | a new method that self-injects hallucinations into a generated response improves halluuutations mitigation. |
Evaluating and Mitigating Object Hallucination in Large Vision-Language Models: Can They Still See Removed Objects? (2025.naacl-long)
Copied to clipboard
| Challenge: | LVLMs often mistakenly determine objects as present in images where they do not exist . authors propose a new benchmark to evaluate object hallucinations by removing objects from images and asking the model whether it can still see the removed objects. |
| Approach: | They propose a benchmark to evaluate object hallucinations by removing objects from images . they propose oDPO, a direct preference optimization objective based on visual objects . |
| Outcome: | The proposed benchmark reduces the likelihood of object hallucinations by removing objects from images and asking the model whether it can still see the removed objects. |
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large vision-language models produce hallucinations that compromise output reliability. |
| Approach: | They propose a dual-stage framework for mitigating hallucinations without performance degradation . they propose semantic-aware component disentanglement and interpretable parameter updates . |
| Outcome: | The proposed model reduces hallucinations by 23.4% while maintaining 97.4% of general generative capability. |