Joint Multimodal Preference Optimization for Fine-Grained Visual-Textual Alignment (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has focused on addressing multimodal hallucinations in Large Vision-Language Models (LVLMs) however, these methods lack fine-grained visual contrast mechanisms and rely on single-margin optimization. |
| Approach: | They propose a framework that integrates text-conditioned preference loss with visual ranking-based objective. |
| Outcome: | The proposed framework improves cross-modal alignment and fine-grained visual grounding. |
Similar Papers
Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization (2025.emnlp-main)
Copied to clipboard
Jiulong Wu, Zhengliang Shi, Shuaiqiang Wang, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao, Min Zhang
| Challenge: | Existing preference alignment methods focus on aligning model responses with human preferences while neglecting image-text modality alignment. |
| Approach: | They propose Entity-centric Multimodal Preference Optimization to improve modality alignment . they use open-source instruction datasets to automatically construct high-quality preference data . |
| Outcome: | The proposed approach reduces hallucination rates by 80.4% on Object HalBench and 52.6% on MM HalBech. |
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu
| Challenge: | emergence of large Vision Language Models (VLMs) has broadened the capabilities of single-modal Large Language Model (LLM) but VLMs are prone to significant hallucinations, especially in the form of cross-modal inconsistencies. |
| Approach: | They propose a new alignment framework that leverages image retrieval to integrate both textual and visual preference signals. |
| Outcome: | The proposed framework mitigates hallucinations more effectively than previous methods . it maintains robustness and scalability across a wide range of VLM sizes and architectures . |
ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances have extended DPO to multimodal scenarios, achieving strong performance. |
| Approach: | They propose to use a sentence-level preference optimization technique to optimize individual sentences for more precise preference optimization without additional models or parameters. |
| Outcome: | Experiments show that Adaptive Sentence-level Preference Optimization significantly improves the alignment of multimodal models. |
Comparing Bad Apples to Good Oranges Aligning Large Language Models via Joint Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that acquiring human preferences by comparing generations is not effective for large language models. |
| Approach: | They propose a preference optimization objective that elicits preferences jointly over the instruction-response pairs. |
| Outcome: | The proposed approach outperforms prior preference optimizations by 5.2% and 3.3% in summarization and open-ended dialogue datasets. |
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are known to hallucinate, which limits their practical applications. |
| Approach: | They propose a method that uses three types of preference pairs to target hallucinations from their diverse forms and causes. |
| Outcome: | The proposed method surpasses most state-of-the-art methods and shows potential for further improvements. |
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing large vision-language models suffer from hallucination due to over-reliance on the Large Language Model (LLM) backbone. |
| Approach: | They propose a method to improve visual context learning by using a large-scale preference learning algorithm to improve hallucination. |
| Outcome: | The proposed method improves on human-annotated hallucination datasets. |
mDPO: Conditional Preference Optimization for Multimodal Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. |
| Approach: | They propose a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference. |
| Outcome: | The proposed method significantly improves performance on two multimodal LLMs of different sizes and three widely used benchmarks. |
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Vision-Language Models (VLMs) tend to neglect image content and over-rely on language-model priors, resulting in errors in visually grounded tasks and hallucinations. |
| Approach: | They propose a novel finetuning objective that steers the model toward capturing important visual details and aligning them with corresponding text tokens. |
| Outcome: | The proposed method achieves up to 22% reduction in hallucinations and significant gains in vision-centric and general tasks while maintaining or improving the model's general abilities. |
Beyond Surface Features: Advancing Medical Vision-Language Alignment via Dynamic Evidence-Guided Preference Optimization (2026.acl-long)
Copied to clipboard
Zixuan Huang, Zhihong Zhu, Xiaolong Liu, Yanchao Hao, Manman Zhang, Zheng Wei, Bowen Xing, Xian Wu, Ye Li, Fen Miao, Yefeng Zheng
| Challenge: | Existing preference-based methods for medical large vision-Language Models face limitations in medical settings . existing methods are limited by overfitting to superficial cues and pseudo convergence of the preference signal. |
| Approach: | They propose a framework that enables evidence-aware and adaptive preference learning for Med-LVLMs. |
| Outcome: | The proposed framework improves evidence-aware and adaptive preference learning for Med-LVLMs. |
Benchmarking Direct Preference Optimization for Medical Large Vision–Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) are gaining traction in clinical tasks such as diagnostic support, report generation, and medical question answering. |
| Approach: | They present a systematic evaluation of nine DPO variants applied to two leading medical LVLMs. |
| Outcome: | The proposed model improves alignment and reduces severe hallucinations, but yields inconsistent gains over supervised fine-tuning. |