Beyond Surface Features: Advancing Medical Vision-Language Alignment via Dynamic Evidence-Guided Preference Optimization (2026.acl-long)
Copied to clipboard
Zixuan Huang, Zhihong Zhu, Xiaolong Liu, Yanchao Hao, Manman Zhang, Zheng Wei, Bowen Xing, Xian Wu, Ye Li, Fen Miao, Yefeng Zheng
| Challenge: | Existing preference-based methods for medical large vision-Language Models face limitations in medical settings . existing methods are limited by overfitting to superficial cues and pseudo convergence of the preference signal. |
| Approach: | They propose a framework that enables evidence-aware and adaptive preference learning for Med-LVLMs. |
| Outcome: | The proposed framework improves evidence-aware and adaptive preference learning for Med-LVLMs. |
Similar Papers
Benchmarking Direct Preference Optimization for Medical Large Vision–Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) are gaining traction in clinical tasks such as diagnostic support, report generation, and medical question answering. |
| Approach: | They present a systematic evaluation of nine DPO variants applied to two leading medical LVLMs. |
| Outcome: | The proposed model improves alignment and reduces severe hallucinations, but yields inconsistent gains over supervised fine-tuning. |
Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have seen remarkable progress for medical decision-making, however, they are designated for specific classification or generative tasks and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing. |
| Approach: | They propose a framework that tackles discriminative and generative multimodal medical tasks using multimodal alignment, instruction tuning and routing. |
| Outcome: | The proposed model can achieve superior performance to or on par with state-of-the-art baselines while only requiring 30%-50% of activated model parameters. |
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing large vision-language models suffer from hallucination due to over-reliance on the Large Language Model (LLM) backbone. |
| Approach: | They propose a method to improve visual context learning by using a large-scale preference learning algorithm to improve hallucination. |
| Outcome: | The proposed method improves on human-annotated hallucination datasets. |
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu
| Challenge: | emergence of large Vision Language Models (VLMs) has broadened the capabilities of single-modal Large Language Model (LLM) but VLMs are prone to significant hallucinations, especially in the form of cross-modal inconsistencies. |
| Approach: | They propose a new alignment framework that leverages image retrieval to integrate both textual and visual preference signals. |
| Outcome: | The proposed framework mitigates hallucinations more effectively than previous methods . it maintains robustness and scalability across a wide range of VLM sizes and architectures . |
Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods rely on inference-time interventions, which are limited in attention adaptation or require additional supervision. |
| Approach: | They propose a framework for automatic attention alignment tuning that leverages weak labels from SAM and selectively modifies visually-critical attention heads to improve alignment while minimizing interference. |
| Outcome: | The proposed framework outperforms state-of-the-art models on medical VQA and report generation benchmarks. |
EAGLE: Expert-Guided Self-Enhancement for Preference Alignment in Pathology Large Vision-Language Model (2025.acl-long)
Copied to clipboard
Meidan Ding, Jipeng Zhang, Wenxuan Wang, Haiqin Zhong, Xiaoqin Wang, Xinheng Lyu, Wenting Chen, Linlin Shen
| Challenge: | Recent advances in Large Vision Language Models (LVLMs) show promise for pathological diagnosis, yet their application in clinical settings faces critical challenges of multimodal hallucination and biased responses. |
| Approach: | They propose a framework that integrates medical expertise into preference alignment. |
| Outcome: | The proposed framework outperforms existing pathological LVLMs while maintaining pathological accuracy. |
Self-Training Large Language and Vision Assistant for Medical Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for collecting medical data are expensive and time-consuming. |
| Approach: | They propose a method to train a large-scale LVLM capable of auto-generating medical visual instruction data to improve data efficiency. |
| Outcome: | The proposed method shows that it performs well across three major visual question answering (VQA) benchmarks. |
mDPO: Conditional Preference Optimization for Multimodal Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. |
| Approach: | They propose a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference. |
| Outcome: | The proposed method significantly improves performance on two multimodal LLMs of different sizes and three widely used benchmarks. |
Joint Multimodal Preference Optimization for Fine-Grained Visual-Textual Alignment (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has focused on addressing multimodal hallucinations in Large Vision-Language Models (LVLMs) however, these methods lack fine-grained visual contrast mechanisms and rely on single-margin optimization. |
| Approach: | They propose a framework that integrates text-conditioned preference loss with visual ranking-based objective. |
| Outcome: | The proposed framework improves cross-modal alignment and fine-grained visual grounding. |
Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization (2025.emnlp-main)
Copied to clipboard
Jiulong Wu, Zhengliang Shi, Shuaiqiang Wang, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao, Min Zhang
| Challenge: | Existing preference alignment methods focus on aligning model responses with human preferences while neglecting image-text modality alignment. |
| Approach: | They propose Entity-centric Multimodal Preference Optimization to improve modality alignment . they use open-source instruction datasets to automatically construct high-quality preference data . |
| Outcome: | The proposed approach reduces hallucination rates by 80.4% on Object HalBench and 52.6% on MM HalBech. |