Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Medical Large Vision-Language Models (Med-LVLMs) lack visual localization in medical images, which is crucial for abnormality detection and interpretation. |
| Approach: | They propose a medical abnormalities unveiling method based on a Medical Abnormalities Unveiler dataset and propose 'abnormal-aware instruction tuning' and 'abbnormal-Aware Reward' method generates diagnoses based upon identified abnormal areas in medical images. |
| Outcome: | The proposed method outperforms existing medical large vision-language models in identifying and understanding medical abnormalities and improves generalization capability. |
Similar Papers
Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale (2024.emnlp-main)
Copied to clipboard
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, Benyou Wang
| Challenge: | Multimodal large language models (MLLMs) lack visual knowledge in medical applications due to data privacy concerns and high annotation costs. |
| Approach: | They refined medical image-text pairs from PubMed and employed MLLMs (GPT-4V) to denoise and reformat the data. |
| Outcome: | The proposed model significantly improves the MMMU Health & Medicine track and shows that it can be used in multimodal scenarios. |
Self-Training Large Language and Vision Assistant for Medical Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for collecting medical data are expensive and time-consuming. |
| Approach: | They propose a method to train a large-scale LVLM capable of auto-generating medical visual instruction data to improve data efficiency. |
| Outcome: | The proposed method shows that it performs well across three major visual question answering (VQA) benchmarks. |
Medical Vision-Language Pre-Training for Brain Abnormalities (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing vision-language models lack expertise for medical applications due to the scarcity and complexity of data. |
| Approach: | They propose a pipeline to collect medical image-text aligned data for pretraining from public resources such as PubMed and build a high-performance vision-language model tailored to specific medical tasks. |
| Outcome: | The proposed model is based on a large brain image-text dataset and will be released to the public. |
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data. |
| Approach: | They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages. |
| Outcome: | The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks. |
Beyond Surface Features: Advancing Medical Vision-Language Alignment via Dynamic Evidence-Guided Preference Optimization (2026.acl-long)
Copied to clipboard
Zixuan Huang, Zhihong Zhu, Xiaolong Liu, Yanchao Hao, Manman Zhang, Zheng Wei, Bowen Xing, Xian Wu, Ye Li, Fen Miao, Yefeng Zheng
| Challenge: | Existing preference-based methods for medical large vision-Language Models face limitations in medical settings . existing methods are limited by overfitting to superficial cues and pseudo convergence of the preference signal. |
| Approach: | They propose a framework that enables evidence-aware and adaptive preference learning for Med-LVLMs. |
| Outcome: | The proposed framework improves evidence-aware and adaptive preference learning for Med-LVLMs. |
LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models (2025.findings-naacl)
Copied to clipboard
Zhenyue Qin, Yu Yin, Dylan Campbell, Xuansheng Wu, Ke Zou, Ninghao Liu, Yih Chung Tham, Xiuzhen Zhang, Qingyu Chen
| Challenge: | Existing benchmarks for large vision-language models (LVLMs) are limited to ophthalmology-specific applications. |
| Approach: | They introduce a large-scale multimodal ophthalmology benchmark consisting of 21,993 instances across five ocular imaging modalities and 13 state-of-the-art LVLM representatives from closed-source, open-source and medical domains. |
| Outcome: | The proposed model shows significant performance drop in ophthalmology compared to other domains. |
Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging (2025.acl-long)
Copied to clipboard
Zhenyang Cai, Junying Chen, Rongsheng Wang, Weihong Wang, Yonglin Deng, Dingjie Song, Yize Chen, Zixu Zhang, Benyou Wang
| Challenge: | Current research suggests that multitask training outperforms single-task as different tasks can benefit each other, but they often overlook the internal relationships within these tasks. |
| Approach: | They employ compositional generalization (CG) to examine the generalization of multimodal large language models in medical imaging. |
| Outcome: | The proposed model can understand unseen medical images and is able to perform CG across classification and detection tasks. |
MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets and models fail to consider critical aspects of medical diagnostics, authors argue . MMXU enables multi-image questions incorporating both current and historical patient data. |
| Approach: | They propose a dataset for MedVQA that focuses on identifying changes in specific regions between two patient visits. |
| Outcome: | The proposed dataset improves diagnostic accuracy by 20% by integrating historical data. |
Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. |
| Approach: | They propose to use a multiple granularity attribute-centric benchmark and training mixture to evaluate LVLMs’ fine-grained visual comprehension ability. |
| Outcome: | The proposed model improves on LLaVa-1.5, InstructBLIP and GPT-4V and demonstrates that they struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. |