Benchmarking and Mitigating the Impact of Noisy User Prompts in Medical VLMs via Cross-Modal Reflection (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing medical vision-language models follow user-provided prompts blindly, a new study finds . current models are noisy, causing problems with reliability in real-world interactions . |
| Approach: | They propose a method to evaluate the influence of clinical prompts on medical vision-language models . they use cross-modal reflection chain-of-thought to train the model to produce reasoning paths . |
| Outcome: | The proposed method significantly improves the robustness against noisy prompts . existing Med-VLMs follow user-provided prompts blindly, the authors show . |
Similar Papers
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
MedThink: A Rationale-Guided Framework for Explaining Medical Visual Question Answering (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models for medical visual question answering are limited in their interpretation and interpretation . a semi-automated annotation process is used to streamline data preparation and build new benchmark datasets . |
| Approach: | They propose a semi-automated annotation process to streamline data preparation and build new benchmark Med-VQA datasets. |
| Outcome: | The proposed method achieves an accuracy of 83.5% on R-RAD, 86.3% on RSLAKE and 87.2% on RPath. |
Few shot chain-of-thought driven reasoning to prompt LLMs for open-ended medical question answering (2024.findings-emnlp)
Copied to clipboard
Saeel Nachane, Ojas Gramopadhye, Prateek Chanda, Ganesh Ramakrishnan, Kshitij Jadhav, Yatin Nandwani, Dinesh Raghu, Sachindra Joshi
| Challenge: | Large Language models (LLMs) are increasingly utilized in the healthcare sector for query-related tasks. |
| Approach: | They propose a modified version of the MedQA-USMLE dataset, named MEDQA-OPEN, which contains open-ended medical questions without options to mimic clinical scenarios. |
| Outcome: | The proposed approach outperforms the state-of-the-art 5-shot CoT-based prompt by exploring multiple differential diagnoses and narrowing down to a final diagnosis using MCQ-ELIMINATIVE. |
Evaluating Large Vision Language Models on Bangla Medical Visual Question Answering (2026.findings-acl)
Copied to clipboard
Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Mahir Jawad, Anam Borhan Uddin, Md Fahim, Md Farhad Alam Bhuiyan
| Challenge: | Recent advances in Large Language Models and Large Vision Language Model (LVLMs) have demonstrated promising capabilities in complex reasoning tasks, but low-resource contexts like Bangla are underexplored. |
| Approach: | They propose a multilingual medical visual question answering dataset using Bangla. |
| Outcome: | The proposed model performs well on generalized visual tasks but struggles with fine-grained diagnostic reasoning, achieving low accuracy in specialized categories. |
Multi-modal Concept Alignment Pre-training for Generative Medical Visual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Medical Visual Question Answering (Med-VQA) aims to provide accurate answers to questions regarding medical images, a task particularly challenging for open-ended questions. |
| Approach: | They propose a multi-modal concept alignment pre-training approach for generative Med-VQA that leverages a knowledge graph sourced from medical image-caption datasets and the Unified Medical Language System. |
| Outcome: | The proposed approach significantly outperforms existing methods on a set of benchmark datasets and shows high efficiency and knowledge-image alignment capability. |
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks. |
| Approach: | They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases . |
| Outcome: | The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases. |
When Background Matters: Breaking Medical Vision Language Models by Transferable Attack (2026.acl-long)
Copied to clipboard
| Challenge: | Existing medical attacks focus on secondary objectives such as model stealing or adversarial fine-tuning, while transferable attacks from natural images introduce visible distortions that clinicians can easily detect. Existing transferable adversarials are less effective in the medical domain. |
| Approach: | They propose a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. |
| Outcome: | The proposed method induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. |
Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have demonstrated impressive performance on existing medical visual question answering benchmarks. |
| Approach: | They evaluate large multimodal models that perform worse than random guessing on medical questions . authors suggest more robust evaluation methods to ensure reliability of LMMs . |
| Outcome: | a new study shows that large multimodal models perform worse than random guessing on medical visual question answering benchmarks. |
Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that Vision-Language Models have robust multimodal reasoning capabilities, but their robustness against textual misinformation remains under-explored. |
| Approach: | They propose to use visual-question-answering (VQA) prompts to generate persuasive prompts that deliberately conflict with visual evidence to test their models. |
| Outcome: | The proposed framework shows that models are vulnerable to misleading prompts, and show an average performance drop of over 48.2% after only one round of persuasive conversation. |
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Pretrained biomedical vision–language models perform well on average but often degrade on challenging modalities. |
| Approach: | They propose a prompt-learning framework that improves cross-domain generalization without extensive backbone fine-tuning. |
| Outcome: | BioVLM learns a diverse prompt bank and introduces dynamic prompt selection . it can combine sparse few-shot evidence with rich LLM semantic priors . bioVLM achieves state-of-the-art on 11 MedMNIST+ 2D datasets based on the proposed framework . |