Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods rely on inference-time interventions, which are limited in attention adaptation or require additional supervision. |
| Approach: | They propose a framework for automatic attention alignment tuning that leverages weak labels from SAM and selectively modifies visually-critical attention heads to improve alignment while minimizing interference. |
| Outcome: | The proposed framework outperforms state-of-the-art models on medical VQA and report generation benchmarks. |
Similar Papers
Guiding Medical Vision-Language Models with Diverse Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations (2025.naacl-long)
Copied to clipboard
| Challenge: | Current vision-language models lack the ability to focus on specific areas designated by humans . a new framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation is proposed to improve visual prompt-guided fine-tuning. |
| Approach: | They propose to use visual prompts to guide and enhance formation of region-specific attention. |
| Outcome: | The proposed framework outperforms state-of-the-art large vision-language models on medical datasets. |
Beyond Surface Features: Advancing Medical Vision-Language Alignment via Dynamic Evidence-Guided Preference Optimization (2026.acl-long)
Copied to clipboard
Zixuan Huang, Zhihong Zhu, Xiaolong Liu, Yanchao Hao, Manman Zhang, Zheng Wei, Bowen Xing, Xian Wu, Ye Li, Fen Miao, Yefeng Zheng
| Challenge: | Existing preference-based methods for medical large vision-Language Models face limitations in medical settings . existing methods are limited by overfitting to superficial cues and pseudo convergence of the preference signal. |
| Approach: | They propose a framework that enables evidence-aware and adaptive preference learning for Med-LVLMs. |
| Outcome: | The proposed framework improves evidence-aware and adaptive preference learning for Med-LVLMs. |
Don’t Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Vision Language Models suffer from hallucinations, attributing incorrect or misleading features to images. |
| Approach: | They propose a test-time approach that recalibrates the influence of blind tokens . they identify blind token by analyzing layer-wise attention distributions over image tokens. |
| Outcome: | The proposed approach reduces hallucinations in large vision language models . it uses a contrastive decoding strategy to balance the influence of blind tokens . |
A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment (2025.acl-long)
Copied to clipboard
Jean-Philippe Corbeil, Amin Dada, Jean-Michel Attendu, Asma Ben Abacha, Alessandro Sordoni, Lucas Caccia, Francois Beaulieu, Thomas Lin, Jens Kleesiek, Paul Vozila
| Challenge: | Large language models such as GPT-4 have limited their deployment in clinical settings . a novel framework for adapting SLMs into high-performing clinical models is needed . |
| Approach: | They propose a framework for adapting large language models into high-performing clinical models . they pre-instruct experts on relevant medical and clinical corpora and model merging . |
| Outcome: | The proposed framework outperforms the existing model on the CLUE+ benchmark on medical entities and radiology reports. |
Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have seen remarkable progress for medical decision-making, however, they are designated for specific classification or generative tasks and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing. |
| Approach: | They propose a framework that tackles discriminative and generative multimodal medical tasks using multimodal alignment, instruction tuning and routing. |
| Outcome: | The proposed model can achieve superior performance to or on par with state-of-the-art baselines while only requiring 30%-50% of activated model parameters. |
Looking Beyond Text: Reducing Language Bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) have been criticized for their language bias. |
| Approach: | They propose to use a dual-attention mechanism to construct separate attention for visual and text inputs to enhance integration of visual inputs across models. |
| Outcome: | Experiments show that the proposed model debiases LVLMs from their language bias, enhancing visual comprehension and reducing hallucinations without additional resources. |
MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Medical Vision-Language Models are predominantly trained on professional literature, limiting their ability to communicate findings in the lay register required for patient-centered care. |
| Approach: | They propose a multimodal benchmark dedicated to expert-lay semantic alignment that enforces strict semantic equivalence by integrating unified medical language system (UMS) Concept Unique Identifiers (CUIs) with micro-level entity constraints. |
| Outcome: | The proposed benchmark enforces strict semantic equivalence by integrating unified medical language system (UMLS) Concept Unique Identifiers (CUIs) with micro-level entity constraints. |
That’s the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Data (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent multimodal models induce soft local alignments between image regions and sentences. |
| Approach: | They compare alignments from a state-of-the-art multimodal model for EHR with human annotations that link image regions to sentences. |
| Outcome: | The proposed models induce soft local alignments between image regions and sentences . the text has an often weak or unintuitive influence on attention, the authors found . |
Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision–Language Models (VLMs) have demonstrated strong capabilities in tasks that require joint understanding of text and images. |
| Approach: | They propose a strategy that incorporates head-wise attention perturbation via continuous multiplicative noise coupled with a visual-guided loss focused on vision-sensitive text tokens to promote a more balanced attention distribution. |
| Outcome: | The proposed approach outperforms baseline models on three benchmarks and consistently outperformed the baseline model. |
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Pretrained biomedical vision–language models perform well on average but often degrade on challenging modalities. |
| Approach: | They propose a prompt-learning framework that improves cross-domain generalization without extensive backbone fine-tuning. |
| Outcome: | BioVLM learns a diverse prompt bank and introduces dynamic prompt selection . it can combine sparse few-shot evidence with rich LLM semantic priors . bioVLM achieves state-of-the-art on 11 MedMNIST+ 2D datasets based on the proposed framework . |