Multi-modal Concept Alignment Pre-training for Generative Medical Visual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Medical Visual Question Answering (Med-VQA) aims to provide accurate answers to questions regarding medical images, a task particularly challenging for open-ended questions. |
| Approach: | They propose a multi-modal concept alignment pre-training approach for generative Med-VQA that leverages a knowledge graph sourced from medical image-caption datasets and the Unified Medical Language System. |
| Outcome: | The proposed approach significantly outperforms existing methods on a set of benchmark datasets and shows high efficiency and knowledge-image alignment capability. |
Similar Papers
MedThink: A Rationale-Guided Framework for Explaining Medical Visual Question Answering (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models for medical visual question answering are limited in their interpretation and interpretation . a semi-automated annotation process is used to streamline data preparation and build new benchmark datasets . |
| Approach: | They propose a semi-automated annotation process to streamline data preparation and build new benchmark Med-VQA datasets. |
| Outcome: | The proposed method achieves an accuracy of 83.5% on R-RAD, 86.3% on RSLAKE and 87.2% on RPath. |
Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to visual question answering (VQA) are not suitable for real-world applications. |
| Approach: | They propose a supervised multi-modal domain adaptation method for visual question answering in images that exploits supervised domain adaptation. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the benchmark VQA 2.0 and VizWiz datasets. |
MLeVLM: Improve Multi-level Progressive Capabilities based on Multimodal Large Language Model for Medical Visual Question Answering (2024.findings-acl)
Copied to clipboard
Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hanpin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, Yu Huang
| Challenge: | Existing MVQA models ignore multi-level progressive capabilities due to unspecific data and plain architecture. |
| Approach: | They propose a multi-level visual language model for medical visual question answering (MVQA) which covers multi- level questions and answers as well as reasoning processes from visual clues to semantic cognition. |
| Outcome: | The proposed model outperforms existing medical multimodal large language models on a multi-level instruction dataset and a feature alignment module. |
Seeing Beyond: Enhancing Visual Question Answering with Multi-Modal Retrieval (2025.coling-industry)
Copied to clipboard
| Challenge: | Multi-modal Large language models still suffer from model hallucination and lack of specific knowledge when answering challenging questions. |
| Approach: | They propose to use a multi-modal retrieval augmented generation method to integrate knowledge from all modalities into a model to enable alignment between query and knowledge. |
| Outcome: | The proposed method achieves significant performance improvement on the VQA dataset. |
Seeing Is Believing! towards Knowledge-Infused Multi-modal Medical Dialogue Generation (2024.lrec-main)
Copied to clipboard
Abhisek Tiwari, Shreyangshu Bera, Preeti Verma, Jaithra Varma Manthena, Sriparna Saha, Pushpak Bhattacharyya, Minakshi Dhar, Sarbajeet Tiwari
| Challenge: | Existing models of disease diagnosis using AI do not use knowledge infusion. |
| Approach: | They propose a transformer-based, knowledge-infused multi-modal medical dialogue generation framework . they propose 'discourse-aware' image identifier that recognizes signs and their severity . |
| Outcome: | The proposed model outperforms state-of-the-art models by 7.84% in the english language. |
MedEx: Enhancing Medical Question-Answering with First-Order Logic based Reasoning and Knowledge Injection (2025.coling-main)
Copied to clipboard
| Challenge: | Existing knowledge triples are ineffective in medical question-answering because of superfluous data and inability to capture complex relationships between symptoms and treatments. |
| Approach: | They propose a first-order logical reasoning model that uses First-Order Logic to model intricate relationships between diseases and treatments. |
| Outcome: | The proposed model captures the interplay of symptoms, diseases, and treatments, enhancing context comprehension. |
Cross-Modal Retrieval Augmentation for Multi-Modal Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. |
| Approach: | They propose a retrieval-augmented multi-modal transformer architecture for embedding images and captions in the same space. |
| Outcome: | The proposed approach improves visual question answering over strong baselines and hot-swapping indices. |
Mulan: A Multi-Level Alignment Model for Video Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on visual-language alignment at the video level, but they do not account for fine-grained semantic interaction between video and text. |
| Approach: | They propose a multi-level Alignment Model for Video Question Answering that establishes alignment between visual and textual modalities at the object-level, frame-level and video-level. |
| Outcome: | The proposed model outperforms state-of-the-art methods even with a small amount of extra visual-language pre-training data and a reduced number of trainable parameters. |
In Factuality: Efficient Integration of Relevant Facts for Visual Question Answering (2021.acl-short)
Copied to clipboard
| Challenge: | Current Visual Question Answering (VQA) models are trained on labelled data that may be insufficient to learn complex knowledge representations. |
| Approach: | They propose a method to integrate external knowledge into a visual pre-trained model by integrating facts extracted from a knowledge base. |
| Outcome: | The proposed method outperforms baseline models on the KVQA dataset benchmark by 19% and shows that it is weaker than previous models. |
Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have seen remarkable progress for medical decision-making, however, they are designated for specific classification or generative tasks and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing. |
| Approach: | They propose a framework that tackles discriminative and generative multimodal medical tasks using multimodal alignment, instruction tuning and routing. |
| Outcome: | The proposed model can achieve superior performance to or on par with state-of-the-art baselines while only requiring 30%-50% of activated model parameters. |