Seeing Beyond: Enhancing Visual Question Answering with Multi-Modal Retrieval (2025.coling-industry)
Copied to clipboard
| Challenge: | Multi-modal Large language models still suffer from model hallucination and lack of specific knowledge when answering challenging questions. |
| Approach: | They propose to use a multi-modal retrieval augmented generation method to integrate knowledge from all modalities into a model to enable alignment between query and knowledge. |
| Outcome: | The proposed method achieves significant performance improvement on the VQA dataset. |
Similar Papers
WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering rely on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs) |
| Approach: | They propose a multi-modal RAG framework that assigns VLMs two specialized agents: a Refiner and an Inspector. |
| Outcome: | Experiments on EVQA, InfoSeek, and M2KR show that the proposed framework achieves state-of-the-art performance with significant improvements in both retrieval accuracy and answer quality. |
Cross-Modal Retrieval Augmentation for Multi-Modal Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. |
| Approach: | They propose a retrieval-augmented multi-modal transformer architecture for embedding images and captions in the same space. |
| Outcome: | The proposed approach improves visual question answering over strong baselines and hot-swapping indices. |
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering lack multimodal retrieval . large language models (LLMs) have demonstrated remarkable generalization and reasoning capabilities in text-based systems. |
| Approach: | They propose a multimodal vision-language retrieval-augmented generation system that harmonizes multiple modalities and modality to enhance retrieval. |
| Outcome: | The proposed system achieves state-of-the-art retrieval performance and competitive answers on InfoSeek and Encyclopedic-VQA benchmarks. |
Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge-Aware Visual Question Answering about Entity tasks require two separate steps to generate accurate answers. |
| Approach: | They propose a multi-level information RAG approach that enhances answer generation through entity retrieval and query expansion. |
| Outcome: | The proposed approach improves answer generation through entity retrieval and query expansion. |
RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing ranking methods rely on small encoder-based ranking models, which are incompatible with modern decoder--based generative large language models (LLMs) Existing methods based on small LLaVA rankers are incompatible with advanced LLMs. |
| Approach: | They propose a framework that combines learning-to-rank methods with generative permutation-enhanced ranking techniques. |
| Outcome: | The proposed framework improves on two benchmarks, WebQA and MultiModalQA, showing significant improvements over baselines. |
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to Visual Question Answering lack synergistic potential of scene graphs and scene graph. |
| Approach: | They propose a retrieval-and-fusion pipeline that fuses scene graphs and commonsense graphs to enable multi-modal reasoning. |
| Outcome: | Experiments on FVQA 2.0+ and MVQA benchmarks show that KG-ViP outperforms existing methods. |
In Factuality: Efficient Integration of Relevant Facts for Visual Question Answering (2021.acl-short)
Copied to clipboard
| Challenge: | Current Visual Question Answering (VQA) models are trained on labelled data that may be insufficient to learn complex knowledge representations. |
| Approach: | They propose a method to integrate external knowledge into a visual pre-trained model by integrating facts extracted from a knowledge base. |
| Outcome: | The proposed method outperforms baseline models on the KVQA dataset benchmark by 19% and shows that it is weaker than previous models. |
MDocRAG-RL: Empowering Multi-Modal Document RAG via Complex Visual Reasoning with Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RAG systems produce suboptimal embeddings and naively insert images into context without adequate visual perception, limiting reasoning capabilities. |
| Approach: | They propose a novel RAG framework for complex visual reasoning that integrates multimodal large language models with external knowledge to enhance retrieval efficiency. |
| Outcome: | The proposed framework achieves state-of-the-art performance on multiple benchmarks. |
EchoSight: Advancing Visual-Language Models with Wiki Knowledge (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge-based visual question answering systems struggle with these tasks due to limited integration of external knowledge. |
| Approach: | They propose a framework that enables large language models to answer visual questions requiring encyclopedic knowledge. |
| Outcome: | The proposed framework improves retrieval outcomes and accuracy of knowledge-based visual question answering tasks. |
Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to integrate multimodal knowledge in a modality-agnostic manner can be sub-optimal. |
| Approach: | They propose a modality-aware integration with large language models (LLMs) that leverages multimodal knowledge for both image understanding and knowledge reasoning. |
| Outcome: | The proposed model is able to bridge a tight inter-modal exchange while preserving insightful intra-modal learning. |