WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering rely on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs) |
| Approach: | They propose a multi-modal RAG framework that assigns VLMs two specialized agents: a Refiner and an Inspector. |
| Outcome: | Experiments on EVQA, InfoSeek, and M2KR show that the proposed framework achieves state-of-the-art performance with significant improvements in both retrieval accuracy and answer quality. |
Similar Papers
Seeing Beyond: Enhancing Visual Question Answering with Multi-Modal Retrieval (2025.coling-industry)
Copied to clipboard
| Challenge: | Multi-modal Large language models still suffer from model hallucination and lack of specific knowledge when answering challenging questions. |
| Approach: | They propose to use a multi-modal retrieval augmented generation method to integrate knowledge from all modalities into a model to enable alignment between query and knowledge. |
| Outcome: | The proposed method achieves significant performance improvement on the VQA dataset. |
EchoSight: Advancing Visual-Language Models with Wiki Knowledge (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge-based visual question answering systems struggle with these tasks due to limited integration of external knowledge. |
| Approach: | They propose a framework that enables large language models to answer visual questions requiring encyclopedic knowledge. |
| Outcome: | The proposed framework improves retrieval outcomes and accuracy of knowledge-based visual question answering tasks. |
Multi-Level Information Retrieval Augmented Generation for Knowledge-based Visual Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge-Aware Visual Question Answering about Entity tasks require two separate steps to generate accurate answers. |
| Approach: | They propose a multi-level information RAG approach that enhances answer generation through entity retrieval and query expansion. |
| Outcome: | The proposed approach improves answer generation through entity retrieval and query expansion. |
MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent knowledge-based visual question answering approaches miss visual information captured by captions and cannot fully utilize the visual information required to answer the question. |
| Approach: | They propose a framework that extracts visual information from an image and prompts an LLM to extract query-specific knowledge from the extracted textual information. |
| Outcome: | Empirical results show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets. |
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering lack multimodal retrieval . large language models (LLMs) have demonstrated remarkable generalization and reasoning capabilities in text-based systems. |
| Approach: | They propose a multimodal vision-language retrieval-augmented generation system that harmonizes multiple modalities and modality to enhance retrieval. |
| Outcome: | The proposed system achieves state-of-the-art retrieval performance and competitive answers on InfoSeek and Encyclopedic-VQA benchmarks. |
Cross-Modal Retrieval Augmentation for Multi-Modal Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. |
| Approach: | They propose a retrieval-augmented multi-modal transformer architecture for embedding images and captions in the same space. |
| Outcome: | The proposed approach improves visual question answering over strong baselines and hot-swapping indices. |
ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on image-based question answering (QA) but ignore the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. |
| Approach: | They propose a novel multi-agent RAG framework tailored for complex reasoning across visual documents that employs a Gaussian Mixture Model (GMM)-based hybrid strategy to handle multi-modal retrieval. |
| Outcome: | The proposed framework outperforms existing methods by over 10% on the competitive ViDoSeek benchmark. |
MDocRAG-RL: Empowering Multi-Modal Document RAG via Complex Visual Reasoning with Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RAG systems produce suboptimal embeddings and naively insert images into context without adequate visual perception, limiting reasoning capabilities. |
| Approach: | They propose a novel RAG framework for complex visual reasoning that integrates multimodal large language models with external knowledge to enhance retrieval efficiency. |
| Outcome: | The proposed framework achieves state-of-the-art performance on multiple benchmarks. |
VLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a significant drawback of Vision-language Models is their reliance on static training data, leading to outdated information and limited contextual awareness. |
| Approach: | They propose a framework with knowledge-enhanced reranking and noise-injected training to improve the VLM's ranking ability. |
| Outcome: | The proposed framework is based on a simple yet effective instruction template and is able to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. |
RG-VQA: Leveraging Retriever-Generator Pipelines for Knowledge Intensive Visual Question Answering (2025.findings-emnlp)
Copied to clipboard
Settaluri Lakshmi Sravanthi, Pulkit Agarwal, Debjyoti Mondal, Rituraj Singh, Subhadarshi Panda, Ankit Mishra, Kiran Pradeep, Srihari K B, Godawari Sudhakar Rao, Pushpak Bhattacharyya
| Challenge: | Existing methods to improve the reasoning capabilities of VQA systems are limited due to complexity of graph neural networks and end-to-end training. |
| Approach: | They propose a method to integrate Dense Passage Retrievers with Vision Language Models to boost the reasoning capabilities of VQA systems. |
| Outcome: | The proposed method outperforms human accuracy and GPT-4 in the ScienceQA dataset. |