Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-based visual question answering systems rely on Concept-Net and Wikipedia to obtain external knowledge. |
| Approach: | They propose a visual retriever-reader pipeline that uses a natural language knowledge base and a Visual retriever to retrieve relevant knowledge. |
| Outcome: | The proposed method significantly improves the visual retriever-reader pipeline on the OK-VQA benchmark. |
Similar Papers
RG-VQA: Leveraging Retriever-Generator Pipelines for Knowledge Intensive Visual Question Answering (2025.findings-emnlp)
Copied to clipboard
Settaluri Lakshmi Sravanthi, Pulkit Agarwal, Debjyoti Mondal, Rituraj Singh, Subhadarshi Panda, Ankit Mishra, Kiran Pradeep, Srihari K B, Godawari Sudhakar Rao, Pushpak Bhattacharyya
| Challenge: | Existing methods to improve the reasoning capabilities of VQA systems are limited due to complexity of graph neural networks and end-to-end training. |
| Approach: | They propose a method to integrate Dense Passage Retrievers with Vision Language Models to boost the reasoning capabilities of VQA systems. |
| Outcome: | The proposed method outperforms human accuracy and GPT-4 in the ScienceQA dataset. |
A Simple Baseline for Knowledge-Based Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies emphasize the importance of incorporating both explicit and implicit knowledge to answer questions requiring external knowledge. |
| Approach: | They propose a pipeline that incorporates both explicit and implicit knowledge . their method is training-free and does not require access to external databases or APIs . |
| Outcome: | The proposed method achieves state-of-the-art accuracy on OK-VQA and A-OK-VQ datasets. |
Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | a framework that leverages the visual-language model to select key knowledge retrieved by DPR and answer questions improves performance of the baseline on the open-domain Knowledge-based VQA benchmark, OK-VQA. |
| Approach: | They propose a framework that leverages visual-language models to retrieve related knowledge . they use dense passage retrieval to retrieve knowledge related to visual-linguistics . |
| Outcome: | The proposed framework significantly improves the baseline on the open-domain Knowledge-based VQA benchmark, OK-VQA. |
Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing outsideknowledge visual question answering systems lack retrieved knowledge and supervision is weak . |
| Approach: | They propose an Entity-Focused Retrieval model that provides stronger supervision during training and recognizes question-relevant entities to help retrieve more specific knowledge. |
| Outcome: | The proposed model achieves superior retrieval performance on the currently largest outside-knowledge VQA dataset. |
Retrieval Augmented Visual Question Answering with Outside Knowledge (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent systems that use Dense Passage Retrieval (DPR) for visual question answering use Denses trained separately from answer generation. |
| Approach: | They propose a joint training scheme that integrates DPR with answer generation to improve the performance of OK-VQA systems. |
| Outcome: | The proposed model outperforms existing systems with strong Dense Passage Retrieval (DPR) it yields significant improvements in answer quality and computation required for training. |
ConceptBert: Concept-Aware Representation for Visual Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. |
| Approach: | They propose an algorithm which learns a joint Concept-Vision-Language embedding for questions which require common sense knowledge from external structured content. |
| Outcome: | The proposed model is based on the Outer Knowledge-VQA and VQA datasets. |
Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to visual question answering (VQA) are not suitable for real-world applications. |
| Approach: | They propose a supervised multi-modal domain adaptation method for visual question answering in images that exploits supervised domain adaptation. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the benchmark VQA 2.0 and VizWiz datasets. |
WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering rely on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs) |
| Approach: | They propose a multi-modal RAG framework that assigns VLMs two specialized agents: a Refiner and an Inspector. |
| Outcome: | Experiments on EVQA, InfoSeek, and M2KR show that the proposed framework achieves state-of-the-art performance with significant improvements in both retrieval accuracy and answer quality. |
WeaQA: Weak Supervision via Captions for Visual Question Answering (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for training visual question answering models rely on datasets with human-annotated image-quest-answer triplets. |
| Approach: | They propose a method to train models with synthetic Q-A pairs generated procedurally from captions. |
| Outcome: | The proposed method trains models with synthetic Q-A pairs generated from captions on three VQA benchmarks. |
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to visual question answering represent images using pre-trained CNNs . but they rarely provide any insight, apart from the answer, into the VQA process . |
| Approach: | They propose to break up the end-to-end VQA into two steps: explaining and reasoning . they first extract attributes and generate descriptions as explanations for an image . a reasoning module utilizes these explanations in place of the image to infer an answer . |
| Outcome: | The proposed system achieves comparable performance with baselines, but with added benefits of explanability and the ability to improve with higher quality explanations. |