Papers with OK-VQA
KAT: A Knowledge Augmented Transformer for Vision-and-Language (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge retrieval and answer prediction have left open questions about the quality and relevance of the retrieved knowledge and how the reasoning processes over implicit and explicit knowledge should be integrated. |
| Approach: | They propose a Knowledge Augmented Transformer which integrates both implicit and explicit knowledge in an encoder-decoder architecture while simultaneously reasoning over both knowledge sources during answer generation. |
| Outcome: | The proposed model achieves a strong state-of-the-art (+6% absolute) on the open-domain multimodal task of OK-VQA. |
Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge (2023.findings-acl)
Copied to clipboard
Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, Bing Xiang
| Challenge: | Open-ended Visual Question Answering (VQA) requires models to reason over visual and natural language inputs using world knowledge. |
| Approach: | They propose a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. |
| Outcome: | The proposed pipeline expands the knowledge coverage from in-domain training data by 4.1% on OK-VQA, without additional computation cost. |
Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge in natural language tasks. |
| Approach: | They propose a framework that enables LLMs to ask relevant questions to uncover more details in the image, along with filters for refining the generated information. |
| Outcome: | The proposed framework boosts the performance of baseline methods by 2.15% on OK-VQA and achieves consistent improvements across different LLMs. |
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to knowledge-intensive visual question answering lack mechanisms to revise misdirected reasoning. |
| Approach: | They propose a framework that progressively constructs a structured reasoning trajectory . they use dual-scope queries to retrieve diverse knowledge from heterogeneous knowledge bases . |
| Outcome: | The proposed framework improves retrieval recall and end-to-end answer accuracy. |