Zero-shot Visual Question Answering with Language Model Feedback (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge-based visual question answering are based on pre-trained language models. |
| Approach: | They propose a language model guided captioning approach that leverages a pre-trained language model to generate captions for an image to help answer a visual question. |
| Outcome: | The proposed method outperforms several competing methods on the knowledge-based VQA task and achieves comparable results to a fine-tuned VLP model. |
Similar Papers
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches require substantial adaptation of pretrained language models for vision-language reasoning tasks. |
| Approach: | They propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. |
| Outcome: | The proposed framework outperforms the Flamingo model on VQAv2 and GQA by 8.5%. |
Modularized Zero-shot VQA with Pre-trained Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on zero-shot visual question answering does not explicitly consider multi-step reasoning chains, making them less interpretable compared with a decomposition-based approach. |
| Approach: | They propose a modularized zero-shot network that explicitly decomposes questions into sub reasoning steps and is highly interpretable. |
| Outcome: | The proposed model decomposes questions into sub reasoning steps and is highly interpretable. |
Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Visual question answering (VQA) is a task that requires an understanding of both the image and the question to provide a natural language answer. |
| Approach: | They propose a multimodal framework that leverages language guidance to answer questions more accurately. |
| Outcome: | The proposed framework improves on the multi-choice question-answering task using CLIP and BLIP models. |
Knowledge Generation for Zero-shot Knowledge-based VQA (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent knowledge-based visual question answering methods do not explicitly show the knowledge needed to answer the questions and therefore lack interpretability. |
| Approach: | They propose a method which generates knowledge from an LLM and incorporates it into a zero-shot manner. |
| Outcome: | The proposed method performs better than previous zero-shot K-VQA methods on two benchmarks and is generally relevant and helpful. |
Modular Visual Question Answering via Code Generation (2023.acl-short)
Copied to clipboard
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein
| Challenge: | a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills . |
| Approach: | They propose a framework that formulates visual question answering as modular code generation. |
| Outcome: | The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation. |
Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering (2020.coling-main)
Copied to clipboard
| Challenge: | Existing text VQA systems generate an answer by selecting from optical character recognition (OCR) texts or a fixed vocabulary. |
| Approach: | They propose a localization-aware answer prediction network that generates the answer and predicts a bounding box as evidence of the generated answer. |
| Outcome: | The proposed network outperforms existing methods on three benchmark datasets for the text VQA task by a noticeable margin. |
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models (LMs) have a surprising ability to perform zero-shot learning. |
| Approach: | They propose to fine-tune pre-trained language models to optimize the zero-shot learning objective by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering format. |
| Outcome: | The proposed model outperforms a same-sized QA model and the previous SOTA zero-shot learning system on unseen tasks. |
Pre-trained Language Models Can be Fully Zero-Shot Learners (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to pre-trained language models require fine-tuning on labeled datasets or manually constructing proper prompts. |
| Approach: | They propose a nonparametric prompting PLM for fully zero-shot language understanding . they compare it to previous methods for text classification and text entailment . |
| Outcome: | The proposed method outperforms previous methods on diverse tasks. |
A Simple Baseline for Knowledge-Based Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies emphasize the importance of incorporating both explicit and implicit knowledge to answer questions requiring external knowledge. |
| Approach: | They propose a pipeline that incorporates both explicit and implicit knowledge . their method is training-free and does not require access to external databases or APIs . |
| Outcome: | The proposed method achieves state-of-the-art accuracy on OK-VQA and A-OK-VQ datasets. |
Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge (2023.findings-acl)
Copied to clipboard
Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, Bing Xiang
| Challenge: | Open-ended Visual Question Answering (VQA) requires models to reason over visual and natural language inputs using world knowledge. |
| Approach: | They propose a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. |
| Outcome: | The proposed pipeline expands the knowledge coverage from in-domain training data by 4.1% on OK-VQA, without additional computation cost. |