Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge in natural language tasks. |
| Approach: | They propose a framework that enables LLMs to ask relevant questions to uncover more details in the image, along with filters for refining the generated information. |
| Outcome: | The proposed framework boosts the performance of baseline methods by 2.15% on OK-VQA and achieves consistent improvements across different LLMs. |
Similar Papers
Learning to Ask Denotative and Connotative Questions for Knowledge-based VQA (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have attracted increasing attention due to their prominent performance on various tasks. |
| Approach: | They propose to let LLMs learn to ask informative questions to collect visual information . they introduce concepts of denotation and connotation to promote image and question understanding . |
| Outcome: | The proposed model can generate high-quality questions and efficiently collect required information without expensive training or annotations. |
When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision and language models have demonstrated remarkable performance in visual question answering tasks. |
| Approach: | They introduce a framework to optimize L-VLMs by leveraging unlabeled images . they conduct extensive experiments on four diverse VQA benchmarks . |
| Outcome: | The proposed framework improves L-VLMs on four visual question answering benchmarks. |
Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Visual question answering (VQA) is a task that requires an understanding of both the image and the question to provide a natural language answer. |
| Approach: | They propose a multimodal framework that leverages language guidance to answer questions more accurately. |
| Outcome: | The proposed framework improves on the multi-choice question-answering task using CLIP and BLIP models. |
Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods address this issue by introducing an auxiliary task such as visual grounding, cycle consistency, or debiasing. |
| Approach: | They propose a data augmentation pipeline to turn “known” knowledge into training examples for VQA. |
| Outcome: | The proposed model can handle multi-modal information and is based on human-annotated examples. |
Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for dialogue-to-image retrieval are constrained by pre-trained vision language models. |
| Approach: | They leverage the reasoning capabilities of large language models to predict potential features in images to be shared based on dialogue context. |
| Outcome: | The proposed method outperforms existing methods significantly in terms of Recall@k. |
From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Tasks (2025.naacl-long)
Copied to clipboard
Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, Jieping Ye
| Challenge: | Large visionlanguage models (LVLMs) are a powerful visual-language reasoning tool. |
| Approach: | They propose to integrate attention analysis with LLaVA-CAM to determine interactions between visual representations. |
| Outcome: | The proposed approach can be used to determine interactions between visual representations. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |
A Simple Baseline for Knowledge-Based Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies emphasize the importance of incorporating both explicit and implicit knowledge to answer questions requiring external knowledge. |
| Approach: | They propose a pipeline that incorporates both explicit and implicit knowledge . their method is training-free and does not require access to external databases or APIs . |
| Outcome: | The proposed method achieves state-of-the-art accuracy on OK-VQA and A-OK-VQ datasets. |
Modality-Aware Integration with Large Language Models for Knowledge-Based Visual Question Answering (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to integrate multimodal knowledge in a modality-agnostic manner can be sub-optimal. |
| Approach: | They propose a modality-aware integration with large language models (LLMs) that leverages multimodal knowledge for both image understanding and knowledge reasoning. |
| Outcome: | The proposed model is able to bridge a tight inter-modal exchange while preserving insightful intra-modal learning. |
Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Existing research addresses ambiguous visual questions by rephrasing questions, but it fails to address the inherently interactive nature of user interactions with visual language models (VLMs). Existing studies focus on re-phrase questions, and lack of a benchmark to assess VLMs’ capacity for resolving ambiguities through interaction. |
| Approach: | They propose a visual question answering task that provides a natural language answer to a question based on a given image and an automated pipeline to generate ambiguity-clarification question pairs. |
| Outcome: | The proposed benchmark targets three common categories of ambiguity in visual question answering (VQA) context and encompasses various VQA scenarios. |