Uncovering the Full Potential of Visual Grounding Methods in VQA (2024.acl-long)
Copied to clipboard
| Challenge: | Visual Grounding (VG) methods in VQA aim to strengthen a model's reliance on question-relevant visual information. |
| Approach: | They propose to strengthen a model's reliance on question-relevant visual information by using a visual grounding method that is based on a question-related visual input. |
| Outcome: | The proposed methods can be much more effective when evaluation conditions are corrected. |
Similar Papers
Measuring Faithful and Plausible Visual Grounding in VQA (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lack of visual grounding (VG) in VQA systems can manifest in over-reliance on irrelevant image parts or a disregard for the visual modality entirely. |
| Approach: | They propose a new metric that measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
| Outcome: | The proposed metric measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
A negative case analysis of visual grounding methods for VQA (2020.acl-main)
Copied to clipboard
| Challenge: | Existing Visual Question Answering (VQA) methods exploit dataset biases and spurious statistical correlations instead of producing correct answers for the right reasons. |
| Approach: | They propose to incorporate visual cues to better ground VQA models . they also propose a regularization effect which prevents over-fitting to linguistic priors . |
| Outcome: | The proposed method outperforms existing methods on the Visual Question Answering (VQA) dataset. |
ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is increasingly used in diverse applications where models must provide accurate answers and explanations that humans can easily understand and verify. |
| Approach: | They propose a unified prototypical framework that learns question-aware prototypes that serve as reasoning anchors and applies spatially constrained matching to ensure that the selected evidence is coherent and semantically relevant. |
| Outcome: | The proposed framework yields faithful, fine-grained explanations while maintaining competitive accuracy. |
CommVQA: Situating Visual Question Answering in Communicative Contexts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current visual question answering models are trained on image-question pairs in isolation, but the questions people ask are dependent on their informational needs and prior knowledge about the image content. |
| Approach: | They propose a visual question-answer-as-question dataset that contains 1000 images and 8,949 question-announcer pairs to evaluate how situating images within naturalistic contexts shapes visual questions. |
| Outcome: | The proposed dataset contains 1000 images and 8,949 question-answer pairs. |
What’s Different between Visual Question Answering for Machine “Understanding” Versus for Accessibility? (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing benchmarking datasets for visual question answering focus on machine "understanding" but it remains unclear how progress on those datasets corresponds to improvements in this real-world use case. |
| Approach: | They evaluate the visual question answering task by evaluating a variety of VQA models. |
| Outcome: | The proposed model can achieve high scores on tasks thought to require human-like comprehension, including image tagging and captioning. |
Multi-grained Attention with Object-level Grounding for Visual Question Answering (P19-1)
Copied to clipboard
| Challenge: | Current approaches to visual question answering train attention models from coarse-grained associations between sentences and images, which fail on small objects or uncommon concepts. |
| Approach: | They propose a multi-grained attention method that learns explicit word-object correspondence by word-level attention complementary to the sentence-image association. |
| Outcome: | The proposed method achieves competitive performance with state-of-the-art models on the VQA benchmark. |
Where is this coming from? Making groundedness count in the evaluation of Document VQA models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Document Visual Question Answering (VQA) models have come close to or matching human performance on some benchmarks. |
| Approach: | They propose a method that accounts for the semantic and multimodal groundedness of a model’s outputs and can be parameterized so that users can configure the score according to their preferences. |
| Outcome: | The proposed method produces scores that are a better indicator of a model’s robustness and tends to give higher rewards to better-calibrated answers. |
Selectively Answering Visual Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are capable of visual question answering (VQA) with unprecedented accuracy. |
| Approach: | They propose a calibration score that can be used to quantify uncertainty in visual question answering models. |
| Outcome: | The proposed calibration score is better calibrated than in text-only models for in-context learning. |
Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods address this issue by introducing an auxiliary task such as visual grounding, cycle consistency, or debiasing. |
| Approach: | They propose a data augmentation pipeline to turn “known” knowledge into training examples for VQA. |
| Outcome: | The proposed model can handle multi-modal information and is based on human-annotated examples. |
‘Just because you are right, doesn’t mean I am wrong’: Overcoming a bottleneck in development and evaluation of Open-Ended VQA tasks (2021.eacl-main)
Copied to clipboard
Man Luo, Shailaja Keyur Sampat, Riley Tallman, Yankai Zeng, Manuha Vancha, Akarshan Sajja, Chitta Baral
| Challenge: | Existing visual question answering datasets assume only one ground truth answer for each question. |
| Approach: | They propose alternative answer sets (AAS) of ground-truth answers to address this limitation . they modify top VQA solvers to support multiple plausible answers for a question . |
| Outcome: | The proposed approach improves on the GQA dataset and shows that it is more efficient than previous approaches. |