Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instances (2022.coling-1)
Copied to clipboard
| Challenge: | Existing VQA models rely on the superficial correlation between question type and frequent answers to make predictions, without really understanding the input. |
| Approach: | They propose a training framework that explicitly encourages the VQA model to distinguish between superficially similar instances. |
| Outcome: | The proposed framework achieves state-of-the-art performance on VQA-CP v2 . it explicitly encourages the model to distinguish between the superficially similar instances . |
Similar Papers
WeaQA: Weak Supervision via Captions for Visual Question Answering (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for training visual question answering models rely on datasets with human-annotated image-quest-answer triplets. |
| Approach: | They propose a method to train models with synthetic Q-A pairs generated procedurally from captions. |
| Outcome: | The proposed method trains models with synthetic Q-A pairs generated from captions on three VQA benchmarks. |
Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that biased samples can be brittle for VQA models . however, the improvements on OOD data severely sacrifice the performance on the in-distribution (ID) data. |
| Approach: | They propose a contrastive learning approach that exploits biased samples for unbiased information that contributes to reasoning. |
| Outcome: | The proposed method achieves competitive performance on the OOD dataset while maintaining robustness on the ID dataset. |
A negative case analysis of visual grounding methods for VQA (2020.acl-main)
Copied to clipboard
| Challenge: | Existing Visual Question Answering (VQA) methods exploit dataset biases and spurious statistical correlations instead of producing correct answers for the right reasons. |
| Approach: | They propose to incorporate visual cues to better ground VQA models . they also propose a regularization effect which prevents over-fitting to linguistic priors . |
| Outcome: | The proposed method outperforms existing methods on the Visual Question Answering (VQA) dataset. |
Learning to Contrast the Counterfactual Samples for Robust Visual Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods of generating counterfactual samples are not fully utilized in the task of Visual Question Answering (VQA). |
| Approach: | They propose a self-supervised contrastive learning mechanism to learn the relationship between original samples, factual samples and counterfactual samples. |
| Outcome: | The proposed method surpasses state-of-the-art models on the VQA-CP dataset, a diagnostic benchmark for assessing the VQ model’s robustness. |
Can Pre-training help VQA with Lexical Variations? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) models are closing the gap between oracle performance and model robustness. |
| Approach: | They propose to use language & cross-modal pre-training to investigate the robustness of VQA models towards lexical variations. |
| Outcome: | The proposed model architectures and training techniques improve the performance of the VQA-Rephrasings dataset on rephrased questions. |
Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods address this issue by introducing an auxiliary task such as visual grounding, cycle consistency, or debiasing. |
| Approach: | They propose a data augmentation pipeline to turn “known” knowledge into training examples for VQA. |
| Outcome: | The proposed model can handle multi-modal information and is based on human-annotated examples. |
Modular Visual Question Answering via Code Generation (2023.acl-short)
Copied to clipboard
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein
| Challenge: | a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills . |
| Approach: | They propose a framework that formulates visual question answering as modular code generation. |
| Outcome: | The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation. |
Compressing and Debiasing Vision-Language Pre-Trained Models for Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on VQA models have found that they suffer from dataset biases and inefficient memory footprints. |
| Approach: | They investigate whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. |
| Outcome: | The proposed compression and debiasing pipelines outperform the debiased full VLPs on VQA tasks. |
Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Existing research addresses ambiguous visual questions by rephrasing questions, but it fails to address the inherently interactive nature of user interactions with visual language models (VLMs). Existing studies focus on re-phrase questions, and lack of a benchmark to assess VLMs’ capacity for resolving ambiguities through interaction. |
| Approach: | They propose a visual question answering task that provides a natural language answer to a question based on a given image and an automated pipeline to generate ambiguity-clarification question pairs. |
| Outcome: | The proposed benchmark targets three common categories of ambiguity in visual question answering (VQA) context and encompasses various VQA scenarios. |
Can We Learn Question, Answer, and Distractors All from an Image? A New Task for Multiple-choice Visual Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies focus on generating QADs from image and question, but a novel task is needed to generate meaningful questions, correct answers, and challenging distractors. |
| Approach: | They propose a task to generate QADs from images and encode images together . they use contrastive learning to ensure consistency of QAD generated and tested . |
| Outcome: | Empirical evaluations on the benchmark dataset validate the performance of the proposed task. |