Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond (2023.findings-emnlp)
Copied to clipboard
Zhecan Wang, Long Chen, Haoxuan You, Keyang Xu, Yicheng He, Wenhao Li, Noel Codella, Kai-Wei Chang, Shih-Fu Chang
| Challenge: | Existing studies have examined dataset biases in VQA benchmarks with short-phrase answers Multiple-choice Question with the LONG Answers (VCR, VLEP, etc.) |
| Approach: | They propose to use Adversarial Data Synthesis (ADS) to generate synthetic training and debiased evaluation data and introduce Intra-sample Counterfactual Training (ICT) to assist models in utilizing synthesized training data. |
| Outcome: | The proposed approach improves model performance even in domain-shifted scenarios. |
Similar Papers
A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning (2022.aacl-main)
Copied to clipboard
| Challenge: | Large-scale, pretrained vision-language models are growing in popularity due to impressive performance on downstream tasks with minimal finetuning. |
| Approach: | They propose to apply ranking metrics to image-text representations to investigate bias measures and debiasing methods to reduce various bias measures. |
| Outcome: | The proposed model reduces bias measures with minimal degradation to image-text representations. |
Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets (N18-1)
Copied to clipboard
| Challenge: | Visual question answering datasets are a form of (visual) Turing test that artificial intelligence should strive to achieve. |
| Approach: | They propose automatic procedures to remedy design deficiencies in visual question answering datasets . they propose to use a set of decoys to re-construct decoying answers for two popular Visual QA datasets. |
| Outcome: | The proposed procedures improve the performance of the proposed datasets. |
Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that datasets contain incidental correlations created by idiosyncrasies in the data collection process. |
| Approach: | They propose a method that detects and ignores dataset-specific correlations by introducing a new method that makes them conditionally independent. |
| Outcome: | The proposed method detects and ignores these kinds of dataset-specific correlations, and does not require the bias to be known in advance. |
Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that biased samples can be brittle for VQA models . however, the improvements on OOD data severely sacrifice the performance on the in-distribution (ID) data. |
| Approach: | They propose a contrastive learning approach that exploits biased samples for unbiased information that contributes to reasoning. |
| Outcome: | The proposed method achieves competitive performance on the OOD dataset while maintaining robustness on the ID dataset. |
Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. |
| Approach: | They build upon existing benchmarks to create five datasets containing mismatched image-text pairs and examine how they reason over visual and textual data . |
| Outcome: | The proposed model reasoned over visual and textual data in real-world applications but not in the visual and visual descriptions. |
When and Why Does Bias Mitigation Work? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural models exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. |
| Approach: | They propose to use model debiasing techniques to pressure models away from spurious features and to use them to learn useful representations instead. |
| Outcome: | The proposed methods increase models' reliance on hidden biases instead of learning robust features that help them solve a task. |
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (D19-1)
Copied to clipboard
| Challenge: | Recent advances in neural models exploit dataset-specific patterns that do not generalize well to out-of-domain or adversarial settings. |
| Approach: | They propose to train a model to be more robust to domain shift if it has prior knowledge of dataset biases. |
| Outcome: | The proposed model can be more robust to domain shift if it has prior knowledge of dataset biases. |
Learning to Contrast the Counterfactual Samples for Robust Visual Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods of generating counterfactual samples are not fully utilized in the task of Visual Question Answering (VQA). |
| Approach: | They propose a self-supervised contrastive learning mechanism to learn the relationship between original samples, factual samples and counterfactual samples. |
| Outcome: | The proposed method surpasses state-of-the-art models on the VQA-CP dataset, a diagnostic benchmark for assessing the VQ model’s robustness. |
Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. |
| Approach: | They propose a method to attack MRC models by exposing statistical biases in a RACE dataset and propose an augmented training method that can greatly reduce models’ statistical bias. |
| Outcome: | The proposed method can reduce models’ statistical biases from human-level performance to chance-level. |
BiasDora: Exploring Hidden Biased Associations in Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on social biases focus on a limited set of documented associations, such as gender-profession or race-crime. |
| Approach: | They propose to examine hidden, implicit bias associations across 9 bias dimensions by probing VLMs to uncover hidden, unexamined associations. |
| Outcome: | The proposed methods reveal that biases vary in negativity, toxicity, and extremity. |