Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods to learn biases from the perspective of model components are limited by their complexity and performance. |
| Approach: | They propose a framework that incorporates causal mediation analysis to measure and map the pathways of bias generation and propagation within vision-language and multimodal tasks. |
| Outcome: | The proposed framework is applicable to a wide range of vision-language and multimodal tasks and reduces bias by 22.03% and 9.04% in the MSCOCO and PASCAL-SENTENCE datasets. |
Similar Papers
Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. |
| Approach: | They build upon existing benchmarks to create five datasets containing mismatched image-text pairs and examine how they reason over visual and textual data . |
| Outcome: | The proposed model reasoned over visual and textual data in real-world applications but not in the visual and visual descriptions. |
Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing vision-language models focus on salient attributes but ignore contextualized nuances, resulting in gender bias. |
| Approach: | They propose a task-agnostic generation framework to mitigate gender bias in vision-language models. |
| Outcome: | The proposed framework can mitigate gender bias in vision-language models . it yields all-sided but gender-obfuscated narratives, which prevents concentration on localized image features, especially gender attributes. |
Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have facilitated the development of Multimodal LLMs. |
| Approach: | They propose a causal framework to interpret unimodal biases in visual question answering problems and a framework to integrate information from different modalities and mitigate biase. |
| Outcome: | The proposed framework analyzes visual question answering (VQA) problems to assess their impact on predictions. |
Biases Propagate in Encoder-based Vision-Language Models: A Systematic Analysis From Intrinsic Measures to Zero-shot Retrieval Outcomes (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing encoder-based vision-language models (VLMs) contain intrinsic biases that manifest in biased outputs. |
| Approach: | They propose a framework to measure intrinsic bias propagation by correlating intrinsic bias with extrinsic bias in zero-shot text-to-image and image-totext retrieval. |
| Outcome: | The proposed framework shows that larger/better-performing models exhibit greater bias propagation, raising concerns given the trend towards increasingly complex AI models. |
A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have highlighted the existence of social biases within large vision and language models. |
| Approach: | They propose a framework for systematically evaluating gender, race, and age biases in vision-language models with respect to professions. |
| Outcome: | The proposed framework covers all supported inference modes of the recent vision-language models, including image-to-text, text-to image, and image- to-image. |
BiasDora: Exploring Hidden Biased Associations in Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on social biases focus on a limited set of documented associations, such as gender-profession or race-crime. |
| Approach: | They propose to examine hidden, implicit bias associations across 9 bias dimensions by probing VLMs to uncover hidden, unexamined associations. |
| Outcome: | The proposed methods reveal that biases vary in negativity, toxicity, and extremity. |
Identifying and Mitigating Annotation Bias in Natural Language Understanding using Causal Mediation Analysis (2024.findings-acl)
Copied to clipboard
Sitiporn Sae Lim, Can Udomcharoenchaikit, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Current NLU models obtain state-of-the-art accuracy on in-distribution benchmarks, but they use annotation bias to make predictions, negatively affecting the models' generalizability. |
| Approach: | They apply causal mediation analysis to gauge how much each component mediates annotation biases and use causal-grounded masking and gradient unlearning to mitigate bias. |
| Outcome: | The proposed methods improve the model's robustness against annotation bias even after employing other training-time debiasing techniques. |
A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning (2022.aacl-main)
Copied to clipboard
| Challenge: | Large-scale, pretrained vision-language models are growing in popularity due to impressive performance on downstream tasks with minimal finetuning. |
| Approach: | They propose to apply ranking metrics to image-text representations to investigate bias measures and debiasing methods to reduce various bias measures. |
| Outcome: | The proposed model reduces bias measures with minimal degradation to image-text representations. |
VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies on VLM bias focus on portrait-style images and gender-occupation associations . existing studies ignore broader and more complex social stereotypes and their implied harm . |
| Approach: | They propose a large-scale VQA benchmark for evaluating bias in vision-language models . they use a question-answering framework that spans factuality, perception, stereotyping, and decision making . |
| Outcome: | The proposed framework examines bias in vision-language models using 30M+ images . findings reveal subtle, multifaceted, and surprising stereotypical patterns . |
Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that Vision-Language Models have robust multimodal reasoning capabilities, but their robustness against textual misinformation remains under-explored. |
| Approach: | They propose to use visual-question-answering (VQA) prompts to generate persuasive prompts that deliberately conflict with visual evidence to test their models. |
| Outcome: | The proposed framework shows that models are vulnerable to misleading prompts, and show an average performance drop of over 48.2% after only one round of persuasive conversation. |