See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models often rely on a single modality rather than treating and utilizing them equally, leading to dominance of a specific modality on the overall performance. |
| Approach: | They propose a framework to mitigate dominant modality bias by adjusting the gradient of KL divergence based on each modality's contribution and aligning task directions in a non-conflicting manner. |
| Outcome: | The proposed framework mitigates dominant modality bias on UPMC Food-101, Hateful Memes, and MM-IMDb datasets. |
Similar Papers
Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language models overemphasize linguistic priors, leading to modality bias. |
| Approach: | They propose a vision-language aggregation framework that mitigates modality bias in TAL by preserving vision as the dominant signal while adaptively exploiting language only when beneficial. |
| Outcome: | Experiments on THUMOS14 show that the proposed model outperforms state-of-the-art models by up to 3.2% mAP. |
Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that vision-language models have modality gaps that persist even in well-aligned models. |
| Approach: | They propose a modality-dominance score to measure and leverage modality gaps . they propose automatic interpretability metrics to evaluate these features in a scalable manner . |
| Outcome: | The proposed framework allows for training-free probing and editing methods for understanding model perception across genders and generating adversarial examples. |
Imbalanced Gradients in RL Post-Training of Multi-Task LLMs (2026.findings-eacl)
Copied to clipboard
Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, Yonathan Efroni
| Challenge: | Large-gradient tasks can achieve similar or even much lower learning gains than small-grading ones. |
| Approach: | They show that large-gradient tasks can achieve lower learning gains than small-grading ones . large-grade tasks can accomplish similar or even lower learning gain than small grade ones if they are large . |
| Outcome: | The proposed approach fails when certain tasks produce larger gradients . Large-gradient tasks can achieve lower learning gains than small-gradent ones . |
Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement (2025.findings-naacl)
Copied to clipboard
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Taha Kass-Hout, Furong Huang, Cao Xiao
| Challenge: | Existing methods for visual and language alignment depend on external models or data, leading to uncontrollable and unstable results. |
| Approach: | They propose a framework that enhances visual and language alignment without external dependencies by incorporating an in-context self-critic mechanism that constructs preference pairs for tuning. |
| Outcome: | The proposed framework outperforms existing methods and improves performance on 14 hallucination and comprehensive benchmarks. |
BiasDora: Exploring Hidden Biased Associations in Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on social biases focus on a limited set of documented associations, such as gender-profession or race-crime. |
| Approach: | They propose to examine hidden, implicit bias associations across 9 bias dimensions by probing VLMs to uncover hidden, unexamined associations. |
| Outcome: | The proposed methods reveal that biases vary in negativity, toxicity, and extremity. |
Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. |
| Approach: | They build upon existing benchmarks to create five datasets containing mismatched image-text pairs and examine how they reason over visual and textual data . |
| Outcome: | The proposed model reasoned over visual and textual data in real-world applications but not in the visual and visual descriptions. |
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021.naacl-main)
Copied to clipboard
| Challenge: | PTLMs can exhibit biases against protected groups in a host of modeling tasks . but, fine-tuned LMs may propagate bias to downstream classifiers . |
| Approach: | They propose to use upstream bias mitigation techniques to reduce bias on downstream tasks by fine-tuning an upstream model and applying it to a downstream model. |
| Outcome: | The proposed model reduces bias on hate speech detection, toxicity detection and coreference resolution tasks over bias factors. |
Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent (2025.findings-emnlp)
Copied to clipboard
Junda Wu, Yuxin Xiong, Xintong Li, Yu Xia, Ruoyu Wang, Yu Wang, Tong Yu, Sungchul Kim, Ryan A. Rossi, Lina Yao, Jingbo Shang, Julian McAuley
| Challenge: | Existing fine-tuning and continual learning methods compress visual representations and emphasize task alignment over visual retention. |
| Approach: | They propose a modality-decoupled gradient descent (MDGD) that regulates gradient updates to preserve effective rank of visual features and explicitly disentangles visual learning from task-specific alignment. |
| Outcome: | The proposed model reduces visual forgetting and improves visual retention . it disentangles visual learning from task-specific alignment and preserves effective rank . |
Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions. |
| Approach: | They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding. |
| Outcome: | The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information. |
Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing fine-tuning algorithms for vision-language models are restricted by patient privacy concerns and can contain imperceptible noise. |
| Approach: | They propose a framework to mitigate adversarial noise and mitigate upstream noise during fine-tuning. |
| Outcome: | The proposed framework improves model robustness and transferability while decreasing noise levels negatively impact downstream performance. |