Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that Vision-Language Models have robust multimodal reasoning capabilities, but their robustness against textual misinformation remains under-explored. |
| Approach: | They propose to use visual-question-answering (VQA) prompts to generate persuasive prompts that deliberately conflict with visual evidence to test their models. |
| Outcome: | The proposed framework shows that models are vulnerable to misleading prompts, and show an average performance drop of over 48.2% after only one round of persuasive conversation. |
Similar Papers
Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions. |
| Approach: | They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding. |
| Outcome: | The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information. |
Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. |
| Approach: | They build upon existing benchmarks to create five datasets containing mismatched image-text pairs and examine how they reason over visual and textual data . |
| Outcome: | The proposed model reasoned over visual and textual data in real-world applications but not in the visual and visual descriptions. |
Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chart question answering (CQA) is a crucial area of Visual Language Understanding. |
| Approach: | They evaluate the robustness and consistency of current Visual Language Models on a dataset encompassing diverse question categories and chart formats. |
| Outcome: | The proposed models handle varying levels of chart and question complexity and are robust across different visual representations of the same underlying data. |
When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision and language models have demonstrated remarkable performance in visual question answering tasks. |
| Approach: | They introduce a framework to optimize L-VLMs by leveraging unlabeled images . they conduct extensive experiments on four diverse VQA benchmarks . |
| Outcome: | The proposed framework improves L-VLMs on four visual question answering benchmarks. |
Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have facilitated the development of Multimodal LLMs. |
| Approach: | They propose a causal framework to interpret unimodal biases in visual question answering problems and a framework to integrate information from different modalities and mitigate biase. |
| Outcome: | The proposed framework analyzes visual question answering (VQA) problems to assess their impact on predictions. |
NegVQA: Can Vision Language Models Understand Negation? (2025.findings-acl)
Copied to clipboard
| Challenge: | NegVQA is a visual question answering (VQA) benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Approach: | They propose a visual question answering benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Outcome: | The proposed model fails to correctly interpret negation, leading to critical errors in interactive AI systems. |
Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods to learn biases from the perspective of model components are limited by their complexity and performance. |
| Approach: | They propose a framework that incorporates causal mediation analysis to measure and map the pathways of bias generation and propagation within vision-language and multimodal tasks. |
| Outcome: | The proposed framework is applicable to a wide range of vision-language and multimodal tasks and reduces bias by 22.03% and 9.04% in the MSCOCO and PASCAL-SENTENCE datasets. |
Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs. |
| Approach: | They propose large vision-Language Models to augment LLMs with visual inputs. |
| Outcome: | The proposed models condition generated text on both an input image and a visual prompt, enabling a variety of use cases such as visual question answering and multimodal chat. |
MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks address single tables or non-visual data, leaving a critical gap . MTabVQA comprises 3,745 complex question-answer pairs . |
| Approach: | They propose a benchmark specifically designed for multi-tabular visual question answering that measures the ability to parse diverse table images and correlate information across them. |
| Outcome: | The proposed benchmarks show that fine-tuning VLMs with MTabVQA-Instruct significantly improves their reasoning abilities. |
Benchmarking Deflection and Hallucination in Large Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks overlook conflicts between visual and textual evidence and the importance of generating deflections when incomplete knowledge is retrieved. |
| Approach: | They propose a dynamic curation pipeline that preserves benchmark difficulty over time . they propose 'vlm-DeflectionBench' benchmark to probe model behaviour under conflicting evidence . |
| Outcome: | The proposed benchmarks overlook conflicts between visual and textual evidence and are prone to obsolescence . the proposed benchmark is based on 2,775 samples spanning diverse retrieval settings . |