Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Vision Language Models (LVLMs) have shown impressive performance on various vision-language tasks. |
| Approach: | They propose a benchmark framework for evaluating Visual Variation Robustness of Large Vision Language Models that incorporates automated evaluation dataset generation and principled metrics for thorough robustness assessment. |
| Outcome: | The proposed framework identifies a vulnerability to visual variations affecting even advanced models that excel at complex vision-language tasks but significantly underperform on simple tasks like object recognition. |
Similar Papers
VLind-Bench: Measuring Language Priors in Large Vision-Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Vision-Language Models suffer from a problem known as language prior . such language priors can lead to undesirable biases and hallucinations when dealing with images that are out of distribution. |
| Approach: | They propose a benchmark to measure the language priors of Large Vision-Language Models. |
| Outcome: | The proposed benchmark is the first specifically designed to measure the language priors, or blindness, of LVLMs. |
MVP-Bench: Can Large Vision-Language Models Conduct Multi-level Visual Perception Like Humans? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LVLMs perform visual perception at multiple levels, but they are not able to perform multi-level tasks. |
| Approach: | They propose a visual–language benchmark to evaluate LVLMs' perceptions . they use manipulated images to examine how LVLs can perform multi-level tasks . |
| Outcome: | The proposed model performs poorly on high-level perception tasks, the authors show . they also show that current models do not generalize in understanding semantics of synthetic images . |
An Examination of the Compositionality of Large Generative Vision-Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on the compositionality of vision-language models (VLMs) however, the performance of GVLMs in multimodal compositional reasoning remains under-explored. |
| Approach: | They propose a syntactical bias score to quantify GVLMs' syntaktical bias . they propose 'SADE' task to assess GVLs's robustness against inclination toward syntical correctness. |
| Outcome: | The proposed benchmarks are based on evaluation metrics and current benchmarks. |
VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on object hallucinations, focusing on object outputs . current evaluation methods struggle to address subtle semantic distinctions between outputs and reference data . |
| Approach: | They propose a multi-dimensional benchmark covering objects, attributes, and relations . they propose metric that generalizes CHAIR metric and incorporates faithfulness and coverage . |
| Outcome: | The proposed evaluation framework is more comprehensive and better correlated with humans than existing evaluation methods. |
Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing understanding and reasoning abilities in graphbased tasks focus on specific graph types or tasks, posing challenges in designing versatile systems suitable for various tasks and graphs across diverse domains. |
| Approach: | They propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through three self-supervised learning tasks. |
| Outcome: | Extensive evaluations on 14 LVLMs reveal that LVLs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information. |
Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have seen a significant increase in research interest and real-world applications, including healthcare, autonomous systems, and security. |
| Approach: | They propose novel approaches to enhance model robustness through prompt engineering by suggesting adversarial perturbations or rephrasing questions. |
| Outcome: | The proposed approaches improve model robustness against strong image-based attacks such as Auto-PGD. |
Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Vision-language Models have been shown to be highly capable but lacking basic visual understanding skills. |
| Approach: | They propose to examine the limitations of vision-language models on visual tasks by constructing a series of tests that probe which components of design may be lacking. |
| Outcome: | The proposed tests compare VLMs to other models on visual encoders, intermediate vision-language projection and LLM-decoder outputs. |
Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs. |
| Approach: | They propose large vision-Language Models to augment LLMs with visual inputs. |
| Outcome: | The proposed models condition generated text on both an input image and a visual prompt, enabling a variety of use cases such as visual question answering and multimodal chat. |
EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for accelerating Large Vision-Language Models lack comprehensive evaluation across diverse backbones, benchmarks, and metrics. |
| Approach: | They propose EffiVLM-BENCH framework for evaluating absolute performance and generalization and loyalty. |
| Outcome: | The proposed framework offers insights into optimal strategies for accelerating LVLMs. |
VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies evaluate efficiency robustness of vision-language models under unrealistic assumptions, requiring access to model architecture and parameters. |
| Approach: | They propose a novel approach to evaluate VLM efficiency robustness in a realistic black-box setting. |
| Outcome: | The proposed approach generates adversarial images with imperceptible perturbations, increasing the computational cost by up to 128.47%. |