Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | evaluators of long-context vision language models (VLMs) have not kept up with the rapid development of open-weight long-constraint language models. |
| Approach: | They propose a dynamic benchmark generator for evaluating long-context reasoning in vision language models. |
| Outcome: | The proposed model can ignore irrelevant information when answering queries, showing that current models lack this capability. |
Similar Papers
Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects (2026.findings-acl)
Copied to clipboard
Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li
| Challenge: | Large Vision-Language Models are hindered by a systemic efficiency barrier known as visual token dominance. |
| Approach: | They propose a systematic taxonomy of efficiency techniques structured around the inference lifecycle . they examine visual encoding, prefilling, and decoding to understand bottlenecks . |
| Outcome: | The proposed techniques reveal how upstream decisions dictate downstream bottlenecks . the proposed techniques include hybrid compression and modality-aware decoding . |
What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks often include a mixture of reasoning questions, making it difficult to truly assess VLMs’ causal reasoning abilities. |
| Approach: | They propose two new benchmarks specifically designed to isolate and rigorously evaluate VLMs’ causal reasoning abilities. |
| Outcome: | The proposed benchmarks show that vision-language models perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. |
CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models have shown impressive ability in various language tasks, especially with their emergent in-context learning capability. |
| Approach: | They propose a causal reasoning benchmark for multi-modal in-context learning from large vision-language models that incorporates visual inputs. |
| Outcome: | The proposed model outperforms existing models on three visual causal reasoning tasks and demonstrates their strengths and weaknesses. |
NegVQA: Can Vision Language Models Understand Negation? (2025.findings-acl)
Copied to clipboard
| Challenge: | NegVQA is a visual question answering (VQA) benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Approach: | They propose a visual question answering benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Outcome: | The proposed model fails to correctly interpret negation, leading to critical errors in interactive AI systems. |
Benchmarking Deflection and Hallucination in Large Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks overlook conflicts between visual and textual evidence and the importance of generating deflections when incomplete knowledge is retrieved. |
| Approach: | They propose a dynamic curation pipeline that preserves benchmark difficulty over time . they propose 'vlm-DeflectionBench' benchmark to probe model behaviour under conflicting evidence . |
| Outcome: | The proposed benchmarks overlook conflicts between visual and textual evidence and are prone to obsolescence . the proposed benchmark is based on 2,775 samples spanning diverse retrieval settings . |
Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies leverage large language models (LLMs) in LVQA benchmarks, achieving exceptional performance while relying on vision language models to convert all visual content into natural language. |
| Approach: | They propose a modular and training-free framework that leverages large language models to generate a small subset of informative frames tailored to each question. |
| Outcome: | The proposed framework achieves state-of-the-art performance among similar models across four benchmark LVQA datasets: EgoSchema, NExT-QA, IntentQA, VideoMME. |
Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has explored unimodal biases in visual question answering, but the problem of selection bias in Multiple-Choice Question Answering (MCQA) remains underexplored. |
| Approach: | They propose a method that mitigates bias without retraining and is compatible with frozen LVLMs. |
| Outcome: | The proposed method mitigates bias without retraining and is compatible with frozen LVLMs. |
Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) generate contextually relevant responses by jointly interpreting visual and textual inputs. |
| Approach: | They propose a method to classify whether an input token is visually grounded by reinterpreting question prompts or replacing the detected absent tokens during generation. |
| Outcome: | The proposed method mitigates the models’ tendency to falsely presume the visual presence of text input and its generality across various LVLMs. |
Logic Haystacks: Probing LLMs’ Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent large language models claim long context windows, but evaluations often involve simple retrieval tasks or synthetic tasks padded with irrelevant text. |
| Approach: | They use grammars to generate simplified English with logical representations to create long input text while controlling its semantics. |
| Outcome: | The proposed model performs better with realistic distractors than with standard models. |
Do Video Language Models really understand the video contexts? (2025.naacl-srw)
Copied to clipboard
| Challenge: | Recent advances in VideoQA performance have shown that visual language models are effective but the processes of understanding and reasoning in VLMs remain under-explored. |
| Approach: | They propose a framework that incorporates a fine-grained question generation and answering process to measure how well VLMs understand video question answering tasks. |
| Outcome: | The proposed framework incorporates a fine-grained question generation and answering process to measure how well the responses generated by VLMs align with what the model understands. |