MagicBench: Diagnosing Visual Agency Loss and Semantic Dependency in Multimodal LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | MLLMs assume linguistic context invariably enhances visual understanding . a diagnostic benchmark is used to evaluate ML models under hierarchical linguistic interference . |
| Approach: | They propose a diagnostic benchmark to evaluate MLLMs under hierarchical linguistic interference. |
| Outcome: | The proposed benchmark compared 402 videos with a physical constraint set to evaluate MLLMs under hierarchical linguistic interference. |
Similar Papers
ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction (2026.findings-acl)
Copied to clipboard
Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma
| Challenge: | Empirical evaluations on state-of-the-art MLLMs reveal a significant performance gap . ML models lack the fine-grained cross-modal reasoning required to bridge visual discontinuities. |
| Approach: | They propose a benchmark that renders fragmented documents directly from Markdown to facilitate evaluation of VRDU tasks. |
| Outcome: | The proposed benchmark renders fragmented documents directly from Markdown. |
MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities. |
| Approach: | They propose a competition-based benchmark framework specifically designed to assess LLMs within multi-agent environments. |
| Outcome: | The proposed framework enhances the LLMs’ abilities in navigating complex social and cognitive dimensions by over threefold between the strongest and weakest LLM models. |
Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have facilitated the development of Multimodal LLMs. |
| Approach: | They propose a causal framework to interpret unimodal biases in visual question answering problems and a framework to integrate information from different modalities and mitigate biase. |
| Outcome: | The proposed framework analyzes visual question answering (VQA) problems to assess their impact on predictions. |
VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models have demonstrated remarkable capabilities across vision-language tasks, but their performance as embodied agents needs further exploration. |
| Approach: | They propose a framework to evaluate multimodal large language models as zero-shot agents . they find that enhancing prevalent agents with Chain-of-Thought reasoning and self-reflection leads to an unexpected performance decrease. |
| Outcome: | The proposed framework enables comparisons and component-level ablations across diverse MLLM architectures, agent designs, and navigation tasks. |
GOBench: Stage-Wise Diagnostics and the Visual Paradox in Multimodal Graph Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks fail to represent multimodal problem specifications, score outcomes only and cannot localize where failures occur along the modeling pipeline. |
| Approach: | They propose a Graph Optimization benchmark that aligns multiple modalities with solver-derived oracles and a diagnostic protocol that evaluates intermediate artifacts as well as end results. |
| Outcome: | Graph Optimization benchmark (GOBench) evaluates intermediate artifacts as well as end results . vision reliably increases inference cost, while reliability impact is regime-dependent . current benchmarks fail to represent multimodal problem specifications, fail to localize failures . |
Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLMs (2025.findings-emnlp)
Copied to clipboard
Chenkun Tan, Pengyu Wang, Shaojun Zhou, Botian Jiang, Zhaowei Li, Dong Zhang, Xinghao Wang, Yaqian Zhou, Xipeng Qiu
| Challenge: | Recent advances in multimodal large language models focus on improving performance . however, language prior conflict leads to suboptimal vision-language alignment . |
| Approach: | They propose a method to decouple the alignment process from language prior interference . they use a proxy LLM to detach from language interference during pretraining . |
| Outcome: | The proposed method improves training performance and generalizes training data. |
Tiny Scales, Great Challenges: The Limits of Multimodal LLMs in Scale Recognition (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on a single type of quantity or a specific format, lacking a comprehensive evaluation of scale recognition capabilities. |
| Approach: | They propose a visual scale recognition benchmark built using images from COCO, Open Images, and Flickr to evaluate scale recognition capabilities of multimodal large language models. |
| Outcome: | The proposed model achieves 42.60% accuracy, lower than the 97.40% of humans. |
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models (2026.findings-acl)
Copied to clipboard
Mohammad Mahdi Abootorabi, Omid Ghahroodi, Anas Madkoor, Marzia Nouri, Doratossadat Dastgheib, Ehsaneddin Asgari
| Challenge: | Existing evaluations focus on piecemeal or disconnected tasks, obscuring critical cognitive weaknesses and providing little insight for targeted improvement. |
| Approach: | They propose a bilingual, cognitively human-grounded multimodal benchmark for VLMs that evaluates six levels of cognition through carefully designed image–question–answer tasks. |
| Outcome: | The proposed framework ensures scalability, cultural inclusivity, and linguistic fidelity. |
Express What You See: Can Multimodal LLMs Decode Visual Ciphers with Intuitive Semiosis Comprehension? (2025.findings-acl)
Copied to clipboard
| Challenge: | Traditional VQA benchmarks encounter a modality gap and over-reliance on language priors, whereas human cognition excels at intuitive semiosis, associating abstract visual symbols to linguistic semantics. |
| Approach: | They propose a task of generating abstract linguistics from emoji sequence images, where such reasoning underpins critical applications in cryptography. |
| Outcome: | The proposed model can generate abstract linguistics from emoji sequence images, challenging MLLMs’ reasoning of decoding complex semantics of visual ciphers. |
MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)
Copied to clipboard
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored. |
| Approach: | They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios. |
| Outcome: | The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC. |