Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation (2025.findings-acl)
Copied to clipboard
Qiji Zhou, YiFan Gong, Guangsheng Bao, Hongjie Qiu, Jinqiang Li, Xiangrong Zhu, Huajian Zhang, Yue Zhang
| Challenge: | Existing multimodal benchmarks often overlook counterfactual reasoning, which is crucial for robust video understanding. |
| Approach: | They propose a multidimensional multimodal benchmark that systematically evaluates MLLMs across the abstract-concrete and perception-cognition dimensions. |
| Outcome: | The proposed model decomposes complex queries into structured sub-questions, enabling fine-grained reasoning analysis. |
Similar Papers
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos (2023.emnlp-main)
Copied to clipboard
Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Chandra, Marjorie Freedman, Ralph Weischedel, Nanyun Peng
| Challenge: | despite its importance, there are few datasets that cover multimodal counterfactual reasoning . a dataset focusing on this area is limited because of its limited coverage over synthetic environments . |
| Approach: | They develop a video question answering dataset that provides questions on multimodal reasoning . they ask questions about counterfactual hypotheses over visual events . |
| Outcome: | The proposed dataset shows a significant performance gap between models and humans . it provides questions that span physical, social, and temporal dimensions . |
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2025.acl-long)
Copied to clipboard
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig
| Challenge: | Recent advances in multimodal large language models have led to progress in tackling complex reasoning tasks that combine textual and visual information. |
| Approach: | They introduce a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. |
| Outcome: | The proposed model performs lower on MMMU-Pro than on the previous benchmark, ranging from 16.8% to 26.9%. |
Multimodal Causal Reasoning Benchmark: Challenging Multimodal Large Language Models to Discern Causal Links Across Modalities (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing MLLMs lack robustness in multimodal causal reasoning compared to their performance in textual settings. |
| Approach: | They propose a novel multimodal chain-of-thought (CoT) reasoning benchmark that leverages siamese images and text pairs to challenge MLLMs. |
| Outcome: | The proposed benchmark leverages siamese images and text pairs to challenge MLLMs. |
Probing Audio-Visual Reasoning in Multimodal Language Models through the Lens of Audio (2026.acl-long)
Copied to clipboard
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, Xiangyu Yue
| Challenge: | Recent multimodal large language models lack robust audio-visual integration ability and performance on DeafTest is highly correlated with AV-Odyssey accuracy. |
| Approach: | They propose a benchmarking tool that integrates audio-visual reasoning with audio-video cues to infer solutions. |
| Outcome: | The proposed model performs well on DeafTest, but lacks audio perception in simple audio tasks. |
Beyond Single View: A Comprehensive Benchmark for Medical Multimodal Large Language Models on Multi-Image Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal large language models are limited to multiview diagnostics. |
| Approach: | They propose a benchmark specifically designed for medical multi-image understanding that evaluates MLLMs across four dimensions. |
| Outcome: | The proposed model performs better in multi-image contexts than open-source models . the model perform better when processing increased visual loads than closed-source ones . |
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues (2026.findings-acl)
Copied to clipboard
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Moore Wang, Yongqian Wen, Yuanxing Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, Yonghong Lin, An Ping, Shihao Li, Yanghai Wang, Tianhao Peng, Jiaheng Liu
| Challenge: | Existing evaluation benchmarks for Multimodal Large Language Models (MLLMs) focus on single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. |
| Approach: | They propose a video understanding benchmark for MLLMs in multi-turn dialogues that assesses six core competencies that focus on perceptivity and interactivity. |
| Outcome: | The MT-Video-Bench evaluates 1,000 multi-turn dialogues from diverse domains and reveals significant performance discrepancies and limitations in handling multi-turned video dialogues. |
TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-rich image understanding benchmarks lack scale and fragmented scenarios . a new full-image structured output format is proposed to enable fine-grained evaluation of perception and reasoning capabilities. |
| Approach: | They propose a large-scale, multilingual benchmark that includes over 100,000 annotations and 22,000 question-answer pairs. |
| Outcome: | The proposed framework provides a comprehensive platform for developing and evaluating next-generation multimodal AI systems. |
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal large language models fail to capture complexity and traceability of reasoning processes . SciVQR includes domain-specific visuals and challenges models to combine visual comprehension with reasoning. |
| Approach: | They propose a multimodal benchmark for scientific reasoning covering 54 subfields . SciVQR includes domain-specific visuals and challenges models to combine visual comprehension with reasoning . |
| Outcome: | SciVQR evaluates 54 subfields in mathematics, physics, chemistry, geography, astronomy, and biology . the results highlight the need for improved multi-step reasoning and integration of interdisciplinary knowledge . |
RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks (2025.emnlp-industry)
Copied to clipboard
Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda, Hansa Meghwani, Jyotika Singh, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Existing evaluation methods do not explicitly measure this distinction, hindering effective dataset curation and real-world focused model development. |
| Approach: | They introduce a region-based score to quantify a dataset's reliance on global versus local visual information. |
| Outcome: | The proposed model-based score systematically compares model performance on image patches versus full images to determine if tasks require holistic image understanding or can be solved with partial or localized visual cues. |
From Detection to Understanding: Multi-Turn Reasoning for Video Misinformation Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on binary veracity judgments and do not evaluate process-level justifications for misinformation models. |
| Approach: | They propose a video misinformation analysis benchmark that assesses reasoning in video misinterpretation. |
| Outcome: | The proposed framework improves reasoning accuracy and explanation quality compared to existing models . it covers 12 fine-grained deception categories and progresses from perceptual attribution to intent and persuasion analysis. |