MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks rarely isolate how much visual information contributes to reasoning . a growing collection of benchmarks has catalyzed rapid progress in multimodal reasoning - but how much it contributes remains unclear . |
| Approach: | They propose a university-level multimodal mathematical reasoning benchmark to quantify the effect of visual input. |
| Outcome: | The proposed benchmark disentangles and quantifies the effect of visual input on multimodal reasoning models. |
Similar Papers
The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights (2025.acl-long)
Copied to clipboard
Yufang Liu, Yao Du, Tao Ji, Jianing Wang, Yang Liu, Yuanbin Wu, Aimin Zhou, Mengdi Zhang, Xunliang Cai
| Challenge: | Existing models that leverage visual information do not improve math reasoning performance . authors suggest that visual information is important for multimodal reasoning . |
| Approach: | They propose a dataset to require image reliance for problem-solving and challenge models with similar, yet distinct, images that change the correct answer. |
| Outcome: | The proposed model performance is unaffected by changes to or removal of images in the dataset. |
Can Vision-Language Models Solve Visual Math Equations? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) perform well on textual equations, but fail on visually grounded counterparts. |
| Approach: | They propose to decompose visual equation solving into symbolic equation solving and visual recognition into two core components to understand this gap. |
| Outcome: | The proposed models perform well on textual equations, but fail on visual grounded ones. |
We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? (2025.acl-long)
Copied to clipboard
Runqi Qiao, Qiuna Tan, Guanting Dong, MinhuiWu MinhuiWu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma GongQue, Shanglin Lei, YiFan Zhang, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Xiao Zong, Yida Xu, Peiqing Yang, Zhimin Bao, Muxi Diao, Chen Li, Honggang Zhang
| Challenge: | Existing benchmarks focus more on end-to-end performance, but neglect the underlying principles of knowledge acquisition and generalization. |
| Approach: | They propose a benchmark specifically designed to explore the problem-solving principles by decomposing 6.5K visual math problems into 10.9K step-level questions for evaluation. |
| Outcome: | The proposed benchmark covers 6.5K visual math problems and 10.9K step-level questions spanning 5 layers of knowledge granularity and 67 hierarchical knowledge concepts. |
Can Vision-Language Models Evaluate Handwritten Math? (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language Models (VLMs) have significantly enhanced the ability to interpret both textual and visual data. |
| Approach: | They propose a benchmark to assess VLMs’ ability to detect, localize and correct errors in handwritten mathematical content. |
| Outcome: | The proposed benchmark covers over 2,200 handwritten math solutions from 609 manually curated problems from grades 7-12 with intentionally introduced perturbations. |
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have demonstrated their strong performance on IQ test questions, achieving high scores across many languages. |
| Approach: | They propose a dataset to evaluate cognitive multimodal reasoning and problem-solving skills of large models. |
| Outcome: | The proposed dataset contains 2,728 multiple-choice questions and 4,642 images spanning 26 categories. |
TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models have been explored for visual programming, but performance is unclear . most prior work focuses on visual programming for productivity . |
| Approach: | They propose a visual programming benchmark that uses visual programming to evaluate VLMs. |
| Outcome: | The proposed model improves on GPT-5, GPT-4o, and Qwen2-VL-72B on real-world tasks by 20% . the proposed model is based on 823 visual programming tasks in the Turtle Graphics domain . |
OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Models (2026.acl-long)
Copied to clipboard
Qiguang Chen, Chengyu Luan, Jiajun Wu, Qiming Yu, Yi Yang, Yizhuo Li, Jingqi Tong, Xiachong Feng, Libo Qin, Wanxiang Che
| Challenge: | Existing multimodal reasoning benchmarks for large vision-language models emphasize single-image analysis and fail to exploit contextual information across multiple images. |
| Approach: | They propose a benchmark to evaluate Olympiad-level reasoning when evidence is distributed over multiple images. |
| Outcome: | The proposed model outperforms existing models on bi-image Olympiads and Gemini-3-Pro on multimodal Olympiad-level reasoning tasks. |
Beyond Visual Understanding Introducing PARROT-360V for Vision Language Model Benchmarking (2025.coling-industry)
Copied to clipboard
| Challenge: | Current benchmarks for evaluating Vision Language Models (VLMs) often fail to thoroughly assess these models’ abilities to understand complex visual and textual content. |
| Approach: | They propose a benchmark that features 2487 visual puzzles designed to test VLMs on complex visual reasoning tasks. |
| Outcome: | The PARROT-360V Benchmark features 2487 visual puzzles designed to test VLMs on complex visual reasoning tasks. |
A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges (2025.findings-acl)
Copied to clipboard
Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, Xuming Hu
| Challenge: | This survey provides **the first comprehensive analysis of mathematical reasoning in the era of multimodal large language models** . integrating large language model with mathematical reasoning tasks is becoming significant as AI advances . |
| Approach: | They review over 200 studies published since 2021 and examine the state-of-the-art developments in Math-LLMs . they identify five major challenges hindering the realization of AGI in this domain . |
| Outcome: | The authors examine the state-of-the-art developments in Math-LLMs with a focus on multimodal settings. |
MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models (2026.acl-long)
Copied to clipboard
Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu, Zhiqi Huang
| Challenge: | Recent advances in vision-language models have improved performance in multi-modal learning. |
| Approach: | They propose a multi-modal benchmark that embeds a single coherent reasoning error in 1997 samples. |
| Outcome: | The proposed benchmark is based on a set of 1997 samples embedding a single coherent reasoning error. |