CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models (2025.coling-main)
Copied to clipboard
Zhongzhi Li, Ming-Liang Zhang, Pei-Jie Wang, Jian Xu, Rui-Song Zhang, Yin Fei, Zhi-Long Ji, Jin-Feng Bai, Zhen-Ru Pan, Jiaxin Zhang, Cheng-Lin Liu
| Challenge: | Large language models excel in various language tasks, while large multimodal models effectively handle visual-language problems. |
| Approach: | They propose to use a multimodal multimodal model evaluation benchmark to evaluate model performance in Chinese K12 classrooms. |
| Outcome: | The proposed model evaluation tool is integrated with the CMMaTH dataset. |
Similar Papers
CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. |
| Approach: | They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings. |
| Outcome: | The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models. |
HighMATH: Evaluating Math Reasoning of Large Language Models in Breadth and Depth (2025.findings-emnlp)
Copied to clipboard
Yan Liu, Minghui Zhang, Bojian Xiong, Yifan Xiao, Yinong Sun, Yating Mei, Longyu Zeng, Jingchao Yang, Yang Wang, Deyi Xiong
| Challenge: | a gap in math models' accuracy has been widened with the development of large language models (LLMs) . a new study aims to bridge this gap by evaluating a set of high-level math reasoning models . |
| Approach: | They propose to evaluate large language models on existing math benchmarks to bridge this gap . they collect 5,293 problems from Chinese senior high school mathematics exams . |
| Outcome: | The proposed model is based on o1-like models and a high-level model. |
Agri-CM3: A Chinese Massive Multi-modal, Multi-level Benchmark for Agricultural Understanding and Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks lack comprehensive evaluations, particularly in multi-level reasoning, making it difficult to identify model limitations. |
| Approach: | They propose to use Agri-CM3 to assess multi-level reasoning in agricultural management by integrating multiple data modalities. |
| Outcome: | The Agri-CM3 benchmark includes 3,939 images and 15,901 multi-level multiple-choice questions with detailed explanations. |
InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning (2025.findings-emnlp)
Copied to clipboard
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, Quanzeng You
| Challenge: | InfiMM-WebMath-40B is a dataset of interleaved image-text documents . it consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
| Approach: | InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents . it contains 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
| Outcome: | InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents . it consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens . |
MMTutorBench: The First Multimodal Benchmark for AI Math Tutoring (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for AI math tutoring largely overlook these skills. |
| Approach: | They evaluate 12 leading multimodal large language models and find clear performance gaps between them. |
| Outcome: | The proposed benchmarks show that they can solve 770 problems and provide diagnostics and guidance to students step by step. |
GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets focus on primary perception abilities and commonsense knowledge, or have low level of text comprehension difficulty, which are insufficient to reflect comprehensive capabilities of large vision-language models. |
| Approach: | They propose a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO) which sets human-level requirements for the model’s abilities, including perception, understanding, knowledge and reasoning. |
| Outcome: | The proposed model derives from native Chinese context and sets human-level requirements for its abilities, including perception, understanding, knowledge and reasoning. |
EduMARS: Can Vision-Language Models Grade Like Teachers? Benchmarking Multimodal, Rubric-Based Assessment on Chinese K-12 Answers (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for automated grading of student work fail to evaluate real student responses . existing models fail to assess real student work, especially on cognitively demanding tasks . |
| Approach: | They propose a multimodal benchmark for rubric-aligned evaluation of real Chinese K-12 student answers. |
| Outcome: | The proposed model improves performance and interpretability of existing models on EduMARS . existing models fail to perform on real-world, cognitively demanding tasks, authors say . |
E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, Min Yang
| Challenge: | despite the rapid development of Large Language Models, there is no dedicated benchmark for evaluating LLMs in Chinese K-12 education. |
| Approach: | They propose to develop a benchmark specifically tailored for Chinese K-12 education. |
| Outcome: | EVAL is the first evaluation benchmark specifically tailored for Chinese K-12 education. |
PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Numerical reasoning is ubiquitous in scientific research and financial analysis, but few benchmarks evaluate them by integrating numerical processing and mathematical reasoning. |
| Approach: | They propose a numerically-integrated hierarchical benchmark with 27,215 questions derived from 7,404 math word problems that spans 4 key cognitive aspects, 14 subcategories, and 2 modalities. |
| Outcome: | The proposed model improves Qwen-2.5 score with SOLVE and IRPO training. |
GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) and multi-modal models (MMs) have demonstrated remarkable capabilities in problem-solving, but their proficiency in tackling geometry math problems has not been thoroughly evaluated. |
| Approach: | They propose a benchmark to evaluate the performance of large language models and multi-modal models in solving geometry math problems. |
| Outcome: | The proposed model achieves 55.67% accuracy on main subset but only 6.00% accuracy on hard subset. |