MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning (2026.findings-eacl)
Copied to clipboard
Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Existing benchmarks primarily focus on English or a narrow subset of high-resource languages, leaving significant gaps in assessing multilingual and cross-lingual mathematical reasoning. |
| Approach: | They propose a parallel multilingual benchmark for mathematical problem solving and reasoning that encompasses 2,890 parallel Bangla-English gold standard artifacts. |
| Outcome: | The proposed model encompasses 2,890 parallel Bangla-English gold standard artifacts, totaling 30K aligned question–answer pairs across thirteen languages, representing high-, medium-, and low-resource linguistic settings. |
Similar Papers
MMATH: A Multilingual Benchmark for Mathematical Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a benchmark for multilingual complex reasoning spans 374 high-quality math problems across 10 typologically diverse languages. |
| Approach: | They propose a benchmark for multilingual complex reasoning across 10 languages . they show reasoning in English and answering in target languages can enhance performance . |
| Outcome: | The proposed benchmark demonstrates that models with high-quality reasoning can perform in multiple languages. |
Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research focuses on developing powerful large language models for mathematical reasoning within monolingual languages. |
| Approach: | They propose to use translation to build powerful multilingual math reasoning models . they propose different training strategies to build xMR LLMs that outperform open-source LLM . |
| Outcome: | The proposed model outperforms open-source LLMs and surpasses ChatGPT in few-shot scenarios. |
Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new benchmark for evaluating the mathematical reasoning on large language models is being developed . popularity of reasoning benchmarks is leading to performance saturation and training set contamination. |
| Approach: | They introduce a benchmark for evaluating the mathematical reasoning on large language models . they find that models struggle with Mathador-LM, scoring lower than average 3rd graders . |
| Outcome: | The proposed benchmark improves performance on large language models . it also reduces test-set leakage into training data, a new study shows . |
LLM Parameters for Math Across Languages: Shared or Separate? (2026.acl-srw)
Copied to clipboard
Behzad Shomali, Luisa Victor, Tim Selbach, Ali Hamza Bashir, David Berghaus, Joachim Koehler, Mehdi Ali, Markus Frey
| Challenge: | Existing research on large language models (LLMs) has focused on performance or representational properties, but it remains unclear whether these differences reflect language-specific parameters or a shared mechanism. |
| Approach: | They propose to localize and compare model parameters that support mathematical reasoning across languages. |
| Outcome: | The proposed analysis shows that the model parameters in English and lower-resource languages exhibit partial cross-lingual overlap with systematic language-dependent differences. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs (2025.emnlp-main)
Copied to clipboard
Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou
| Challenge: | Recent advances in large language models showcase varied multilingual capabilities across tasks . previous assessments focused on fundamental natural language processing (NLP) or isolated capability-specific tasks. |
| Approach: | They propose a multilingual multitask benchmark to assess multilingual capabilities . they use a large-scale benchmark covering fundamental and capability-specialized datasets . |
| Outcome: | The proposed benchmark compares models and tasks across languages and tasks and examines knowledge transfer from English to other languages. |
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (2025.emnlp-main)
Copied to clipboard
Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, Felix Juefei-Xu, Foutse Khomh, Osamu Yoshie, Qingyu Chen, Douglas Teodoro, Nan Liu, Randy Goebel, Lei Ma, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
| Challenge: | Existing large language model evaluation benchmarks focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. |
| Approach: | They propose a comprehensive benchmark covering 29 languages, built on an English benchmark. |
| Outcome: | The MMLU-ProX is a comprehensive benchmark covering 29 languages, built on an English benchmark. |
Beyond Accuracy: Alignment and Error Detection across Languages in the Bi-GSM8K Math-Teaching Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in LLMs have significantly improved mathematical problem-solving, with models like GPT-4 achieving human-level performance. |
| Approach: | They propose a bilingual English-Korean dataset enriched with teacher solutions, student solutions, and annotations marking students’ initial errors. |
| Outcome: | The proposed model achieves high agreement with human judgments and lower latency and resource usage than commercial APIs, demonstrating strong computational efficiency. |
Multi-lingual Functional Evaluation for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Multilingual competence in large language models is often evaluated via static data benchmarks such as Belebele, M-MMLU and M-GSM. |
| Approach: | They extend existing functional benchmark templates from English to five additional languages that span the range of resources available for NLP: French, Spanish, Hindi, Arabic and Yoruba. |
| Outcome: | The proposed models are translated from English to French, Spanish, Hindi, Arabic and Yoruba. |
MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark (2024.findings-acl)
Copied to clipboard
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, Kai Chen
| Challenge: | Recent advances in large language models have showcased significant improvements in mathematics, but traditional benchmarks like GSM8k offer a unidimensional perspective. |
| Approach: | MathBench is a benchmark that rigorously assesses the mathematical capabilities of large language models. |
| Outcome: | MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. |