Evaluating LLMs’ Mathematical and Coding Competency through Ontology-guided Interventions (2025.findings-acl)
Copied to clipboard
| Challenge: | Current large language models have shown impressive performance on logical reasoning benchmarks . however, the true depth of their competencies and robustness in reasoning tasks remains an open question . |
| Approach: | They propose a general ontology of perturbations and a semi-automatic method to apply perturbations to arithmetic reasoning and code generation datasets to test their LLMs' capabilities. |
| Outcome: | The proposed model outperforms existing models on arithmetic reasoning and code generation tasks. |
Similar Papers
GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance across various mathematical reasoning benchmarks. |
| Approach: | They introduce an adversarial grade school math dataset and explore whether LLMs can be more robust when questions are slightly changed. |
| Outcome: | The proposed method generates and verifies each intermediate thought based on its reasoning goal and calculation result. |
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024.eacl-srw)
Copied to clipboard
| Challenge: | a survey examines the landscape of mathematical problem-solving techniques . large language models have proven to be potent assets in unraveling nuances of mathematical reasoning . |
| Approach: | They examine the evolution of Large Language Models (LLMs) for solving mathematical problems . they examine the spectrum of LLM-oriented techniques proposed for solving math problems - and their challenges . |
| Outcome: | The survey examines the spectrum of proposed LLM-oriented techniques in solving math problems. |
Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) achieve impressive performance on complex benchmarks yet sometimes fail on basic math reasoning. |
| Approach: | They propose a benchmark to evaluate the efficiency of reasoning in large language models . they formalize the accuracy-verbosity tradeoff and introduce the overthinking score . |
| Outcome: | The proposed model performs well on complex benchmarks but fails on basic math reasoning . the proposed model generates 18 more tokens while achieving lower accuracy . |
MARIO: MAth Reasoning with code Interpreter Output - A Reproducible Pipeline (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack mathematical reasoning, a hurdle on the path to true artificial general intelligence. |
| Approach: | They propose a protocol for fine-tuning large language models with a Python code interpreter to enhance the text analysis of the LLMs. |
| Outcome: | The proposed protocol improves the performance of a 7B-parameter LLM on the GSM8K and MATH datasets while allowing for an outlier-free value model-based inference method. |
From A and B to A+B: Can Large Language Models Solve Compositional Math Problems? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies that create problem variants by adding perturbations to a single problem focus on the interaction between problems. |
| Approach: | They propose a pipeline with 98.2% accuracy to combine two original problems with a logical connection and to evaluate LLMs' generalization ability on the compositional problems. |
| Outcome: | The proposed pipeline can combine two original problems with a logical connection to get a new math problem and evaluate its compositional generalization on the compositional problems. |
Exposing the Achilles’ Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus on final accuracy, neglecting the critical aspect of reasoning capabilities. |
| Approach: | They propose to evaluate LLMs’ abilities to detect and correct reasoning mistakes by using rule-based methods and smaller language models. |
| Outcome: | The proposed model outperforms existing models such as GPT-4o and GPT4 in both accuracy and accuracy, but lacks data contamination and memorization concerns. |
LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated strong performance across various natural language processing tasks, but their proficiency in mathematical reasoning remains a key challenge. |
| Approach: | They propose a process-oriented framework to evaluate LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
| Outcome: | The proposed framework evaluates LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
1,729 vs. 1729: The Effect of Scripts and Formats on LLM Numeracy (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive proficiency in basic arithmetic, but little attention has been given to how they perform when numerical expressions deviate from the prevailing conventions present in their training corpora. |
| Approach: | They investigate numerical reasoning across a wide range of numeral scripts and formats . they show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats despite the underlying mathematical reasoning being identical . |
| Outcome: | The proposed methods can narrow the gap between LLMs and human models when they deviate from prevailing numerical conventions. |
Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable potential in assisting with mathematical reasoning on different downstream tasks. |
| Approach: | They propose two new tools for autoformalizing real-world mathematical definitions from Wikipedia and arXiv papers. |
| Outcome: | The proposed methods improve definitions by up to 16% and undefined errors by 43%. |
Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations focus on problem-solving from examiner perspective, overlooking a dual perspective of examiner regarding error identification and correction. |
| Approach: | They propose to use an annotated dataset to evaluate large language models from the examiner perspective and to use diverse prompts to evaluate eleven representative LLMs. |
| Outcome: | The proposed model outperforms all models while LLaMA-2-7B has comparable abilities to closed-source models GPT-3.5 and Gemini Pro. |