| Challenge: | Existing numerical reasoning models are too weak for downstream tasks like fact-checking . FERMAT evaluates models on number understanding, mathematical operations, and training dependency . |
| Approach: | They propose a multi-view evaluation set for numerical reasoning in English that evaluates models on key numerical reasoning aspects instead of reporting a single score on a whole dataset. |
| Outcome: | FERMAT evaluates models on number understanding, mathematical operations, and training dependency. |
Similar Papers
Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent benchmarks have assessed language models' numerical abilities . limitations include tokenization and representation of numbers in text, hallucination, and a lack of numerical commonsense knowledge. |
| Approach: | They propose a hierarchical taxonomy for numerical reasoning skills that includes representation, number sense, manipulation, and complex reasoning. |
| Outcome: | The proposed model outperforms other models on the tabular Natural Language Inference task. |
Self-training Language Models for Arithmetic Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work improves the reasoning capabilities of language models by scaling training data to more diverse or complex collections, but reaching further improvements becomes exceedingly expensive. |
| Approach: | They propose to use implicit feedback to improve models' reasoning capabilities by training from implicit feedback. |
| Outcome: | The proposed model can reach a correct result in +13.9% and +25.9% more cases than previous models, underlining the importance of actuality of self-training feedback. |
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022.acl-long)
Copied to clipboard
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan
| Challenge: | Existing AI systems fail to perform basic mathematical reasoning when presented in a slightly different manner. |
| Approach: | They propose a multi-task benchmark that evaluates the performance of AI systems on eight different tasks that at their core require simple arithmetic understanding. |
| Outcome: | The proposed benchmark compares the performance of AI systems on eight different tasks. |
Enhancing Numerical Reasoning with the Guidance of Reliable Reasoning Processes (2024.acl-long)
Copied to clipboard
| Challenge: | Numerical reasoning is an essential ability for NLP systems to handle numeric information. |
| Approach: | They propose a numerical reasoning method that generates reliable reasoning processes by decomposing the answer formula and aim to train models to generate the process with synthesized data. |
| Outcome: | The proposed method improves on all five datasets with an average improvement of 1.8% compared with baselines and gpt-3.5-turbo. |
Do Language Models Understand Measurements? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on numerical reasoning over text (NRoT) tests PLMs to understand numbers in contexts where numbers are an integral part of the context. |
| Approach: | They propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in probing tasks. |
| Outcome: | The proposed model distinguishes between numbers and units, which leads to significant improvement in probing tasks. |
MathPrompter: Mathematical Reasoning using Large Language Models (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) can be attributed to massive scaling of Large Language Models (LLMs). |
| Approach: | They propose a technique that improves performance of Large Language Models (LLMs) on arithmetic problems along with increased reliance in the predictions. |
| Outcome: | The proposed technique improves performance on arithmetic problems and increases confidence in the output results. |
Predicate-Guided Generation for Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that Prolog-MATH generates 81.3% solution coverage on Deepseek-V3 . |
| Approach: | They propose a curated corpus to support mathematical reasoning in large language models . they propose supervised fine-tuning followed by GRPO training to address problems that Deepseek-V3 fails to solve. |
| Outcome: | The proposed pipeline achieves 81.3% solution coverage on the Deepseek-V3 training set. |
PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Numerical reasoning is ubiquitous in scientific research and financial analysis, but few benchmarks evaluate them by integrating numerical processing and mathematical reasoning. |
| Approach: | They propose a numerically-integrated hierarchical benchmark with 27,215 questions derived from 7,404 math word problems that spans 4 key cognitive aspects, 14 subcategories, and 2 modalities. |
| Outcome: | The proposed model improves Qwen-2.5 score with SOLVE and IRPO training. |
Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models (2025.findings-acl)
Copied to clipboard
Haoyang Li, Xuejia Chen, Zhanchao Xu, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Li Qing, Lei Chen
| Challenge: | Existing benchmarks focus on linguistic competence or structured mathematical problem-solving, neglecting fundamental numerical reasoning required in real-world scenarios. |
| Approach: | They propose a benchmark to evaluate numerical capabilities for large language models . they use a dataset to assess number recognition, arithmetic operations, contextual retrieval, comparison, summary, and multi-step reasoning. |
| Outcome: | The proposed benchmark evaluates six fundamental numerical capabilities: number recognition, arithmetic operations, contextual retrieval, comparison, summary, and multi-step reasoning. |
Towards Table-to-Text Generation with Numerical Reasoning (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown improvement in generating descriptive text from structured data. |
| Approach: | They propose a framework for numerical table-to-text generation based on numerical reasoning . they use a pre-trained model and a copy mechanism to fine-tune the models to produce fluent text . |
| Outcome: | The proposed framework lacks fidelity to the table contents and is based on a pre-trained model and a copy mechanism. |