Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel at complex math but fail on basic addition, raising the question of whether they grasp rules or are merely reproducing patterns. |
| Approach: | They systematically probe LLMs’ understanding of two-integer addition by testing three crucial properties: commutativity (A+B=B+A), representation invariance via symbolic remapping and consistent accuracy scaling with operand length. |
| Outcome: | The proposed models achieve high numeric accuracy but fail basic addition tasks. |
Similar Papers
From A and B to A+B: Can Large Language Models Solve Compositional Math Problems? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies that create problem variants by adding perturbations to a single problem focus on the interaction between problems. |
| Approach: | They propose a pipeline with 98.2% accuracy to combine two original problems with a logical connection and to evaluate LLMs' generalization ability on the compositional problems. |
| Outcome: | The proposed pipeline can combine two original problems with a logical connection to get a new math problem and evaluate its compositional generalization on the compositional problems. |
Limitations of Language Models in Arithmetic and Symbolic Induction (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that large pretrained Language Models (LMs) can perform remarkably well on a range of NLP tasks but they have limitations on basic symbolic manipulation tasks such as copy, reverse, and addition. |
| Approach: | They propose to use explicit positional markers, fine-grained computation steps, and LMs with callable programs to teach large pretrained Language Models. |
| Outcome: | The proposed model can perform 100% accuracy in OOD and repeating symbols. |
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |
The Impact of Language on Arithmetic Proficiency: A Multilingual Investigation with Cross-Agent Checking Computation (2024.naacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) have garnered significant attention over the past year . previous studies have evaluated LLMs' performance in solving math word problems, but there is little discussion on whether they comprehend the operations they generate. |
| Approach: | They challenge the notion that arithmetic is language-independent and compare models with cross-agent collaborations to find significant limitations in their performance. |
| Outcome: | The proposed model outperforms collaborative approaches in basic arithmetic tasks. |
1,729 vs. 1729: The Effect of Scripts and Formats on LLM Numeracy (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive proficiency in basic arithmetic, but little attention has been given to how they perform when numerical expressions deviate from the prevailing conventions present in their training corpora. |
| Approach: | They investigate numerical reasoning across a wide range of numeral scripts and formats . they show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats despite the underlying mathematical reasoning being identical . |
| Outcome: | The proposed methods can narrow the gap between LLMs and human models when they deviate from prevailing numerical conventions. |
Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks (2024.acl-short)
Copied to clipboard
| Challenge: | Despite the generality and far-reaching consequences of large language models, there are still significant limitations making it difficult to apply them to certain tasks. |
| Approach: | They show that large language models can perform arithmetic tasks more robustly when conditioned on all of the correct higher-order digits. |
| Outcome: | The proposed model can predict the first digit of n-digit by m-digit multiplication without chain of thought reasoning, but in practice it fails to correctly predict the last digit on n digit by 1-digit multiplikation . |
GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance across various mathematical reasoning benchmarks. |
| Approach: | They introduce an adversarial grade school math dataset and explore whether LLMs can be more robust when questions are slightly changed. |
| Outcome: | The proposed method generates and verifies each intermediate thought based on its reasoning goal and calculation result. |
Probing for Arithmetic Errors in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a controlled setting of 3-digit addition, we show that simple probes can accurately decode both the model’s output and the correct answer from hidden states. |
| Approach: | They extend their analysis to structured chain-of-thought traces on addition-only GSM8K problems and find that probes trained on simple arithmetic generalize well to this more complex setting, revealing consistent internal representations. |
| Outcome: | The proposed probes can predict model correctness with over 90% accuracy on addition-only GSM8K problems and guide selective re-prompting of erroneous reasoning steps with minimal disruption to correct outputs. |
Memory or Reasoning? Explore How LLMs Compute Mixed Arithmetic Expressions (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve complex multi-step math reasoning problems, but their internal implementation is limited. |
| Approach: | They propose to use a "C**ausal **E**ffect **D**riven **F**ine-tuning method" to improve LLMs' reasoning ability. |
| Outcome: | The proposed method improves the model's reasoning ability by enhancing key components that are used to execute mixed arithmetic calculations. |
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs (2025.findings-acl)
Copied to clipboard
Guhao Feng, Kai Yang, Yuntian Gu, Xinyue Ai, Shengjie Luo, Jiacheng Sun, Di He, Zhenguo Li, Liwei Wang
| Challenge: | Despite the success of transformer-based large language models, understanding and enhancing their mathematical capabilities remains a significant challenge. |
| Approach: | They propose to use numerical precision as a key factor that influences LLMs' effectiveness in arithmetical tasks to determine their effectiveness. |
| Outcome: | The proposed models perform better in arithmetic tasks than transformer-based models with standard numerical precision. |