Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency (2023.acl-long)
Copied to clipboard
| Challenge: | Numeracy is the most prevalent form of non-linguistic information embedded in textual corpora. |
| Approach: | They propose a framework for non-linguistic skill injection for LLMs that incorporates information-theoretic interventions and skill-specific losses to enable the learning of strict arithmetic reasoning. |
| Outcome: | The proposed model outperforms the state-of-the-art on injected non-linguistic skills and on linguistic knowledge retention with a fraction of the non-language training data (1/4) and zero additional synthetic linguistic training data. |
Similar Papers
Evaluating Numeracy of Language Models as a Natural Language Inference Task (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enhanced their capabilities to solve mathematical problems, but other aspects of numeracy remain underexplored. |
| Approach: | They propose to frame numeracy as a Natural Language Inference task to assess the models’ ability to understand both numbers and language contexts. |
| Outcome: | The proposed model outperforms smaller models in arithmetic tasks, indicating that mathematical reasoning cannot be generalized to other numeracy skills such as number comparison and normalization. |
Limitations of Language Models in Arithmetic and Symbolic Induction (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that large pretrained Language Models (LMs) can perform remarkably well on a range of NLP tasks but they have limitations on basic symbolic manipulation tasks such as copy, reverse, and addition. |
| Approach: | They propose to use explicit positional markers, fine-grained computation steps, and LMs with callable programs to teach large pretrained Language Models. |
| Outcome: | The proposed model can perform 100% accuracy in OOD and repeating symbols. |
Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks (2024.acl-short)
Copied to clipboard
| Challenge: | Despite the generality and far-reaching consequences of large language models, there are still significant limitations making it difficult to apply them to certain tasks. |
| Approach: | They show that large language models can perform arithmetic tasks more robustly when conditioned on all of the correct higher-order digits. |
| Outcome: | The proposed model can predict the first digit of n-digit by m-digit multiplication without chain of thought reasoning, but in practice it fails to correctly predict the last digit on n digit by 1-digit multiplikation . |
Towards Robust Evaluation of Unlearning in LLMs via Data Transformations (2024.findings-emnlp)
Copied to clipboard
Abhinav Joshi, Shaswati Saha, Divyaksh Shukla, Sriram Vema, Harsh Jhamtani, Manas Gaur, Ashutosh Modi
| Challenge: | Large Language Models (LLMs) have shown to be a great success in a wide range of applications ranging from regular NLP-based use cases to AI agents. |
| Approach: | They examine the robustness of existing MUL techniques for their ability to enable leakage-proof forgetting in LLMs. |
| Outcome: | The proposed methods can be used to enable leakage-proof forgetting in LLMs. |
1,729 vs. 1729: The Effect of Scripts and Formats on LLM Numeracy (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive proficiency in basic arithmetic, but little attention has been given to how they perform when numerical expressions deviate from the prevailing conventions present in their training corpora. |
| Approach: | They investigate numerical reasoning across a wide range of numeral scripts and formats . they show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats despite the underlying mathematical reasoning being identical . |
| Outcome: | The proposed methods can narrow the gap between LLMs and human models when they deviate from prevailing numerical conventions. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
The Impact of Language on Arithmetic Proficiency: A Multilingual Investigation with Cross-Agent Checking Computation (2024.naacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) have garnered significant attention over the past year . previous studies have evaluated LLMs' performance in solving math word problems, but there is little discussion on whether they comprehend the operations they generate. |
| Approach: | They challenge the notion that arithmetic is language-independent and compare models with cross-agent collaborations to find significant limitations in their performance. |
| Outcome: | The proposed model outperforms collaborative approaches in basic arithmetic tasks. |
Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel at complex math but fail on basic addition, raising the question of whether they grasp rules or are merely reproducing patterns. |
| Approach: | They systematically probe LLMs’ understanding of two-integer addition by testing three crucial properties: commutativity (A+B=B+A), representation invariance via symbolic remapping and consistent accuracy scaling with operand length. |
| Outcome: | The proposed models achieve high numeric accuracy but fail basic addition tasks. |
GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance across various mathematical reasoning benchmarks. |
| Approach: | They introduce an adversarial grade school math dataset and explore whether LLMs can be more robust when questions are slightly changed. |
| Outcome: | The proposed method generates and verifies each intermediate thought based on its reasoning goal and calculation result. |
Language Models Learn Universal Representations of Numbers and Here’s Why You Should Care (2026.acl-long)
Copied to clipboard
Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
| Challenge: | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Approach: | They show that large language models often converge to accurate input embedding for numbers, based on sinusoidal representations. |
| Outcome: | The proposed representations are strikingly systematic, and are interchangeable in a large swathe of experimental setups. |