Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models are notoriously inclined to make factual errors in tasks requiring arithmetic computation. |
| Approach: | They propose to combine existing chain-of-thought datasets into a unified format that can be used to train and evaluate open-source calculator-using models. |
| Outcome: | The proposed model doubles the accuracy of generating correct results compared to baseline models. |
Similar Papers
Arithmetic Reasoning with LLM: Prolog Generation & Permutation (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing work has shown that large language models can generate arithmetic and commonsense reasoning, but they are not native to mathematical operations and symbolic manipulations. |
| Approach: | They propose to use large language models to generate Prolog programs to solve math problems using a code interpreter to generate arithmetic and symbolic formulas. |
| Outcome: | The proposed model outperforms CoT generation in the GSM8K benchmark across three LLMs. |
NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs are not able to handle numerals and units of measurement, but they can be improved by introducing perturbations. |
| Approach: | They propose to analyze existing LLMs on processing numerals and units of measurement by perturbing their datasets. |
| Outcome: | The proposed model improves on ancient Chinese arithmetic problems and can handle numeral and measurement conversions. |
Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition (2022.lrec-1)
Copied to clipboard
| Challenge: | Large Language Models such as GPT-3 have demonstrated on-the-fly reasoning capabilities in NLP tasks, but they struggle with arithmetic operations. |
| Approach: | They propose a Transformer Language Model that decomposes numbers in units, tens, and so on . they introduce a pipeline that allows them to perform arithmetic operations between decomposed numbers . |
| Outcome: | The proposed model improves accuracy in addition, subtractions and multiplication tasks by 63% . the model is fine-tuned to perform arithmetic operations between decomposed numbers . |
Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective (2025.acl-long)
Copied to clipboard
Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Mahmoud Khademi, Hany Hassan Awadalla, Junjie Wang, Yujiu Yang, Furu Wei
| Challenge: | Existing work shows that LLMs rely on single-paradigm reasoning that limits their effectiveness across diverse tasks. |
| Approach: | They propose a new framework that integrates multiple reasoning paradigms to enable synergistic collaboration. |
| Outcome: | The proposed model outperforms current SOTA models in theorem proving tasks and the MATH benchmark in arithmetic tasks. |
LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers (2023.emnlp-main)
Copied to clipboard
Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, Roger Levy
| Challenge: | Logical reasoning is an important task for artificial intelligence, says a new study . many prompting-based strategies to enable large language models fail in subtle and unpredictable ways. |
| Approach: | They propose to reformulate logical reasoning tasks by leveraging large language models . they use a modular neurosymbolic programming approach to translate premises and conclusions from natural language to logic . |
| Outcome: | The proposed approach outperforms open-source models on FOLIO and ProofWriter while showing distinct failure modes. |
Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency (2023.acl-long)
Copied to clipboard
| Challenge: | Numeracy is the most prevalent form of non-linguistic information embedded in textual corpora. |
| Approach: | They propose a framework for non-linguistic skill injection for LLMs that incorporates information-theoretic interventions and skill-specific losses to enable the learning of strict arithmetic reasoning. |
| Outcome: | The proposed model outperforms the state-of-the-art on injected non-linguistic skills and on linguistic knowledge retention with a fraction of the non-language training data (1/4) and zero additional synthetic linguistic training data. |
Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step (2023.acl-long)
Copied to clipboard
| Challenge: | Symbolic Chain-of-thought Distillation (SCoTD) is a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. |
| Approach: | They propose a method to train a smaller student model on rationalizations from a larger teacher model. |
| Outcome: | The proposed method improves the performance of a student model in supervised and few-shot settings and especially for challenge sets. |
A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers (2024.naacl-long)
Copied to clipboard
| Challenge: | evaluating the generalisability of Transformers to out-of-distribution mathematical reasoning problems is a challenge for many open-source models. |
| Approach: | They propose a method for generating and perturbing detailed derivations of equations at scale, aided by a symbolic engine, and compare their results to sequence classification tasks. |
| Outcome: | The proposed framework outperforms GPT-4, GPT-3.5 and a canon of fine-tuned BERT models in classification tasks . perturbations to input reasoning can reduce their performance by up to 80 F1 points . |
Limitations of Language Models in Arithmetic and Symbolic Induction (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that large pretrained Language Models (LMs) can perform remarkably well on a range of NLP tasks but they have limitations on basic symbolic manipulation tasks such as copy, reverse, and addition. |
| Approach: | They propose to use explicit positional markers, fine-grained computation steps, and LMs with callable programs to teach large pretrained Language Models. |
| Outcome: | The proposed model can perform 100% accuracy in OOD and repeating symbols. |
From A and B to A+B: Can Large Language Models Solve Compositional Math Problems? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies that create problem variants by adding perturbations to a single problem focus on the interaction between problems. |
| Approach: | They propose a pipeline with 98.2% accuracy to combine two original problems with a logical connection and to evaluate LLMs' generalization ability on the compositional problems. |
| Outcome: | The proposed pipeline can combine two original problems with a logical connection to get a new math problem and evaluate its compositional generalization on the compositional problems. |