LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics (2026.findings-eacl)
Copied to clipboard
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Shusaku Sone, Masaya Taniguchi, Ana Brassard, Keisuke Sakaguchi, Kentaro Inui
| Challenge: | Specifically, we examine when the LLMs’ answer is (pre)determined, especially before the CoT begins or after, and how strongly the information from CoT specifically has a causal effect on the final answer. |
| Approach: | They examine when the LLMs’ answer is (pre)determined, especially before the CoT begins or after, and how strongly the information from CoT specifically has a causal effect on the final answer. |
| Outcome: | The proposed model can generate reasoning chains while generating the reasoning chain on the fly. |
Similar Papers
Over-Reasoning and Redundant Calculation of Large Language Models (2024.eacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve problems step-by-step, but it is unclear whether they know when to use CoT and whether they are always necessary. |
| Approach: | They propose to use LLMs to generate redundant calculations and reasoning on a manually constructed math QA dataset, GSM8K-Zero. |
| Outcome: | The proposed model generates redundant calculations and reasoning on a manually constructed math QA dataset, but it is unclear whether it is necessary to use CoT reasoning. |
How Likely Do LLMs with CoT Mimic Human Reasoning? (2025.coling-main)
Copied to clipboard
| Challenge: | Using chain-of-thought to elicit reasoning capabilities is not always effective and accurate. |
| Approach: | They compare the reasoning process of LLMs with humans to understand the causal chain . they find that LLM deviates from the ideal causal chain, resulting in spurious correlations . |
| Outcome: | The proposed method does not improve performance or accurately represent reasoning processes in LLMs. |
Memory or Reasoning? Explore How LLMs Compute Mixed Arithmetic Expressions (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve complex multi-step math reasoning problems, but their internal implementation is limited. |
| Approach: | They propose to use a "C**ausal **E**ffect **D**riven **F**ine-tuning method" to improve LLMs' reasoning ability. |
| Outcome: | The proposed method improves the model's reasoning ability by enhancing key components that are used to execute mixed arithmetic calculations. |
Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are often evaluated on math word problems . however, such metrics conflate two distinct sub-skills: abstract formulation and arithmetic computation. |
| Approach: | They propose to use Final-answer-based metrics to evaluate large language models on math word problems to conflate two distinct sub-skills: abstract formulation and arithmetic computation. |
| Outcome: | The proposed model performance is bottlenecked by arithmetic computation and not abstract formulation, the study shows. |
Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) take advantage of step-by-step reasoning instructions . negation is a core linguistic phenomenon that is difficult to process . |
| Approach: | They examine the step-by-step reasoning ability of large language models with a focus on negation . negation is a core linguistic phenomenon that is difficult to process . |
| Outcome: | The proposed models perform better when using chain-of-thought prompting . the results highlight unique limitations in each LLM family . |
Arithmetic Reasoning with LLM: Prolog Generation & Permutation (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing work has shown that large language models can generate arithmetic and commonsense reasoning, but they are not native to mathematical operations and symbolic manipulations. |
| Approach: | They propose to use large language models to generate Prolog programs to solve math problems using a code interpreter to generate arithmetic and symbolic formulas. |
| Outcome: | The proposed model outperforms CoT generation in the GSM8K benchmark across three LLMs. |
Large Language Models are Better Reasoners with Self-Verification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to solve complex natural language processing tasks require multiple steps to verify the answers. |
| Approach: | They propose to use chain of thought prompting to solve reasoning tasks with large language models. |
| Outcome: | The proposed method can improve reasoning performance on arithmetic, commonsense, and logical reasoning datasets. |
A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent work has examined final-answer accuracy in multilingual settings, but the behavior of thinking traces, i.e., the intermediate steps that lead to the final answer, remains underexplored. |
| Approach: | They propose to measure language compliance, answer accuracy, and answer consistency when LRMs are explicitly instructed or prompt-hacked to think in a target language. |
| Outcome: | The proposed model improves in English and other high-resource languages while relying on traces to varying degrees. |
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2023.acl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). |
| Approach: | They propose to use Chain-of-Thought (CoT) prompting to encourage the LLM to generate intermediate rationales for solving a problem by providing a series of reasoning steps in the demonstrations. |
| Outcome: | The proposed model can generate coherent lines of reasoning even with invalid demonstrations while still generating coherent lines during inference. |
DeCoT: Debiasing Chain-of-Thought for Knowledge-Intensive Tasks in Large Language Models via Causal Intervention (2024.acl-long)
Copied to clipboard
| Challenge: | In large language models, external knowledge is required to augment their internal knowledge through prompts, but this does not guarantee that LLMs can identify and use relevant information in the prompts to conduct chain-of-thought reasoning. |
| Approach: | They propose a structural causal model to formally explain the internal knowledge bias of large language models (LLMs) they review the chain-of-thought (CoT) prompting from a causal perspective and find that biased information from pretrained models can impair LLMs’ reasoning abilities. |
| Outcome: | The proposed model enables more accurate CoT reasoning and enhances LLM generation on knowledge-intensive tasks. |