Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve mathematical reasoning require extensive datasets for training or depend on few-shot methods that compromise computational accuracy. |
| Approach: | They propose a training-free adaptation framework that efficiently equips general-purpose pre-trained language models with enhanced mathematical reasoning capabilities. |
| Outcome: | The proposed framework outperforms Qwen2.5-72B-Math-Instruct on MMLU-STEM with a score of 90.9%, compared to 87.3%. |
Similar Papers
Learning Multi-Step Reasoning by Solving Arithmetic Tasks (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies have demonstrated large LMs’ impressive performance in solving math problems, but such ability seems only to emerge from models with abundant parameters. |
| Approach: | They propose to continuously pre-train LMs with the capabilities of multi-step reasoning by continuously pretraining them on a synthetic dataset MsAT. |
| Outcome: | The proposed method improves LMs' multi-step reasoning abilities on four math word problem datasets. |
Enhancing Mathematical Reasoning in LLMs by Stepwise Correction (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Best-of-N decoding methods often lead to incorrect solutions . a novel method is proposed to help large language models identify and revise incorrect steps in their generated reasoning paths. |
| Approach: | They propose a method that helps large language models identify and revise incorrect steps in their generated reasoning paths. |
| Outcome: | The proposed method outperforms the state-of-the-art Best-ofN decoding method by +2.4 and reduces token consumption by 77.8%. |
First-Step Advantage: Importance of Starting Right in Multi-Step Math Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Language models can solve complex reasoning tasks better by learning to generate rationales for their predictions. |
| Approach: | They propose to use a larger model to guide smaller models to start . this allows them to generate rationales for their predictions when correct . |
| Outcome: | The proposed method improves performance on multistep reasoning datasets over multiple smaller models. |
Multilingual Reasoning via Self-training (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have introduced eclectic strategies to improve reasoning beyond English, but these methods are related to specific language that is not always optimal for reasoning. |
| Approach: | They propose a modular approach that instructs models to structure reasoning passages in a different problem space and then self-refines their capabilities to deliver step-wise reasoning passage. |
| Outcome: | The proposed approach achieves significant improvements in multilingual reasoning of various models and task, with improved reasoning consistency across languages. |
Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verification (2025.acl-long)
Copied to clipboard
Chengwu Liu, Ye Yuan, Yichun Yin, Yan Xu, Xin Xu, Zaoyu Chen, Yasheng Wang, Lifeng Shang, Qun Liu, Ming Zhang
| Challenge: | Chain-of-Thought prompting is a de facto method to elicit reasoning capabilities from large language models (LLMs). |
| Approach: | They propose a step-aware formal verification framework Safe to address hallucinations in CoT prompting . they propose 'formal step' as a benchmark for step correctness theorem proving with 30,809 formal statements. |
| Outcome: | The proposed framework shows significant performance improvement while offering interpretable and verifiable evidence. |
OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with maintaining accuracy throughout multiple reasoning steps, especially in mathematical reasoning where an error in earlier steps can propagate to subsequent ones and ultimately leading to an incorrect answer. |
| Approach: | They propose an Outcome-supervised Value Model (OVM) that employs outcome supervision for training a value model, which prioritizes steps that lead to accurate conclusions. |
| Outcome: | The proposed model performs better on two multi-step reasoning datasets, GSM8K and Game of 24. |
The Impact of Reasoning Step Length on Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Long reasoning steps in LLMs improve reasoning abilities, but the correlation between their effectiveness and the length of reasoning steps remains largely unknown. |
| Approach: | They conducted experiments that expand and compress the rationale reasoning steps within CoT demonstrations while keeping all other factors constant. |
| Outcome: | The results show that lengthening the reasoning steps in prompts significantly enhances LLMs’ reasoning abilities across multiple datasets. |
Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances on prompting and post-training have enabled LLMs to perform step-wise reasoning tasks, but they tend to explore unproductive solution paths without effective backtracking or strategy adjustment. |
| Approach: | They propose a framework that empowers LLMs to “think about how to think” and dynamically adapts reasoning strategies in real-time. |
| Outcome: | The proposed framework outperforms previous SOTA methods by 9-12% in accuracy while reducing inference time by 28-35% under the same compute budget. |
Training in Step-by-Step Formal Reasoning Improves Pronominal Reasoning in Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Large reasoning models are limited to formal reasoning, i.e., math, code, and logic. |
| Approach: | They evaluate a set of large reasoning models on a dataset for pronoun resolution and fidelity. |
| Outcome: | The results show that distilling step-by-step formal reasoning improves pronoun resolution and fidelity. |
Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Advanced models such as OpenAI o1 exhibit impressive problem-solving capabilities, but they may still falter on more complex problems, making errors that disrupt their reasoning paths. |
| Approach: | They propose a framework that encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model’s overall problem-solving performance. |
| Outcome: | The proposed framework improves reasoning performance on multi-step reasoning tasks such as math word problems and science-based exam questions. |