Learning to Reason via Self-Iterative Process Feedback for Small Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing SLMs’ reasoning depend on costly external signals, resulting in SLM overly confident with limited supervision signals. |
| Approach: | They propose to fine-tune and align SLMs using positive and negative feedback signals and introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models. |
| Outcome: | The proposed method improves Gemma-2B's performance on GSM8K and MBPP, and out-of-domain generalization capabilities on MMLU_Math and HumanEval. |
Similar Papers
Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to align reasoning abilities between Large Language Models and Smaller Language Model are supervised fine-tuning and preference optimization. |
| Approach: | They propose a method that elicits Smaller Language Models to self-improve their reasoning abilities via preference optimization. |
| Outcome: | The proposed method outperforms Instruction-tuning on commonsense and math reasoning tasks on common and math scenarios. |
SIKeD: Self-guided Iterative Knowledge Distillation for Mathematical Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can generate intermediate reasoning process for multistep reasoning tasks. |
| Approach: | They propose a distillation method that teaches the model to approach a task using different strategies and the model uses its self-generated on-policy outputs to choose the most suitable strategy. |
| Outcome: | The proposed method significantly outperforms distillation techniques on large models of different sizes. |
Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies focus on enhancing large-scale language models' reasoning abilities, but the research question of how to GSM8K Performance vs. computational cost remains. |
| Approach: | They propose to train small-scale language models with their own outputs to avoid relying on large models' outputs. |
| Outcome: | The proposed approach outperforms baseline models with comparable sizes while minimizing the required compute. |
Small Language Models Need Strong Verifiers to Self-Correct Reasoning (2024.findings-acl)
Copied to clipboard
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
| Challenge: | Existing studies show that large language models can self-correct their outputs by generating a critique and revising it based on the critique. |
| Approach: | They propose a pipeline that prompts small language models to collect self-correction data that supports the training of self-refinement abilities. |
| Outcome: | The proposed pipeline improves the self-correction abilities of two models on five datasets spanning math and commonsense reasoning. |
LightReasoner: Can Small Language Models Teach Large Language Models Reasoning? (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable progress in reasoning, but are resource-intensive and require large curated datasets. |
| Approach: | They propose a framework that leverages the behavioral divergence between a stronger expert model and a weaker amateur model. |
| Outcome: | The proposed framework improves accuracy by up to 28.1% while reducing time consumption by 90% and tuning token usage by 99%. |
Large Language Models Can Self-Improve (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have excellent performance in various tasks, but fine-tuning requires extensive supervision. |
| Approach: | They propose to use a pre-trained Large Language Model to generate rationale-augmented answers for unlabeled questions and fine-tune the LLM using those self-generated solutions as target outputs. |
| Outcome: | The proposed approach improves the general reasoning ability of a 540B-parameter LLM without any ground truth label. |
Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models can solve complex reasoning tasks with Chain-of-Thought Prompting. |
| Approach: | They propose a training method where the LLM is tasked to explore the first wrong step within the rationale and use such signals as fine-grained rewards for further improvement. |
| Outcome: | The proposed model improves on the GSM8K and MATH test sets by 11.57% and 2.89% on average compared to supervised fine-tuning (SFT). |
SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve numerical and logical reasoning of Large Language Models are limited . existing approaches rely on prompt engineering and pretrained knowledge to ensure correctness . |
| Approach: | They propose to train LLMs with process-based reasoning using a dynamic value margin . they use the Bellman optimality equation to derive a value margin for step-level preference optimization . |
| Outcome: | The proposed method is equivalent to on-policy policy gradient methods under constrained reward functions. |
Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable abilities, but they invariably generate flawed responses. |
| Approach: | They propose a self-correction approach that instructs VLMs to refine their outputs by allowing them to learn from their self-generated self-reference data without external feedback. |
| Outcome: | The proposed approach enables VLMs to learn from their self-generated self-correction data without relying on external feedback, facilitating self-improvement. |
Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Empirical evaluations on eight recent LLMs reveal that DRPO significantly enhances alignment performance, enabling base models to outperform their SFT/RLHF-tuned counterparts. |
| Approach: | They propose a tuning-free approach to self-alignment called Dynamic Rewarding with Prompt Optimization (DRPO) it leverages a dynamic rewarding mechanism to identify and rectify alignment weaknesses . |
| Outcome: | The proposed approach outperforms existing methods and is highly adaptable to various alignment challenges. |