Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods train small language models to learn long rationales in one iteration. |
| Approach: | They propose a method that uses a heuristic search to divide rationale into internal chunks . they propose CWT, which uses CWt to focus SLM on learning from only one chunk per iteration. |
| Outcome: | The proposed method can guide a large language model (LLM) in reasoning tasks. |
Similar Papers
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning (2025.findings-acl)
Copied to clipboard
Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, Xiaoyu Shen
| Challenge: | Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought prompting. |
| Approach: | They examine the factors influencing CoT distillation including granularity, format and teacher model. |
| Outcome: | The proposed model is based on four teacher models and seven student models across seven mathematical and commonsense reasoning datasets. |
Mixed Distillation Helps Smaller Language Models Reason Better (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated impressive multiple step-by-step reasoning capabilities in recent NLP reasoning tasks. |
| Approach: | They propose a mixed distillation framework that distills multiple step-by-step reasoning abilities into smaller language models (SLMs) they leverage LLMs to generate multiple step by step reasoning rationales by sampling automatically. |
| Outcome: | The proposed framework outperforms existing models on SVAMP, GSM8K and ASDIV, while a single model generated by MD exceeds the comprehensive performance of two individual CoT and PoT distilled models. |
Aligning Large and Small Language Models via Chain-of-Thought Reasoning (2024.eacl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting empowers Large Language Models to solve complex reasoning tasks in a step-wise manner. |
| Approach: | They propose a method for aligning and transferring reasoning abilities between larger and smaller Language Models by using CoT-Demonstrations. |
| Outcome: | The proposed method outperforms baselines on question-answering and mathematical reasoning benchmarks. |
Capture the Key in Reasoning to Enhance CoT Distillation Generalization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing distillation methods for Large Language Models (LLMs) focus on fine-tuning student SLMs on correct data, resulting in students struggling to learn the key instead of analyzing mistakes according to correct solutions. |
| Approach: | They propose a method that exposes key reasoning steps rather than simple fine-tuning students' CoTs data by using a set of prompts with similar reasoning paths but divergent conclusions. |
| Outcome: | The proposed method improves student SLMs' ability to learn key reasoning steps rather than fine-tuning them on teacher data. |
Distilling Reasoning Capabilities into Smaller Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | a step-by-step reasoning approach like chain of thought has proved to be effective in eliciting reasoning abilities in large language models. |
| Approach: | They propose a knowledge distillation approach that leverages CoT reasoning capabilities of larger models and distills them into smaller models. |
| Outcome: | The proposed scheme boosts the performance of smaller models over 70% on multiple reasoning datasets. |
Self-Training Elicits Concise Reasoning in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Chain-of-thought reasoning has enabled large language models to use additional computation through intermediate tokens to solve complex tasks, but current models often generate more tokens than necessary to accomplish the task, incurring extraneous inference costs. |
| Approach: | They propose to fine-tune models with self-generated concise reasoning paths obtained by best-of-N sampling and few-shot conditioning in task-specific settings to elicit concise reasoning. |
| Outcome: | The proposed method reduces output tokens by 30% on GSM8K and MATH while maintaining average accuracy. |
Small Models Struggle to Learn from Strong Reasoners (2025.findings-acl)
Copied to clipboard
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, Radha Poovendran
| Challenge: | a small learning gap exists between large and small language models . long CoT data and large model responses are not beneficial for small models - a problem that may be due to the small student model's ability to handle distribution shifts. |
| Approach: | They propose a mix distillation strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models. |
| Outcome: | The proposed strategy outperforms training on large and small models on short CoT and small model CoT. |
Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing frameworks for guiding a language model in reasoning tasks are limited by their tendency to generate low-quality rationales that are repetitive and vacuous. |
| Approach: | They propose a framework that leverages a lightweight language model for guiding a black-box large LM in reasoning tasks. |
| Outcome: | The proposed framework outperforms baselines in answer prediction accuracy. |
CAC-CoT: Connector-Aware Compact Chain-of-Thought for Efficient Reasoning Data Synthesis Across Dual-System Cognitive Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Long chain-of-thought (CoT) prompting often slows or even degrades performance on fast, intuitive "System-1" tasks. |
| Approach: | They introduce a method that deliberately restricts reasoning to a small, fixed set of connector phrases, steering the model toward concise and well-structured explanations. |
| Outcome: | The method achieves 85% on GSM8K and 40% on GPQA while also surpassing the baseline by over 20%. |
PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models excel in various tasks, but their huge size and inaccessibility of parameters present challenges for practical deployment. |
| Approach: | They propose to use CoT data to distill task-specific ability from large language models to smaller models . they use reasoning programs to suppress errors in distilled data and improve distillation quality . |
| Outcome: | The proposed model outperforms LLMs on arithmetic reasoning, symbolic reasoning, and general ability. |