Learning from Diverse Reasoning Paths with Routing and Collaboration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies suggest that the reasoning abilities of large language models (LLMs) grows with model size and pre-training data. |
| Approach: | They propose to combine quality filtering, conditional routing, and cooperative peer teaching to transfer knowledge from powerful teacher models to compact and transparent students. |
| Outcome: | Experiments show that QR-Distill is superior to traditional methods. |
Similar Papers
DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on distilled lightweight LLMs have focused on transferring knowledge from a larger model (the teacher) to a smaller model (sector). |
| Approach: | They propose a family of distilled, lightweight LLMs derived from Qwen2.5 models. |
| Outcome: | Experimental results show that the distilled models have significantly stronger instruction-following capabilities than the original models. |
Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have proven to be highly effective in addressing a wide range of complex tasks. |
| Approach: | They propose a method that asks teachers to identify and explain student’s mistakes and then asks them to provide customized instruction learning data. |
| Outcome: | The proposed method reduces the chance of teachers guessing incorrectly with flawed rationales, improving instructional data quality. |
Beyond One-Step Distillation: Bridging the Capacity Gap in Small Language Models via Multi-Step Knowledge Transfer (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel across diverse tasks but remain too large for efficient on-device deployment. |
| Approach: | They revisit multi-step knowledge distillation as an effective remedy . they demonstrate that MSKD improves ROUGE-L and perplexity over single-step approaches . |
| Outcome: | The proposed approach improves ROUGE-L and perplexity over single-step approaches . large language models are too large for efficient on-device deployment, the authors show . |
Mentor-KD: Making Small Language Models Better Multi-step Reasoners (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive emergent capabilities by leveraging Chain-of-Thought (CoT) prompting. |
| Approach: | They propose a Knowledge Distillation approach which transfers multi-step reasoning ability of Large Language Models (LLMs) to smaller LMs by fine-tuning language models of multi- step rationales generated by LLM teachers. |
| Outcome: | The proposed method is able to transfer multi-step reasoning ability of LLMs to smaller LMs while addressing data quality and soft label provision. |
Mixed Distillation Helps Smaller Language Models Reason Better (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated impressive multiple step-by-step reasoning capabilities in recent NLP reasoning tasks. |
| Approach: | They propose a mixed distillation framework that distills multiple step-by-step reasoning abilities into smaller language models (SLMs) they leverage LLMs to generate multiple step by step reasoning rationales by sampling automatically. |
| Outcome: | The proposed framework outperforms existing models on SVAMP, GSM8K and ASDIV, while a single model generated by MD exceeds the comprehensive performance of two individual CoT and PoT distilled models. |
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing curation-based approaches to inference are inefficient and fail to adapt dynamically, leading to redundant sampling and missed opportunities for complementary reasoning. |
| Approach: | They propose a collaborative multi-teacher decoding framework that performs step-wise reasoning synthesis guided by predictive perplexity–based scoring and beam search. |
| Outcome: | The proposed framework generates higher-quality reasoning data and achieves student-level results, demonstrating that fine-grained collaboration yields structured, efficient, and robust reasoning distillation. |
QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on smaller, task-specific models enhanced by distilling knowledge from LLMs, but the diversity and quality of negative knowledge remains understudied. |
| Approach: | They propose a quality-guided contrastive rationale distillation framework that aims to enhance reasoning capabilities through contrastive knowledge learning. |
| Outcome: | The proposed method consistently outperforms existing distillation techniques yielding higher-quality rationales. |
KNOT: Knowledge Distillation Using Optimal Transport for Solving NLP Tasks (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge Distillation using Optimal Transport (KNOT) aims to distill the natural language semantic knowledge from multiple teacher networks to a student network. |
| Approach: | They propose to distill natural language semantic knowledge from multiple teacher networks to a student network by learning to minimize the optimal transport cost of its assigned probability distribution over the labels to the weighted sum of probabilities predicted by the (local) teacher models. |
| Outcome: | The proposed method shows improvements in the global model’s SD performance over the baseline across three NLP tasks while performing on par with Entropy-based distillation on standard accuracy and F1 metrics. |
Co-training and Co-distillation for Quality Improvement and Compression of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) compresses expensive pre-trained language models . however, most smaller models fail to surpass performance of larger model . |
| Approach: | They propose a framework that co-trains two models while mutually distilling knowledge to improve performance and inference speed together. |
| Outcome: | The proposed framework outperforms the original larger model by 1.66 on the GLUE benchmark. |
Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction tuning aims to align large language models (LLMs) with open-domain instructions and human-preferred responses. |
| Approach: | They propose a multi-round distillation framework that uses an oracle LLM to select instructions that are difficult for a student LLM. |
| Outcome: | The proposed framework outperforms large language models and user-tuned models on several widely recognized benchmarks and multiple student LLMs. |