KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Empirical evaluation shows that our approach yields superior performance in both standard task metrics and large language model (LLM)-based evaluation. |
| Approach: | They propose a K-step return estimation method for reinforcement learning (RL)-based knowledge distillation in text generation tasks using the Bellman Optimality Equation. |
| Outcome: | The proposed method performs better on standard task metrics and large language model evaluations on three text generation tasks. |
Similar Papers
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models (2023.acl-short)
Copied to clipboard
Jianyi Zhang, Aashiq Muhamed, Aditya Anantharaman, Guoyin Wang, Changyou Chen, Kai Zhong, Qingjun Cui, Yi Xu, Belinda Zeng, Trishul Chilimbi, Yiran Chen
| Challenge: | Knowledge distillation (KD) is an effective compression technique to derive a smaller student model from a larger teacher model by transferring the knowledge embedded in the teacher's network. |
| Approach: | They propose a framework and loss function that preserves the semantic similarities of teacher and student training examples to enable the student to retrieve from the knowledge base effectively. |
| Outcome: | The proposed framework preserves the semantic similarities of teacher and student training examples to achieve state-of-the-art performance on the GLUE benchmark. |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models have demonstrated remarkable performance in various NLP tasks, but are typically computationally expensive and difficult to be deployed in resource-constrained environments. |
| Approach: | They propose a knowledge distillation method based on a reward function induced from large language models. |
| Outcome: | The proposed method outperforms traditional methods on multiple datasets and tasks. |
Beyond One-Step Distillation: Bridging the Capacity Gap in Small Language Models via Multi-Step Knowledge Transfer (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel across diverse tasks but remain too large for efficient on-device deployment. |
| Approach: | They revisit multi-step knowledge distillation as an effective remedy . they demonstrate that MSKD improves ROUGE-L and perplexity over single-step approaches . |
| Outcome: | The proposed approach improves ROUGE-L and perplexity over single-step approaches . large language models are too large for efficient on-device deployment, the authors show . |
Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a promising solution for large language models, but their deployment remains computationally expensive. |
| Approach: | They propose a framework which iteratively balances training data within a fixed computational budget and enables the transfer of knowledge from expensive teacher LLMs to smaller student models. |
| Outcome: | The proposed framework achieves state-of-the-art performance across diverse long-tailed datasets, enhancing both the efficiency and efficacy of the distilled models. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) enables the compression of large language models (LLMs) conventional methods suffer from training-inference mismatches and suboptimal performance due to expensive student-generated outputs. |
| Approach: | They propose a method that combines a CL strategy and adaptive loss design to reduce training mismatches and suboptimal performance. |
| Outcome: | L2M-KD outperforms existing white-box KD methods on instruction-following tasks . it outperformed existing methods, achieving superior student model performance with reduced overhead . |
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training (2023.acl-long)
Copied to clipboard
| Challenge: | Modern Natural Language Generation models come with massive computational and storage requirements. |
| Approach: | They propose a method that applies word-level knowledge distillation to multiple PTs generated by both teacher and student. |
| Outcome: | The proposed techniques can be used to compress natural language models while preserving their performance. |
Mentor-KD: Making Small Language Models Better Multi-step Reasoners (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive emergent capabilities by leveraging Chain-of-Thought (CoT) prompting. |
| Approach: | They propose a Knowledge Distillation approach which transfers multi-step reasoning ability of Large Language Models (LLMs) to smaller LMs by fine-tuning language models of multi- step rationales generated by LLM teachers. |
| Outcome: | The proposed method is able to transfer multi-step reasoning ability of LLMs to smaller LMs while addressing data quality and soft label provision. |