Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a powerful tool for deep learning applications. |
| Approach: | They propose a method which defines a smoother training path for the student by following the training footprints of the teacher rather than solely relying on distilling from a single mature fully-trained teacher. |
| Outcome: | The proposed technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem. |
Similar Papers
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques for neural machine translation lack special treatment on the top-1 information, which is limiting the potential of KD. |
| Approach: | They propose a method to distill knowledge from top-1 predictions of teachers and a technique to infuse more additional knowledge by distilling on the data without ground-truth targets. |
| Outcome: | The proposed method outperforms the vanilla word-level KD and outperfies the existing methods on three different students with different capacity gaps. |
Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation (2025.acl-long)
Copied to clipboard
Guoqiang Gong, Jiaxing Wang, Jin Xu, Deping Xiang, Zicheng Zhang, Leqi Shen, Yifeng Zhang, JunhuaShu JunhuaShu, ZhaolongXing ZhaolongXing, Zhen Chen, Pengzhang Liu, Ke Zhang
| Challenge: | Knowledge distillation (KD) compresses large language models into lightweight versions called student models. |
| Approach: | They propose to align the entire feature dynamics between teacher and student models by using two additional loss terms to achieve this. |
| Outcome: | The proposed method matches the entire feature dynamics between teacher and student models rather than just the final states. |
Distillation Traps and Guards: A Calibration Knob for LLM Distillability (2026.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks. |
| Approach: | They propose a method that allows teachers to control their distillability via reinforcement fine-tuning (RFT) they propose to use tail noise, off-policy instability, and the teacher–student gap to improve KD. |
| Outcome: | The proposed method outperforms SFT and KD baselines and can be used to protect teachers and students from bottlenecks. |
Cost-effective Distillation of Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. |
| Approach: | They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture. |
| Outcome: | The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets. |
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intermediate layer matching are limited due to huge over-parameterization . |
| Approach: | They propose to match intermediate layers of teacher and student in output space via attention-based layer projection. |
| Outcome: | The proposed method outperforms existing methods on GLUE tasks. |
Annealing Knowledge Distillation (2021.eacl-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a powerful model compression technique for deep neural networks. |
| Approach: | They propose a method to feed the rich information provided by teacher’s soft-targets incrementally and more efficiently by annealing the teacher output incrementally. |
| Outcome: | The proposed method can be used on image classification and NLP language inference tasks with BERT-based models on the GLUE benchmark. |
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints. |
| Approach: | They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S . |
| Outcome: | The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers. |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |
Maximizing the Effectiveness of Larger BERT Models for Compression (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for capturing large BERT models as teachers do not fully exploit the potential advantages of larger teachers. |
| Approach: | They propose a method that leverages a pretrained teacher model to guide the training of a lightweight student model to enhance knowledge transfer. |
| Outcome: | The proposed method enhances knowledge transfer by leveraging a pretrained teacher model to guide the training of a lightweight student model. |