Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications. |
| Approach: | They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model. |
| Outcome: | The proposed method can achieve consistent and significant performance gains across all model types and sizes. |
Similar Papers
On the Generalization vs Fidelity Paradox in Knowledge Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. |
| Approach: | They propose to use knowledge distillation to compress large language models into smaller ones while preserving performance. |
| Outcome: | The proposed technique improves the performance of smaller models by 10% while providing only marginal benefits for larger models. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |
Dynamic Knowledge Distillation for Pre-trained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods conduct knowledge distillation statically, e.g., student model aligns output distribution to teacher model on pre-defined training dataset. |
| Approach: | They propose a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency . they find it is promising and provide discussions on potential future directions towards more efficient methods . |
| Outcome: | The proposed method can boost student model performance while accelerating training . the proposed method reduces memory usage and accelerates model inference . |
Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) enables the compression of large language models (LLMs) conventional methods suffer from training-inference mismatches and suboptimal performance due to expensive student-generated outputs. |
| Approach: | They propose a method that combines a CL strategy and adaptive loss design to reduce training mismatches and suboptimal performance. |
| Outcome: | L2M-KD outperforms existing white-box KD methods on instruction-following tasks . it outperformed existing methods, achieving superior student model performance with reduced overhead . |
Self-Evolution Knowledge Distillation for LLM-based Machine Translation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation strategies for large language models minimize output distributions between student and teacher models indiscriminately for each token. |
| Approach: | They propose a distillation strategy that integrates teacher and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process. |
| Outcome: | The proposed method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets. |
Cost-effective Distillation of Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. |
| Approach: | They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture. |
| Outcome: | The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets. |
Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression. |
| Approach: | They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM . |
| Outcome: | The proposed model can be used to distill a 1.9B student model using a teacher LLM. |
ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models (2023.acl-short)
Copied to clipboard
Jianyi Zhang, Aashiq Muhamed, Aditya Anantharaman, Guoyin Wang, Changyou Chen, Kai Zhong, Qingjun Cui, Yi Xu, Belinda Zeng, Trishul Chilimbi, Yiran Chen
| Challenge: | Knowledge distillation (KD) is an effective compression technique to derive a smaller student model from a larger teacher model by transferring the knowledge embedded in the teacher's network. |
| Approach: | They propose a framework and loss function that preserves the semantic similarities of teacher and student training examples to enable the student to retrieve from the knowledge base effectively. |
| Outcome: | The proposed framework preserves the semantic similarities of teacher and student training examples to achieve state-of-the-art performance on the GLUE benchmark. |
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) has emerged as a popular method for compressing large language models due to high inference costs and memory requirements. |
| Approach: | They propose a method that integrates the teacher model during the student's sequence generation to reduce misguidance from the teacher. |
| Outcome: | Experiments on three model families and five instruction-following datasets show that SWITCH surpasses traditional methods, especially in the generation of long sequential data. |