| Challenge: | Existing large language models (LLMs) have strong generalization abilities due to their huge model capacities. |
| Approach: | They propose a dual-space knowledge distillation framework that unifies the output spaces of the two models for KD. |
| Outcome: | The proposed framework outperforms existing white-box KD frameworks on task-agnostic instruction-following benchmarks and can automatically align representations of two models with different vocabularies. |
Similar Papers
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |
Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications. |
| Approach: | They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model. |
| Outcome: | The proposed method can achieve consistent and significant performance gains across all model types and sizes. |
Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression. |
| Approach: | They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM . |
| Outcome: | The proposed model can be used to distill a 1.9B student model using a teacher LLM. |
On the Generalization vs Fidelity Paradox in Knowledge Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. |
| Approach: | They propose to use knowledge distillation to compress large language models into smaller ones while preserving performance. |
| Outcome: | The proposed technique improves the performance of smaller models by 10% while providing only marginal benefits for larger models. |
Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment (2025.coling-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques for large language models are causing difficulties for student models to learn multi-modal probability distributions. |
| Approach: | They propose a ranking loss-based knowledge distillation method that encourages consistency of the ranking of peak predictions between teacher and student models. |
| Outcome: | The proposed method improves student models' ability to learn multi-modal distributions. |
DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing cross-tokenizer distillation methods are limited by suboptimal alignment across sequence and vocabulary levels. |
| Approach: | They propose a cross-tokenizer distillation framework that enhances token-wise distillation . they use dual-space entropy-based weighting to achieve precise sequence-level alignment . |
| Outcome: | The proposed framework outperforms state-of-the-art methods in large language models but has high computational and memory costs. |
Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have been successful in NLP tasks, but their large size and long inference time limit their deployment in real-time applications. |
| Approach: | They propose a meta-teacher model that captures transferable knowledge across domains and passes it to students. |
| Outcome: | The proposed model can distill large teacher models into small student models with guidance from the meta-teacher. |
f-Divergence Minimization for Sequence-Level Knowledge Distillation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches focus on minimizing a generalized f-divergence function. |
| Approach: | They propose a framework which formulates sequence-level knowledge distillation as minimizing a generalized f-divergence function. |
| Outcome: | The proposed framework outperforms existing methods and reduces intractable divergence to word-level losses. |
Maximizing the Effectiveness of Larger BERT Models for Compression (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for capturing large BERT models as teachers do not fully exploit the potential advantages of larger teachers. |
| Approach: | They propose a method that leverages a pretrained teacher model to guide the training of a lightweight student model to enhance knowledge transfer. |
| Outcome: | The proposed method enhances knowledge transfer by leveraging a pretrained teacher model to guide the training of a lightweight student model. |
Sinkhorn Distance Minimization for Knowledge Distillation (2024.lrec-main)
Copied to clipboard
Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, Houqiang Li
| Challenge: | Existing knowledge distillation methods investigate divergence measures but fail to deliver effective supervision when few distribution overlap exists between teacher and student. |
| Approach: | They propose a knowledge distillation method that exploits the Sinkhorn distance to ensure a nuanced assessment of the disparity between teacher and student distributions. |
| Outcome: | The proposed method outperforms state-of-the-art methods on all kinds of LLMs with encoder-only, encoder decoder, and decoded architectures. |