Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor

Challenge:	Knowledge distillation is a technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student).
Approach:	They propose a factorized form of the knowledge distillation objective for structured prediction which is tractable for many typical choices of the teacher and student models.
Outcome:	The proposed model is able to transfer knowledge between teacher and student models without loss of accuracy under four different scenarios.

Similar Papers

Overview

Challenge

Approach

Outcome

Distilling Knowledge for Search-based Structured Prediction (P18-1)

Copied to clipboard

Yijia Liu, Wanxiang Che, Huaipeng Zhao, Bing Qin, Ting Liu

Semantic Text Processing

Challenge:	Existing studies have focused on the performance of structured prediction models, but they are often limited by the ambiguities of the reference policy.
Approach:	They propose to distill an ensemble of multiple models trained with different initializations into a single model and use it to explore the search space.
Outcome:	The proposed model outperforms the greedy models on two typical search-based structured prediction tasks and achieves 1.32 in LAS and 2.65 in BLEU over strong baselines.

Multi-Granularity Structural Knowledge Distillation for Language Model Compression (2022.acl-long)

Copied to clipboard

Chang Liu, Chongyang Tao, Jiazhan Feng, Dongyan Zhao

Semantic Text Processing

Challenge:	Existing methods to transfer knowledge to a small model are not enough to represent the rich semantics of a text.
Approach:	They propose to distill the knowledge to a student hierarchically across layers using a large teacher-student framework.
Outcome:	Experimental results show that the proposed method outperforms distillation methods on GLUE benchmark.

Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs (2025.acl-long)

Copied to clipboard

Anshumann Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee

Semantic Text Processing

Challenge:	Knowledge distillation is a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached.
Approach:	They propose an importance-sampling-based method which provides unbiased estimates, preserves the gradient in expectation, and requires storing significantly sparser logits.
Outcome:	The proposed method enables faster training of student models with marginal overhead (10%) compared to cross-entropy based training, while maintaining competitive performance compared with full distillation.

Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation (2023.acl-long)

Copied to clipboard

Songming Zhang, Yunlong Liang, Shuaibo Wang, Yufeng Chen, Wenjuan Han, Jian Liu, Jinan Xu

Language Models

Challenge:	Existing knowledge distillation techniques for neural machine translation lack special treatment on the top-1 information, which is limiting the potential of KD.
Approach:	They propose a method to distill knowledge from top-1 predictions of teachers and a technique to infuse more additional knowledge by distilling on the data without ground-truth targets.
Outcome:	The proposed method outperforms the vanilla word-level KD and outperfies the existing methods on three different students with different capacity gaps.

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)

Copied to clipboard

Chenglong Wang, Yi Lu, Yongyu Mu, Yimin Hu, Tong Xiao, Jingbo Zhu

Semantic Text Processing

Challenge:	Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model.
Approach:	They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation.
Outcome:	The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets.

Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast—Choose Three (2020.emnlp-main)

Copied to clipboard

Steven Reich, David Mueller, Nicholas Andrews

Semantic Text Processing

Challenge:	Modern neural networks do not always produce wellcalibrated predictions . post-hoc calibration methods require a held-out calibration dataset, which may not be available in all circumstances.
Approach:	They validate ensemble distillation framework for producing well-calibrated structured prediction models without the prohibitive inference-time cost of ensembles.
Outcome:	The proposed framework produces well-calibrated predictions without the prohibitive inference-time cost of ensembles.

AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)

Copied to clipboard

Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Rui Wang

Language Models

Challenge:	Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information.
Approach:	They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model.
Outcome:	The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods.

Multi-Grained Knowledge Distillation for Named Entity Recognition (2021.naacl-main)

Copied to clipboard

Xuan Zhou, Xiao Zhang, Chenyang Tao, Junya Chen, Bing Xu, Wei Wang, Jing Xiao

Green & Sustainable NLP

Challenge:	Pre-trained big models have delivered top performance in Seq2seq modeling, but their deployments in real-world applications are often hindered by excessive computations and memory demands.
Approach:	They propose a distillation scheme to efficiently transfer knowledge from big models to their cheaper counterparts.
Outcome:	The proposed scheme maximizes the assimilation of knowledge from the teacher model to the student model.

f-Divergence Minimization for Sequence-Level Knowledge Distillation (2023.acl-long)

Copied to clipboard

Yuqiao Wen, Zichao Li, Wenyu Du, Lili Mou

Language Models

Challenge:	Existing knowledge distillation approaches focus on minimizing a generalized f-divergence function.
Approach:	They propose a framework which formulates sequence-level knowledge distillation as minimizing a generalized f-divergence function.
Outcome:	The proposed framework outperforms existing methods and reduces intractable divergence to word-level losses.

Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model (2022.emnlp-main)

Copied to clipboard

Dongkyu Lee, Zhiliang Tian, Yingxiu Zhao, Ka Chun Cheung, Nevin Zhang

Responsible & Trustworthy NLP

Challenge:	Existing knowledge distillation schemes focus on a teacher as a source of knowledge and a gauge to detect miscalibration of a student.
Approach:	They propose a method that uses a teacher model as a source of knowledge and a model as an error detector to detect miscalibration of a student.
Outcome:	The proposed scheme improves model generalization and significantly lowers calibration error.