TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing (2020.acl-demos)
Copied to clipboard
| Challenge: | Large pre-trained language models have hundreds of millions of parameters and take several gigabytes of memory to train and inference. |
| Approach: | They propose an open-source knowledge distillation toolkit designed for natural language processing that provides a set of predefined distillation methods and can be extended with custom code. |
| Outcome: | The proposed method is comparable with or even higher than the public distilled BERT models with similar numbers of parameters. |
Similar Papers
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for transferring knowledge from BERT into a model with large parameters are not efficient due to their large-scale and high computational cost. |
| Approach: | They propose a sentence representation approximating oriented distillation framework that can distill pre-trained BERT into a simple LSTM based model without specifying tasks. |
| Outcome: | The proposed model outperforms other distillation methods and larger models on multiple NLP tasks with efficiency well-improved. |
Distillation of encoder-decoder transformers for sequence labelling (2023.findings-eacl)
Copied to clipboard
| Challenge: | despite the strong trend in NLP to explore the use of large language models, there is still limited work evaluating prompting and decoding mechanisms for SL tasks. |
| Approach: | They propose a hallucination-free framework for sequence tagging that is especially suited for distillation. |
| Outcome: | The proposed framework performs well across multiple sequence labelling datasets and in a few-shot learning scenario. |
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model. |
| Approach: | They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. |
| Outcome: | The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets. |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |
Meta-Learning Adaptive Knowledge Distillation for Efficient Biomedical Natural Language Processing (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods have been proposed to reduce the size of large models for biomedical natural language processing tasks. |
| Approach: | They propose a meta-learning approach which adaptively learns parameters that enable optimal rate of knowledge exchange between teacher and student models from the distillation data during knowledge distillation. |
| Outcome: | The proposed method improves the performance of knowledge distillation methods on two biomedical natural language processing tasks. |
Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. |
| Approach: | They propose to use teacher training data for model compression . they investigate six tasks and find they can achieve between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
| Outcome: | The proposed solution achieves between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
| Challenge: | In this paper, we present EasyDistill, a comprehensive toolkit designed for effective black-box and white-box knowledge distillation (KD) of large language models. |
| Approach: | They propose a toolkit for effective black-box and white-box knowledge distillation (KD) of large language models (LLMs). |
| Outcome: | The framework offers data synthesis, supervised fine-tuning, ranking optimization, and reinforcement learning techniques specifically tailored for KD scenarios. |
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing knowledge distillation frameworks for language models are limited by memory and the use of complex distillation methods on larger-scale PLMs. |
| Approach: | They propose a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods. |
| Outcome: | The proposed framework can support distillation on larger-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs. |