Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods rely on intermediate layer features and golden labels, which require aligned model architecture and labeled data respectively. |
| Approach: | They propose a general language model distillation method that performs two-stage word prediction distillation and vocabulary compression, which is simple and shows extremely strong performance. |
| Outcome: | The proposed method outperforms 25 state-of-the-art methods on the SuperGLUE benchmark, achieving an average score that surpasses the best method by 3%. |
Similar Papers
Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) are prone to overfitting to training datasets . recent advances in NLP have shown that using PLMs such as BERT and RoBERTa on downstream tasks is effective. |
| Approach: | They propose a consistency-regularized knowledge distillation method which mitigates overfitting of existing methods. |
| Outcome: | The proposed method outperforms existing methods on the GLUE benchmark and synthetic datasets. |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing multilingual models still underperform individual monolingual models due to model capacity limitations. |
| Approach: | They propose to distill the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). |
| Outcome: | The proposed model outperforms strong baseline models and teacher models on 4 multilingual tasks with 25 datasets and has stronger zero-shot generalizability. |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing knowledge distillation frameworks for language models are limited by memory and the use of complex distillation methods on larger-scale PLMs. |
| Approach: | They propose a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods. |
| Outcome: | The proposed framework can support distillation on larger-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs. |
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)
Copied to clipboard
Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart
| Challenge: | Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping. |
| Approach: | They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models. |
| Outcome: | The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets. |
Accurate Knowledge Distillation via n-best Reranking (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies using sequencelevel knowledge distillation (KD) have adopted this approach. |
| Approach: | They propose to utilize n-best reranking to enhance Sequence-Level Knowledge Distillation by utilizing a diverse set of models with different inductive biases, objective functions or architectures to pick the highest-quality hypotheses as labels. |
| Outcome: | The proposed approach is validated on the WMT’21 German English and Chinese english translation tasks. |
Contrastive Distillation on Intermediate Representations for Language Model Compression (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to compress language models use a simple L_2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. |
| Approach: | They propose a method that uses knowledge distillation to distill knowledge through intermediate layers of the teacher via a contrastive objective. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark. |
Towards General-Domain Word Sense Disambiguation: Distilling Large Language Model into Compact Disambiguator (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Word Sense Disambiguation rely heavily on manually annotated data, which limits coverage and generalization. |
| Approach: | They propose a framework that leverages large language models as knowledge distillers to build silver-standard WSD corpora by combining generation-based distillation and annotation-based disambiguation. |
| Outcome: | The proposed framework outperforms existing methods on general-domain benchmarks by 50% on the most challenging test set and by 1000 times fewer parameters. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |