Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for transferring knowledge from BERT into a model with large parameters are not efficient due to their large-scale and high computational cost. |
| Approach: | They propose a sentence representation approximating oriented distillation framework that can distill pre-trained BERT into a simple LSTM based model without specifying tasks. |
| Outcome: | The proposed model outperforms other distillation methods and larger models on multiple NLP tasks with efficiency well-improved. |
Similar Papers
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
Patient Knowledge Distillation for BERT Model Compression (D19-1)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have proven to be highly effective for natural language processing tasks, but the high demand for computing resources hinders their application in practice. |
| Approach: | They propose to compress an original large model (teacher) into an equally-effective lightweight shallow network (student) Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy. |
| Outcome: | The proposed model reduces the computational cost of training models using the teacher model into a lightweight shallow network. |
TinyBERT: Distilling BERT for Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive and difficult to efficiently execute on resource-restricted devices. |
| Approach: | They propose a Transformer distillation method that performs Transformer distillations at pre-training and task-specific learning stages. |
| Outcome: | The proposed method accelerates inference and reduces model size while maintaining accuracy. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Large-scale pretrained language models have led to significant improvements in Natural Language Processing, but they come at the cost of high computational and storage requirements. |
| Approach: | They propose to distill knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets. |
| Outcome: | The proposed approach improves on the SST-2, MRPC, YELP-2, and TREC-6 datasets. |
Task-agnostic Distillation of Encoder-Decoder Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing distillation methods that focus on encoder-only LMs fail to handle the distillation of encoder decoder LM. |
| Approach: | They propose a method that finetunes pretrained language models (LMs) they propose 'MiniEnD' that allows for task-agnostic distillation of LMs. |
| Outcome: | The proposed distillation method is generally effective and competitive compared to other alternatives. |
Cost-effective Distillation of Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. |
| Approach: | They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture. |
| Outcome: | The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets. |
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based pre-training models like BERT are computationally expensive and limited to resource-constrained devices. |
| Approach: | They propose a method which ternarizes the weights in a fine-tuned BERT model. |
| Outcome: | The proposed method outperforms the other methods on the GLUE and SQUAD benchmarks while being 14.9x smaller. |
Extremely Small BERT Models from Mixed-Vocabulary Training (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods cannot be directly applied to train student models with reduced vocabulary and embedding dimensions. |
| Approach: | They propose a method to align teacher and student embeddings via mixed-vocabulary training. |
| Outcome: | The proposed method compresses BERT-LARGE to a task-agnostic model with smaller vocabulary and hidden dimensions, which is an order of magnitude smaller than other distilled models. |
Dataset Distillation with Attention Labels for Fine-tuning BERT (2023.acl-short)
Copied to clipboard
| Challenge: | Specifically, we propose to introduce attention labels, which can efficiently distill the knowledge from the original dataset and transfer it to the transformer models via attention probabilities. |
| Approach: | They propose to introduce attention labels which can efficiently distill the knowledge from the original dataset and transfer it to the transformer models via attention probabilities. |
| Outcome: | The proposed methods perform impressively in four different NLP tasks and achieve 93.2% accuracy in AGNews, which is 98.5% of the original dataset even with only one sample per class and only one gradient step. |