Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Domain Classification (DC) and Intent Classification/Named Entity Recognition (ICNER) are the most common methods for reducing teacher-student knowledge into manageable sizes for low-latency downstream applications. |
| Approach: | They investigate whether distillation from a generic LM benefits downstream tasks . a domain classification and a task-specific data set are used to fine tune the model . |
| Outcome: | The proposed model improves across tasks and test sets when only task-specific data is used. |
Similar Papers
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model. |
| Approach: | They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. |
| Outcome: | The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets. |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Large-scale pretrained language models have led to significant improvements in Natural Language Processing, but they come at the cost of high computational and storage requirements. |
| Approach: | They propose to distill knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets. |
| Outcome: | The proposed approach improves on the SST-2, MRPC, YELP-2, and TREC-6 datasets. |
Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models are used to boost their performance on downstream tasks . pretraining with in-domain texts requires considerable in- domain data and training resources . |
| Approach: | They propose a domain knowledge transferring framework for pre-trained language models without additional in-domain pretraining. |
| Outcome: | The proposed framework extracts domain knowledge from an existing in-domain pretrained language model and transfers it to other PLMs by applying knowledge distillation. |
Causal Distillation for Language Models (2022.naacl-main)
Copied to clipboard
Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah Goodman
| Challenge: | Distillation efforts have led to language models that are more compact and efficient without serious drops in performance. |
| Approach: | They propose to augment distillation with a third objective that encourages the student model to imitate the causal dynamics of the teacher through a distillation interchange intervention training objective (DIITO). |
| Outcome: | The proposed method lowers perplexity on the WikiText-103M corpus and improves on the GLUE benchmark, SQuAD, and CoNLL-2003. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models are often inefficient for real-world deployment due to expensive inference costs. |
| Approach: | They propose to use knowledge distillation to transfer the knowledge of the original model to a smaller, more efficient student model. |
| Outcome: | The proposed method is the best for multi-lingual and multilingual student architectures. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
Rethinking Task-Specific Knowledge Distillation: Contextualized Corpus as Better Textbook (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation use a two-stage paradigm: general distillation with a task-agnostic general corpus and task-specific distillation using augmented task- specific corpus. |
| Approach: | They propose a contextualized corpus that contextualizes task corpus with large-scale general corpus through relevance-based text retrieval to improve student learning. |
| Outcome: | The proposed model improves on the GLUE benchmark and shows that it is better than generalized corpus and augmented task-specific corpus. |