Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Large-scale pretrained language models have led to significant improvements in Natural Language Processing, but they come at the cost of high computational and storage requirements. |
| Approach: | They propose to distill knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets. |
| Outcome: | The proposed approach improves on the SST-2, MRPC, YELP-2, and TREC-6 datasets. |
Similar Papers
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training (2023.acl-long)
Copied to clipboard
| Challenge: | Modern Natural Language Generation models come with massive computational and storage requirements. |
| Approach: | They propose a method that applies word-level knowledge distillation to multiple PTs generated by both teacher and student. |
| Outcome: | The proposed techniques can be used to compress natural language models while preserving their performance. |
Distilling Knowledge Learned in BERT for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT have revolutionized the state of the art in many language understanding tasks. |
| Approach: | They propose a conditional masked language modeling approach to fine tune BERT on target generation tasks by imposing global sequence-level supervision on conventional Seq2Seq models. |
| Outcome: | The proposed model outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
TinyBERT: Distilling BERT for Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive and difficult to efficiently execute on resource-restricted devices. |
| Approach: | They propose a Transformer distillation method that performs Transformer distillations at pre-training and task-specific learning stages. |
| Outcome: | The proposed method accelerates inference and reduces model size while maintaining accuracy. |
Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits. |
| Approach: | They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target. |
| Outcome: | The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain. |
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases. |
| Approach: | They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time. |
| Outcome: | The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings. |
Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data (2025.findings-naacl)
Copied to clipboard
Juanhui Li, Sreyashi Nag, Hui Liu, Xianfeng Tang, Sheikh Muhammad Sarwar, Limeng Cui, Hansu Gu, Suhang Wang, Qi He, Jiliang Tang
| Challenge: | Large Language Models (LLMs) have demonstrated superior language understanding abilities in many real-world NLP applications. |
| Approach: | They propose a learning-based sample selection method that incorporates signals from both teacher and student to enhance model performance. |
| Outcome: | The proposed method improves model performance across datasets with higher data efficiency. |
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023.findings-acl)
Copied to clipboard
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
| Challenge: | Deploying large language models (LLMs) is difficult because they are memory inefficient and compute-intensive for practical applications. |
| Approach: | They propose a mechanism that fine tunes or distills small models that outperform LLMs . they use human labels to fine tune models or LLM-generated labels to train models . |
| Outcome: | The proposed method outperforms LLMs by using fewer training examples compared to few-shot prompted models using substantially smaller model sizes. |
Accurate Knowledge Distillation via n-best Reranking (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies using sequencelevel knowledge distillation (KD) have adopted this approach. |
| Approach: | They propose to utilize n-best reranking to enhance Sequence-Level Knowledge Distillation by utilizing a diverse set of models with different inductive biases, objective functions or architectures to pick the highest-quality hypotheses as labels. |
| Outcome: | The proposed approach is validated on the WMT’21 German English and Chinese english translation tasks. |