Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that curriculum learning facilitates dialogue generation tasks while knowledge distillation can yield significant performance boosts for student models. |
| Approach: | They propose a combination of curriculum learning and knowledge distillation for dialogue generation models . they cluster training cases according to their complexity and employ an adversarial training strategy . |
| Outcome: | The proposed model improves compared with baselines. |
Similar Papers
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model. |
| Approach: | They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. |
| Outcome: | The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets. |
Diversifying Neural Dialogue Generation via Negative Distillation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to generate generic responses are ignoring low-frequency but generic responses and bringing low- frequency but meaningless responses. |
| Approach: | They propose a negative training paradigm that reminds dialogue models not to generate high-frequency responses during training. |
| Outcome: | The proposed method outperforms previous methods in the generic response problem while minimizing low-frequency but meaningless responses. |
Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has focused on training open-domain dialogue models using unpaired data. |
| Approach: | They propose a data-level distillation method for training open-domain dialogue models by utilizing unpaired data. |
| Outcome: | The proposed method produces high-quality dialogue pairs with diverse contents, and can improve competitive baselines. |
One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing knowledge distillation models require large computing resources and long inference time to perform. |
| Approach: | They propose a one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning. |
| Outcome: | The proposed method achieves better results with fewer parameters and extremely high speedup ratios on three sentiment classification tasks. |
On Knowledge distillation from complex networks for response prediction (N19-1)
Copied to clipboard
| Challenge: | Recent advances in Question Answering have led to the development of very complex models . however, these models are expensive in space and time and require limited resources . |
| Approach: | They propose to use simple models which learn to emulate characteristics of a teacher network . they use a 12GB Tesla K80 GPU to restrict the maximum length of the input document . |
| Outcome: | The proposed model can perform better on a Holl-E dialog dataset. |
DistillCSE: Distilled Contrastive Learning for Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to sentence embeddings are based on contrastive learning (CL) . |
| Approach: | They propose a framework which performs contrastive learning under the self-training paradigm with knowledge distillation and propose 'Group-P shuffling strategy' and averaging logits from multiple teacher components. |
| Outcome: | The proposed framework outperforms many strong baseline methods and yields a new state-of-the-art performance. |
Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation address the teacher–student capacity gap by mixing teacher and student distributions in the distillation target or using curriculum learning to sequence training from easy to hard examples. |
| Approach: | They propose a white-box KD framework that co-designs curriculum scheduling and target mixing through a unified difficulty-aware principle. |
| Outcome: | The proposed framework outperforms existing methods while reducing training runtime by over 10%. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor Network (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches for language models have overlooked the difficulty of training examples. |
| Approach: | They propose a framework that controls difficulty of training examples during pre-training by a tutor network. |
| Outcome: | The proposed framework outperforms state-of-the-art KD methods with student models on the GLUE benchmark. |