| Challenge: | Existing approaches to reading comprehension systems are vulnerable to adversarial attacks. |
| Approach: | They propose to use knowledge distillation to transfer knowledge from an ensemble to a single model. |
| Outcome: | The proposed methods outperform the teacher on adversarial datasets and NarrativeQA benchmarks. |
Similar Papers
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model. |
| Approach: | They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. |
| Outcome: | The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets. |
Distilling Knowledge for Search-based Structured Prediction (P18-1)
Copied to clipboard
| Challenge: | Existing studies have focused on the performance of structured prediction models, but they are often limited by the ambiguities of the reference policy. |
| Approach: | They propose to distill an ensemble of multiple models trained with different initializations into a single model and use it to explore the search space. |
| Outcome: | The proposed model outperforms the greedy models on two typical search-based structured prediction tasks and achieves 1.32 in LAS and 2.65 in BLEU over strong baselines. |
Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques for neural machine translation lack special treatment on the top-1 information, which is limiting the potential of KD. |
| Approach: | They propose a method to distill knowledge from top-1 predictions of teachers and a technique to infuse more additional knowledge by distilling on the data without ground-truth targets. |
| Outcome: | The proposed method outperforms the vanilla word-level KD and outperfies the existing methods on three different students with different capacity gaps. |
Efficient Transformer Knowledge Distillation: A Performance Review (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Pretrained transformer language models have been gaining popularity in the field of natural language processing . however, there is no study into the intersection of these two fields . |
| Approach: | They propose a method to extract knowledge from transformers to produce high-performing efficient attention models with low costs. |
| Outcome: | The proposed model compression method preserves up to 98.6% of original model performance across short-context tasks and up to 95.8% on long-concept Named Entity Recognition tasks while decreasing inference times by up to 57%. |
Noisy Self-Knowledge Distillation for Text Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to summarize text using a single reference and noisy datasets are ill-suited to summarising on single reference datasets. |
| Approach: | They propose to use self-knowledge distillation to improve text summarization by generating smoothed labels for students and teachers to reduce model uncertainty. |
| Outcome: | The proposed framework improves on pretrained and non-pretrained models on three benchmarks. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks. |
| Approach: | They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions. |
| Outcome: | The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively. |
GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved remarkable performance in OpenQA, but for practical deployment, knowledge distillation is crucial to maintain high performance while operating under computational constraints. |
| Approach: | They propose an algorithm to perform unsupervised knowledge distillation without the guidance of labels to achieve 99.5% of performance. |
| Outcome: | The proposed algorithm achieves 99.5% of performance in a commercial question-answering system. |
Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation approaches to NMT often rely on heuristics when deciding which teacher layers to distill from. |
| Approach: | They propose an approach to align student attention heads with their teacher counterparts by heuristics to solve a feature mapping problem. |
| Outcome: | The proposed strategy shows gains of +3.61 and +0.63 BLEU points for WMT-2022 DeDsb and WMT-2014 EnDe compared to baselines. |
One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing knowledge distillation models require large computing resources and long inference time to perform. |
| Approach: | They propose a one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning. |
| Outcome: | The proposed method achieves better results with fewer parameters and extremely high speedup ratios on three sentiment classification tasks. |