| Challenge: | Existing neural machine translation models have a deep structure with large amounts of parameters, making them hard to train. |
| Approach: | They propose an online method to generate a teacher model from checkpoints . they show steady improvement over a strong self-attention-based baseline system . |
| Outcome: | The proposed method improves on-the-fly on several datasets and language pairs. |
Similar Papers
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints. |
| Approach: | They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S . |
| Outcome: | The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers. |
Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks. |
| Approach: | They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions. |
| Outcome: | The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively. |
A Self-Distillation Recipe for Neural Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Neural Machine Translation (NMT) have been proven effective in improving the performance of computer vision tasks without pre-training a teacher. |
| Approach: | They propose a rank-order augmented Pearson correlation loss and an iterative distillation method to prevent the discrepancy of predictions between the student and a stronger teacher from disturbing the training. |
| Outcome: | The proposed method can lead to significant improvements over the strong Transformer baseline on low/middle/high-resource tasks, obtaining comparable or better performance with fewer layers. |
Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to train high-quality NMT models in bilingually low-resource scenarios are limited by the scarcity of parallel sentence-pairs. |
| Approach: | They propose to distill the knowledge of teacher models to a single student model by using knowledge distillation. |
| Outcome: | The proposed approach achieves up to +0.9 BLEU score improvements compared to strong baselines. |
Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation schemes focus on a teacher as a source of knowledge and a gauge to detect miscalibration of a student. |
| Approach: | They propose a method that uses a teacher model as a source of knowledge and a model as an error detector to detect miscalibration of a student. |
| Outcome: | The proposed scheme improves model generalization and significantly lowers calibration error. |
Checkpoint Reranking: An Approach to Select Better Hypothesis for Neural Machine Translation Systems (2020.acl-srw)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has produced excellent results in the field of machine translation due to generation of high-quality translations for different language pairs. |
| Approach: | They propose a method of re-ranking the outputs of Neural Machine Translation systems by focusing on the decoder's ability to generate distinct tokens and without the use of any language model or data. |
| Outcome: | The proposed method achieves translation improvement up to +0.16 BLEU points over baseline. |
Unifying the Convergences in Multilingual Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multilingual neural machine translation are overfitting and inconsistency is ignored . |
| Approach: | They propose a training strategy that picks up language-specific best checkpoints for each language pair to teach the current model on the fly. |
| Outcome: | The proposed training strategy alleviates convergence inconsistency and achieves state-of-the-art on language pairs. |
Nearest Neighbor Knowledge Distillation for Neural Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | k-nearest-neighbor machine translation (kNN-MT) is a state-of-the-art machine translation technique . however, it requires conducting kNN searches for each decoding step, which increases the cost of decoding . |
| Approach: | They propose to move the time-consuming kNN search forward to the preprocessing phase and introduce k Nearest Neighbor Knowledge Distillation (kNN-KD) that trains the base NMT model to directly learn the knowledge of kN. |
| Outcome: | The proposed method improves over the state-of-the-art model while maintaining the same training and decoding speed as the standard model. |
Distilling Knowledge for Search-based Structured Prediction (P18-1)
Copied to clipboard
| Challenge: | Existing studies have focused on the performance of structured prediction models, but they are often limited by the ambiguities of the reference policy. |
| Approach: | They propose to distill an ensemble of multiple models trained with different initializations into a single model and use it to explore the search space. |
| Outcome: | The proposed model outperforms the greedy models on two typical search-based structured prediction tasks and achieves 1.32 in LAS and 2.65 in BLEU over strong baselines. |
Exploring All-In-One Knowledge Distillation Framework for Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods only obtain one lightweight student each time . this could be resource-intensive and resulting in multiple students not being optimally utilized . |
| Approach: | They propose a knowledge distillation framework which generates multiple satisfactory students at once. |
| Outcome: | The proposed framework generates multiple satisfactory students at once. |