| Challenge: | Existing studies have focused on language knowledge transfer from pretrained models to neural machine translation models. |
| Approach: | They propose to use masked language pretraining to efficiently transfer bidirectional language knowledge to NMT models. |
| Outcome: | The proposed method can significantly improve machine translation performance and achieve competitive or even better results than previous methods. |
Similar Papers
Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | End-to-end speech translation models can be trained to leverage source text . however, since the input modalities are different, it is difficult to leverage the source text successfully. |
| Approach: | They propose to leverage source transcriptions via pre-training and joint training with ASR and NMT tasks. |
| Outcome: | The proposed model predicts paraphrased transcriptions as an auxiliary task with a single decoder. |
Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks. |
| Approach: | They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions. |
| Outcome: | The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
A Self-Distillation Recipe for Neural Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Neural Machine Translation (NMT) have been proven effective in improving the performance of computer vision tasks without pre-training a teacher. |
| Approach: | They propose a rank-order augmented Pearson correlation loss and an iterative distillation method to prevent the discrepancy of predictions between the student and a stronger teacher from disturbing the training. |
| Outcome: | The proposed method can lead to significant improvements over the strong Transformer baseline on low/middle/high-resource tasks, obtaining comparable or better performance with fewer layers. |
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints. |
| Approach: | They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S . |
| Outcome: | The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers. |
Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on how to effectively exploit bidirectional global contexts in neural machine translation models. |
| Approach: | They propose a Confidence Based Bidirectional Global Context Aware training framework for NMT . they incorporate bidirectional global context to the NMT model on unconfidently-predicted target words . |
| Outcome: | The proposed framework improves the NMT model on three large-scale translation datasets by +1.02, +0.57 BLEU scores. |
Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models are used to boost their performance on downstream tasks . pretraining with in-domain texts requires considerable in- domain data and training resources . |
| Approach: | They propose a domain knowledge transferring framework for pre-trained language models without additional in-domain pretraining. |
| Outcome: | The proposed framework extracts domain knowledge from an existing in-domain pretrained language model and transfers it to other PLMs by applying knowledge distillation. |
Exploring All-In-One Knowledge Distillation Framework for Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods only obtain one lightweight student each time . this could be resource-intensive and resulting in multiple students not being optimally utilized . |
| Approach: | They propose a knowledge distillation framework which generates multiple satisfactory students at once. |
| Outcome: | The proposed framework generates multiple satisfactory students at once. |
Beyond One-Step Distillation: Bridging the Capacity Gap in Small Language Models via Multi-Step Knowledge Transfer (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel across diverse tasks but remain too large for efficient on-device deployment. |
| Approach: | They revisit multi-step knowledge distillation as an effective remedy . they demonstrate that MSKD improves ROUGE-L and perplexity over single-step approaches . |
| Outcome: | The proposed approach improves ROUGE-L and perplexity over single-step approaches . large language models are too large for efficient on-device deployment, the authors show . |
Self-Evolution Knowledge Distillation for LLM-based Machine Translation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation strategies for large language models minimize output distributions between student and teacher models indiscriminately for each token. |
| Approach: | They propose a distillation strategy that integrates teacher and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process. |
| Outcome: | The proposed method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets. |