MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown their strong ability in the field of machine translation, yet they suffer from high computational cost and latency. |
| Approach: | They propose a framework which transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. |
| Outcome: | The proposed framework transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. |
Similar Papers
Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel on English NLU tasks, yet struggle to extend their NLU capabilities to underrepresented languages. |
| Approach: | They integrate machine translation models (MT) directly into LLM backbones via sample-efficient self-distillation. |
| Outcome: | The proposed model outperforms translation-test models on 127 low-resource languages. |
Beyond One-Step Distillation: Bridging the Capacity Gap in Small Language Models via Multi-Step Knowledge Transfer (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel across diverse tasks but remain too large for efficient on-device deployment. |
| Approach: | They revisit multi-step knowledge distillation as an effective remedy . they demonstrate that MSKD improves ROUGE-L and perplexity over single-step approaches . |
| Outcome: | The proposed approach improves ROUGE-L and perplexity over single-step approaches . large language models are too large for efficient on-device deployment, the authors show . |
Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to train high-quality NMT models in bilingually low-resource scenarios are limited by the scarcity of parallel sentence-pairs. |
| Approach: | They propose to distill the knowledge of teacher models to a single student model by using knowledge distillation. |
| Outcome: | The proposed approach achieves up to +0.9 BLEU score improvements compared to strong baselines. |
CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech-totext translation (ST) data are limited due to the limited resources. |
| Approach: | They propose a knowledge distillation framework for speech translation that integrates knowledge from machine translation and decouples knowledge from non-target class knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art models on a benchmark dataset. |
Distilling Rule-based Knowledge into Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have broadened their applicability across diverse realworld scenarios. |
| Approach: | They propose to encode rule-based knowledge into large language models by using strong in-context abilities to extract the knowledge from the textual rules and then explicitly encode it into the parameters of LLMs. |
| Outcome: | The proposed learning paradigm is much more efficient than example-based learning in both sample size and generalization ability. |
Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks. |
| Approach: | They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions. |
| Outcome: | The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively. |
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive performance across numerous NLP tasks, but fine-tuning them for Machine Translation (MT) often introduces catastrophic forgetting, compromising the broad general abilities of LLMs and introducing potential security risks. |
| Approach: | They propose a method that harnesses the strong generative capabilities of Large Language Models to create rationales for training data, which are then "replayed" to prevent forgetting. |
| Outcome: | The proposed approach harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting. |
Multilingual Neural Machine Translation: Can Linguistic Hierarchies Help? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual Neural Machine Translation (MNMT) trains a single model that supports translation between multiple languages . transferring knowledge from a diverse set of languages degrades the translation performance due to negative transfer. |
| Approach: | They propose a hierarchical knowledge distillation approach to train multilingual models . they use typological features and phylogeny to overcome negative transfer issue . |
| Outcome: | The proposed approach avoids negative transfer effect by capitalising on language groups generated according to typological features and phylogeny of languages. |
Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have been successful in NLP tasks, but their large size and long inference time limit their deployment in real-time applications. |
| Approach: | They propose a meta-teacher model that captures transferable knowledge across domains and passes it to students. |
| Outcome: | The proposed model can distill large teacher models into small student models with guidance from the meta-teacher. |
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints. |
| Approach: | They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S . |
| Outcome: | The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers. |