Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is the preliminary step for training non-autoregressive translation models, but it can lose important information for translating low-frequency words. |
| Approach: | They propose a knowledge distillation method which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data. |
| Outcome: | Extensive experiments on eight WMT benchmarks show that monolingual KD outperforms the standard KD by improving low-frequency word translation without introducing any computational cost. |
Similar Papers
Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation models. |
| Approach: | They propose to use knowledge distillation to generate training data for non-autoregressive translation models by leveraging pretraining. |
| Outcome: | The proposed approach achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16 Romanian-English datasets. |
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |
Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications. |
| Approach: | They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model. |
| Outcome: | The proposed method can achieve consistent and significant performance gains across all model types and sizes. |
Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. |
| Approach: | They propose to use teacher training data for model compression . they investigate six tasks and find they can achieve between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
| Outcome: | The proposed solution achieves between 75% and 92% of the teacher’s classification score while compressing the model 30 times. |
Dual-Space Knowledge Distillation for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) have strong generalization abilities due to their huge model capacities. |
| Approach: | They propose a dual-space knowledge distillation framework that unifies the output spaces of the two models for KD. |
| Outcome: | The proposed framework outperforms existing white-box KD frameworks on task-agnostic instruction-following benchmarks and can automatically align representations of two models with different vocabularies. |
Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques for neural machine translation lack special treatment on the top-1 information, which is limiting the potential of KD. |
| Approach: | They propose a method to distill knowledge from top-1 predictions of teachers and a technique to infuse more additional knowledge by distilling on the data without ground-truth targets. |
| Outcome: | The proposed method outperforms the vanilla word-level KD and outperfies the existing methods on three different students with different capacity gaps. |
Dual-teacher Knowledge Distillation for Low-frequency Word Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation models are trained on parallel corpora with unbalanced word frequency distribution, resulting in high-frequency words being ignored. |
| Approach: | They propose to employ a low-frequency teacher model that excels in translating low- frequency words to guide the learning of the student model. |
| Outcome: | The proposed method achieves +0.64 BLEU improvements over the state-of-the-art method on the low-frequency translation task while maintaining the translation quality of high-frequency words. |
Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have been successful in NLP tasks, but their large size and long inference time limit their deployment in real-time applications. |
| Approach: | They propose a meta-teacher model that captures transferable knowledge across domains and passes it to students. |
| Outcome: | The proposed model can distill large teacher models into small student models with guidance from the meta-teacher. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Beyond the Mode: Sequence-Level Distillation of Multilingual Translation Models for Low-Resource Language Pairs (2025.findings-naacl)
Copied to clipboard
Aarón Galiano-Jiménez, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Víctor M. Sánchez-Cartagena
| Challenge: | Existing multilingual pre-trained models for low-resource languages have outperformed those trained from scratch for low resources due to high hardware requirements. |
| Approach: | They propose to use beam search to decode the whole output distribution of the teacher to improve student learning. |
| Outcome: | The proposed methods improve student model performance and reduce gender bias amplification common to beam search based methods. |