Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation models. |
| Approach: | They propose to use knowledge distillation to generate training data for non-autoregressive translation models by leveraging pretraining. |
| Outcome: | The proposed approach achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16 Romanian-English datasets. |
Similar Papers
Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is the preliminary step for training non-autoregressive translation models, but it can lose important information for translating low-frequency words. |
| Approach: | They propose a knowledge distillation method which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data. |
| Outcome: | Extensive experiments on eight WMT benchmarks show that monolingual KD outperforms the standard KD by improving low-frequency word translation without introducing any computational cost. |
Dual-teacher Knowledge Distillation for Low-frequency Word Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation models are trained on parallel corpora with unbalanced word frequency distribution, resulting in high-frequency words being ignored. |
| Approach: | They propose to employ a low-frequency teacher model that excels in translating low- frequency words to guide the learning of the student model. |
| Outcome: | The proposed method achieves +0.64 BLEU improvements over the state-of-the-art method on the low-frequency translation task while maintaining the translation quality of high-frequency words. |
How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior work suggests that distilled training data is less complex than manual translations. |
| Approach: | They propose to use sequence-level knowledge distillation to match autoregressive models' translation quality. |
| Outcome: | The proposed model can match translation quality of autoregressive models with distilled training data. |
Accurate Knowledge Distillation via n-best Reranking (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies using sequencelevel knowledge distillation (KD) have adopted this approach. |
| Approach: | They propose to utilize n-best reranking to enhance Sequence-Level Knowledge Distillation by utilizing a diverse set of models with different inductive biases, objective functions or architectures to pick the highest-quality hypotheses as labels. |
| Outcome: | The proposed approach is validated on the WMT’21 German English and Chinese english translation tasks. |
Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to lexically constrained neural machine translation suffer from high latency. |
| Approach: | They propose a plug-in algorithm for non-autoregressive translation for this problem . they propose ACT to familiarize the model with the source-side context of constraints . |
| Outcome: | The proposed model improves over the backbone constrained NAT model in constraint preservation and translation quality, especially for rare constraints. |
Scaling Low-Resource MT via Synthetic Data Generation with LLMs (2025.emnlp-main)
Copied to clipboard
Ona de Gibert, Joseph Attieh, Teemu Vahtola, Mikko Aulamo, Zihao Li, Raúl Vázquez, Tiancheng Hu, Jörg Tiedemann
| Challenge: | a recent study has shown that LLM-generated synthetic data can improve low-resource machine translation performance . traditional data augmentation techniques like back-translation preserve the human-written target and synthesize the other . |
| Approach: | They construct a document-level synthetic corpus from English Europarl and extend it via pivoting to 147 additional language pairs. |
| Outcome: | The proposed model can significantly improve low-resource machine translation performance even when noisy. |
Progressive Multi-Granularity Training for Non-Autoregressive Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Non-autoregressive translation models are weak at learning high-mode knowledge, argues a new study . despite the improved learning difficulty, there are still complicated word orders and structures in the synthetic sentences, making the NAT performance sub-optimal. |
| Approach: | They propose to train non-autoregressive translation models to learn fine-grained lower-mode knowledge . they break down sentence-level examples into three types and increase granularities . |
| Outcome: | The proposed method improves phrase translation accuracy and model reordering ability against strong NAT baselines. |
Integrating Translation Memories into Non-Autoregressive Machine Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Non-autoregressive machine translation (NAT) has made great progress, but most studies focus on standard translation tasks. |
| Approach: | They propose to train an edit-based NAT model with a Translation Memory (TM) they propose to modify the data presentation and introduce an extra deletion operation to reduce decoding load. |
| Outcome: | The proposed model performs on par with an autoregressive approach while reducing the decoding load. |
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |
Revisiting Non-Autoregressive Translation at Scale (2023.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments on two advanced NAT models show scaling can improve translation performance. |
| Approach: | They empirically examine the impact of scaling on NAT behaviors on a large-scale WMT dataset. |
| Outcome: | The proposed model can achieve comparable performance with the scaling model while maintaining the superiority of decoding speed with standard NAT models. |