Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |
Similar Papers
Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems (2022.naacl-main)
Copied to clipboard
| Challenge: | Efficient machine translation models are commercially important as they can increase inference speeds, reduce costs and carbon emissions. |
| Approach: | They compare NAR models with autoregressive models to evaluate their performance . they point out flaws in evaluation methodology and argue for consistent evaluation . |
| Outcome: | The proposed model is faster on GPUs, but slower under more realistic usage conditions. |
An Empirical Study of Iterative Refinements for Non-autoregressive Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Iterative non-autoregressive (NAR) models have recently demonstrated impressive performance in varied generation tasks, surpassing the autoregressive Transformer. |
| Approach: | They propose a strategy to conduct efficient refinements without performance declines by using two simple metrics to identify potential problems existing in current refinement processes. |
| Outcome: | The proposed model outperforms the autoregressive Transformer by around one BLEU on average. |
Exploring Non-Autoregressive Text Style Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text style transfer use autoregressive decoding, but they are slow and low parallelizability. |
| Approach: | They propose a base NAR model by directly adapting the common training scheme from its AutoRegressive counterpart. |
| Outcome: | The proposed model sacrifices performance due to lack of conditional dependence between output tokens . knowledge distillation, contrastive learning, and iterative decoding are employed to improve the model . |
Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) neural machine translation models require a conditional independence assumption on target sequences, resulting in less informative learning signals. |
| Approach: | They propose a model-agnostic multi-task learning framework to provide more informative learning signals for NAR models under conventional MLE training. |
| Outcome: | The proposed framework improves accuracy of multiple NAR baselines without additional decoding overhead. |
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translation models are slow to learn the dependency between output tokens. |
| Approach: | They propose to use fully non-autoregressive neural machine translation (NAT) to predict tokens with single forward of neural networks. |
| Outcome: | The proposed model achieves state-of-the-art results on three translation benchmarks with comparable performance to autoregressive and iterative NAT systems. |
Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive models have boosted the efficiency of neural machine translation, but their performance is significantly worse than that of autoregressive counterparts. |
| Approach: | They propose to incorporate syntactic and semantic structures among natural languages into a non-autoregressive Transformer for the task of neural machine translation. |
| Outcome: | The proposed model achieves faster speed and keeps translation quality compared with other models. |
A Study of Non-autoregressive Model for Sequence Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) models generate all tokens in parallel, resulting in faster generation speed compared to autoregressive models. |
| Approach: | They propose to use knowledge distillation and source-target alignment to bridge the gap between NAR and autoregressive models in various tasks. |
| Outcome: | The proposed techniques can speed up NAR models in some tasks but not all . the proposed techniques reduce target token dependency while allowing for faster inference . |
How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior work suggests that distilled training data is less complex than manual translations. |
| Approach: | They propose to use sequence-level knowledge distillation to match autoregressive models' translation quality. |
| Outcome: | The proposed model can match translation quality of autoregressive models with distilled training data. |
Hint-Based Training for Non-Autoregressive Machine Translation (D19-1)
Copied to clipboard
| Challenge: | AutoRegressive Translation models have to generate tokens sequentially during decoding and thus suffer from high inference latency. |
| Approach: | They propose to use hidden states and word alignments to help train NART models. |
| Outcome: | The proposed model improves on the WMT14 En-De and De-En datasets but is faster in inference than the current models. |
Non-Autoregressive Neural Machine Translation: A Call for Clarity (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models require a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Approach: | They propose to use a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Outcome: | The proposed models improve translation quality and speed under third-party testing environments. |