End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification (D18-1)
Copied to clipboard
| Challenge: | Autoregressive decoding is the only part of sequence-to-sequence models that prevents massive parallelization at inference time. |
| Approach: | They propose a non-autoregressive architecture based on connectionist temporal classification . they conduct experiments on the WMT English-Romanian and English-German datasets . |
| Outcome: | The proposed model achieves a significant speedup over autoregressive models . the model can be trained end-to-end and maintains translation quality comparable to other models compared to autoregression models based on connectionist temporal classification . |
Similar Papers
Context-Aware Non-Autoregressive Document-Level Translation with Sentence-Aligned Connectionist Temporal Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies employ autoregressive translation (AT) methods to encode sentences . however, the AT methods struggle with error accumulation when the length of sentences increases. |
| Approach: | They propose a context-aware non-autoregressive framework with the sentence-aligned connectionist temporal classification loss for document-level neural machine translation. |
| Outcome: | The proposed framework achieves 46X speedup on three benchmarks compared to strong baselines. |
CTC-based Non-autoregressive Speech Translation (2023.acl-long)
Copied to clipboard
Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, Anxiang Ma, Jingbo Zhu
| Challenge: | End-to-end speech translation (E2E ST) and non-autoregressive (NAR) generation are promising in language and speech processing for their advantages of less error propagation and low latency. |
| Approach: | They develop a model that uses connectionist temporal classification to predict the source and target texts. |
| Outcome: | The proposed model achieves an average BLEU score of 29.5 with a speed-up of 5.67. |
Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive models have boosted the efficiency of neural machine translation, but their performance is significantly worse than that of autoregressive counterparts. |
| Approach: | They propose to incorporate syntactic and semantic structures among natural languages into a non-autoregressive Transformer for the task of neural machine translation. |
| Outcome: | The proposed model achieves faster speed and keeps translation quality compared with other models. |
Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems (2022.naacl-main)
Copied to clipboard
| Challenge: | Efficient machine translation models are commercially important as they can increase inference speeds, reduce costs and carbon emissions. |
| Approach: | They compare NAR models with autoregressive models to evaluate their performance . they point out flaws in evaluation methodology and argue for consistent evaluation . |
| Outcome: | The proposed model is faster on GPUs, but slower under more realistic usage conditions. |
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translation models are slow to learn the dependency between output tokens. |
| Approach: | They propose to use fully non-autoregressive neural machine translation (NAT) to predict tokens with single forward of neural networks. |
| Outcome: | The proposed model achieves state-of-the-art results on three translation benchmarks with comparable performance to autoregressive and iterative NAT systems. |
Non-Autoregressive Neural Machine Translation: A Call for Clarity (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models require a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Approach: | They propose to use a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Outcome: | The proposed models improve translation quality and speed under third-party testing environments. |
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |
Semi-Autoregressive Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to neural machine translation are typically autoregressive but suffer from low parallelizability and thus slow at decoding long sequences. |
| Approach: | They propose a semi-autoregressive Transformer model for fast sequence generation that keeps the autoregressive property in global but relieves in local . |
| Outcome: | The proposed model achieves 5.58 speedup while maintaining 88% translation quality, significantly better than previous non-autoregressive methods. |
Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Using connectionist temporal classification (CTC) for speech-to-text translation is counter-intuitive due to its monotonicity assumption. |
| Approach: | They propose to build a non-autoregressive speech-to-text translation model using connectionist temporal classification (CTC) their work shows transformer encoders can change the word order and points out the future research direction that needs to be explored more on non-Autoregressives speech translation. |
| Outcome: | The proposed model improves translation performance by using transformer encoders. |
Non-autoregressive Streaming Transformer for Simultaneous Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous machine translation models are trained to strike a balance between latency and translation quality. |
| Approach: | They propose a non-autoregressive streaming Transformer which generates blank tokens and decodes repetitive tokens to adjust its READ/WRITE strategy flexibly. |
| Outcome: | The proposed model outperforms previous strong autoregressive models on various benchmarks on siMT. |