Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (D18-1)
Copied to clipboard
| Challenge: | Despite its success, neural autoregressive modeling has its weakness in decoding, i.e., finding the most likely sequence. |
| Approach: | They propose a conditional non-autoregressive neural sequence model based on iterative refinement based upon latent variable models and conditional denoising autoencoders. |
| Outcome: | The proposed model significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart. |
Similar Papers
An Empirical Study of Iterative Refinements for Non-autoregressive Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Iterative non-autoregressive (NAR) models have recently demonstrated impressive performance in varied generation tasks, surpassing the autoregressive Transformer. |
| Approach: | They propose a strategy to conduct efficient refinements without performance declines by using two simple metrics to identify potential problems existing in current refinement processes. |
| Outcome: | The proposed model outperforms the autoregressive Transformer by around one BLEU on average. |
Non-Autoregressive Models for Fast Sequence Generation (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Autoregressive (AR) models can only generate target sequence word-by-word due to the AR mechanism and suffer from slow inference. |
| Approach: | This tutorial provides an introduction to non-autoregressive sequence generation. |
| Outcome: | This tutorial explains how to generate non-autoregressive sequence generation models. |
Non-Autoregressive Sequence Generation (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Non-autoregressive sequence generation (NAR) models generate output sequences in parallel to speed up generation process. |
| Approach: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to generate the entire or partial output sequences in parallel to speed up the generation process . |
| Outcome: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to reduce the performance gap between state-of-the-art models due to lack of modeling power . |
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (D19-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are autoregressive, meaning they factor the joint probability of the output sequence into the product of probabilities over the next to-ken. |
| Approach: | They propose a non-autoregressive sequence generation model using latent variables . they use generative flow to model complex distributions using neural networks . |
| Outcome: | The proposed model performs comparable to state-of-the-art models and has constant decoding time w.r.t the sequence length. |
Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive inference procedures that refine in token space often require computational overhead. |
| Approach: | They propose an efficient inference procedure that iteratively refines translation purely in the continuous space using a latent variable instead of the latent variables. |
| Outcome: | The proposed procedure is twice as efficient and more effective than the existing EM-like inference procedure. |
Variational Autoregressive Decoder for Neural Response Generation (D18-1)
Copied to clipboard
| Challenge: | Existing variational Bayesian models generate responses from a single latent variable, which is not sufficient to model high variability in responses. |
| Approach: | They propose a conditional variable auto-encoder that sequentially introduces latent variables to condition the generation of each word in the response sequence. |
| Outcome: | Empirical results show that the proposed model improves on state-of-the-art models on Opensubtitle and Reddit datasets. |
Deep Equilibrium Non-Autoregressive Sequence Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | et al., 2017) is the most prevailing neural architecture for sequence-to-sequence learning. |
| Approach: | They propose to solve for the equilibrium state of NAR models with black-box root-finding solvers and back-propagate through the equilibrium point via implicit differentiation with constant memory. |
| Outcome: | The proposed framework can converge to a more accurate prediction on four WMT benchmarks. |
End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification (D18-1)
Copied to clipboard
| Challenge: | Autoregressive decoding is the only part of sequence-to-sequence models that prevents massive parallelization at inference time. |
| Approach: | They propose a non-autoregressive architecture based on connectionist temporal classification . they conduct experiments on the WMT English-Romanian and English-German datasets . |
| Outcome: | The proposed model achieves a significant speedup over autoregressive models . the model can be trained end-to-end and maintains translation quality comparable to other models compared to autoregression models based on connectionist temporal classification . |
Non-Autoregressive Neural Machine Translation: A Call for Clarity (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models require a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Approach: | They propose to use a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Outcome: | The proposed models improve translation quality and speed under third-party testing environments. |
Jointly Masked Sequence-to-Sequence Model for Non-Autoregressive Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | masked language models have been used for natural language processing tasks but few studies have adopted it in the sequence-to-sequence models. |
| Approach: | They propose to combine encoder and decoder to train a masked sequence-to-sequence model . they propose to train the encoder more rigorously by masking the encoded input . |
| Outcome: | The proposed model achieves 27.69/32.24 BLEU scores on English-German/German-English tasks with 5+ times speed up compared with an autoregressive model. |