Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing non-AutoRegressive (NAR) text generation models lack proper pre-training, making them far behind pre-trained autoregressive models. |
| Approach: | They propose a novel pre-training task to promote prediction consistency in non-autoregressive (NAR) generation. |
| Outcome: | The proposed model outperforms existing pre-trained models and achieves 17 times speedup in throughput. |
Similar Papers
Non-Autoregressive Text Generation with Pre-trained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Autoregressive generation models generate tokens in a left-to-right, token-by-token fashion, resulting in lag in inference. |
| Approach: | They propose to use BERT as the backbone of a non-autoregressive generation model for greatly improved performance. |
| Outcome: | The proposed model outperforms existing non-autoregressive models and achieves competitive performance with many strong autoregressive model. |
An Empirical Study of Iterative Refinements for Non-autoregressive Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Iterative non-autoregressive (NAR) models have recently demonstrated impressive performance in varied generation tasks, surpassing the autoregressive Transformer. |
| Approach: | They propose a strategy to conduct efficient refinements without performance declines by using two simple metrics to identify potential problems existing in current refinement processes. |
| Outcome: | The proposed model outperforms the autoregressive Transformer by around one BLEU on average. |
Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing non-autoregressive (NAR) models fail to generate specified entity names in up to 40% of responses and produce OOV errors. |
| Approach: | They propose a constrained decoding algorithm for Directed Acyclic T5 model which offers lexical, vocabulary and length control. |
| Outcome: | The proposed model significantly improves on Schema Guided Dialogue and DART datasets, establishing strong results for Task-Oriented Dialog and Data-to-Text NLG. |
Exploring Non-Autoregressive Text Style Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text style transfer use autoregressive decoding, but they are slow and low parallelizability. |
| Approach: | They propose a base NAR model by directly adapting the common training scheme from its AutoRegressive counterpart. |
| Outcome: | The proposed model sacrifices performance due to lack of conditional dependence between output tokens . knowledge distillation, contrastive learning, and iterative decoding are employed to improve the model . |
Non-Autoregressive Neural Machine Translation: A Call for Clarity (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive translation models require a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Approach: | They propose to use a single forward pass to generate the output sequence instead of iteratively producing each predicted token. |
| Outcome: | The proposed models improve translation quality and speed under third-party testing environments. |
Non-Autoregressive Translation by Learning Target Categorical Codes (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive text generation models still fall behind in translation quality . authors propose a model that learns implicitly categorical codes as latent variables . |
| Approach: | They propose a non-autoregressive Transformer model that implicitly categorizes latent variables into decoding . they find it improves translation quality by introducing more informative decoder inputs . |
| Outcome: | The proposed model achieves comparable or better performance in machine translation tasks than strong baselines. |
JANUS: Joint Autoregressive and Non-autoregressive Training with Auxiliary Loss for Sequence Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train autoregressive and non-autoregressive models only consider relevance of model parameters, ignoring correlations between the two manners. |
| Approach: | They propose a joint autoregressive and non-autoregressive training method using aUxiliary losS to enhance the model performance in both AR and NAR manners simultaneously. |
| Outcome: | The proposed method improves the model performance in both AR and NAR manners and reduces the inference latency. |
Non-Autoregressive Sequence Generation (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Non-autoregressive sequence generation (NAR) models generate output sequences in parallel to speed up generation process. |
| Approach: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to generate the entire or partial output sequences in parallel to speed up the generation process . |
| Outcome: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to reduce the performance gap between state-of-the-art models due to lack of modeling power . |
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |
Releasing the Capacity of GANs in Non-Autoregressive Image Captioning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive (NAR) models suffer from their inherent multi-modality problem. |
| Approach: | They propose an Adversarial Non-autoregressive Transformer for Image Captioning that improves model performance by modifying model structure to be compatible with contrastive learning. |
| Outcome: | The proposed model achieves 26.72 times faster than the autoregressive model on the MSCOCO dataset. |