Non-Autoregressive Text Generation with Pre-trained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Autoregressive generation models generate tokens in a left-to-right, token-by-token fashion, resulting in lag in inference. |
| Approach: | They propose to use BERT as the backbone of a non-autoregressive generation model for greatly improved performance. |
| Outcome: | The proposed model outperforms existing non-autoregressive models and achieves competitive performance with many strong autoregressive model. |
Similar Papers
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing non-AutoRegressive (NAR) text generation models lack proper pre-training, making them far behind pre-trained autoregressive models. |
| Approach: | They propose a novel pre-training task to promote prediction consistency in non-autoregressive (NAR) generation. |
| Outcome: | The proposed model outperforms existing pre-trained models and achieves 17 times speedup in throughput. |
ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text generation use auto-regressive (AR) methods, but inefficient inference is a problem. |
| Approach: | They propose an efficient and effective PLM to explicitly model the token dependency during NAR text generation. |
| Outcome: | The proposed model outperforms existing models on three text generation tasks while achieving 10 times faster inference speedup. |
Non-Autoregressive Sequence Generation (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Non-autoregressive sequence generation (NAR) models generate output sequences in parallel to speed up generation process. |
| Approach: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to generate the entire or partial output sequences in parallel to speed up the generation process . |
| Outcome: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to reduce the performance gap between state-of-the-art models due to lack of modeling power . |
Dynamic and Efficient Inference for Text Generation via BERT Family (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve performance of pre-trained language models are limited due to large-scale parameters and the universal autoregressive decoding paradigm. |
| Approach: | They propose a novel fine-tuning method which can make a single pre-trained model support Dynamic and Efficient infERence and achieve an adaptive trade-off between model performance and latency. |
| Outcome: | The proposed method achieves higher BLEU scores than the strong autoregressive Transformer model on translation tasks with 3 12 times speedup and faster inference speed compared with the BART model on four GLGE benchmark tasks. |
A Study of Non-autoregressive Model for Sequence Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) models generate all tokens in parallel, resulting in faster generation speed compared to autoregressive models. |
| Approach: | They propose to use knowledge distillation and source-target alignment to bridge the gap between NAR and autoregressive models in various tasks. |
| Outcome: | The proposed techniques can speed up NAR models in some tasks but not all . the proposed techniques reduce target token dependency while allowing for faster inference . |
Exploring Non-Autoregressive Text Style Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text style transfer use autoregressive decoding, but they are slow and low parallelizability. |
| Approach: | They propose a base NAR model by directly adapting the common training scheme from its AutoRegressive counterpart. |
| Outcome: | The proposed model sacrifices performance due to lack of conditional dependence between output tokens . knowledge distillation, contrastive learning, and iterative decoding are employed to improve the model . |
NAG-NER: a Unified Non-Autoregressive Generation Framework for Various NER Tasks (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing models for general NER tasks require entities to be generated in a predefined order, causing error propagation and inefficient decoding. |
| Approach: | They propose a non-autoregressive generation framework for general NER tasks that generates entities as a set instead of a sequence, avoiding error propagation and inefficient decoding. |
| Outcome: | The proposed model outperforms state-of-the-art models on three benchmark NER datasets and two of our proprietary NER tasks. |
Tree-Structured Non-Autoregressive Decoding for Sequence-to-Sequence Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive Transformers suffer from high inference latency due to sequential token generation. |
| Approach: | They propose a tree-structured non-autoregressive decoding paradigm that bridges autoregressive and non-automatic decoding. |
| Outcome: | The proposed paradigm outperforms autoregressive and non-autoregressive decoding in machine translation and paraphrase generation. |
Non-Autoregressive Models for Fast Sequence Generation (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Autoregressive (AR) models can only generate target sequence word-by-word due to the AR mechanism and suffer from slow inference. |
| Approach: | This tutorial provides an introduction to non-autoregressive sequence generation. |
| Outcome: | This tutorial explains how to generate non-autoregressive sequence generation models. |
An Empirical Study of Iterative Refinements for Non-autoregressive Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Iterative non-autoregressive (NAR) models have recently demonstrated impressive performance in varied generation tasks, surpassing the autoregressive Transformer. |
| Approach: | They propose a strategy to conduct efficient refinements without performance declines by using two simple metrics to identify potential problems existing in current refinement processes. |
| Outcome: | The proposed model outperforms the autoregressive Transformer by around one BLEU on average. |