POS-Constrained Parallel Decoding for Non-autoregressive Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing non-autoregressive generation systems face multimodality problem due to conditionally independent decoding. |
| Approach: | They propose to incorporate linguistic structure into NAG inference instead of teacher AG . they propose a method that provides a specific POS sequence to constrain the NAG model . |
| Outcome: | The proposed method improves NAG models on four text generation tasks to a greater extent compared to knowledge distillation. |
Similar Papers
A Study of Non-autoregressive Model for Sequence Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) models generate all tokens in parallel, resulting in faster generation speed compared to autoregressive models. |
| Approach: | They propose to use knowledge distillation and source-target alignment to bridge the gap between NAR and autoregressive models in various tasks. |
| Outcome: | The proposed techniques can speed up NAR models in some tasks but not all . the proposed techniques reduce target token dependency while allowing for faster inference . |
Tree-Structured Non-Autoregressive Decoding for Sequence-to-Sequence Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive Transformers suffer from high inference latency due to sequential token generation. |
| Approach: | They propose a tree-structured non-autoregressive decoding paradigm that bridges autoregressive and non-automatic decoding. |
| Outcome: | The proposed paradigm outperforms autoregressive and non-autoregressive decoding in machine translation and paraphrase generation. |
Non-Autoregressive Sequence Generation (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Non-autoregressive sequence generation (NAR) models generate output sequences in parallel to speed up generation process. |
| Approach: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to generate the entire or partial output sequences in parallel to speed up the generation process . |
| Outcome: | This tutorial provides a thorough introduction and review of non-autoregressive sequence generation . it aims to reduce the performance gap between state-of-the-art models due to lack of modeling power . |
NAG-NER: a Unified Non-Autoregressive Generation Framework for Various NER Tasks (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing models for general NER tasks require entities to be generated in a predefined order, causing error propagation and inefficient decoding. |
| Approach: | They propose a non-autoregressive generation framework for general NER tasks that generates entities as a set instead of a sequence, avoiding error propagation and inefficient decoding. |
| Outcome: | The proposed model outperforms state-of-the-art models on three benchmark NER datasets and two of our proprietary NER tasks. |
Non-Autoregressive Text Generation with Pre-trained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Autoregressive generation models generate tokens in a left-to-right, token-by-token fashion, resulting in lag in inference. |
| Approach: | They propose to use BERT as the backbone of a non-autoregressive generation model for greatly improved performance. |
| Outcome: | The proposed model outperforms existing non-autoregressive models and achieves competitive performance with many strong autoregressive model. |
Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing non-autoregressive (NAR) models fail to generate specified entity names in up to 40% of responses and produce OOV errors. |
| Approach: | They propose a constrained decoding algorithm for Directed Acyclic T5 model which offers lexical, vocabulary and length control. |
| Outcome: | The proposed model significantly improves on Schema Guided Dialogue and DART datasets, establishing strong results for Task-Oriented Dialog and Data-to-Text NLG. |
Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators (2022.emnlp-main)
Copied to clipboard
| Challenge: | Non-autoregressive (NAR) neural machine translation models require a conditional independence assumption on target sequences, resulting in less informative learning signals. |
| Approach: | They propose a model-agnostic multi-task learning framework to provide more informative learning signals for NAR models under conventional MLE training. |
| Outcome: | The proposed framework improves accuracy of multiple NAR baselines without additional decoding overhead. |
latent-GLAT: Glancing at Latent Variables for Parallel Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in text generation have limited applications due to multimodality problem. |
| Approach: | They propose a method which uses latent variables to capture word categorical information and invoke an advanced curriculum learning technique to overcome multi-modality problem. |
| Outcome: | The proposed method outperforms strong baselines without an autoregressive model, which further broadens the application scenarios of the parallel decoding paradigm. |
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |
Non-Autoregressive Models for Fast Sequence Generation (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Autoregressive (AR) models can only generate target sequence word-by-word due to the AR mechanism and suffer from slow inference. |
| Approach: | This tutorial provides an introduction to non-autoregressive sequence generation. |
| Outcome: | This tutorial explains how to generate non-autoregressive sequence generation models. |