Enconter: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer (2021.eacl-main)
Copied to clipboard
| Challenge: | Autoregressive language models do not perform well under hard lexical constraints as they lack fine control of content generation process. |
| Approach: | They propose a new insertion transformer that considers hard lexical constraints and imposes rules over objects in the generated text. |
| Outcome: | The proposed model outperforms baseline models in several performance metrics rendering it more suitable in practical applications. |
Similar Papers
POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models cannot be directly employed to generate text under specified lexical constraints. |
| Approach: | They propose a method for insertion-based text generation that inserts tokens between existing tokens in a parallel manner. |
| Outcome: | The proposed method is intuitive and interpretable on Wikipedia and Yelp datasets. |
Towards More Efficient Insertion Transformer with Fractional Positional Encoding (2023.eacl-main)
Copied to clipboard
| Challenge: | Empirical studies on text generation tasks demonstrate the effectiveness of insertion-based models. |
| Approach: | They propose a reusable positional encoding scheme for insertion transformers that allows reusing representations calculated in previous steps. |
| Outcome: | Empirical studies show that the proposed model reduces the time required to generate a token and improves decoding efficiency. |
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing non-AutoRegressive (NAR) text generation models lack proper pre-training, making them far behind pre-trained autoregressive models. |
| Approach: | They propose a novel pre-training task to promote prediction consistency in non-autoregressive (NAR) generation. |
| Outcome: | The proposed model outperforms existing pre-trained models and achieves 17 times speedup in throughput. |
NAG-NER: a Unified Non-Autoregressive Generation Framework for Various NER Tasks (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing models for general NER tasks require entities to be generated in a predefined order, causing error propagation and inefficient decoding. |
| Approach: | They propose a non-autoregressive generation framework for general NER tasks that generates entities as a set instead of a sequence, avoiding error propagation and inefficient decoding. |
| Outcome: | The proposed model outperforms state-of-the-art models on three benchmark NER datasets and two of our proprietary NER tasks. |
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large pre-trained language models for textual data have an unconstrained output space . when fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. |
| Approach: | They propose a method for constraining auto-regressive decoders of language models through incremental parsing. |
| Outcome: | The proposed method can find valid output sequences by rejecting inadmissible tokens . it can be used on Spider and CoSQL text-to-SQl translation tasks . |
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2021.tacl-1)
Copied to clipboard
| Challenge: | Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer while speeding up decoding dramatically compared to constrained beam search. |
| Approach: | They propose an Edit-Based TransfOrmer with Repositioning that integrates lexical preferences into output sequences by iterative editing hypotheses. |
| Outcome: | The proposed model uses soft lexical constraints more effectively than the Levenshtein Transformer while speeding up decoding dramatically compared to constrained beam search. |
Transformer-based Lexically Constrained Headline Generation (2021.emnlp-main)
Copied to clipboard
Kosuke Yamada, Yuta Hitomi, Hideaki Tamori, Ryohei Sasano, Naoaki Okazaki, Kentaro Inui, Koichi Takeda
| Challenge: | Existing automatic headline generation methods cannot include a given phrase in the generated headline. |
| Approach: | They propose a Transformer-based method that guarantees to include a given phrase in a generated headline. |
| Outcome: | The proposed method achieves ROUGE scores comparable to previous methods with Japanese news corpus. |
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020.acl-main)
Copied to clipboard
Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, Salim Roukos
| Challenge: | Existing approaches to generating text from AMRs focus on training sequence-to-sequence or graph-tosequent models on annotated data. |
| Approach: | They propose a strong pre-trained language model with cycle consistency-based re-scoring to generate AMR text. |
| Outcome: | The proposed model outperforms existing methods on the English LDC2017T10 dataset. |
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)
Copied to clipboard
| Challenge: | Unsupervised pre-training of large neural models has revolutionized Natural Language Processing. |
| Approach: | They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint. |
| Outcome: | The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion. |
ITER: Iterative Transformer-based Entity Recognition and Relation Extraction (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in NLP generate structured information in an autoregressive manner, causing low throughput . authors propose an efficient encoder-based relation extraction model that performs the task in three parallelizable steps. |
| Approach: | They propose an efficient encoder-based relation extraction model that performs the task in three parallelizable steps. |
| Outcome: | The proposed model achieves state-of-the-art on two datasets and is faster than existing models. |