Mention Flags (MF): Constraining Transformer-based Text Generators (2021.acl-long)
Copied to clipboard
| Challenge: | Constrained decoding algorithms produce hypotheses satisfying all constraints, but they are computationally expensive and can lower the generated text quality. |
| Approach: | They propose a Mention Flag mechanism which traces whether lexical constraints are satisfied in outputs of an S2S decoder. |
| Outcome: | The proposed models maintain higher constraint satisfaction and text quality than baseline models and other constrained decoding algorithms. |
Similar Papers
PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models (LM) based on transformers generate plausible long texts . a discriminator-guided approach allows to apply constraints more finely and dynamically. |
| Approach: | They propose to use a discriminator-guided approach to generate constrained texts without fine-tuning the LM. |
| Outcome: | The proposed method is easier and cheaper to train than fine-tuning the LM. |
Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for text-to-text generation do not explicitly focus on important concepts in the input and output. |
| Approach: | They propose a framework to automatically extract, denoise, and enforce important input concepts as lexical constraints. |
| Outcome: | The proposed framework performs comparably or better than its unconstrained counterpart on automatic metrics and receives better ratings in the human evaluation. |
POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models cannot be directly employed to generate text under specified lexical constraints. |
| Approach: | They propose a method for insertion-based text generation that inserts tokens between existing tokens in a parallel manner. |
| Outcome: | The proposed method is intuitive and interpretable on Wikipedia and Yelp datasets. |
Constrained Decoding for Computationally Efficient Named Entity Recognition Taggers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Named entity recognition models use a conditional random field as the final layer . current work eschews prior knowledge of how the span encoding scheme works . |
| Approach: | They propose to constrain the output to suppress illegal transitions to train a tagger with a cross-entropy loss twice as fast as a CRF. |
| Outcome: | The proposed model trains twice as fast as a CRF with statistically insignificant differences in F1 . the proposed model is open source and can be used in PyTorch and TensorFlow. |
Training Neural Machine Translation to Apply Terminology Constraints (P19-1)
Copied to clipboard
| Challenge: | Existing methods to integrate domain terminology into neural machine translation (NMT) are brittle when tested in real-world situations. |
| Approach: | They propose a method to inject custom terminology into neural machine translation at run time by using the target side of terminology entries whose source side match the input as decoding-time constraints. |
| Outcome: | The proposed method is faster than state-of-the-art decoding and more efficient than constraint-free decoding. |
Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsing (2023.tacl-1)
Copied to clipboard
| Challenge: | Sequence-to-Sequence (S2S) models have been successful on text generation tasks . however, learning complex structures with S2S models remains challenging . |
| Approach: | They propose to use constrained decoding to model part-of-speech tagging, named entity recognition, constituency, and dependency parsing tasks with 3 lexically diverse linearization schemas and corresponding constrained coding methods. |
| Outcome: | The proposed methods outperform the state-of-the-art on four core tasks. |
Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging (2022.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks, but they fail to achieve state-of-the-art (SOTA) performance. |
| Approach: | They propose a Guassian HMM variant for unsupervised POS tagging that incorporates contexualized word representations into the decoder. |
| Outcome: | The proposed model outperforms state-of-the-art models on Penn Treebank and multilingual Universal Dependencies treebank v2.0. |
Enconter: Entity Constrained Progressive Sequence Generation via Insertion-based Transformer (2021.eacl-main)
Copied to clipboard
| Challenge: | Autoregressive language models do not perform well under hard lexical constraints as they lack fine control of content generation process. |
| Approach: | They propose a new insertion transformer that considers hard lexical constraints and imposes rules over objects in the generated text. |
| Outcome: | The proposed model outperforms baseline models in several performance metrics rendering it more suitable in practical applications. |
Diagnosing Transformers in Task-Oriented Semantic Parsing (2021.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows transformer-based parsers struggle with disambiguating intents/slots and producing syntactically valid frames. |
| Approach: | They propose to use seq2seq transformers to map textual utterances to semantic frames . they propose to model transformer-based parsers across monolingual and multilingual settings . |
| Outcome: | The proposed parsers struggle with disambiguating intents/slots and produce syntactically valid frames. |
Transformer and seq2seq model for Paraphrase Generation (D19-56)
Copied to clipboard
| Challenge: | Existing methods for generating paraphrases fall into one of these broad categories -rule-based, seq2seq, deep generative models and a varied combination. |
| Approach: | They propose a framework that combines transformer and sequence-to-sequence models for better quality of generated paraphrases. |
| Outcome: | The proposed framework improves on two datasets-QUORA and MSCOCO using transformer and sequence-to-sequence models. |