The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank (2022.findings-naacl)
Copied to clipboard
| Challenge: | a natural language generation system can be used to create text at the end of a passage . fill in the blank (FITB) is a task of inserting text into a specified position in a text . |
| Approach: | They evaluate the feasibility of using a single model to perform both tasks . they show that models pre-trained with a FitB-style objective are capable of both tasks. |
| Outcome: | The proposed model can perform both fill in the blank and continuation tasks. |
Similar Papers
Enabling Language Models to Fill in the Blanks (2020.acl-main)
Copied to clipboard
| Challenge: | Infilling is the task of predicting missing spans of text at any position in a document. |
| Approach: | They propose a framework which can be used to infill entire sentences . they train off-the-shelf LMs on sequences containing concatenation of masked text . |
| Outcome: | The proposed approach can infill entire sentences on short stories, scientific abstracts, and lyrics. |
Blank Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches focus on adapting left-to-right language models for text infilling. |
| Approach: | They propose a model that generates sequences by dynamically creating and filling in blanks. |
| Outcome: | Experiments on style transfer and damaged ancient text restoration show that the proposed model outperforms baseline models in terms of accuracy and fluency. |
The Importance of Generation Order in Language Modeling (D18-1)
Copied to clipboard
| Challenge: | Neural language models are universally autoregressive, generating sentences one token at a time from left to right. |
| Approach: | They propose a two-pass language model that generates partially-filled sentences and fills in missing tokens. |
| Outcome: | The proposed model produces partially-filled sentences and fills in missing tokens. |
Sentence-Level Content Planning and Style Specification for Neural Text Generation (D19-1)
Copied to clipboard
| Challenge: | Recent advances in text generation systems often produce incoherent and unfaithful outputs . a novel automated text generation system takes into account content selection, text planning, and surface realization. |
| Approach: | They propose an end-to-end trained two-step text generation model that considers sentence-level content planners and language styles. |
| Outcome: | The proposed model outperforms competing models in three domains with diverse topics and varying language styles. |
Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to pretrain language models are limited by one-size-fits-all vocabulary . embeddings of mismatch tokens can be efficiently initialized in downstream tasks . |
| Approach: | They propose to extend pretrain-finetune pipeline with an embedding transfer step . plug-and-play embeddable generator is introduced to generate any input token . |
| Outcome: | The proposed approach allows for more efficient and better performed NLG models. |
Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing strategies to teach pre-trained models to generate simple texts are inadequate. |
| Approach: | They propose a continued pre-training strategy to teach pre-trained models to generate simple texts by randomly masking text spans in ordinary texts. |
| Outcome: | The proposed strategy improves on lexical simplification, sentence simplification and document-level simplification tasks over existing models. |
MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent pre-trained language models have produced impressive results, but there is still a gap between human written texts and machine-generated outputs. |
| Approach: | They propose a multi-task training strategy for long text generation grounded on the cognitive theory of writing. |
| Outcome: | The proposed model achieves better results on three open-ended generation tasks than baselines. |
Addressing the Training-Inference Discrepancy in Discrete Diffusion for Text Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing discrete diffusion models for text generation have a discrepancy between training and inference. |
| Approach: | They propose a training schema that considers two-step diffusion processes and a scheduling technique that gradually increases the probability of using self-generated text as training progresses. |
| Outcome: | The proposed training schema and scheduling technique improve diffusion models on four widely used datasets. |
Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for text generation ignore faithfulness between generated text and table . current methods ignore faithfulity, leading to generated information that goes beyond table content . |
| Approach: | They propose a Transformer-based generation framework to enforce faithfulness between generated text and table . they propose metric to evaluate faithfulness and automatic metric for automatic generating . |
| Outcome: | The proposed framework outperforms state-of-the-art methods in automatic evaluations and human evaluations. |
Leveraging Context Information for Natural Question Generation (N18-2)
Copied to clipboard
| Challenge: | Existing work for natural question generation ignores the input passage or hard-codes answer positions. |
| Approach: | They propose a model that matches the answer with the passage before generating a question. |
| Outcome: | The proposed model outperforms the state-of-the-art model using rich features. |