| Challenge: | Missing sentence generation fosters a wide range of applications in natural language generation . Developing models for sentence infilling can potentially facilitate many text generation applications . |
| Approach: | They propose a framework to decouple the problem from natural language processing . they propose generating missing sentences that can syntactically and semantically bridge context . |
| Outcome: | The proposed model learns a sentence representation and generates 'missing sentences' the proposed model can be used for document auto-completion and meeting note expansion . |
Similar Papers
Enabling Language Models to Fill in the Blanks (2020.acl-main)
Copied to clipboard
| Challenge: | Infilling is the task of predicting missing spans of text at any position in a document. |
| Approach: | They propose a framework which can be used to infill entire sentences . they train off-the-shelf LMs on sequences containing concatenation of masked text . |
| Outcome: | The proposed approach can infill entire sentences on short stories, scientific abstracts, and lyrics. |
TIGS: An Inference Algorithm for Text Infilling with Gradient Search (P19-1)
Copied to clipboard
| Challenge: | Text infilling is an under-explored challenge in the field of text generation. |
| Approach: | They propose an iterative inference algorithm based on gradient search that can be broadly applied to any sequence generative model for text infilling tasks. |
| Outcome: | The proposed method performs well on three different text infilling tasks with different mask ratios and mask strategies compared with five state-of-the-art methods. |
Varying Sentence Representations via Condition-Specified Routers (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing sentences cannot account for different aspects of semantic similarity between two sentences. |
| Approach: | They propose a transformer-style framework that generates conditioned sentences . they propose 'conditional' STS, which measures similarity between two sentences based on condition sentences - a task that requires a sentence embedding model capable of generating distinct representations for the same sentence under different conditions. |
| Outcome: | The proposed framework is superior to existing models on two condition sentences . it can generate conditioned sentences while maintaining model parameters and computational efficiency . |
A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations (N19-1)
Copied to clipboard
| Challenge: | Empirically, the model with the best performing syntactic and semantic representations gives rise to the most disentangled representations. |
| Approach: | They propose a generative model that uses latent variables to learn a sentence that uses both latent and latent representations. |
| Outcome: | The proposed model achieves better disentanglement between semantic and syntactic representations by training with multiple losses, including losses that exploit aligned paraphrastic sentences and word-order information. |
Sentence Bottleneck Autoencoders from Transformer Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for pretraining a language model on text have been used for building models in NLP, but they do not work for sentence representations derived from pretrainer models based on tokens or basic pooling operations. |
| Approach: | They propose to build a sentence-level autoencoder from a pretrained transformer language model. |
| Outcome: | The proposed model achieves better quality than previous methods on text similarity and style transfer tasks while using fewer parameters than large pretrained models. |
Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence (2021.acl-long)
Copied to clipboard
| Challenge: | Existing generation models struggle to maintain a coherent event sequence throughout the generated text. |
| Approach: | They propose a long text generation model which can represent prefix sentences at sentence level and discourse level in the decoding process. |
| Outcome: | The proposed model can generate more coherent texts than state-of-the-art models. |
Natural Language Deduction with Incomplete Information (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for reasoning given incomplete information are inadequate . current approaches to reasoning are based on latent reasoning by large language models . |
| Approach: | They propose a system that generates a natural language "proof" by abductively inferring a premise from another premise and a conclusion. |
| Outcome: | The proposed system can handle the underspecified setting where not all premises are stated at the outset; additional assumptions need to be materialized to prove a claim. |
A Bilingual Generative Transformer for Semantic Sentence Embedding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embeddable space indicates closeness of semantics between the sentences. |
| Approach: | They propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. |
| Outcome: | The proposed model outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. |
Text Compression for Efficient Language Generation (2025.naacl-srw)
Copied to clipboard
| Challenge: | Existing models rely on sub-word tokens for text generation, but there is no evidence for a more efficient way to generate text. |
| Approach: | They propose a hierarchical transformer language model capable of text generation by compressing text into sentence embeddings and employing a sentence attention mechanism. |
| Outcome: | The proposed model achieves an up to an order of magnitude improvement in FLOPs efficiency and a threefold increase in runtime speed compared to equally-sized models in the low-size regime. |
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |