A-TIP: Attribute-aware Text Infilling via Pre-trained Language Model (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for text infilling focus on the infill length of blanks and attribute relevance, but attribute-aware content can be more useful. |
| Approach: | They propose an attribute-aware text infilling method via a Pre-trained language model which contains a text in filling component and a plug-and-play discriminator. |
| Outcome: | The proposed method improves attribute relevance without decreasing text fluency on three open-source datasets. |
Similar Papers
Enabling Language Models to Fill in the Blanks (2020.acl-main)
Copied to clipboard
| Challenge: | Infilling is the task of predicting missing spans of text at any position in a document. |
| Approach: | They propose a framework which can be used to infill entire sentences . they train off-the-shelf LMs on sequences containing concatenation of masked text . |
| Outcome: | The proposed approach can infill entire sentences on short stories, scientific abstracts, and lyrics. |
Attribute Alignment: Controlling Text Generation from Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can generate text with sentiment polarity or specific topics without changing the original model parameters. |
| Approach: | They propose a method for controlling text generation by aligning disentangled attribute representations. |
| Outcome: | The proposed method shows large performance gains while maintaining diversity and fluency. |
APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu
| Challenge: | Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement. |
| Approach: | They propose a method that involves tuning a small set of soft prompts for pre-trained language models. |
| Outcome: | The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark. |
Tailor: A Soft-Prompt-Based Approach to Attribute-Based Controlled Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work focuses on generating sentences satisfying pre-specified attributes such as topic and sentiment, yet suffers from increases in storage and inference time. |
| Approach: | They propose a method that uses a pre-trained continuous vector to generate a fixed pre-trainable language model to satisfy a specified attribute. |
| Outcome: | The proposed model can achieve improvements on eleven attribute-specific generation tasks with 0.08% extra training parameters. |
Empowering Character-level Text Infilling by Eliminating Sub-Tokens (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for character-level infilling relied on predicting sub-tokens, but this strategy was ineffective. |
| Approach: | They propose a method to fill-in-the-mid with Starting and Ending character constraints that avoids predicting sub-tokens in inference. |
| Outcome: | The proposed method surpasses existing methods and offers significant performance gains. |
TIGS: An Inference Algorithm for Text Infilling with Gradient Search (P19-1)
Copied to clipboard
| Challenge: | Text infilling is an under-explored challenge in the field of text generation. |
| Approach: | They propose an iterative inference algorithm based on gradient search that can be broadly applied to any sequence generative model for text infilling tasks. |
| Outcome: | The proposed method performs well on three different text infilling tasks with different mask ratios and mask strategies compared with five state-of-the-art methods. |
TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models (2022.acl-demo)
Copied to clipboard
| Challenge: | Large pre-trained language models have been used for many NLP tasks but computational resources are limited. |
| Approach: | They propose an open-source model pruning toolkit for pre-trained language models . they propose a self-supervised pruning method that can be applied without labeled data. |
| Outcome: | The proposed pruning method reduces model size without retraining the model and speeds up inference speed on the common CPU and GPU devices. |
INSET: Sentence Infilling with INter-SEntential Transformer (2020.acl-main)
Copied to clipboard
| Challenge: | Missing sentence generation fosters a wide range of applications in natural language generation . Developing models for sentence infilling can potentially facilitate many text generation applications . |
| Approach: | They propose a framework to decouple the problem from natural language processing . they propose generating missing sentences that can syntactically and semantically bridge context . |
| Outcome: | The proposed model learns a sentence representation and generates 'missing sentences' the proposed model can be used for document auto-completion and meeting note expansion . |
RecGPT: Generative Pre-training for Text-based Recommendation (2024.acl-short)
Copied to clipboard
| Challenge: | Existing models for text-based recommendation lack data sparsity and flexibility to capture fluctuations in user preferences over time. |
| Approach: | They present the first domain-adapted and fully-trained large language model for text-based recommendation. |
| Outcome: | The proposed model outperforms baseline models on rating prediction and sequential recommendation tasks. |
MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only large language models are increasingly being adapted for bidirectional modeling . however, their reliance on causal attention restricts their effectiveness in tasks that require understanding of bidirectional context. |
| Approach: | They propose a method to adapt decoder-only large language models to generate robust representations and infill missing text spans. |
| Outcome: | The proposed method surpasses strong decoders on token-level and sentence-level representation learning tasks and generates contextually appropriate text infills without excessive repetition of words or phrases. |