Improving Text Auto-Completion with Next Phrase Prediction (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Language models such as GPT-2 require considerable training effort to adapt to specific writing domains (e.g., medical). |
| Approach: | They propose an intermediate training strategy that encourages language models to complete partial queries with enriched phrases and eventually improve their text auto-completion performance. |
| Outcome: | The proposed approach outperforms baselines in auto-completion tasks for email and academic-writing domains with only around 1.2B tokens. |
Similar Papers
Subword Language Model for Query Auto-Completion (D19-1)
Copied to clipboard
| Challenge: | Current neural query auto-completion systems rely on character-level language models but they slow down when queries are long. |
| Approach: | They propose to use subword language models for query auto-completion . they propose to implement a retrace algorithm and a method for approximate marginalization . |
| Outcome: | The proposed model achieves 2.5 times faster while maintaining similar quality to character-level baseline. |
Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task (2020.emnlp-main)
Copied to clipboard
| Challenge: | despite the success of contextualized language models, language models cannot capture textual coherence of a long, multi-sentence document. |
| Approach: | They propose a paragraph completion task that predicts masked sentences in a sentence . they propose SSPlanner that predict what to say first and guides the pretrained model . |
| Outcome: | The proposed model outperforms baseline generation models on the paragraph completion task in automatic and human evaluation. |
Personalized Language Model for Query Auto-Completion (P18-2)
Copied to clipboard
| Challenge: | Query auto-completion (QAC) is a search engine feature that suggests completed queries as the user types . recent work suggests personalization of the recurrent layer to generate personalized completions. |
| Approach: | They propose to use a recurrent neural network language model to generate personalized completions for search engines. |
| Outcome: | The proposed model can generate personalized completions for users not seen during training. |
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future (P19-1)
Copied to clipboard
| Challenge: | Common language models typically predict the next word given a past context. |
| Approach: | They propose a method that aligns the given context and the following phrase . they define syntactic heights and phrase segmentation rules to enable it to learn . |
| Outcome: | The proposed model outperforms strong baseline models on Wikitext-103 dataset. |
Multi-Stage Pre-training for Low-Resource Domain Adaptation (2020.emnlp-main)
Copied to clipboard
Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, Todd Ward
| Challenge: | Existing approaches to transfer learning target data to in-domain text . prior work has adapted pre-trained LMs to specific domains . |
| Approach: | They extend the vocabulary of a pretrained language model with domain-specific terms to create synthetic tasks that help it transfer to downstream tasks. |
| Outcome: | The proposed approaches show significant performance gains on extractive reading comprehension, document ranking and duplicate question detection tasks. |
Personalized neural language models for real-world query auto completion (N18-3)
Copied to clipboard
| Challenge: | Existing popularity-based methods for query auto completion (QAC) are ineffective in predicting unseen queries. |
| Approach: | They propose to use real-world data to build an end-to-end system that can predict unseen queries by integrating user information. |
| Outcome: | The proposed methods improve on two separate datasets while increasing diversity while scalability. |
RecGPT: Generative Pre-training for Text-based Recommendation (2024.acl-short)
Copied to clipboard
| Challenge: | Existing models for text-based recommendation lack data sparsity and flexibility to capture fluctuations in user preferences over time. |
| Approach: | They present the first domain-adapted and fully-trained large language model for text-based recommendation. |
| Outcome: | The proposed model outperforms baseline models on rating prediction and sequential recommendation tasks. |
Toward Better Storylines with Sentence-Level Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Rather than modeling fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. |
| Approach: | They propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. |
| Outcome: | The proposed model can focus on longer range dependencies, crucial for multi-sentence coherence. |
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. |
| Approach: | They propose a framework that integrates Language Learning Tasks alongside standard next-token prediction to stimulate the acquisition of morphological, syntactic, and semantic knowledge. |
| Outcome: | The proposed framework improves performance on linguistic competence benchmarks while maintaining competitive performance on reasoning tasks. |
Incorporating Syntactic Knowledge into Pre-trained Language Model using Optimization for Overcoming Catastrophic Forgetting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models lack syntactic knowledge for many tasks that handle complex or long sentences. |
| Approach: | They propose to use pre-trained language models to incorporate syntactic knowledge into a model by adding additional syntatic knowledge to the model. |
| Outcome: | The proposed model can be easily applied to downstream tasks that require syntactic knowledge. |