| Challenge: | Current neural query auto-completion systems rely on character-level language models but they slow down when queries are long. |
| Approach: | They propose to use subword language models for query auto-completion . they propose to implement a retrace algorithm and a method for approximate marginalization . |
| Outcome: | The proposed model achieves 2.5 times faster while maintaining similar quality to character-level baseline. |
Similar Papers
Personalized neural language models for real-world query auto completion (N18-3)
Copied to clipboard
| Challenge: | Existing popularity-based methods for query auto completion (QAC) are ineffective in predicting unseen queries. |
| Approach: | They propose to use real-world data to build an end-to-end system that can predict unseen queries by integrating user information. |
| Outcome: | The proposed methods improve on two separate datasets while increasing diversity while scalability. |
Personalized Language Model for Query Auto-Completion (P18-2)
Copied to clipboard
| Challenge: | Query auto-completion (QAC) is a search engine feature that suggests completed queries as the user types . recent work suggests personalization of the recurrent layer to generate personalized completions. |
| Approach: | They propose to use a recurrent neural network language model to generate personalized completions for search engines. |
| Outcome: | The proposed model can generate personalized completions for users not seen during training. |
Rethinking Word-Level Auto-Completion in Computer-Aided Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for word-level auto-completion (WLAC) do not meet the criterion of good auto-completes. |
| Approach: | They propose a measurable criterion to address the question: what kind of words are good auto-completions? they propose an approach to enhance WLAC performance by promoting adherence to the cri-terion. |
| Outcome: | The proposed approach outperforms the top-performing system submitted to the WLAC shared tasks in WMT2022 while using significantly smaller model sizes. |
Subword-Delimited Downsampling for Better Character-Level Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword-level models are expensive in terms of time and computation, but character-level model with downsampling component can be used for machine translation. |
| Approach: | They propose a character-level downsampling method which is informed by subwords to improve model performance. |
| Outcome: | The proposed method outperforms existing methods and shows that it can be done without sacrificing quality. |
Improving Text Auto-Completion with Next Phrase Prediction (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Language models such as GPT-2 require considerable training effort to adapt to specific writing domains (e.g., medical). |
| Approach: | They propose an intermediate training strategy that encourages language models to complete partial queries with enriched phrases and eventually improve their text auto-completion performance. |
| Outcome: | The proposed approach outperforms baselines in auto-completion tasks for email and academic-writing domains with only around 1.2B tokens. |
AutoRE: Document-Level Relation Extraction with Large Language Models (2024.acl-demos)
Copied to clipboard
| Challenge: | Existing methods for relation extraction are limited to Sentence-level Relation Extraction (SentRE) tasks. |
| Approach: | They propose an end-to-end DocRE model that adopts a novel RE extraction paradigm named RHF (Relation-Head-Facts) Unlike existing approaches, AutoRE does not rely on the assumption of known relation options, making it more reflective of real-world scenarios. |
| Outcome: | The proposed model surpasses TAG by 10.03% and 9.03% on the dev and test set. |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
Neural Text Normalization with Subword Units (N19-2)
Copied to clipboard
| Challenge: | Text normalization (TN) is an important step in conversational systems. |
| Approach: | They frame text normalization as a machine translation task and tackle it with sequence-to-sequence models. |
| Outcome: | The proposed model normalizes written text to its spoken form to facilitate speech recognition and text-to-speech synthesis. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
On the Importance of Word Boundaries in Character-level Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural Machine Translation models typically use a fixed-size lexical vocabulary . subword segmentation methods rely on statistical heuristics that lack any linguistic notion . |
| Approach: | They propose a hierarchical decoding architecture for character-level NMT using subwords . they propose fewer parameters and a more efficient approach to perform translation at the level of words . |
| Outcome: | The proposed model can reach higher translation accuracy than the subword-level model with fewer parameters while maintaining longer-distance contextual and grammatical dependencies. |