| Challenge: | Existing methods for learning word embedding assume there are enough occurrences for each word in the corpus to accurately estimate the representation of words. |
| Approach: | They propose to fit a representation function to predict an oracle embedding vector based on limited contexts. |
| Outcome: | The proposed model outperforms existing methods in constructing an accurate embedding for OOV words and improves downstream tasks when the embeddable is utilized. |
Similar Papers
Robust Backed-off Estimation of Out-of-Vocabulary Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to solving out-of-vocabulary (OOV) words use subwords to represent oov words with a bag of subword. |
| Approach: | They propose a method to estimate oov word embeddings by referring to pre-trained word embeds for known words with similar surfaces to target ov words. |
| Outcome: | The proposed method improves word similarity tasks and biomedical tasks even with weak baselines. |
Representation Learning for Unseen Words by Bridging Subwords to Semantic Networks (2020.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained word embeddings only include words that appeared in corpora where pre-tried embedds are learned. |
| Approach: | They propose a method to represent out-of-vocabulary words using subword information and knowledge. |
| Outcome: | The proposed method improves performance over baselines that only use subwords or knowledge to represent OOV words. |
Enhancing Out-of-Vocabulary Estimation with Subword Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to learn OOV word representations use advanced architectures like attention on the context of the word, but they tend to use simple structures like ngram addition or character based convolutional neural networks (CNN) |
| Approach: | They propose a transformer-based OOV estimation model that uses attention mechanisms on both the context and the subwords to learn OOV representations. |
| Outcome: | The proposed model outperforms current state-of-the-art models on OOV representations based on attention mechanisms on the context and subwords . |
Handling Out-Of-Vocabulary Problem in Hangeul Word Embeddings (2021.eacl-main)
Copied to clipboard
| Challenge: | Word embedding is considered an essential factor in improving the performance of various Natural Language Processing (NLP) models. |
| Approach: | They propose a Hangeul word embedding model that infers original word embeds from typos while maintaining high performance. |
| Outcome: | The proposed model performs well against typos while maintaining high performance. |
Imputing Out-of-Vocabulary Embeddings with LOVE Makes LanguageModels Robust with Little Cost (2022.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art NLP systems are brittle when faced with Out-ofVocabulary words . we present a framework that extends word embeddings and makes them robust to OOV . |
| Approach: | They propose a framework that extends existing word embeddings and makes them robust to OOV. |
| Outcome: | The proposed model performs better on original datasets and corrupted variants than previous competitors. |
Using dependency parsing for few-shot learning in distributional semantics (2022.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for few-shot learning use dependency parsing information to learn meaning of rare words based on limited amount of context sentences. |
| Approach: | They propose dependency parsing for few-shot learning to learn meaning of rare words . they use word embedding models as background spaces for few shot learning . |
| Outcome: | The proposed methods enhance the additive baseline model by using dependencies. |
Learning to Generate Word Representations using Subword Information (C18-1)
Copied to clipboard
| Challenge: | Existing word-based approaches to learning word representations are blind to subword information in words. |
| Approach: | They propose a character-based word representation approach to learn word representations from characters. |
| Outcome: | The proposed model outperforms baseline models that regard words as atomic units . the proposed model achieves 18.5% improvement on average in perplexity for morphologically rich languages . |
Improving Word Embeddings through Iterative Refinement of Word- and Character-level Models (2020.coling-main)
Copied to clipboard
| Challenge: | Embedding of rare and out-of-vocabulary words is an important open NLP problem . standard embedding models are not useful for recommending jobs to users with rare or unseen words . |
| Approach: | They propose to train a character-level neural network to reproduce word embeddings . they then use the model to assign vectors to any input string, including rare words . |
| Outcome: | The proposed method outperforms existing methods on word similarity data sets and can be applied to job title normalization in the e-recruitment domain. |
Subword-based Compact Reconstruction of Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing word-based word embeddings are based on subword information and memory-shared embeddables. |
| Approach: | They propose a method for reconstructing pre-trained word embeddings using subword information using memory-shared embedds and a variant of the key-value-query self-attention mechanism. |
| Outcome: | The proposed method can imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets. |
Evaluating Sub-word Embeddings in Cross-lingual Models (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches to learning sub-word embeddings for out-of-vocabulary words have not considered sub- word embedds in cross-lingual models. |
| Approach: | They propose to use sub-word embeddings to form cross-lingual embeddables for out-of-vocabulary (OOV) words for which no embeddibles are available. |
| Outcome: | The proposed bilingual lexicon induction task shows that sub-word embeddings can be leveraged to form cross-lingual embeddables for OOV words. |