| Challenge: | Existing word embeddings techniques have a fixed vocabulary, i.e., they can only provide vectors over a finite set of common words that appear frequently in a given corpus. |
| Approach: | They propose a subword-level word vector generation model that views words as bags of character n-grams and provides good vectors for rare or unseen words. |
| Outcome: | The proposed model performs state-of-the-art in English word similarity task and in joint prediction of part-of speech tag and morphosyntactic attributes in 23 languages. |
Similar Papers
PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embeddings assume fixed finite-size vocabularies, hindering their ability to provide useful word representations for out-of-vocaulary words. |
| Approach: | They propose a model that generalizes word embeddings without extra contextual information . they use the spellings of words to model subword segmentation and compute subword-based compositional word embeds. |
| Outcome: | The proposed model can generate meaningful subword segmentations without any source of explicit morphological knowledge. |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Subword-based Compact Reconstruction of Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing word-based word embeddings are based on subword information and memory-shared embeddables. |
| Approach: | They propose a method for reconstructing pre-trained word embeddings using subword information using memory-shared embedds and a variant of the key-value-query self-attention mechanism. |
| Outcome: | The proposed method can imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Subword Attention and Post-Processing for Rare and Unknown Contextualized Embeddings (2024.findings-naacl)
Copied to clipboard
| Challenge: | Word embeddings are useful, but struggle on rare and unknown words. |
| Approach: | They propose a rare/unknown embedding architecture that focuses on contextualized representations. |
| Outcome: | The proposed architecture improves performance in most intrinsic and downstream tasks. |
COIN – an Inexpensive and Strong Baseline for Predicting Out of Vocabulary Word Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | Word embedding models only include terms that occur a sufficient number of times in training corpora. |
| Approach: | They propose a method for predicting word embeddings for out of vocabulary terms using word2vec. |
| Outcome: | The proposed method surpasses several methods on benchmark tasks and is inexpensive to compute. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Reusing Weights in Subword-Aware Neural Language Models (N18-1)
Copied to clipboard
| Challenge: | a statistical language model assigns a probability to a sequence of words . data sparsity is a major problem in building traditional n-gram language models . |
| Approach: | They propose several ways to reuse subword embeddings and other weights in subword-aware neural language models. |
| Outcome: | The proposed techniques do not benefit a competitive character-aware model . but they show significant reductions in model sizes and performance. |
Enhancing Word Embeddings with Knowledge Extracted from Lexical Resources (2020.acl-srw)
Copied to clipboard
| Challenge: | In this paper, we present an effective method for semantic specialization of word vector representations. |
| Approach: | They propose a method for semantic specialization of word vector representations using BabelNet. |
| Outcome: | The proposed method improves on word similarity and dialog state tracking tasks. |