Aleksandra Piktus, Necati Bora Edizel, Piotr Bojanowski, Edouard Grave, Rui Ferreira, Fabrizio Silvestri
| Challenge: | Existing word embeddings have limited applicability to malformed texts . misspellings are frequent and embeddable for words that have not been observed at training time . |
| Approach: | They propose a method to learn word embeddings that are resilient to misspellings . they use FastText with subwords to train embeddables on a new dataset . |
| Outcome: | The proposed method is tested on a publicly available dataset. |
Similar Papers
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Subword-based Compact Reconstruction of Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing word-based word embeddings are based on subword information and memory-shared embeddables. |
| Approach: | They propose a method for reconstructing pre-trained word embeddings using subword information using memory-shared embedds and a variant of the key-value-query self-attention mechanism. |
| Outcome: | The proposed method can imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets. |
A Simple Approach to Learning Unsupervised Multilingual Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on unsupervised cross-lingual embeddings in the bilingual setting has given the impetus to learning a shared embeddable space for several languages. |
| Approach: | They propose to solve two sub-problems together to learn a shared embedding space for several languages. |
| Outcome: | The proposed approach outperforms existing methods in bilingual lexicon induction, cross-lingual word similarity, multilingual document classification, and multilingual dependency parsing tasks. |
Do Word Embeddings Capture Spelling Variation? (2020.coling-main)
Copied to clipboard
| Challenge: | Using word embeddings, we analyze spelling variation in word embeds trained on Twitter and Reddit data. |
| Approach: | They propose a new perspective on the analysis of word embeddings by focusing on spelling variation. |
| Outcome: | The proposed analysis shows that word embeddings encode spelling variation patterns of various types to some extent, even when trained using the skipgram model. |
Deconstructing word embedding algorithms (2020.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are reliable feature representations of words used in many NLP tasks today. |
| Approach: | They propose to deconstruct Word2vec, GloVe and others into a common form . they propose to generalize several word embedding algorithms into . a low rank embedder framework is proposed to generalise the algorithms into one common form. |
| Outcome: | The proposed framework can be used to make word embeddings more performant. |
Addressing Noise in Multidialectal Word Embeddings (P18-2)
Copied to clipboard
| Challenge: | Dialectal Arabic (DA) is problematically noisy and lacks a large corpus of non-noisy words. |
| Approach: | They propose to use word embedding tools to maximize the informative content leveraged in each training sentence and analyze methods for representing disparate dialects in one embeddable space. |
| Outcome: | The proposed methods improve performance on low and high frequency words while preserving accuracy on low frequency forms. |
Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)
Copied to clipboard
| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
Tiny Word Embeddings Using Globally Informed Reconstruction (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for word embedding reconstruction use only local information of subwords and pre-trained word embeds. |
| Approach: | They propose a global loss function that uses words other than the target word to improve word embedding reconstruction by a factor of 200. |
| Outcome: | The proposed method reduces the model size of pre-trained word embeddings by a factor of 200 while preserving its quality. |
Joint Learning of Sense and Word Embeddings (L18-1)
Copied to clipboard
| Challenge: | Existing methods for learning lower-dimensional representations of words using unlabelled data learn a single representation for a word, ignoring the different senses of that word (polysemy). |
| Approach: | They propose a method that jointly learns sense-aware word embeddings using both unlabelled and sense-tagged text corpora. |
| Outcome: | The proposed method outperforms competing methods on word similarity and short-text classification benchmark datasets. |