PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embeddings assume fixed finite-size vocabularies, hindering their ability to provide useful word representations for out-of-vocaulary words. |
| Approach: | They propose a model that generalizes word embeddings without extra contextual information . they use the spellings of words to model subword segmentation and compute subword-based compositional word embeds. |
| Outcome: | The proposed model can generate meaningful subword segmentations without any source of explicit morphological knowledge. |
Similar Papers
Generalizing Word Embeddings using Bag of Subwords (D18-1)
Copied to clipboard
| Challenge: | Existing word embeddings techniques have a fixed vocabulary, i.e., they can only provide vectors over a finite set of common words that appear frequently in a given corpus. |
| Approach: | They propose a subword-level word vector generation model that views words as bags of character n-grams and provides good vectors for rare or unseen words. |
| Outcome: | The proposed model performs state-of-the-art in English word similarity task and in joint prediction of part-of speech tag and morphosyntactic attributes in 23 languages. |
A Systematic Study of Leveraging Subword Information for Learning Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing word representation models for morphologically rich languages use subword-level information, but their systematic comparative analysis across typologically diverse languages and tasks is still missing. |
| Approach: | They propose a framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components. |
| Outcome: | The proposed framework allows for easy experimentation with different segmentation and composition components, as well as advanced techniques based on position embeddings and self-attention. |
GRaMPa: Subword Regularisation by Skewing Uniform Segmentation Distributions with an Efficient Path-counting Markov Model (2025.acl-long)
Copied to clipboard
| Challenge: | Subword regularisations are known to be stochastic, but only a handful of possible segmentations are sampled. |
| Approach: | They propose to randomise word segmentations from a subword tokeniser instead of randomising them by weighting paths in an unweighted segmentation graph. |
| Outcome: | The proposed method outperforms existing methods on token-level tasks with spelling errors. |
Embedding Words as Distributions with a Bayesian Skip-gram Model (C18-1)
Copied to clipboard
| Challenge: | Rather than assuming that word embeddings are fixed across the entire text collection, we generate them from word-specific prior densities for each word. |
| Approach: | They propose a method for embedding words as probability densities in a low-dimensional space from a word-specific prior density for each occurrence of a given word. |
| Outcome: | The proposed method can encode word as a distribution on a range of benchmarks and is comparable to Gaussian embeddings. |
Subword-based Compact Reconstruction of Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing word-based word embeddings are based on subword information and memory-shared embeddables. |
| Approach: | They propose a method for reconstructing pre-trained word embeddings using subword information using memory-shared embedds and a variant of the key-value-query self-attention mechanism. |
| Outcome: | The proposed method can imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets. |
From Text to Lexicon: Bridging the Gap between Word Embeddings and Lexical Resources (C18-1)
Copied to clipboard
| Challenge: | Distributional word representations are omnipresent in modern NLP. |
| Approach: | They propose to combine lemmatization and part of speech (POS) typing to improve word embedding performance. |
| Outcome: | The proposed methods improve word embedding performance on verbs and verbs. |
Segmentation-free compositional n-gram embedding (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding models depend on word segmentation, but this method is difficult when corpora written in noisy or unsegmented languages. |
| Approach: | They propose a new method that models words, phrases and sentences seamlessly without word segmentation. |
| Outcome: | The proposed method is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. |
Conditional Word Embedding and Hypothesis Testing via Bayes-by-Backprop (D18-1)
Copied to clipboard
| Challenge: | Whether word's meaning varies across contexts has become a major focus of research in recent years. |
| Approach: | They propose a word embedding model that incorporates document covariates to estimate conditional word embeds. |
| Outcome: | The proposed model estimates word embedding distributions based on document covariates . if word embeds are statistically significant, hypothesis tests can be performed . |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Lexically Grounded Subword Segmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Statistical word segmentation algorithms have remained a thorn in the side of many researchers. |
| Approach: | They propose to use unsupervised morphological analysis with Morfessor as pre-tokenization and an algebraic method for obtaining subword embeddings grounded in a word embeddable space. |
| Outcome: | The proposed methods improve morphological plausibility and Rényi efficiency on part-of-speech tagging and machine translation tasks. |