| Challenge: | Existing approaches to learn word embedding on a corpus with only a few million tokens are limited to low-resource languages. |
| Approach: | They propose to use a sparse co-occurrence matrix to factorize the co-existence matrix and validate the proposed approaches in four different languages. |
| Outcome: | The proposed model is validated in four different languages. |
Similar Papers
Dirichlet-Smoothed Word Embeddings for Low-Resource Settings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing count-based word embeddings are superseded by machine-learning methods like word2vec and GloVe, but in many settings there is not much text data available. |
| Approach: | They propose to use positive pointwise mutual information (PPMI) weighted co-occurrence matrices to compute word embeddings from a corpus using large amounts of text data. |
| Outcome: | The proposed method outperforms word2vec and the state-of-the-art for low-resource settings and obtains competitive results for Maltese and Luxembourgish. |
Improving Low Compute Language Modeling with In-Domain Embedding Initialisation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train language models on in-domain data are limited. |
| Approach: | They propose to initialise and freeze in-domain embeddings to provide a useful representation of rare words in English . they find that the standard configuration is not optimal when rare words are present . |
| Outcome: | The proposed approach improves language modeling by providing a useful representation of rare words in English. |
Learning Word Embeddings for Data Sparse and Sentiment Rich Data Sets (N18-4)
Copied to clipboard
| Challenge: | Existing word embeddings for sentiment analysis are limited in domain specific applications . generic word embeds are poor initialization for tasks on domain specific data sets. |
| Approach: | They propose to use word embeddings adapted for domain specific data sets in sentiment classification applications. |
| Outcome: | The proposed algorithms learn word embeddings on sparse and sentiment rich data sets. |
A Simple Approach to Learning Unsupervised Multilingual Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on unsupervised cross-lingual embeddings in the bilingual setting has given the impetus to learning a shared embeddable space for several languages. |
| Approach: | They propose to solve two sub-problems together to learn a shared embedding space for several languages. |
| Outcome: | The proposed approach outperforms existing methods in bilingual lexicon induction, cross-lingual word similarity, multilingual document classification, and multilingual dependency parsing tasks. |
Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models (P19-1)
Copied to clipboard
| Challenge: | Existing methods that map word embeddings into a common space without any parallel data or pre-training have been proposed that are limited in resources and perform poorly under resource-poor conditions. |
| Approach: | They propose a model that maps monolingual word embeddings into a common space without any parallel data and generates multilingual embeddables without any pre-training. |
| Outcome: | The proposed model outperforms existing methods on word alignment tasks on low-resource conditions and with limited resources. |
Joint Learning of Sense and Word Embeddings (L18-1)
Copied to clipboard
| Challenge: | Existing methods for learning lower-dimensional representations of words using unlabelled data learn a single representation for a word, ignoring the different senses of that word (polysemy). |
| Approach: | They propose a method that jointly learns sense-aware word embeddings using both unlabelled and sense-tagged text corpora. |
| Outcome: | The proposed method outperforms competing methods on word similarity and short-text classification benchmark datasets. |
Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures (2022.lrec-1)
Copied to clipboard
| Challenge: | a paper aims to evaluate embedding similarity, stability and reliability in low-resource settings . it uses corpus similarity measures before training to predict properties of embeddables . |
| Approach: | They use corpus similarity measures before training to predict properties of embeddings . they then apply the same measures to low-resource settings by modelling reliability . authors hope to use this method to evaluate low-source languages with limited corpus size . |
| Outcome: | The paper shows that it is possible to predict downstream embedding similarity using upstream corpus similarity measures . the main finding is that the measures remain robust on small amounts of training data . |
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing methods to learn cross-lingual word embeddings have failed in more realistic scenarios . a fully unsupervised initialization and a robust self-learning algorithm are needed to improve the existing methods. |
| Approach: | They propose an unsupervised initialization method that exploits structural similarity of embeddings and a robust self-learning algorithm that iteratively improves it. |
| Outcome: | The proposed method achieves the best published results in standard datasets even surpassing previous supervised systems. |
Unsupervised Cross-Lingual Representation Learning (P19-4)
Copied to clipboard
| Challenge: | a comprehensive survey of cutting-edge weakly-supervised and unsupervised cross-lingual word representations is presented . |
| Approach: | This tutorial provides a comprehensive survey of recent work on weakly-supervised and unsupervised cross-lingual word representations. |
| Outcome: | This tutorial provides a comprehensive survey of cutting-edge weakly-supervised and unsupervised word representations. |
TEMA: Token Embeddings Mapping for Enriching Low-Resource Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Low-resource languages, that is, languages that do not have a massive amount of text, risk being almost excluded from the possibility of having good NLP applications. |
| Approach: | They propose an algorithm that maps token embeddings of a richly pre-trained model to a poorly trained model and creates a more complex model. |
| Outcome: | The proposed model reduces perplexity and is competitive or better for the most semantic tasks. |