| Challenge: | Distributed word representations, or word vectors, have been used in natural language processing for many tasks. |
| Approach: | They propose to use the encyclopedia Wikipedia and the common crawl corpus to train distributed word representations on large corpora and use them in downstream tasks. |
| Outcome: | The proposed model performs very well on 10 languages for which evaluation dataset exists. |
Similar Papers
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Give your Text Representation Models some Love: the Case for Basque (2020.lrec-1)
Copied to clipboard
Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, Eneko Agirre
| Challenge: | Word embeddings and pre-trained language models are expensive to train and are often used by small companies and research groups to build their own. |
| Approach: | They propose to use word embeddings and pre-trained language models to build rich representations of text and improve NLP tasks. |
| Outcome: | The proposed models perform better than publicly available versions in downstream NLP tasks for Basque. |
Multilingual Culture-Independent Word Analogy Datasets (2020.lrec-1)
Copied to clipboard
| Challenge: | In text processing, deep neural networks use word embeddings as an input. |
| Approach: | They propose to use benchmark datasets to compare the quality of word embeddings in text processing . they use a word analogy task in Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish . |
| Outcome: | The proposed datasets are culturally independent and cross-lingual for the languages used. |
Subword-level Word Vector Representations for Korean (P18-1)
Copied to clipboard
| Challenge: | Existing research on word vectors for English focuses on decomposing words into subword units and using subwords to improve performance. |
| Approach: | They propose to decompose Korean words into the jamo-level, beyond the character-level . they develop Korean test sets for word similarity and analogy and make them publicly available . |
| Outcome: | The proposed method outperforms word2vec and character-level skip-grams on similarity and analogy tasks and contributes positively toward downstream NLP tasks such as sentiment analysis. |
Can Network Embedding of Distributional Thesaurus Be Combined with Word Vectors for Better Representation? (N18-1)
Copied to clipboard
| Challenge: | Distributed representations of words learned from text have proved to be successful in various natural language processing tasks. |
| Approach: | They propose to embed a distributional thesaurus network into dense word vectors and compare them to state-of-the-art word representations. |
| Outcome: | The proposed representations improve performance against state-of-the-art word representations even without handcrafted lexical resources. |
Pre-training Universal Language Representation (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the cutting-edge representation learning, most language models focus on specific levels of linguistic units. |
| Approach: | They propose a training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by an algorithm for pre-trained language models. |
| Outcome: | The proposed model achieves highest accuracy on analogy tasks in different language levels and significantly improves performance on downstream tasks. |
Leveraging a Bilingual Dictionary to Learn Wolastoqey Word Representations (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing word embeddings for lowresource languages require large corpora of running text to learn high quality representations. |
| Approach: | They leverage a bilingual dictionary to learn Wolastoqey word embeddings by encoding their corresponding English definitions into vector representations using pretrained English word and sequence representation models. |
| Outcome: | The proposed model outperforms baseline models without language-specific training or fine-tuning. |
Unsupervised Cross-Lingual Representation Learning (P19-4)
Copied to clipboard
| Challenge: | a comprehensive survey of cutting-edge weakly-supervised and unsupervised cross-lingual word representations is presented . |
| Approach: | This tutorial provides a comprehensive survey of recent work on weakly-supervised and unsupervised cross-lingual word representations. |
| Outcome: | This tutorial provides a comprehensive survey of cutting-edge weakly-supervised and unsupervised word representations. |
Meaning Representations for Natural Languages: Design, Models and Applications (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation models. |
| Approach: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation models. |
| Outcome: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation models . it also reviews the applications of meaning representation in downstream NLP tasks and real-world applications . |
Generalizing Word Embeddings using Bag of Subwords (D18-1)
Copied to clipboard
| Challenge: | Existing word embeddings techniques have a fixed vocabulary, i.e., they can only provide vectors over a finite set of common words that appear frequently in a given corpus. |
| Approach: | They propose a subword-level word vector generation model that views words as bags of character n-grams and provides good vectors for rare or unseen words. |
| Outcome: | The proposed model performs state-of-the-art in English word similarity task and in joint prediction of part-of speech tag and morphosyntactic attributes in 23 languages. |