| Challenge: | a new test set that measures word embeddings' ability to recognize linguistic regularities is presented in a paper in elijsson, iran . the test sets are a good quality estimator for extrinsic evaluation of word embedded models . |
| Approach: | They propose a test set that measures language models' ability to recognize linguistic regularities in a balanced way. |
| Outcome: | The proposed set is apt at measuring the capabilities of word embedding models. |
Similar Papers
MGAD: Multilingual Generation of Analogy Datasets (L18-1)
Copied to clipboard
| Challenge: | Existing methods for word embedding evaluation are computationally expensive and task-specific. |
| Approach: | They propose a minimally supervised method for generating word embedding evaluation datasets for a large number of languages using existing dependency treebanks and parsers. |
| Outcome: | The proposed method evaluates three popular word embedding algorithms against these datasets and shows that their performance varies between syntactic categories. |
Multilingual Culture-Independent Word Analogy Datasets (2020.lrec-1)
Copied to clipboard
| Challenge: | In text processing, deep neural networks use word embeddings as an input. |
| Approach: | They propose to use benchmark datasets to compare the quality of word embeddings in text processing . they use a word analogy task in Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish . |
| Outcome: | The proposed datasets are culturally independent and cross-lingual for the languages used. |
Evaluation of Greek Word Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are the most popular input for many NLP tasks. |
| Approach: | They propose to use Greek word embeddings as an unsupervised learning tool . they use a Greek word analogy test set and a morphological test collection to evaluate word similarities . |
| Outcome: | The proposed model is able to create meaningful representations of Greek words . the proposed model can be adapted to Greek language and polysemy . |
The Word Analogy Testing Caveat (N18-2)
Copied to clipboard
| Challenge: | a number of word analogy tests are used to evaluate word embeddings . word embeds are used as a proxy for semantics and syntax à la Harris . |
| Approach: | They propose to use word embeddings as a proxy for distributional similarity . they propose to apply a transfer learning approach to word embeds to improve performance . |
| Outcome: | The proposed method improves performance across a wide range of NLP tasks. |
On the Correlation of Word Embedding Evaluation Metrics (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are geometrical representations of word paradigmatics and syntagmatics. |
| Approach: | They propose to investigate evaluation metrics on various datasets to find correlations . they propose a fast solution to select the best word embeddings among many others . |
| Outcome: | The proposed method could be used to select the best word embeddings among many others. |
Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? (2020.lrec-1)
Copied to clipboard
| Challenge: | In recent years, vectors derived from neural network training have replaced count-based distributional semantic models as a de facto standard for word representation in NLP. |
| Approach: | They propose to evaluate count models and word embeddings on thematic fit estimation by taking into account a larger number of parameters and verb roles and introducing dependency-based embedders in the comparison. |
| Outcome: | The proposed model outperforms count models and word embeddings in thematic fit estimation tasks while introducing dependency-based embedders. |
Towards a Gold Standard for Evaluating Danish Word Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing word embedding models resemble semantic similarity solely by distribution, but there seems to be a need for future judgments to measure similarity in full context and along more than a single spectrum. |
| Approach: | They propose a model-agnostic similarity goal standard for evaluating Danish word embeddings based on human judgments made by 42 native speakers of Danish. |
| Outcome: | The goal standard is applied to evaluate Danish word embeddings on 42 native speakers of Danish. |
How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions (P19-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings (CLEs) are used for downstream NLP tasks . CLEs are based on bilingual lexicon induction (BLI) evaluations vary greatly, hindering ability to interpret performance and properties of different CLE models. |
| Approach: | They evaluate CLE models for a large number of language pairs on bilingual lexicon induction and three downstream tasks. |
| Outcome: | The proposed model performance is based on supervised and unsupervised models on bilingual lexicon induction and three downstream tasks. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |