Word Embedding Evaluation in Downstream Tasks and Semantic Analogies (2020.lrec-1)
Copied to clipboard
| Challenge: | Language Models (LMs) are an oft studied area of natural language processing . Word Embeddings (WE) are vector space representations of a vocabulary . |
| Approach: | They evaluate Word Embeddings (WE) models for the Portuguese langauage . results show that a diverse corpus can often outperform a larger, less textually diverse corp. |
| Outcome: | The proposed models outperform a larger, less textually diverse corpus in two tasks . the evaluation shows that a diverse and comprehensive corpus outperformed a smaller, less diverse corp. |
Similar Papers
A Deeper Look into Dependency-Based Word Embeddings (N18-4)
Copied to clipboard
| Challenge: | Word embeddings trained with dependency contexts excel at different tasks, and enhanced dependencies often improve performance. |
| Approach: | They propose to use dependency-based word embeddings to capture semantic similarity rather than relatedness. |
| Outcome: | The results show that word embeddings trained with Universal and Stanford dependencies excel at different tasks and that enhanced dependencies often improve performance. |
Just Rank: Rethinking Evaluation with Word and Sentence Similarities (2022.acl-long)
Copied to clipboard
| Challenge: | Word and sentence similarity tasks are the de facto evaluation method for embeddings. |
| Approach: | They propose a new intrinsic evaluation method called EvalRank which shows a much stronger correlation with downstream tasks. |
| Outcome: | The proposed method shows a much stronger correlation with downstream tasks and is released for future benchmarking purposes. |
What’s in Your Embedding, And How It Predicts Task Performance (C18-1)
Copied to clipboard
| Challenge: | Attempts to find a single technique for general-purpose intrinsic evaluation of word embeddings have so far not been successful. |
| Approach: | They propose a method that quantifies interpretable characteristics of word vector neighborhoods and shows how they correlate with performance on 14 extrinsic and intrinsic task datasets. |
| Outcome: | The proposed approach enables multi-faceted evaluation, parameter search, and generally – a more principled, hypothesis-driven approach to development of distributional semantic representations. |
Is Language Modeling Enough? Evaluating Effective Embedding Combinations (2020.lrec-1)
Copied to clipboard
Rudolf Schneider, Tom Oberhauser, Paul Grundmann, Felix Alexander Gers, Alexander Loeser, Steffen Staab
| Challenge: | specialized embeddings are not available for tasks like entity linking or paragraph classification. |
| Approach: | They evaluate whether universal embeddings can be complemented by specialized embeddables. |
| Outcome: | The proposed embeddings outperform state-of-the-art embeddables without any fine-tuning. |
Finely Tuned, 2 Billion Token Based Word Embeddings for Portuguese (L18-1)
Copied to clipboard
| Challenge: | A distributional semantics model is instrumental to improve the performance of many applications and processing tasks for any language. |
| Approach: | They propose to develop an advanced distributional model for Portuguese with the largest vocabulary and best evaluation scores published so far. |
| Outcome: | The proposed model has the largest vocabulary and the best evaluation scores published so far. |
BioReddit: Word Embeddings for User-Generated Biomedical NLP (D19-62)
Copied to clipboard
| Challenge: | a corpus of medical-themed posts was scrapped from Reddit to train word embeddings on downstream tasks. |
| Approach: | They propose to train word embeddings from a corpus of medical forums from reddit scrapping posts from medical-themed subreddits. |
| Outcome: | The proposed system outperforms embeddings trained on general purpose data or on scientific papers when applied on user-generated content. |
Evaluation of Domain-specific Word Embeddings using Knowledge Resources (L18-1)
Copied to clipboard
| Challenge: | Existing word embeddings capture a range of semantic relations relevant to the interpretation of lexical items, but domain-specific terms are difficult to evaluate because of a lack of statistical clues in the underlying corpus. |
| Approach: | They conduct intrinsic and extrinsic evaluations of both general and domain-specific embeddings and adapt embeddment enhancement methods to provide vector representations for infrequent and unseen terms. |
| Outcome: | The proposed model improves both in the intrinsic evaluation and extrinsic evaluation of the embedding models and their representations of infrequent and unseen terms. |
How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions (P19-1)
Copied to clipboard
| Challenge: | Cross-lingual word embeddings (CLEs) are used for downstream NLP tasks . CLEs are based on bilingual lexicon induction (BLI) evaluations vary greatly, hindering ability to interpret performance and properties of different CLE models. |
| Approach: | They evaluate CLE models for a large number of language pairs on bilingual lexicon induction and three downstream tasks. |
| Outcome: | The proposed model performance is based on supervised and unsupervised models on bilingual lexicon induction and three downstream tasks. |
An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have been shown to improve the performance of neural networks across a wide variety of tasks. |
| Approach: | They propose two new metrics to understand the downstream reliability of word embeddings. |
| Outcome: | The proposed model can improve performance with slight changes to the training data, but it can also fail with multiple neural network architectures. |
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (P18-1)
Copied to clipboard
| Challenge: | a lack of understanding of the properties of sentence embeddings is limiting the use of the techniques. |
| Approach: | They propose 10 probing tasks designed to capture simple linguistic features of sentences . they use three different encoders to train embeddings in eight different ways . |
| Outcome: | The proposed tasks capture key linguistic features of sentences, but they are difficult to infer from them. |