An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have been shown to improve the performance of neural networks across a wide variety of tasks. |
| Approach: | They propose two new metrics to understand the downstream reliability of word embeddings. |
| Outcome: | The proposed model can improve performance with slight changes to the training data, but it can also fail with multiple neural network architectures. |
Similar Papers
Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures (2022.lrec-1)
Copied to clipboard
| Challenge: | a paper aims to evaluate embedding similarity, stability and reliability in low-resource settings . it uses corpus similarity measures before training to predict properties of embeddables . |
| Approach: | They use corpus similarity measures before training to predict properties of embeddings . they then apply the same measures to low-resource settings by modelling reliability . authors hope to use this method to evaluate low-source languages with limited corpus size . |
| Outcome: | The paper shows that it is possible to predict downstream embedding similarity using upstream corpus similarity measures . the main finding is that the measures remain robust on small amounts of training data . |
A Deeper Look into Dependency-Based Word Embeddings (N18-4)
Copied to clipboard
| Challenge: | Word embeddings trained with dependency contexts excel at different tasks, and enhanced dependencies often improve performance. |
| Approach: | They propose to use dependency-based word embeddings to capture semantic similarity rather than relatedness. |
| Outcome: | The results show that word embeddings trained with Universal and Stanford dependencies excel at different tasks and that enhanced dependencies often improve performance. |
Factors Influencing the Surprising Instability of Word Embeddings (N18-1)
Copied to clipboard
| Challenge: | Word embeddings are low-dimensional, dense vector representations that capture semantic properties of words. |
| Approach: | They examine the stability of word embeddings by examining their properties and analyzing their effects on downstream tasks. |
| Outcome: | The results show that even high frequency words exhibit substantial instability, which can have implications for downstream tasks. |
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (N18-2)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks where large-scale parallel corpora cannot be obtained. |
| Approach: | They perform five sets of experiments to analyze when pre-trained word embeddings can be useful in NMT tasks. |
| Outcome: | The embeddings provide gains of up to 20 BLEU points in the most favorable setting. |
On the Correlation of Word Embedding Evaluation Metrics (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are geometrical representations of word paradigmatics and syntagmatics. |
| Approach: | They propose to investigate evaluation metrics on various datasets to find correlations . they propose a fast solution to select the best word embeddings among many others . |
| Outcome: | The proposed method could be used to select the best word embeddings among many others. |
Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? (2020.lrec-1)
Copied to clipboard
| Challenge: | In recent years, vectors derived from neural network training have replaced count-based distributional semantic models as a de facto standard for word representation in NLP. |
| Approach: | They propose to evaluate count models and word embeddings on thematic fit estimation by taking into account a larger number of parameters and verb roles and introducing dependency-based embedders in the comparison. |
| Outcome: | The proposed model outperforms count models and word embeddings in thematic fit estimation tasks while introducing dependency-based embedders. |
Analyzing the Surprising Variability in Word Embedding Stability Across Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are powerful representations that form the foundation of many natural language processing architectures. |
| Approach: | They explore word embedding stability in a wide range of languages to gain insight into their stability. |
| Outcome: | The proposed results provide insights into word embedding stability in English and other languages. |
Exploring the Value of Personalized Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | a subset of words belonging to specific psycholinguistic categories vary more in their representations across users . combining generic and personalized word embeddings yields the best performance . |
| Approach: | They propose personalized word embeddings and compare their performance to generic ones . they show that personalized word representations can be leveraged for improved performance . |
| Outcome: | The proposed model can be used for authorship attribution. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Evaluating Performance of Pre-trained Word Embeddings on Assamese, a Low-resource Language (2024.lrec-main)
Copied to clipboard
| Challenge: | Word embeddings are not explored in high-resource languages such as Assamese, where resources are limited. |
| Approach: | They propose to use assamese pre-trained word embeddings for sequence labeling tasks such as Parts-of-speech and Named Entity Recognition to evaluate their performance. |
| Outcome: | The proposed embeddings outperform the existing methods on Parts-of-speech and Named Entity Recognition tasks. |