| Challenge: | Existing work has shown that word embeddings are distributed in a narrow cone and that centering and projection can improve the accuracy of pre-trained word embeds without requiring additional training data. |
| Approach: | They propose to remove the top principal components from pre-trained word embeddings and center and project them onto principal component vectors to reinstate isotropy in the embeddable space. |
| Outcome: | The proposed method is equivalent to applying a linear autoencoder to minimize the squared L2 reconstruction error. |
Similar Papers
Better Word Embeddings by Disentangling Contextual n-Gram Information (N19-1)
Copied to clipboard
| Challenge: | Pre-trained word vectors are ubiquitous in Natural Language Processing applications. |
| Approach: | They show that word embeddings with bigram and trigram embedds improve unigram embeds . they claim this removes contextual information from unigrammes, resulting in better unigraph embedders . |
| Outcome: | The proposed model outperforms competing models on a wide variety of tasks. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (N18-2)
Copied to clipboard
| Challenge: | Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks where large-scale parallel corpora cannot be obtained. |
| Approach: | They perform five sets of experiments to analyze when pre-trained word embeddings can be useful in NMT tasks. |
| Outcome: | The embeddings provide gains of up to 20 BLEU points in the most favorable setting. |
Leveraging the Structure of Pre-trained Embeddings to Minimize Annotation Effort (2024.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for text classification are based on fine-tuning the representations computed by large language models. |
| Approach: | They propose to exploit structural properties of pre-trained embeddings to spread information . they use a semisupervised strategy to train models with minimal annotation effort . |
| Outcome: | The proposed method outperforms self-training and random walk labels on different datasets. |
More Embeddings, Better Sequence Labelers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work suggests contextual embeddings improve sequence labeling accuracy . but, there is no definite conclusion on whether concatenating different kinds of embeddables is effective . |
| Approach: | They propose a family of contextual embeddings that improves sequence labeling accuracy . they conduct extensive experiments on 3 tasks over 18 datasets and 8 languages . |
| Outcome: | The proposed family of contextual embeddings improves the accuracy of sequence labelers over non-contextual embedders. |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
Tiny Word Embeddings Using Globally Informed Reconstruction (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for word embedding reconstruction use only local information of subwords and pre-trained word embeds. |
| Approach: | They propose a global loss function that uses words other than the target word to improve word embedding reconstruction by a factor of 200. |
| Outcome: | The proposed method reduces the model size of pre-trained word embeddings by a factor of 200 while preserving its quality. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have used probing tasks to verify the presence of linguistic properties in vector representations, but it is unclear whether they can be manipulated to indirectly steer them. |
| Approach: | They validate a geometric mapping technique to transform linguistic properties without tuning . they use a pre-trained multilingual autoencoder to transform three linguistic property . |
| Outcome: | The proposed method can be used without tuning of the pre-trained autoencoder . the results are validated in monolingual and cross-lingual settings . |
Learning Word Meta-Embeddings by Autoencoding (C18-1)
Copied to clipboard
| Challenge: | Existing word embeddings have shown superior performance in numerous Natural Language Processing (NLP) tasks, however, their performances vary significantly across different tasks. |
| Approach: | They propose to combine distributed word embeddings to produce more accurate and complete meta-embeddings of words. |
| Outcome: | The proposed meta-embeddings outperform the state-of-the-art in multiple tasks. |