| Challenge: | A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks. |
| Approach: | They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks. |
| Outcome: | The proposed method leads to state-of-the-art performance on a variety of tasks. |
Similar Papers
Embeddings in Natural Language Processing (2020.coling-tutorials)
Copied to clipboard
| Challenge: | Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts . |
| Approach: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors . |
| Outcome: | This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations . |
Static Word Embeddings for Sentence Semantic Representation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn fixed-length embeddings for sentence semantics require large computational cost, making it difficult to process billions of sentences cost-efficiently or deploy models on resource-constrained devices such as smartphones. |
| Approach: | They propose to extract word embeddings from a pre-trained Sentence Transformer and improve them with sentence-level principal component analysis followed by knowledge distillation or contrastive learning. |
| Outcome: | The proposed model outperforms existing models on sentence semantic tasks and surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. |
Learning Efficient Task-Specific Meta-Embeddings with Word Prisms (2020.coling-main)
Copied to clipboard
| Challenge: | Word embeddings possess different lexical properties depending on the notion of context defined at training time. |
| Approach: | They introduce a meta-embedding method that learns to combine source embeddings according to the task at hand. |
| Outcome: | The proposed method improves performance on six extrinsic evaluations over other methods. |
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2020.acl-main)
Copied to clipboard
| Challenge: | Existing word embeddings combine complementary strengths of their components to achieve unsupervised semantic similarity (STS). |
| Approach: | They propose to ensemble pre-trained sentence encoders into sentence meta-embeddings to achieve unsupervised Semantic Textual Similarity (STS) they adapt dimensionality reduction, generalized Canonical Correlation Analysis and cross-view auto-encoders to their work. |
| Outcome: | The proposed method achieves 3.7% to 6.4% Pearson’s r over single-source word embeddings on the STS Benchmark and on the StS12-STS16 datasets. |
Learning Word Meta-Embeddings by Autoencoding (C18-1)
Copied to clipboard
| Challenge: | Existing word embeddings have shown superior performance in numerous Natural Language Processing (NLP) tasks, however, their performances vary significantly across different tasks. |
| Approach: | They propose to combine distributed word embeddings to produce more accurate and complete meta-embeddings of words. |
| Outcome: | The proposed meta-embeddings outperform the state-of-the-art in multiple tasks. |
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? (2023.findings-eacl)
Copied to clipboard
| Challenge: | obtaining document embeddings at document level is challenging due to computational requirements and lack of appropriate data. |
| Approach: | They compare methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. |
| Outcome: | The proposed methods produce document-level representations from sentences in 8 languages . the results show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit. |
Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings (N18-2)
Copied to clipboard
| Challenge: | Existing methods for producing word embeddings have shown to produce accurate meta-embeddings from pre-trained source embeddables. |
| Approach: | They propose to use arithmetic mean of two distinct word embedding sets to produce an accurate meta-embedding. |
| Outcome: | The proposed method produces meta-embeddings comparable or better than more complex methods. |
Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve text classification performance of pre-trained models have been used to improve their performance. |
| Approach: | They propose a method for improving BERT's performance by using a label embedding technique while keeping almost the same computational cost. |
| Outcome: | The proposed method improves BERT's performance on six text classification benchmark datasets while keeping almost the same computational cost. |
Reusing Weights in Subword-Aware Neural Language Models (N18-1)
Copied to clipboard
| Challenge: | a statistical language model assigns a probability to a sequence of words . data sparsity is a major problem in building traditional n-gram language models . |
| Approach: | They propose several ways to reuse subword embeddings and other weights in subword-aware neural language models. |
| Outcome: | The proposed techniques do not benefit a competitive character-aware model . but they show significant reductions in model sizes and performance. |
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |