Do We Really Need All Those Dimensions? An Intrinsic Evaluation Framework for Compressed Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for compressed text embeddings are either expensive or too simplistic. |
| Approach: | They propose a task-agnostic intrinsic evaluation framework that provides a reliable proxy for downstream performance. |
| Outcome: | The proposed framework provides a reliable proxy for downstream performance. |
Similar Papers
What’s in Your Embedding, And How It Predicts Task Performance (C18-1)
Copied to clipboard
| Challenge: | Attempts to find a single technique for general-purpose intrinsic evaluation of word embeddings have so far not been successful. |
| Approach: | They propose a method that quantifies interpretable characteristics of word vector neighborhoods and shows how they correlate with performance on 14 extrinsic and intrinsic task datasets. |
| Outcome: | The proposed approach enables multi-faceted evaluation, parameter search, and generally – a more principled, hypothesis-driven approach to development of distributional semantic representations. |
Benchmarking Meta-embeddings: What Works and What Does Not (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to build meta-embeddings have been evaluated using a variety of methods and datasets, which makes it difficult to draw meaningful conclusions regarding the merits of each approach. |
| Approach: | They propose a unified framework for a fair and objective meta-embedding evaluation using intrinsic and extrinsic tasks. |
| Outcome: | The proposed framework outperforms existing methods on intrinsic and extrinsic evaluation benchmarks and outperformed existing methods. |
Extreme Model Compression for On-device Natural Language Understanding (2020.coling-industry)
Copied to clipboard
| Challenge: | Xu and Sarikaya et al., 2014) perform word-embedding compression with NLU task learning . their approach achieves a compression rate of 97.4% with less than 3.7% degradation in predictive performance. |
| Approach: | They propose a task-aware, end-to-end compression approach that performs word-embedding compression with NLU task learning. |
| Outcome: | The proposed approach outperforms baselines and achieves 97.4% compression rate with less than 3.7% degradation in predictive performance. |
Evaluation of Domain-specific Word Embeddings using Knowledge Resources (L18-1)
Copied to clipboard
| Challenge: | Existing word embeddings capture a range of semantic relations relevant to the interpretation of lexical items, but domain-specific terms are difficult to evaluate because of a lack of statistical clues in the underlying corpus. |
| Approach: | They conduct intrinsic and extrinsic evaluations of both general and domain-specific embeddings and adapt embeddment enhancement methods to provide vector representations for infrequent and unseen terms. |
| Outcome: | The proposed model improves both in the intrinsic evaluation and extrinsic evaluation of the embedding models and their representations of infrequent and unseen terms. |
Just Rank: Rethinking Evaluation with Word and Sentence Similarities (2022.acl-long)
Copied to clipboard
| Challenge: | Word and sentence similarity tasks are the de facto evaluation method for embeddings. |
| Approach: | They propose a new intrinsic evaluation method called EvalRank which shows a much stronger correlation with downstream tasks. |
| Outcome: | The proposed method shows a much stronger correlation with downstream tasks and is released for future benchmarking purposes. |
FLARE: Task-Agnostic Embedding Model Evaluation via Normalizing Flows (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods based on kernel estimators or Gaussian mixtures fail to model high-dimensional distributions effectively, resulting in unstable rankings. |
| Approach: | They propose a method which uses normalizing flows to estimate information sufficiency in high-dimensional spaces by learning invertible transformations. |
| Outcome: | Experiments on 11 datasets show that FLARE achieves a strong Spearman’s (up to 0.90) with supervised benchmarks, remaining robust even for high-dimensional embeddings (d 3,584). |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025.emnlp-main)
Copied to clipboard
| Challenge: | Embedding-based similarity metrics can be influenced by content dimensions and spurious attributes like the text’s source or language. |
| Approach: | They propose a debiasing algorithm that removes observed confounders from encoder representations and removes them from the encoder. |
| Outcome: | The proposed method improves on out-of-distribution benchmarks and on benchmarks, but performance is not affected. |
Tensorized Embedding Layers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using the Tensor Train decomposition, embeddings layers occupy large portion of model weights, preventing their deployment in limited resource settings. |
| Approach: | They propose a method for parameterizing embedding layers based on the Tensor Train decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. |
| Outcome: | The proposed method can be plugged into any model and trained end-to-end. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |