Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on text embeddings focus less on how information is encoded. |
| Approach: | They find that truncating embedding dimensions causes an increase in performance when removed. |
| Outcome: | The proposed method improves performance across 6 state-of-the-art text encoders and 26 downstream tasks. |
Similar Papers
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025.findings-acl)
Copied to clipboard
| Challenge: | Prompt-based text embedding models generate task-specific embeddables but have thousands of dimensions . dimensionality reductions for embedded text can result in performance degradations of only the first 25% of the dimensions resulting in a very small degradation . |
| Approach: | They investigate how post-hoc dimensionality reduction affects performance of various tasks . they find that embeddings for classification and clustering exhibit lower intrinsic dimensionalities . |
| Outcome: | The proposed model generates task-specific embeddings upon receiving tailored prompts, but has thousands of dimensions and high storage costs. |
The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025.emnlp-main)
Copied to clipboard
| Challenge: | Embedding-based similarity metrics can be influenced by content dimensions and spurious attributes like the text’s source or language. |
| Approach: | They propose a debiasing algorithm that removes observed confounders from encoder representations and removes them from the encoder. |
| Outcome: | The proposed method improves on out-of-distribution benchmarks and on benchmarks, but performance is not affected. |
Understanding the Influence of Synthetic Data for Text Embedders (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in general purpose text embedders have been driven by training on synthetic training data. |
| Approach: | They propose to use GPT-4 to produce high quality synthetic data that expands existing training datasets for embeddings to new tasks. |
| Outcome: | The proposed dataset is high quality and leads to consistent improvements in performance. |
Length-Induced Embedding Collapse in PLM-based Models (2025.acl-long)
Copied to clipboard
| Challenge: | In text embeddings from PLMs are essential for many NLP applications, but performance degrades on longer texts. |
| Approach: | They propose a method which mitigates the phenomenon of Length Collapse . they propose TempScale to ensure more consistent embeddings across different text lengths . |
| Outcome: | The proposed method improves performance on MTEB and LongEmbed by 0.94% on short and 1.10% on long texts. |
On the Dimensionality of Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work focuses on improving the quality of sentence embeddings, but the exploration of sentence dimension is limited. |
| Approach: | They propose a two-step training method where the encoder and pooler are optimized separately to mitigate the overall performance loss in low-dimension scenarios. |
| Outcome: | The proposed method significantly improves the performance of low-dimensional sentence embeddings on seven STS tasks and seven sentence classification tasks. |
Statistical Depth for Ranking and Characterizing Transformer-Based Text Embeddings (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generalized transformer-based text embedding models have produced state of the art performance results on a variety of tasks such as natural language inference (NLI) |
| Approach: | They propose a statistical depth to measure distributions of transformer-based text embeddings and an associated rank sum test to characterize distributions in synthetic and human-generated corpora. |
| Outcome: | The proposed method improves performance over baseline methods on six text classification tasks. |
EmbedTextNet: Dimension Reduction with Weighted Reconstruction and Correlation Losses for Efficient Text Embedding (2023.findings-acl)
Copied to clipboard
| Challenge: | EmbedTextNet is a light add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure. |
| Approach: | They propose an add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure. |
| Outcome: | The proposed network can be appended to an arbitrary language model to generate a compact embedding without any changes in its architecture or training procedure. |
Text Embeddings Reveal (Almost) As Much As Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a vector database of dense text embeddings stores only the text data, not the original text . a multi-step method that iteratively corrects and re-embeds text can recover 92% of 32-token text inputs exactly. |
| Approach: | They propose a method that iteratively corrects and re-embeds text to recover 92% of 32-token text inputs exactly. |
| Outcome: | The proposed method recovers 92% of 32-token text inputs exactly. |
Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality (2021.naacl-main)
Copied to clipboard
| Challenge: | In human-level NLP tasks, the number of observations is often smaller than the standard 768+ hidden state sizes of each layer within transformer-based language models. |
| Approach: | They propose to use dimension reduction methods to fine-tune large models with limited data and to use pre-trained dimension reduction regimes to improve model performance. |
| Outcome: | The proposed model outperforms other models in human-level NLP tasks with a pre-trained dimension reduction regime. |
Do We Really Need All Those Dimensions? An Intrinsic Evaluation Framework for Compressed Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for compressed text embeddings are either expensive or too simplistic. |
| Approach: | They propose a task-agnostic intrinsic evaluation framework that provides a reliable proxy for downstream performance. |
| Outcome: | The proposed framework provides a reliable proxy for downstream performance. |