PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of sentence embedding models rely on static tests like the Massive Text Embedding Benchmark (MTEB) repeated tuning on a fixed suite can inflate reported performance and obscure real-world robustness. |
| Approach: | They propose a dynamic protocol that generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. |
| Outcome: | The proposed protocol generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. |
Similar Papers
MTEB: Massive Text Embedding Benchmark (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing text embeddings are evaluated on a small set of datasets, not covering their possible applications to other tasks. |
| Approach: | They propose a benchmarking framework that evaluates 8 embedding tasks covering 58 datasets and 112 languages. |
| Outcome: | The proposed model is the most comprehensive benchmark of text embeddings to date. |
RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing PLMs suffer from poor robustness in adversarial scenarios, despite their success with unseen samples. |
| Approach: | They propose a self-supervised sentence embedding framework that enhances generalization and robustness in various text representation tasks and against diverse adversarial attacks. |
| Outcome: | The proposed framework improves generalization and robustness in various representation tasks and against diverse adversarial attacks. |
TR-MTEB: A Comprehensive Benchmark and Embedding Model Suite for Turkish Sentence Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | TR-MTEB is the first large-scale, task-diverse benchmark for sentence embedding models for Turkish. |
| Approach: | a new benchmark evaluates sentence embedding models for Turkish . TR-MTEB covers six core tasks and 26 high-quality datasets . |
| Outcome: | The TR-MTEB benchmark covers six core tasks and includes 26 high-quality datasets . the models achieve competitive performance across most tasks and significantly improve on baseline models. |
PL-MTEB: Polish Massive Text Embedding Benchmark (2026.findings-acl)
Copied to clipboard
| Challenge: | Text embeddings are used in many NLP tasks, including document clustering, semantic search, question answering, and classification. |
| Approach: | They introduce the Polish Massive Text Embedding Benchmark (PL-MTEB) it is a comprehensive benchmark for text embeddings in the Polish language. |
| Outcome: | The proposed model is based on 30 different NLP tasks in the Polish language. |
Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework (2026.eacl-long)
Copied to clipboard
Clea Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Taik, Elliot Creager, Golnoosh Farnadi
| Challenge: | Existing studies have studied prompt sensitivity by altering formatting or generating paraphrases with automated techniques. |
| Approach: | They propose a framework for generating controlled paraphrases grounded in user behaviors . they leverage linguistically informed rules and enforce quality through checks on instruction adherence . |
| Outcome: | The proposed framework is able to detect weaknesses in large language models . it leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism. |
A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens (2025.acl-long)
Copied to clipboard
| Challenge: | et al., 2023) show that text embeddings from large language models can be aligned with key tokens in input text. |
| Approach: | They propose a sparse retrieval method based on aligned tokens for large language models . they show that this phenomenon is universal and is not affected by model architecture . |
| Outcome: | The proposed method can achieve 80% of the dense retrieval effect of the same model while reducing the computation significantly. |
FinMTEB: Finance Massive Text Embedding Benchmark (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text embedding benchmarks for financial domains are inadequately addressing the nuanced requirements of specialized domains like finance. |
| Approach: | They propose a finance-adapted embedding model that outperforms general-purpose models . they also introduce a new model, Fin-E5, which is also open-sourced . |
| Outcome: | The proposed framework outperforms general-purpose models on financial embedding tasks. |
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods rely on fixed prompt templates or involve modifications to the model architecture, compromising its generative capabilities. |
| Approach: | They propose a sentence-level direct preference optimization approach that boosts the sentence representations while preserving the generative ability of LLMs. |
| Outcome: | The proposed method improves representations of semantically meaningful vectors without sacrificing generation capability. |
GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings (2025.findings-naacl)
Copied to clipboard
| Challenge: | Training-free embedding methods focus on optimizing embeddable prompts . previous methods have overlooked the benefits of utilizing generative abilities of LLMs - GenEOL . |
| Approach: | They propose a method that leverages pretrained large language models to embed text . they propose generating diverse transformations of a sentence that preserve its meaning . |
| Outcome: | The proposed method outperforms existing training-free embedding methods by 2.85 points on the sentence semantic text similarity (STS) benchmark. |
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on prompt engineering to extract sentence embeddings from large language models (LLMs) but these models are mostly decoder-only and the earlier tokens in the sentence cannot attend to the latter, resulting in biased encoding of sentence information and cascading effects on the final decoded token. |
| Approach: | They propose a plug-and-play and training-free technique that prepends each layer’s decoded sentence embedding to the beginning of the sentence in the next layer’ s input. |
| Outcome: | The proposed technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs while incurring negligible additional inference cost. |