| Challenge: | Text embeddings are used in many NLP tasks, including document clustering, semantic search, question answering, and classification. |
| Approach: | They introduce the Polish Massive Text Embedding Benchmark (PL-MTEB) it is a comprehensive benchmark for text embeddings in the Polish language. |
| Outcome: | The proposed model is based on 30 different NLP tasks in the Polish language. |
Similar Papers
MTEB: Massive Text Embedding Benchmark (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing text embeddings are evaluated on a small set of datasets, not covering their possible applications to other tasks. |
| Approach: | They propose a benchmarking framework that evaluates 8 embedding tasks covering 58 datasets and 112 languages. |
| Outcome: | The proposed model is the most comprehensive benchmark of text embeddings to date. |
FaMTEB: Massive Text Embedding Benchmark in Persian Language (2025.findings-emnlp)
Copied to clipboard
Erfan Zinvandi, Morteza Alikhani, Mehran Sarmadi, Zahra Pourbahman, Sepehr Arvin, Reza Kazemi, Arash Amini
| Challenge: | a comprehensive benchmark for Persian text embeddings is built upon the Massive Text Embedding Benchmark (MTEB) 63 datasets are included in the benchmark, including a novel task of summary retrieval. |
| Approach: | They propose a benchmark for Persian (Farsi) text embeddings built upon the Massive Text Embedding Benchmark. |
| Outcome: | The proposed framework includes 63 datasets spanning seven different tasks . the evaluation datasets were rigorously evaluated by humans and automated systems . |
MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in embedding resources have led to a lack of representation of the Dutch language in multilingual resources. |
| Approach: | They introduce Massive Text Embedding Benchmark for Dutch (MTEB-NL) which includes existing Dutch datasets and newly created ones, covering a wide range of tasks. |
| Outcome: | The proposed models demonstrate strong performance across multiple tasks. |
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation (2026.acl-long)
Copied to clipboard
| Challenge: | Slovak embeddings are core infrastructure for semantic search, retrieval-augmented generation (RAG), clustering, and classification. |
| Approach: | They propose a MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language . they use 31 datasets across 7 task types to evaluate the performance of the models . |
| Outcome: | The proposed model achieves competitive performance with proprietary APIs while remaining locally deployable for RAG . the model is based on 31 datasets across 7 task types and is 4 the depth of existing benchmark for Slovak . |
VN-MTEB: Vietnamese Massive Text Embedding Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | a lack of large-scale test datasets makes it difficult to evaluate AI models before deploying them in real-world projects. |
| Approach: | They propose a Vietnamese benchmark for embedding models that leverages large language models and embeddable models to translate and filter samples from the Massive Multilingual Text Embedding Benchmark. |
| Outcome: | The proposed benchmark outperforms existing models in Vietnamese and English tasks with 41 datasets. |
TR-MTEB: A Comprehensive Benchmark and Embedding Model Suite for Turkish Sentence Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | TR-MTEB is the first large-scale, task-diverse benchmark for sentence embedding models for Turkish. |
| Approach: | a new benchmark evaluates sentence embedding models for Turkish . TR-MTEB covers six core tasks and 26 high-quality datasets . |
| Outcome: | The TR-MTEB benchmark covers six core tasks and includes 26 high-quality datasets . the models achieve competitive performance across most tasks and significantly improve on baseline models. |
The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design (2025.naacl-long)
Copied to clipboard
| Challenge: | Embedding models are used in tasks such as information retrieval and semantic textual similarity. |
| Approach: | They propose a new Russian-focused embedding model called ru-en-RoSBERTa and a benchmark for Russian language . they propose to use the roMTEB benchmark to assess Russian and multilingual models . |
| Outcome: | The proposed model achieves results that are on par with state-of-the-art models in Russian. |
PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of sentence embedding models rely on static tests like the Massive Text Embedding Benchmark (MTEB) repeated tuning on a fixed suite can inflate reported performance and obscure real-world robustness. |
| Approach: | They propose a dynamic protocol that generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. |
| Outcome: | The proposed protocol generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. |
Evaluation of Transfer Learning for Polish with a Text-to-Text Model (2022.lrec-1)
Copied to clipboard
Aleksandra Chrabrowa, Łukasz Dragan, Karol Grzegorczyk, Dariusz Kajtoch, Mikołaj Koszowski, Robert Mroczkowski, Piotr Rybak
| Challenge: | Recent years have brought significant progress in natural language understanding (NLU) and natural language generation (NLG). |
| Approach: | They propose a benchmark for assessing the quality of text-to-text models for Polish . they evaluate the performance of plT5, mT5, Polish BART, and Polish GPT-2 . |
| Outcome: | The proposed model can be fine-tuned on various NLP tasks with a single training objective. |
AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages (2026.eacl-long)
Copied to clipboard
| Challenge: | Text embeddings are an essential building component of several NLP tasks. |
| Approach: | They propose a regional expansion of MTEB covering 59 languages, 14 tasks, and 38 datasets, including six newly added datasets. |
| Outcome: | The proposed model outperforms baselines and mE5 in hate speech detection, intent detection, and emotion classification tasks. |