SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation (2026.acl-long)
Copied to clipboard
| Challenge: | Slovak embeddings are core infrastructure for semantic search, retrieval-augmented generation (RAG), clustering, and classification. |
| Approach: | They propose a MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language . they use 31 datasets across 7 task types to evaluate the performance of the models . |
| Outcome: | The proposed model achieves competitive performance with proprietary APIs while remaining locally deployable for RAG . the model is based on 31 datasets across 7 task types and is 4 the depth of existing benchmark for Slovak . |
Similar Papers
MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in embedding resources have led to a lack of representation of the Dutch language in multilingual resources. |
| Approach: | They introduce Massive Text Embedding Benchmark for Dutch (MTEB-NL) which includes existing Dutch datasets and newly created ones, covering a wide range of tasks. |
| Outcome: | The proposed models demonstrate strong performance across multiple tasks. |
PL-MTEB: Polish Massive Text Embedding Benchmark (2026.findings-acl)
Copied to clipboard
| Challenge: | Text embeddings are used in many NLP tasks, including document clustering, semantic search, question answering, and classification. |
| Approach: | They introduce the Polish Massive Text Embedding Benchmark (PL-MTEB) it is a comprehensive benchmark for text embeddings in the Polish language. |
| Outcome: | The proposed model is based on 30 different NLP tasks in the Polish language. |
MTEB: Massive Text Embedding Benchmark (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing text embeddings are evaluated on a small set of datasets, not covering their possible applications to other tasks. |
| Approach: | They propose a benchmarking framework that evaluates 8 embedding tasks covering 58 datasets and 112 languages. |
| Outcome: | The proposed model is the most comprehensive benchmark of text embeddings to date. |
FaMTEB: Massive Text Embedding Benchmark in Persian Language (2025.findings-emnlp)
Copied to clipboard
Erfan Zinvandi, Morteza Alikhani, Mehran Sarmadi, Zahra Pourbahman, Sepehr Arvin, Reza Kazemi, Arash Amini
| Challenge: | a comprehensive benchmark for Persian text embeddings is built upon the Massive Text Embedding Benchmark (MTEB) 63 datasets are included in the benchmark, including a novel task of summary retrieval. |
| Approach: | They propose a benchmark for Persian (Farsi) text embeddings built upon the Massive Text Embedding Benchmark. |
| Outcome: | The proposed framework includes 63 datasets spanning seven different tasks . the evaluation datasets were rigorously evaluated by humans and automated systems . |
TR-MTEB: A Comprehensive Benchmark and Embedding Model Suite for Turkish Sentence Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | TR-MTEB is the first large-scale, task-diverse benchmark for sentence embedding models for Turkish. |
| Approach: | a new benchmark evaluates sentence embedding models for Turkish . TR-MTEB covers six core tasks and 26 high-quality datasets . |
| Outcome: | The TR-MTEB benchmark covers six core tasks and includes 26 high-quality datasets . the models achieve competitive performance across most tasks and significantly improve on baseline models. |
The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design (2025.naacl-long)
Copied to clipboard
| Challenge: | Embedding models are used in tasks such as information retrieval and semantic textual similarity. |
| Approach: | They propose a new Russian-focused embedding model called ru-en-RoSBERTa and a benchmark for Russian language . they propose to use the roMTEB benchmark to assess Russian and multilingual models . |
| Outcome: | The proposed model achieves results that are on par with state-of-the-art models in Russian. |
FinMTEB: Finance Massive Text Embedding Benchmark (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing text embedding benchmarks for financial domains are inadequately addressing the nuanced requirements of specialized domains like finance. |
| Approach: | They propose a finance-adapted embedding model that outperforms general-purpose models . they also introduce a new model, Fin-E5, which is also open-sourced . |
| Outcome: | The proposed framework outperforms general-purpose models on financial embedding tasks. |
VN-MTEB: Vietnamese Massive Text Embedding Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | a lack of large-scale test datasets makes it difficult to evaluate AI models before deploying them in real-world projects. |
| Approach: | They propose a Vietnamese benchmark for embedding models that leverages large language models and embeddable models to translate and filter samples from the Massive Multilingual Text Embedding Benchmark. |
| Outcome: | The proposed benchmark outperforms existing models in Vietnamese and English tasks with 41 datasets. |
Improving Text Embeddings with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for obtaining text embeddings require complex training pipelines . authors leverage proprietary LLMs to generate diverse synthetic data for text embeds based on 93 languages . |
| Approach: | They propose a method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. |
| Outcome: | The proposed method achieves strong performance on competitive text embedding benchmarks without using any labeled data. |
skLEP: A Slovak General Language Understanding Benchmark (2025.findings-acl)
Copied to clipboard
Marek Suppa, Andrej Ridzik, Daniel Hládek, Tomáš Javůrek, Viktória Ondrejová, Kristína Sásiková, Martin Tamajka, Marian Simko
| Challenge: | skLEP is the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding models. |
| Approach: | They introduce a benchmark specifically designed for evaluating Slovak natural language understanding models. |
| Outcome: | The proposed benchmark covers nine tasks that span token-level, sentence-pair, document-level tasks. |