SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects (2024.eacl-long)
Copied to clipboard
David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, En-Shiun Lee
| Challenge: | despite progress in building multilingual language models evaluation is limited to a few languages with available datasets . despite this, we create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Approach: | They create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). |
| Outcome: | The proposed dataset addresses the lack of evaluation dataset for Natural Language Understanding (NLU) for many languages, it is the first publicly available evaluation dataset. |
Similar Papers
Taxi1500: A Dataset for Multilingual Text Classification in 1500 Languages (2025.naacl-short)
Copied to clipboard
| Challenge: | a large-scale text classification dataset encompassing 1504 languages is needed to address this gap . low-resource languages are often overlooked due to the scarcity of evaluation datasets. |
| Approach: | They propose to use translations of the Bible to construct a large-scale text classification dataset that covers 1504 languages and annotate them using crowdsourcing. |
| Outcome: | The proposed dataset covers 1504 languages and is available to the public. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2022.tacl-1)
Copied to clipboard
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, Angela Fan
| Challenge: | a lack of good evaluation benchmarks hinders progress in low-resource and multilingual machine translation . despite advances in translation quality for a handful of languages, many low-source languages are not even supported by most popular translation engines. |
| Approach: | They propose a high-quality evaluation benchmark for machine translation using 3001 sentences from Wikipedia . they aim to improve evaluation of models on long tail of low-resource languages . |
| Outcome: | The proposed evaluation benchmarks are based on 3001 sentences extracted from Wikipedia . the results show that the models can be used to evaluate multilingual systems . |
Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan (2021.findings-acl)
Copied to clipboard
Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, Marta Villegas
| Challenge: | Multilingual language models have been a crucial breakthrough for under-resourced languages . however, the superiority of language-specific models has already been proven for underresourced ones . |
| Approach: | They propose to build a monolingual monolingual model that is comparable to state-of-the-art large multilingual models. |
| Outcome: | The proposed model consistently outperforms state-of-the-art models across tasks and settings. |
MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation datasets lack cross-lingual alignment, leaving assessments of multilingual capabilities fragmented in both language and skill coverage. |
| Approach: | They propose to use multilingual consistency as a complementary metric to assess performance bottlenecks and guide model improvement. |
| Outcome: | The proposed model lacks cross-lingual alignment and language coverage gaps between state-of-the-art models. |
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, Jörg Tiedemann
| Challenge: | Existing evaluation frameworks focus on English and a handful of high-resource languages, thereby overlooking the realistic performance of large language models in multilingual and lower-resourced scenarios. |
| Approach: | They propose a unified and lightweight framework that integrates 27 benchmarks under a standard ISO 639-3 language identifier system to enable seamless incorporation of new benchmarks. |
| Outcome: | The proposed framework integrates 27 benchmarks under a standard ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks. |
Wiki-40B: Multilingual Language Model Dataset (2020.lrec-1)
Copied to clipboard
| Challenge: | We propose a new multilingual language model benchmark that is composed of 40+ languages spanning several scripts and linguistic families. |
| Approach: | They propose a multilingual language model benchmark composed of 40+ languages . they train monolingual causal language models using a state-of-the-art model . |
| Outcome: | The proposed model is composed of 40+ languages spanning several scripts and linguistic families. |
XNLI: Evaluating Cross-lingual Sentence Representations (D18-1)
Copied to clipboard
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov
| Challenge: | State-of-the-art natural language processing systems rely on annotated data to learn competent models. |
| Approach: | They extend the development and test sets of the Multi-Genre Natural Language Inference Corpus to 14 languages, including Swahili and Urdu. |
| Outcome: | The proposed evaluation set extends the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 14 languages including low-resource languages such as Swahili and Urdu. |
Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have examined the quality of labeled data in non-English languages. |
| Approach: | They annotate how datasets are created, input text and label sources, tools used to build them and what they study. |
| Outcome: | The results show that language-proficient NLP researchers' estimated availability correlates with dataset availability. |
EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios (2026.acl-long)
Copied to clipboard
Bin Xu, Yu Bai, Huashan Sun, Yiguan Lin, Siming Liu, Xinyue Liang, Yaolin Li, Zhuangzhi Dong, Jingren Zhang, Yufan Deng, Xinyu Zou, Yang Gao, Heyan Huang
| Challenge: | Existing benchmarks that focus on knowledge-intensive tasks do not reflect diverse educational scenarios. |
| Approach: | They propose a benchmark that incorporates 9 major scenarios and 4,000 educational contexts. |
| Outcome: | The proposed model performs comparable to state-of-the-art large models on the test set. |