The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

Challenge:	Existing benchmarks for text comprehension only cover 30 languages, but lack of labeled data is a major obstacle to building functional systems in most languages.
Approach:	They present a multiple-choice machine reading comprehension dataset spanning 122 languages . they use it to evaluate the capabilities of multilingual masked language models and large language models .
Outcome:	The proposed dataset enables the evaluation of text models in high-, medium- and low-resource languages.

Similar Papers

Overview

Challenge

Approach

Outcome

BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels (D19-1)

Copied to clipboard

Yimin Jing, Deyi Xiong, Zhen Yan

Multilinguality

Challenge:	Using BiPaR, we build monolingual, multilingual and cross-lingual MRC on novels.
Approach:	They propose a bilingual parallel novel-style machine reading comprehension dataset BiPaR . they collect 3,667 bilingual parallel paragraphs from Chinese and English novels .
Outcome:	The proposed dataset supports multilingual and cross-lingual reading comprehension.

From Multiple-Choice to Extractive QA: A Case Study for English and Arabic (2025.coling-main)

Copied to clipboard

Teresa Lynn, Malik H. Altakrori, Samar M. Magdy, Rocktim Jyoti Das, Chenyang Lyu, Mohamed Nasr, Younes Samih, Kirill Chirkunov, Alham Fikri Aji, Preslav Nakov, Shantanu Godbole, Salim Roukos, Radu Florian, Nizar Habash

Question Answering

Challenge:	Recent years have brought about very fast developments in Natural Language Processing (NLP), but many other languages are overlooked due to limited resources.
Approach:	They propose to repurpose a multilingual BELEBELE dataset for a task of extractive QA in the style of machine reading comprehension.
Outcome:	The proposed approach could be used to extract QA in the style of machine reading comprehension.

BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)

Copied to clipboard

Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, Fei Yuan

Language Models

Challenge:	Existing multilingual benchmarks focus primarily on language understanding tasks.
Approach:	They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages.
Outcome:	Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve.

SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects (2024.eacl-long)

Copied to clipboard

David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, En-Shiun Lee

Information Retrieval

Challenge:	despite progress in building multilingual language models evaluation is limited to a few languages with available datasets . despite this, we create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU).
Approach:	They create a large-scale open-sourced benchmark dataset for topic classification in 205 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU).
Outcome:	The proposed dataset addresses the lack of evaluation dataset for Natural Language Understanding (NLU) for many languages, it is the first publicly available evaluation dataset.

ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models (2026.acl-long)

Copied to clipboard

Emily Chang, Niyati Bafna

Language Models

Challenge:	Existing benchmarks for large language models (LLMs) are restricted to high- or mid-resource languages, and evaluate performance on higher-order tasks in reasoning and generation.
Approach:	They propose a multilingual benchmarking tool to evaluate lexical comprehension and generation abilities of large language models.
Outcome:	The proposed benchmarks cover 2700+ languages and surpasses existing benchmarks in terms of language coverage.

Comprehensive Multi-Dataset Evaluation of Reading Comprehension (D19-58)

Copied to clipboard

Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner

Reasoning

Challenge:	Recent research aims to facilitate training and evaluation on several reading comprehension datasets at the same time.
Approach:	They propose an evaluation server that reports performance on seven diverse reading comprehension datasets and includes synthetic augmentations to test models' ability to handle out-of-domain questions.
Outcome:	The evaluation server performs on seven reading comprehension datasets, and collects and includes synthetic augmentations for these datasets to test models' ability to handle out-of-domain questions.

MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages (2026.findings-acl)

Copied to clipboard

Wenhan Han, Yifan Zhang, Zhixun Chen, null Binbinliu, Mykola Pechenizkiy, Meng Fang, Yin Zheng

Language Models

Challenge:	Existing evaluation datasets lack cross-lingual alignment, leaving assessments of multilingual capabilities fragmented in both language and skill coverage.
Approach:	They propose to use multilingual consistency as a complementary metric to assess performance bottlenecks and guide model improvement.
Outcome:	The proposed model lacks cross-lingual alignment and language coverage gaps between state-of-the-art models.

Multi-lingual Functional Evaluation for Large Language Models (2026.findings-acl)

Copied to clipboard

Victor Ojewale, Inioluwa Deborah Raji, Suresh Venkatasubramanian

Robustness in NLP

Challenge:	Multilingual competence in large language models is often evaluated via static data benchmarks such as Belebele, M-MMLU and M-GSM.
Approach:	They extend existing functional benchmark templates from English to five additional languages that span the range of resources available for NLP: French, Spanish, Hindi, Arabic and Yoruba.
Outcome:	The proposed models are translated from English to French, Spanish, Hindi, Arabic and Yoruba.

Beyond Facts- Benchmarking Distributional Reading Comprehension in Large Language Models (2026.findings-acl)

Copied to clipboard

Pei-Fu Guo, Ya An Tsai, Chun-Chia Hsu, Kai-Xin Chen, Yun-Da Tsai, Kai-Wei Chang, Nanyun Peng, Mi-Yen Yeh, Shou-De Lin

Language Models

Challenge:	Existing reading comprehension benchmarks focus on factual information, but many real-world tasks require distributional knowledge expressed across text.
Approach:	They propose a reading comprehension benchmark for LLMs to evaluate their ability to infer distributional knowledge from natural language.
Outcome:	Experiments with multiple LLMs show that the model outperforms baselines, but performance varies widely across distribution types and characteristics.

Dataset for the First Evaluation on Chinese Machine Reading Comprehension (L18-1)

Copied to clipboard

Yiming Cui, Ting Liu, Zhipeng Chen, Wentao Ma, Shijin Wang, Guoping Hu

Reasoning

Challenge:	Existing reading comprehension datasets are mostly in English .
Approach:	They propose a Chinese reading comprehension dataset to add diversity to existing reading comprehension data . proposed dataset contains cloze-style reading comprehension and user query reading comprehension .
Outcome:	The proposed dataset is based on a Chinese reading comprehension dataset . it includes two types of cloze-style and user query reading comprehension . the proposed dataset hosted the 1st Evaluation on Chinese Machine Reading Comprehension (CMRC-2017)