TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages (2025.acl-long)
Copied to clipboard
Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif Köksal, Samir Rustamov, Duygu Ataman
| Challenge: | preparing native language MMLU benchmarks is costly and limits representativeness of evaluation datasets. |
| Approach: | They propose to use a Turkic language MMLU benchmark to assess massive multitask language understanding capabilities. |
| Outcome: | The proposed benchmarks are based on a Turkic language morphosyntactic and cultural benchmark . the benchmarks evaluate a diverse range of open and proprietary multilingual large language models . |
Similar Papers
TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multiple choice question answering benchmarks employ automatic translation for multilingual evaluation, but this approach is error-prone and potentially introduces culturally biased questions. |
| Approach: | They introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU . they evaluate over 20 LLMs including open-source, closed-source and Turkish-adapted models . |
| Outcome: | The proposed benchmarks evaluate the reasoning, comprehension, and mathematical abilities of large language models. |
KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan (2025.acl-long)
Copied to clipboard
Mukhammed Togmanov, Nurdaulet Mukhituly, Diana Turmakhan, Jonibek Mansurov, Maiya Goloburda, Akhmed Sakip, Zhuohan Xie, Yuxia Wang, Bekassyl Syzdykov, Nurkhan Laiyk, Alham Fikri Aji, Ekaterina Kochmar, Preslav Nakov, Fajri Koto
| Challenge: | Kazakh language remains underrepresented in the field of natural language processing despite the country's population exceeding twenty million . however, there is a lack of dedicated models and benchmark evaluations specifically tailored to Kazakh languages. |
| Approach: | They propose to create a dataset specifically designed for Kazakh language with 23,000 questions sourced from authentic educational materials and manually validated by native speakers and educators. |
| Outcome: | The first MMLU-style dataset specifically designed for Kazakh language. |
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (2025.emnlp-main)
Copied to clipboard
Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, Felix Juefei-Xu, Foutse Khomh, Osamu Yoshie, Qingyu Chen, Douglas Teodoro, Nan Liu, Randy Goebel, Lei Ma, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
| Challenge: | Existing large language model evaluation benchmarks focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. |
| Approach: | They propose a comprehensive benchmark covering 29 languages, built on an English benchmark. |
| Outcome: | The MMLU-ProX is a comprehensive benchmark covering 29 languages, built on an English benchmark. |
A Large-Scale Study of Machine Translation in Turkic Languages (2021.emnlp-main)
Copied to clipboard
Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman, Sherzod Kariev, Francis Tyers, Otabek Abduraufov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverghetta Jr., Bekhzodbek Moydinboyev, Esra Onal, Shaxnoza Pulatova, Ahsan Wahab, Orhan Firat, Sriram Chellappan
| Challenge: | a large corpus covering 22 Turkic languages is included in this paper . low-resource MT evaluation has traditionally focused on European languages due to limitations of available technology and resources. |
| Approach: | They present a case study of the practical application of MT in the Turkic language family . they propose to realize the gains of NMT for Turkic languages under high-resource to extremely low-resourced scenarios. |
| Outcome: | The proposed study shows that the new methods can be used in the Turkic language family . the results highlight bottlenecks in building competitive systems . |
LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models (2026.acl-long)
Copied to clipboard
Jian Gao, Richeng Xuan, Zhaolu Kang, Dingshi Liao, Wenxin Huang, Zongmou Huang, Yangdi Xu, Bowen Qin, Zheqi He, Xi Yang, null Changjinli, Yonghua Lin
| Challenge: | Existing SEA-focused benchmarks miss Lao-specific cultural grounding and linguistic properties. |
| Approach: | They propose a multi-dimensional benchmark for assessing large language models in Lao . they use open-source and held-out subsets to evaluate languages with a hybrid pipeline . |
| Outcome: | LaoBench is the first large-scale, high-quality, and multidimensional benchmark for assessing LLM language understanding and reasoning in Lao. |
KMMLU: Measuring Massive Multitask Language Understanding in Korean (2025.naacl-long)
Copied to clipboard
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, Stella Biderman
| Challenge: | Recent models struggle to show performance over 60%, significantly below the pass mark of the source exams (80%), highlighting the room for improvement. |
| Approach: | They propose to use Korean exams to collect 35,030 questions from an expert-level multiple choice model to capture linguistic and cultural aspects of the Korean language. |
| Outcome: | The proposed benchmark is based on 35,030 questions from original Korean exams. |
Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing Turkish benchmarks lack task diversity or culturally relevant content . Cetvel combines a broad range of discriminative and generative tasks . |
| Approach: | They propose a benchmark to evaluate large language models in Turkish . Cetvel combines a broad range of discriminative and generative tasks . they find that Turkish-centric instruction-tuned models generally underperform . |
| Outcome: | The proposed benchmark covers 23 tasks grouped into seven categories . it shows that Turkish-centric instruction-tuned models underperform relative to multilingual or general-purpose models despite being tailored for the language. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. |
| Approach: | They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings. |
| Outcome: | The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models. |
GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek (2026.findings-acl)
Copied to clipboard
Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang, Michalis Vazirgiannis
| Challenge: | Existing evaluation benchmarks for large language models are limited for Greek . Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. |
| Approach: | They propose a native-sourced benchmark for massive multitask language understanding in Greek . they publicize 16,857 samples and reserve 4,948 samples for a private leaderboard . |
| Outcome: | The proposed model is based on 21,805 multiple-choice questions across 45 subject areas . the model is publicly released and reserved for a private leaderboard . |