SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala (2025.emnlp-main)
Copied to clipboard
Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Large Language Models (LLMs) have been evaluated mostly on global or anglocentric subjects, often neglecting low-resource languages and culturally specific content. |
| Approach: | They evaluate 26 Large Language Models using a multiple-choice question answering benchmark for Sinhala. |
| Outcome: | The new benchmarks show that Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies, but overall performance remains limited. |
Similar Papers
MalayMMLU: A Multitask Benchmark for the Low-Resource Malay Language (2024.findings-emnlp)
Copied to clipboard
Soon Poh, Sze Jue Yang, Jeraelyn Tan, Lawrence Chieng, Jia Tan, Zhenyu Yu, Foong Mun, Chee Seng Chan
| Challenge: | Large Language Models (LLMs) and Large Vision Language Model (LVLMs) exhibit advanced proficiency in language reasoning and comprehension across a wide array of languages. |
| Approach: | They propose to use a multitask language understanding benchmark specifically designed for the Malay language to assess their proficiency. |
| Outcome: | The proposed model performs well in well-resourced languages, but in low-resource languages such as Bahasa Melayu, they are less studied due to a lack of studies and benchmarks. |
BnMMLU: Measuring Massive Multitask Language Understanding in Bengali (2026.findings-acl)
Copied to clipboard
| Challenge: | Large-scale multitask benchmarks have driven rapid progress in language modeling, yet most emphasize low-resource languages like English. |
| Approach: | They propose a benchmark for massive multitask language understanding in Bengali . they use a dataset that preserves mathematical content via MathML and a subset of questions most frequently missed by top systems to stress difficult cases. |
| Outcome: | The proposed benchmark covers 24 model variants across 11 LLM families. |
ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are restricted to high- or mid-resource languages, and evaluate performance on higher-order tasks in reasoning and generation. |
| Approach: | They propose a multilingual benchmarking tool to evaluate lexical comprehension and generation abilities of large language models. |
| Outcome: | The proposed benchmarks cover 2700+ languages and surpasses existing benchmarks in terms of language coverage. |
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models (2026.acl-long)
Copied to clipboard
Yilun Liu, Chunguang Zhao, Mengyao Piao, Lingqi Miao, Shimin Tao, Minggui HE, Chenxin Liu, Zhang Li, null Mahongxia, Jiaxin Guo, Chen Liu, Liqun Deng, Jiansheng Wei, Xiaojun Meng, Fanyi Du, Daimeng Wei, Yanghua Xiao
| Challenge: | Existing multilingual evaluation benchmarks neglect cultural nuances and lack language coverage in subjective tasks. |
| Approach: | They propose a framework that categorizes evaluation tasks into three cultural layers and nine cognitive sub-layers. |
| Outcome: | The proposed framework surpasses prior coverage by up to 111% on 20+ LLMs. |
TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multiple choice question answering benchmarks employ automatic translation for multilingual evaluation, but this approach is error-prone and potentially introduces culturally biased questions. |
| Approach: | They introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU . they evaluate over 20 LLMs including open-source, closed-source and Turkish-adapted models . |
| Outcome: | The proposed benchmarks evaluate the reasoning, comprehension, and mathematical abilities of large language models. |
Sinhala Encoder-only Language Models and Evaluation (2025.acl-long)
Copied to clipboard
Tharindu Ranasinghe, Hansi Hettiarachchi, Nadeesha Chathurangi Naradde Vidana Pathirana, Damith Premasiri, Lasitha Uyangodage, Isuri Nanomi Arachchige, Alistair Plum, Paul Rayson, Ruslan Mitkov
| Challenge: | Recent advances in language models (LMs) have produced excellent results in many NLP tasks, but their effectiveness is highly dependent on available pre-training resources. |
| Approach: | They propose to collect the largest monolingual corpus for Sinhala and compile a benchmark and evaluate LMs on it. |
| Outcome: | The proposed language models outperform the popular multilingual LMs in downstream NLP tasks. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
VMLU Benchmarks: A comprehensive benchmark toolkit for Vietnamese LLMs (2025.acl-long)
Copied to clipboard
Cuc Thi Bui, Nguyen Truong Son, Truong Van Trang, Lam Viet Phung, Pham Nhut Huy, Hoang Anh Le, Quoc Huu Van, Phong Nguyen-Thuan Do, Van Le Tran Truc, Duc Thanh Chau, Le-Minh Nguyen
| Challenge: | The evolution of Large Language Models (LLMs) has underscored the need for benchmarks designed for various languages and cultural contexts. |
| Approach: | They propose to use Vietnamese multitask language understanding (VMLU) benchmarks to assess different capabilities of LLMs, including general knowledge, reading comprehension, reasoning, and conversational skills. |
| Outcome: | The VMLU Benchmarks assess LLMs' general knowledge, reading comprehension, reasoning, and conversational skills. |
P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs (2025.emnlp-main)
Copied to clipboard
Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou
| Challenge: | Recent advances in large language models showcase varied multilingual capabilities across tasks . previous assessments focused on fundamental natural language processing (NLP) or isolated capability-specific tasks. |
| Approach: | They propose a multilingual multitask benchmark to assess multilingual capabilities . they use a large-scale benchmark covering fundamental and capability-specialized datasets . |
| Outcome: | The proposed benchmark compares models and tasks across languages and tasks and examines knowledge transfer from English to other languages. |
MILU: A Multi-task Indic Language Understanding Benchmark (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on English, leaving substantial gaps in assessing LLM capabilities in low-resource and linguistically diverse languages. |
| Approach: | They propose a multi-task indic language understanding benchmark to assess LLMs in low-resource languages. |
| Outcome: | The new benchmark spans 8 domains and 41 subjects across 11 Indic languages, reflecting general and culturally specific knowledge. |