SEA-HELM: Southeast Asian Holistic Evaluation of Language Models (2025.findings-acl)
Copied to clipboard
Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xianbin Yong, Wei Qi Leong, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Yifan Mai, William Chandra Tjhi
| Challenge: | Existing LLM benchmarks are capable of evaluating specific capabilities in English as well as in various mid- to low-resource languages, but a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. |
| Approach: | They propose a holistic linguistic and cultural LLM evaluation suite that emphasizes SEA languages and introduces a leaderboard that allows users to understand models’ multilingual and multicultural performance. |
| Outcome: | The proposed evaluation suite emphasizes SEA languages and supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. |
Similar Papers
SEA-SafeguardBench: Culturally Grounded Safety Benchmark for Southeast Asian Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing multilingual safety benchmarks rely on machine-translated English data, which fails to capture nuances in low-resource languages. |
| Approach: | They propose to use a human-verified safety benchmark for Southeast Asian languages to validate their safety and cultural diversity. |
| Outcome: | The proposed model outperforms existing models in general, in-the-wild, and content generation across eight languages and 21,640 samples across three subsets: general, and in- the-wild. |
SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages (2024.emnlp-main)
Copied to clipboard
Holy Lovenia, Rahmad Mahendra, Salsabil Akbar, Lester James Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno Kampman, Joel Moniz, Muhammad Habibi, Frederikus Hudi, Jann Montalan, Ryan Hadiwijaya, Joanito Lopo, William Nixon, Börje Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Irawan, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Ryanda, Sonny Hermawan, Dan Velasco, Muhammad Kautsar, Willy Hendria, Yasmin Moslem, Noah Flynn, Muhammad Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Tai Chia, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Aji, Sedrick Keh, Genta Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, Samuel Cahyawijaya
| Challenge: | Southeast Asia (SEA) is home to over 1,300 indigenous languages and 671 million people . prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA . |
| Approach: | They propose to provide a resource center that provides standardized corpora in nearly 1,000 SEA languages across three modalities. |
| Outcome: | a new benchmark assesses the quality of AI models on 36 SEA languages across 13 tasks . the results highlight the importance of SEA as a culturally diverse region . |
SeaLLMs - Large Language Models for Southeast Asia (2024.acl-demos)
Copied to clipboard
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, Lidong Bing
| Challenge: | Existing large language models favor high-resource languages, such as English, at the expense of low-resourced and regional languages. |
| Approach: | They propose a series of language models that specifically focuses on Southeast Asian languages. |
| Outcome: | SeaLLM models outperform ChatGPT-3.5 in non-Latin languages by large margins . linguistic disparity impedes access to state-of-the-art AI technologies for non-English-speaking populations . |
SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable performance across various English benchmarks, including both human exam datasets such as MMLU and instruction-following datasets. |
| Approach: | They introduce two new benchmarks to evaluate the capabilities of Large Language Models in Southeast Asian (SEA) application scenarios. |
| Outcome: | The proposed benchmarks show that they can discern LLM performance on SEA language tasks compared to their translated benchmarks. |
SEA-BED: How Do Embedding Models Represent Southeast Asian Languages? (2026.acl-long)
Copied to clipboard
Wuttikorn Ponwitayarat, Peerat Limkonchotiwat, Raymond Ng, Jann Railey Montalan, Thura Aung, Jian Gang Ngui, Yosephine Susanto, William Chandra Tjhi, Panuthep Tasawong, Erik Cambria, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | SEA-BED examines how multilingual text embeddings perform across tasks and languages . performance gaps arise from data coverage, training objectives, and architectural design, authors say . |
| Approach: | They propose a large-scale benchmark covering 10 SEA languages and diverse embedding tasks. |
| Outcome: | The proposed model performs poorly across languages and tasks, but language-task analyses reveal inconsistencies . the results suggest that performance gaps arise from limitations in data coverage, training objectives, and architectural design. |
SEA-Guard: Culturally Grounded Multilingual Safeguard for Southeast Asia (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing safeguard models rely on translation of English datasets, missing regional and cultural nuances. |
| Approach: | They propose a framework to generate culturally grounded safety datasets for Southeast Asia . SEA-Guard family is the first multilingual safeguard model grounded in SEA cultural contexts . |
| Outcome: | The proposed model outperforms existing safeguard models in detecting regionally sensitive content while maintaining strong general safety performance. |
LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models (2026.acl-long)
Copied to clipboard
Jian Gao, Richeng Xuan, Zhaolu Kang, Dingshi Liao, Wenxin Huang, Zongmou Huang, Yangdi Xu, Bowen Qin, Zheqi He, Xi Yang, null Changjinli, Yonghua Lin
| Challenge: | Existing SEA-focused benchmarks miss Lao-specific cultural grounding and linguistic properties. |
| Approach: | They propose a multi-dimensional benchmark for assessing large language models in Lao . they use open-source and held-out subsets to evaluate languages with a hybrid pipeline . |
| Outcome: | LaoBench is the first large-scale, high-quality, and multidimensional benchmark for assessing LLM language understanding and reasoning in Lao. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness . |
| Approach: | They introduce an automatic multilingual framework for evaluating cultural awareness in large language models across languages, regions, and topics. |
| Outcome: | The framework evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. |
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models (2026.acl-long)
Copied to clipboard
Yilun Liu, Chunguang Zhao, Mengyao Piao, Lingqi Miao, Shimin Tao, Minggui HE, Chenxin Liu, Zhang Li, null Mahongxia, Jiaxin Guo, Chen Liu, Liqun Deng, Jiansheng Wei, Xiaojun Meng, Fanyi Du, Daimeng Wei, Yanghua Xiao
| Challenge: | Existing multilingual evaluation benchmarks neglect cultural nuances and lack language coverage in subjective tasks. |
| Approach: | They propose a framework that categorizes evaluation tasks into three cultural layers and nine cognitive sub-layers. |
| Outcome: | The proposed framework surpasses prior coverage by up to 111% on 20+ LLMs. |