Papers by Fajri Koto
Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study examined the potential for cross-cultural transfer of commonsense reasoning . merely 12 culture-specific examples from one country can improve performance in others by 10% on average . |
| Approach: | They evaluate cross-cultural transfer of commonsense reasoning within the arab world . they use in-context learning and demonstration-based reinforcement to evaluate alignment methods . |
| Outcome: | The proposed model can improve performance in cultures with cultural similarities in the Arab world by 10% on average. |
Commonsense Reasoning in Arab Culture (2025.acl-long)
Copied to clipboard
Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, Fajri Koto
| Challenge: | Existing studies on commonsense reasoning in Arabic have relied on machine translations that lack cultural depth and introduce anglocentric biases. |
| Approach: | They propose a commonsense reasoning dataset in Arabic that covers 13 Arab countries. |
| Outcome: | The proposed dataset covers 13 countries across the Gulf, Levant, North Africa, and the Nile Valley. |
Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on single-round inference, but this view is problematic in real-world applications. |
| Approach: | They propose a framework that couples Steering Token Calibration with Semantic Alignment to ensure that LLMs are correctly aligned across gender, race, and sentiment. |
| Outcome: | The proposed framework outperforms baseline methods in achieving precise distributional control in attribute generation tasks. |
LipKey: A Large-Scale News Dataset for Absent Keyphrases Generation and Abstractive Summarization (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work has addressed each element individually, but this study focuses on LipKey, the largest news corpus with human-written abstractive summaries, absent keyphrases, and titles. |
| Approach: | They propose a novel news dataset that consists of highly absent keyphrases . they combine lips keyphrase and TF-IDF to obtain abstractive summaries . |
| Outcome: | The proposed dataset is the largest news corpus with human-written abstractive summaries, absent keyphrases, and titles. |
Qorǵau: Evaluating Safety in Kazakh-Russian Bilingual Contexts (2025.findings-acl)
Copied to clipboard
Maiya Goloburda, Nurkhan Laiyk, Diana Turmakhan, Yuxia Wang, Mukhammed Togmanov, Jonibek Mansurov, Askhat Sametov, Nurdaulet Mukhituly, Minghan Wang, Daniil Orel, Zain Muhammad Mujahid, Fajri Koto, Timothy Baldwin, Preslav Nakov
| Challenge: | Large language models (LLMs) have the potential to generate harmful content, posing risks to users. |
| Approach: | They propose a dataset specifically designed for safety evaluation in Kazakh and Russian . they use a bilingual context in Kazakhstan where both Kazakh (a low-resource language) and Russian (a high-resourced language) |
| Outcome: | The proposed dataset is designed for safety evaluation in Kazakh and Russian . it shows that both multilingual and language-specific LLMs perform better than others . |
Liputan6: A Large-scale Indonesian Dataset for Text Summarization (2020.aacl-main)
Copied to clipboard
| Challenge: | Despite having the fourth largest speaker population in the world, 1 Indonesian is under-represented in NLP. |
| Approach: | They propose to use a large-scale Indonesian summarization dataset to test extractive and abstractive summarizing methods. |
| Outcome: | The proposed methods are compared with multilingual and monolingual BERT-based models. |
Instruction-Guided Poetry Generation in Arabic and Its Dialects (2026.findings-acl)
Copied to clipboard
Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Fajri Koto
| Challenge: | Existing literature on Arabic poetry has focused on analysis tasks such as interpretation or metadata prediction, e.g., rhyme schemes and titles. |
| Approach: | They propose to use a large-scale instruction-based dataset to generate Arabic poetry based on predefined criteria such as style and rhyme . |
| Outcome: | The proposed model can generate poetry that is aligned with user requirements, based on automated metrics and human evaluation with native Arabic speakers. |
NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages (2023.eacl-main)
Copied to clipboard
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, Sebastian Ruder
| Challenge: | In Indonesia, many languages are endangered and some are even extinct due to the unavailability of data resources and benchmarks. |
| Approach: | They propose a high-quality multilingual parallel corpus that covers 10 local languages from Indonesia. |
| Outcome: | The proposed resource includes sentiment and machine translation datasets, and bilingual lexicons. |
FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning (2026.acl-long)
Copied to clipboard
Zhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan, Hachem Madmoun, Fan Zhang, Debopriyo Banerjee, Georgi Nenkov Georgiev, Xueqing Peng, Lingfei Qian, Jimin Huang, Jinyan Su, Aaryamonvikram Singh, Rui Xing, Rania Elbadry, Chen Xu, Haonan Li, Fajri Koto, Ivan Koychev, Tanmoy Chakraborty, Yuxia Wang, Salem Lahlou, Veselin Stoyanov, Sophia Ananiadou, Preslav Nakov
| Challenge: | Existing benchmarks emphasize final numerical answers while neglecting intermediate reasoning steps. |
| Approach: | They propose a symbolic benchmark for verifiable Chain-of-Thought evaluation in finance . FINCHAIN spans 58 topics across 12 financial domains and three difficulty levels . |
| Outcome: | The proposed benchmark aims to bridge symbolic reasoning and factual verification. |
One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia (2022.acl-long)
Copied to clipboard
Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Timothy Baldwin, Jey Han Lau, Sebastian Ruder
| Challenge: | There are more than 700 languages spoken in Indonesia, equal to 10% of the world's languages, second only to Papua New Guinea. |
| Approach: | They focus on the languages spoken in Indonesia, the world's second most linguistically diverse nation, and the fourth most populous nation of the world. |
| Outcome: | The proposed model is based on the languages spoken in Indonesia, the world's second-most linguistically diverse nation, with 273 million people spread over 17,508 islands. |
Unveiling Cultural Blind Spots: Analyzing the Limitations of mLLMs in Procedural Text Comprehension (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks, including text summarization, multi-modal machine translation, and code generation and understanding. |
| Approach: | They propose a benchmark to evaluate mLLMs’ ability to process and reason over culturally diverse procedural texts in multiple languages. |
| Outcome: | The proposed benchmarks show that mLLMs struggle with culturally contextualized procedural content, especially in low-resource languages, and perform better on multiple-choice tasks presented in conversational formats than on direct questions. |
Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon (2024.eacl-long)
Copied to clipboard
| Challenge: | Prior work extended multilingual models to other languages due to the unavailability of labeled and unlabeled training data. |
| Approach: | They use multilingual lexicons to enhance multilingual models capabilities in low-resource languages . they focus on zero-shot sentiment analysis tasks across 34 languages based on a single sentence . |
| Outcome: | The proposed model improves zero-shot performance across 34 languages without using any sentence-level sentiment data. |
LLMs as Cultural Archives: Cultural Commonsense Knowledge Graph Extraction (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models encode rich cultural knowledge, but it remains mostly implicit and unstructured, limiting its interpretability and use. |
| Approach: | They propose an iterative framework for constructing a Cultural Commonsense Knowledge Graph using a prompt-based framework. |
| Outcome: | The proposed framework improves cultural reasoning and story generation on non-English cultures. |
Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling (2026.acl-long)
Copied to clipboard
Alaa Elsetohy, Sama Hadhoud, Haryo Akbarianto Wibowo, Chenxi Whitehouse, Genta Indra Winata, Fajri Koto, Alham Fikri Aji
| Challenge: | Existing benchmarks test reasoning over culturally grounded premises, but translation-parallel benchmarks inherit English-centric scenarios. |
| Approach: | They propose a template-first benchmark that factorizes reasoning type and cultural aspect across question languages. |
| Outcome: | The proposed benchmark factorizes reasoning type and cultural aspect across question languages. |
KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan (2025.acl-long)
Copied to clipboard
Mukhammed Togmanov, Nurdaulet Mukhituly, Diana Turmakhan, Jonibek Mansurov, Maiya Goloburda, Akhmed Sakip, Zhuohan Xie, Yuxia Wang, Bekassyl Syzdykov, Nurkhan Laiyk, Alham Fikri Aji, Ekaterina Kochmar, Preslav Nakov, Fajri Koto
| Challenge: | Kazakh language remains underrepresented in the field of natural language processing despite the country's population exceeding twenty million . however, there is a lack of dedicated models and benchmark evaluations specifically tailored to Kazakh languages. |
| Approach: | They propose to create a dataset specifically designed for Kazakh language with 23,000 questions sourced from authentic educational materials and manually validated by native speakers and educators. |
| Outcome: | The first MMLU-style dataset specifically designed for Kazakh language. |
NusaCrowd: Open Source Initiative for Indonesian NLP Resources (2023.findings-acl)
Copied to clipboard
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono, Ivan Parmonangan, Ika Alfina, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Septiandri, James Jaya, Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindyatama Nityasya, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Haryo Wibowo, Cuk Tho, Ichwanul Karo Karo, Tirana Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sebastian Ruder, Pascale Fung, Herry Sujaini, Sakriani Sakti, Ayu Purwarianti
| Challenge: | Existing NLP research in Indonesian languages has been held back by factors such as language diversity, orthographic variation, resource limitation and other societal challenges. |
| Approach: | They present a collaborative initiative to collect and unify existing resources for Indonesian languages and open access to previously non-public resources. |
| Outcome: | The results show that the datasets are highly reliable and can be used to generate the first zero-shot benchmarks for natural language understanding and generation in Indonesian and the local languages of Indonesia. |
CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. |
| Approach: | They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings. |
| Outcome: | The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models. |
Stereotype Bias in a Bilingual Setting: A Culturally Grounded Evaluation in Kazakhstan (2026.acl-long)
Copied to clipboard
Nurkhan Laiyk, Daniil Orel, Ayana Mussabayeva, Maiya Goloburda, Kamila Kuishibekova, Liya Goloburda, Diana Turmakhan, Preslav Nakov, Yuxia Wang, Fajri Koto
| Challenge: | Stereotype bias in language models is largely understudied in English . language models perform strongly on downstream NLP tasks, but they are pre-trained on large text corpora . |
| Approach: | They use a dataset to assess stereotype bias in language models in Kazakhstan . they find that stereotype bias is most pronounced in code-mixed inputs . |
| Outcome: | The proposed dataset shows that stereotype bias is most pronounced in code-mixed inputs. |
ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic (2024.findings-acl)
Copied to clipboard
Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, Timothy Baldwin
| Challenge: | evaluating language models in Arabic remains challenging due to limited datasets . focus has shift to reasoning and knowledge-intensive tasks due to lack of relevant datasets. |
| Approach: | They propose to use ArabicMMLU to evaluate models' understanding of Arabic . they use 40 tasks and 14,575 multiple-choice questions from school exams in different countries . |
| Outcome: | The ArabicMMLU is the first multi-task language understanding benchmark for the Arabic language . it is based on 40 tasks and 14,575 multiple-choice questions in modern standard Arabic . the models are based in different countries across North Africa, the Levant, and the Gulf regions . |
Top-down Discourse Parsing via Sequence Labelling (2021.eacl-main)
Copied to clipboard
| Challenge: | Discourse analysis is a systematic way to understand how texts are segmented hierarchically into discourse units. |
| Approach: | They propose a top-down approach to discourse parsing that is conceptually simpler than its predecessors. |
| Outcome: | The proposed model eliminates the decoder and reduces the search space for splitting points. |
Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia (2025.acl-long)
Copied to clipboard
Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz, Tack Hwa Wong, Mohammad Rifqi Farhansyah, Thant Thiri Maung, Frederikus Hudi, David Anugraha, Muhammad Ravi Shulthan Habibi, Muhammad Reza Qorib, Amit Agarwal, Joseph Marvin Imperial, Hitesh Laxmichand Patel, Vicky Feliren, Bahrul Ilmi Nasution, Manuel Antonio Rufino, Genta Indra Winata, Rian Adam Rajagede, Carlos Rafael Catalan, Mohamed Fazli Mohamed Imam, Priyaranjan Pattnayak, Salsabila Zahirah Pranida, Kevin Pratama, Yeshil Bangera, Adisai Na-Thalang, Patricia Nicole Monderin, Yueqi Song, Christian Simon, Lynnette Hui Xian Ng, Richardy Lobo Sapan, Taki Hasan Rafi, Bin Wang, null Supryadi, Kanyakorn Veerakanjana, Piyalitt Ittichaiwong, Matthew Theodore Roque, Karissa Vincentio, Takdanai Kreangphet, Phakphum Artkaew, Kadek Hendrawan Palgunadi, Yanzhi Yu, Rochana Prih Hastuti, William Nixon, Mithil Bangera, Adrian Xuan Wei Lim, Aye Hninn Khine, Hanif Muhammad Zhafran, Teddy Ferdinan, Audra Aurora Izzani, Ayushman Singh, Evan Evan, Jauza Akbar Krito, Michael Anugraha, Fenal Ashokbhai Ilasariya, Haochen Li, John Amadeo Daniswara, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Can Udomcharoenchaikit, Fadil Risdian Ansori, Mahardika Krisna Ihsani, Giang Nguyen, Anab Maulana Barik, Dan John Velasco, Rifo Ahmad Genadi, Saptarshi Saha, Chengwei Wei, Isaiah Edri W. Flores, Kenneth Chen Ko Han, Anjela Gail D. Santos, Wan Shen Lim, Kaung Si Phyo, Tim Santos, Meisyarah Dwiastuti, Jiayun Luo, Jan Christian Blaise Cruz, Ming Shan Hee, Ikhlasul Akmal Hanif, M.Alif Al Hakim, Muhammad Rizky Sya’ban, Kun Kerdthaisong, Lester James Validad Miranda, Fajri Koto, Tirana Noor Fatyanosa, Alham Fikri Aji, Jostin Jerico Rosal, Jun Kevin, Robert Wijaya, Onno P. Kampman, Ruochen Zhang, Börje F. Karlsson, Peerat Limkonchotiwat
| Challenge: | Southeast Asia is underrepresented in vision-language research . SEA-VL is an open-source initiative dedicated to developing culturally relevant datasets for SEA languages. |
| Approach: | They propose to use crowdsourced, automated image crawling and synthetic image generation to develop culturally relevant datasets for SEA languages. |
| Outcome: | The proposed datasets capture SEA cultural nuances and contexts better than existing datasets. |
IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety standards are often based on direct translations from English, which overlook key aspects of local communication. |
| Approach: | They propose a high-quality, human-verified safety evaluation dataset tailored for the Indonesian context. |
| Outcome: | The proposed dataset covers formal and colloquial Indonesian, along with three major local languages: Javanese, Sundanese, and Minangkabau. |
Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are adept at question answering and reasoning tasks, but when reasoning in situational context, human expectations vary depending on the relevant cultural common ground. |
| Approach: | They construct and evaluate a dataset for proverb understanding with conversational context for six different languages and their usage within the context. |
| Outcome: | The proposed model is able to reason with proverbs and sayings in conversational contexts. |
Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh (2025.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. |
| Approach: | They propose to open-source a large-scale instruction-following dataset covering key institutional and cultural knowledge relevant to Kazakhstan. |
| Outcome: | The proposed dataset improves LLMs’ understanding of procedural, legal, and structural governance topics. |
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics have been developed and validated for English and other languages . this narrow focus leaves Indian languages largely overlooked, casting doubt on universality of current evaluation practices. |
| Approach: | They propose a large-scale benchmark that compares 26 automatic metrics with human judgments across six major Indian languages. |
| Outcome: | ITEM evaluates alignment of 26 automatic metrics with human judgments across six languages . authors: outliers exert significant impact on metric-human agreement, improve fidelity . they say the results offer critical guidance for advancing metric design and evaluation in Indian languages - a global market for machine translation and text summarization systems. |
What Do Indonesians Really Need from Language Technology? A Nationwide Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite efforts to develop NLP for Indonesia’s 700+ local languages, progress remains costly due to the need for direct engagement with native speakers. |
| Approach: | They conduct a nationwide survey to assess the actual needs of native Indonesian speakers. |
| Outcome: | The findings indicate that addressing language barriers is the most critical priority . concerns around privacy, bias, and the use of public data highlight the need for greater transparency and clear communication to support broader AI adoption. |
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020.coling-main)
Copied to clipboard
| Challenge: | despite being spoken by 200 million people, the Indonesian language is underrepresented in NLP research. |
| Approach: | They propose a dataset for Indonesian that includes seven NLP tasks . they also propose 'indonesian language evaluation Montage' tasks that are based on previous work . |
| Outcome: | The proposed dataset shows that IndoBERT outperforms IndoLEM over most of the tasks. |
Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages (2026.acl-long)
Copied to clipboard
Saeed Almheiri, Bilal Elbouardi, Salsabila Zahirah Pranida, Irina Nikishina, Ashwath Rao B, Parameswari Krishnamurthy, Muhammad Cendekia Airlangga, Rifo Ahmad Genadi, Nguyen Phan Gia Bao, Amir Hossein Yari, Hawau Olamide Toyin, Nurdaulet Mukhituly, Mena Attia, Besher Hassan, Ahmad Fathan Hidayatullah, Tatsuki Kuribayashi, Haonan Li, Suma Bhat, Fajri Koto
| Challenge: | idioms are a major challenge for multilingual NLP because their meanings shift between figurative and literal usage, often requiring context for accurate interpretation. |
| Approach: | They propose a multilingual idiom dataset that provides idiomatic expressions in both sentence-level and conversational contexts. |
| Outcome: | The proposed model performs well with low-resource idioms, but lacks contextual inference. |
Discourse Probing of Pretrained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on probing of pretrained language models has focused on sentence-level syntactic tasks. |
| Approach: | They introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document- level relations. |
| Outcome: | The proposed model performs best in encoder, but only in the encoder layer. |
IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization (2021.emnlp-main)
Copied to clipboard
| Challenge: | In IndoBERTweet, a pretraining model for Indonesian Twitter is extended with domain-specific vocabulary. |
| Approach: | They propose a pretraining model that extends a monolingual Indonesian BERT model with domain-specific vocabulary. |
| Outcome: | The proposed model can be initialized with the average BERT subword embedding five times faster than existing methods for vocabulary adaptation. |
Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia (2025.naacl-industry)
Copied to clipboard
| Challenge: | Using the entire dataset, shuffling answer options introduces instability in the insurance and finance sectors. |
| Approach: | They propose a dataset for evaluation of performance in vocational and professional certification exams in Indonesia. |
| Outcome: | The proposed dataset includes 8,834 multiple-choice questions from 27 large language models across six key sectors. |
Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models based on English datasets do not provide adequate data for evaluating their capabilities beyond English. |
| Approach: | They propose a multi-task language understanding benchmark for Indonesian culture and languages . it measures language proficiency, reasoning abilities and real-world knowledge . |
| Outcome: | The proposed model passes the primary school level in Indonesia, while other models perform at lower levels. |
Nanda Family: Open-Weights Generative Large Language Models for Hindi (2026.eacl-long)
Copied to clipboard
Aaryamonvikram Singh, Debopriyo Banerjee, Dhruv Sahnan, Monojit Choudhury, Shivam Chauhan, Rocktim Jyoti Das, Xudong Han, Haonan Li, Alok Anil Jadhav, Utkarsh Agarwal, Mukund Choudhary, Fajri Koto, Junaid Hamid Bhat, Awantika Shukla, Samujjwal Ghosh, Samta Kamboj, Onkar Pandit, Lalit Pradhan, Rahul Pal, Sunil Kumar Sahu, Parvez Mullah, Ali El Filali, Zainul Abedien Ahmed Quraishi, Neha Sengupta, Gokulakrishnan Ramakrishnan, Rituraj Joshi, Gurpreet Gosal, Avraham Sheinin, Natalia Vassilieva, Preslav Nakov
| Challenge: | Large language models remain predominantly English-centric, which limits their utility for underrepresented languages. |
| Approach: | They propose to extend Llama’s vocabulary with 20% Hindi-specific tokens, thus halving Hindi tokenization fertility while preserving English efficiency. |
| Outcome: | The proposed models outperform open-weight models of comparable size on a 65B-token corpus and bilingual instruction and safety alignment on . a culturally grounded dataset. |
Evaluating the Efficacy of Summarization Evaluation across Languages (2021.findings-acl)
Copied to clipboard
| Challenge: | Using multilingual summarization evaluation methods is more reliable and interpretable than manual methods. |
| Approach: | They propose to use multilingual BERT within BERTScore to evaluate summarization evaluation metrics . they use English datasets that are not representative of modern summarizing systems . |
| Outcome: | The proposed methods perform well across all languages, at a level above that for English. |
Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing quantization methods focus on general metrics like perplexity or accuracy on standard benchmarks. |
| Approach: | They propose a method that preserves fairness- and safety-critical weights during quantization. |
| Outcome: | The proposed method reduces bias and safety degradation without costly retraining or alignment while maintaining trustworthiness while retaining efficiency. |
SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages (2024.emnlp-main)
Copied to clipboard
Holy Lovenia, Rahmad Mahendra, Salsabil Akbar, Lester James Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno Kampman, Joel Moniz, Muhammad Habibi, Frederikus Hudi, Jann Montalan, Ryan Hadiwijaya, Joanito Lopo, William Nixon, Börje Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Irawan, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Ryanda, Sonny Hermawan, Dan Velasco, Muhammad Kautsar, Willy Hendria, Yasmin Moslem, Noah Flynn, Muhammad Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Tai Chia, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Aji, Sedrick Keh, Genta Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, Samuel Cahyawijaya
| Challenge: | Southeast Asia (SEA) is home to over 1,300 indigenous languages and 671 million people . prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA . |
| Approach: | They propose to provide a resource center that provides standardized corpora in nearly 1,000 SEA languages across three modalities. |
| Outcome: | a new benchmark assesses the quality of AI models on 36 SEA languages across 13 tasks . the results highlight the importance of SEA as a culturally diverse region . |
Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues (2026.acl-long)
Copied to clipboard
Muhammad Dehan Al Kautsar, Saeed Almheiri, Momina Ahsan, Bilal Elbouardi, Younes Samih, Sarfraz Ahmad, Amr Keleg, Omar El Herraoui, Kareem Elzeky, Abed Alhakim Freihat, Mohamed Anwar, Zhuohan Xie, Junhong Liang, Mohammad Rustom Al Nasar, Preslav Nakov, Fajri Koto
| Challenge: | Most benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking cultural nuances that naturally arise in dialogues. |
| Approach: | They propose a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both Modern Standard Arabic (MSA) and each country’s respective dialect, spanning 12 daily-life topics and 54 fine-grained subtopics. |
| Outcome: | The proposed model performs worse on all three tasks than the MSA benchmark. |