FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. |
| Approach: | They introduce FormosanBench, a benchmark for evaluating LLMs on low-resource Austronesian languages. |
| Outcome: | The proposed benchmark covers three endangered Formosan languages: Atayal, Amis, and Paiwan . existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. |
Similar Papers
Can Large Language Models Translate Unseen Languages in Underrepresented Scripts? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance in machine translation, but struggle with unseen low-resource languages. |
| Approach: | They propose a benchmark to evaluate translation for Mongolian and Yi using linguistic resources. |
| Outcome: | The proposed model can translate Mongolian (in traditional script) and Yi with the help of linguistic resources, but is limited in its ability to handle these languages effectively. |
MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable performance across various NLP tasks, largely due to their generalisability and ability to perform tasks without additional training. |
| Approach: | They evaluate the performance of 55 publicly available Large Language Models on Maltese, a low-resource language, using a newly introduced benchmark covering 11 discriminative and generative tasks. |
| Outcome: | The proposed models perform poorly on discriminative and generative tasks and smaller fine-tuned models perform better across all tasks. |
Improving Low-Resource Machine Translation for Formosan Languages Using Bilingual Lexical Resources (2024.findings-acl)
Copied to clipboard
| Challenge: | Using bilingual lexicons for low-resource languages can improve machine translation for low resource languages. |
| Approach: | They propose to use bilingual lexicons to improve machine translation for low-resource languages . they use parallel data and bilingual dictionaries to generate pseudo-parallel sentences . |
| Outcome: | The proposed techniques improve translation between Mandarin and Formosan languages and Spanish and Nahuatl, a language pair consisting of languages from completely different language families. |
High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pretrained large language models (LLMs) can bridge the performance gap for under-resourced languages by substantial margins, as measured by both automatic and human evaluations. |
| Approach: | They propose to use pretrained large language models to bridge this gap by automating and evaluating data-to-text generation in under-resourced languages. |
| Outcome: | The proposed model can set the state of the art for under-resourced languages by substantial margins, as measured by both automatic and human evaluations. |
IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models (2025.naacl-long)
Copied to clipboard
David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Salomey Osei, Shamsuddeen Hassan Muhammad, Sokhar Samb, Tadesse Kebede Guge, Tombekai Vangoni Sherman, Pontus Stenetorp
| Challenge: | Large language models (LLMs) are limited to a few high-resource languages . many low-resourced languages are evaluated only on basic text classification tasks . |
| Approach: | They propose to use IrokoBench to evaluate 17 low-resource African languages . they use human-translated benchmark datasets to evaluate zero-shot, few-shot and translate-test settings . |
| Outcome: | The proposed model performs well in English and French, but the highest performing model perform poorly in proprietary models. |
LLMs for Extremely Low-Resource Finno-Ugric Languages (2025.findings-naacl)
Copied to clipboard
| Challenge: | Low-resource languages such as those in the Finno-Ugric family are underrepresented in large language models. |
| Approach: | They propose to develop large language models for extremely low-resource languages . they focus on Vro, Livonian, and Komi, which are underrepresented . |
| Outcome: | The proposed models cover almost the entire cycle of creation, from data collection to instruction tuning and evaluation. |
LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Recent advances in AI can be attributed to the remarkable performance of Large Language Models (LLMs) success of LLMs depends on specific training techniques, such as instruction tuning and prompting . |
| Approach: | They explore the capabilities of Large Language Models (LLMs) in various tasks and languages . they also examine their performance, fine-tuning, instructions tuning, and close vs. open models . |
| Outcome: | The proposed model can be used for speech and multimodal tasks across modalities, languages, and dialects. |
Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Rapid advancements in large language models have highlighted the need for robust evaluation frameworks that assess their core capabilities. |
| Approach: | They propose two benchmarks to assess core capabilities of large language models . current benchmarks for Thai focus mainly on traditional NLP tasks . |
| Outcome: | The proposed benchmarks are based on evaluations of various LLMs with multi-lingual capabilities and are publicly available to encourage further research and development for Thai LLM. |
Batayan: A Filipino NLP benchmark for evaluating Large Language Models (2025.acl-long)
Copied to clipboard
Jann Railey Montalan, Jimson Paulo Layacan, David Demitri Africa, Richell Isaiah S. Flores, Michael T. Lopez Ii, Theresa Denise Magsajo, Anjanette Cayabyab, William Chandra Tjhi
| Challenge: | Recent advances in large language models (LLMs) have demonstrated remarkable capabilities on widely benchmarked high-resource languages. |
| Approach: | They propose a benchmark that systematically evaluates LLMs across three key natural language processing competencies: understanding, reasoning, and generation. |
| Outcome: | The proposed benchmark covers eight tasks covering Tagalog and code-switched Taglish utterances. |
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024.naacl-long)
Copied to clipboard
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Several new LLMs have been introduced necessitating their evaluation on non-English languages. |
| Approach: | They perform a thorough evaluation of the non-English capabilities of SoTA LLMs by comparing them on the same set of multilingual datasets. |
| Outcome: | The proposed model outperforms models on multilingual datasets on 22 languages including low-resource African languages. |