Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024.acl-long)
Copied to clipboard
Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing datasets in the English language are mostly in the realm of instruction fine-tuning . aya dataset, the Aya Collection, and the AYa Evaluation Suite are key resources . |
| Approach: | They aim to build a human-curated instruction-following dataset spanning 65 languages . they work with fluent speakers of languages from around the world to collect natural instances of instructions and completions . |
| Outcome: | The goal is to build a human-curated instruction-following dataset spanning 65 languages. |
Similar Papers
UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset (2024.acl-long)
Copied to clipboard
Haoyu Wang, Shuo Wang, Yukun Yan, Xujia Wang, Zhiyu Yang, Yuzhuang Xu, Zhenghao Liu, Liner Yang, Ning Ding, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Open-source large language models (LLMs) have gained strength across diverse fields, but the majority of studies focus on English. |
| Approach: | They propose a knowledge-grounded data augmentation approach to elicit more language-specific knowledge of LLMs by enhancing their ability to serve users from different countries. |
| Outcome: | The proposed method can prune the language-agnostic supervised fine-tuning dataset without any performance degradation. |
Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasets (2024.findings-emnlp)
Copied to clipboard
Sathish Reddy Indurthi, Wenxuan Zhou, Shamil Chollampatt, Ravi Agrawal, Kaiqiang Song, Lingxiao Zhao, Chenguang Zhu
| Challenge: | Advancements in Large Language Models (LLMs) have significantly enhanced instruction-following capabilities, but most IFT datasets are predominantly in English, limiting model performance in other languages. |
| Approach: | They propose a method for collecting multilingual IFT datasets that preserves linguistic naturalness and ensures prompt diversity. |
| Outcome: | Experiments show that LLMs fine-tuned using this method show significant improvements in generative and discriminative tasks. |
Advancing Language Models through Instruction Tuning: Recent Progress and Challenges (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | tutorial addresses three critical questions within the field of instruction tuning: (1) What are the current focal points in instruction tuning research? (2) What are best practices in training an instruction-following model? (3) What new challenges have emerged? |
| Approach: | This tutorial presents a systematic overview of recent advances in instruction tuning. |
| Outcome: | The tutorial covers different stages in model training: supervised fine-tuning, preference optimization, and reinforcement learning. |
M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to collect instruction fine-tuning data are limited due to their toxicity, privacy and toxicity concerns. |
| Approach: | They propose to use a two-step taxonomy to transform a small set of human written instructions into complex and challenging conversations. |
| Outcome: | M2Lingual has 175K conversations across 70 languages with a balanced mix of high, low and mid-resourced languages. |
Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh (2025.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. |
| Approach: | They propose to open-source a large-scale instruction-following dataset covering key institutional and cultural knowledge relevant to Kazakhstan. |
| Outcome: | The proposed dataset improves LLMs’ understanding of procedural, legal, and structural governance topics. |
SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning (2025.acl-long)
Copied to clipboard
Prabhat Pandey, Rupak Vignesh Swaminathan, K V Vijay Girish, Arunasish Sen, Jian. Xie, Grant Strimel, Andreas Schwarz
| Challenge: | Recent years have witnessed significant advancements in integrating speech and audio capabilities into large language models. |
| Approach: | They propose a 50M-example dataset for instruction fine-tuning and pre-training of speech-text large language models (LLMs) the dataset spans five languages and enables a diverse range of speech understanding and controllable speech generation instructions. |
| Outcome: | The proposed dataset outperforms existing speech-text LLMs on instruction-following benchmarks while achieving competitive performance on foundational speech tasks. |
Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions? (2024.emnlp-main)
Copied to clipboard
Alexander Weber, Klaudia Thellmann, Jan Ebert, Nicolas Flores-Herr, Jens Lehmann, Michael Fromm, Mehdi Ali
| Challenge: | a study of multilingual pre-trained LLMs on parallel instruction-tuning benchmarks shows that instruction-following models can be used across languages by up to 9.9%. |
| Approach: | They conduct an extensive study of the performance of multilingual pre-trained LLMs instruction-tuned on parallel instruction-uning datasets. |
| Outcome: | The proposed model improves cross-lingual instruction following capabilities by 9.9% on a large and mid-sized LLM on parallel instruction-tuning datasets. |
How Many Languages Make Good Multilingual Instruction Tuning? A Case Study on BLOOM (2025.coling-main)
Copied to clipboard
| Challenge: | Many large language models (LLMs) support many languages, while others only support a few, e.g. the Llama series. |
| Approach: | They present a case study on BLOOM to understand three pertinent factors affecting performance: the number of languages, language exposure, and similarity between training and test languages. |
| Outcome: | The proposed model can be used to perform multilingual tasks on 1 to 52 languages. |
Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca (2024.findings-eacl)
Copied to clipboard
| Challenge: | Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. |
| Approach: | They employ a dataset and machine translations of it to form multilingual data and use it to tune LLMs. |
| Outcome: | The proposed model is on par or better than a model for each language, and multilingual tuning with downsampled data is as powerful and robust. |
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2023.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions without human supervision. |
| Approach: | They propose to use a dataset of natural language instructions to generate large datasets with no human supervision. |
| Outcome: | The proposed dataset outperforms open-source models on various benchmarks, and is cost-effective. |