Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh (2025.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. |
| Approach: | They propose to open-source a large-scale instruction-following dataset covering key institutional and cultural knowledge relevant to Kazakhstan. |
| Outcome: | The proposed dataset improves LLMs’ understanding of procedural, legal, and structural governance topics. |
Similar Papers
Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai (2024.acl-srw)
Copied to clipboard
| Challenge: | Xue et al., 2024) have demonstrated that large language models can perform at human level across multitudes of tasks and domains. |
| Approach: | They propose a seed-free framework for generating synthetic instruction-tuning data that incorporates fluency, diversity, and cultural context. |
| Outcome: | The proposed framework achieves competitive performance using only 5,000 instructions compared to state-of-the-art Thai LLMs trained on hundreds of thousands of instructions. |
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024.acl-long)
Copied to clipboard
Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing datasets in the English language are mostly in the realm of instruction fine-tuning . aya dataset, the Aya Collection, and the AYa Evaluation Suite are key resources . |
| Approach: | They aim to build a human-curated instruction-following dataset spanning 65 languages . they work with fluent speakers of languages from around the world to collect natural instances of instructions and completions . |
| Outcome: | The goal is to build a human-curated instruction-following dataset spanning 65 languages. |
WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai (2025.emnlp-main)
Copied to clipboard
Peerat Limkonchotiwat, Pume Tuchinda, Lalita Lowphansirikul, Surapon Nonesung, Panuthep Tasawong, Alham Fikri Aji, Can Udomcharoenchaikit, Sarana Nutanong
| Challenge: | Existing benchmarks for large language models rely on translations, missing cultural and domain specificity. |
| Approach: | They present a human-authored dataset for evaluation and instruction tuning in Thai . findings highlight need for culturally and professionally grounded instruction data . |
| Outcome: | a human-authored dataset for evaluation and instruction tuning in Thai outperforms translation-based models . findings highlight need for culturally and professionally grounded instruction data . |
LongForm: Effective Instruction Tuning with Reverse Instructions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on instruction tuning relies on expensive human annotation and crowd-sourced datasets with alignment issues. |
| Approach: | They propose a method to generate instructions via LLMs from human-written corpus examples using reverse instructions. |
| Outcome: | The proposed method outperforms larger language models without instruction tuning on tasks such as story/recipe generation and long-form question answering. |
RECOST: External Knowledge Guided Data-efficient Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Considering the high computing power overhead, data-efficient instruction tuning is proposed to reduce the training data size. |
| Approach: | They propose a framework to improve instruction tuning by integrating external knowledge into a single pipeline. |
| Outcome: | The proposed method achieves better results with only 1% of the full dataset. |
Dynamics of Instruction Fine-Tuning for Chinese Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Instruction tuning is a burgeoning method to elicit the general intelligence of Large Language Models. |
| Approach: | They investigate the effects of data quantity, model size, and data construction methods on instruction tuning for Chinese LLMs. |
| Outcome: | The proposed model includes over 40,000 high-quality instruction instances covering ten underlying abilities. |
Demystifying Instruction Mixing for Fine-tuning Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Instruction tuning is effective for aligning large language models with human instructions, but the procedure to optimizing the mixing of instruction datasets is still unclear. |
| Approach: | They categorize instructions into three primary types: NLP downstream tasks, coding, and general chat. |
| Outcome: | The proposed method improves performance of large language models (LLMs) but it is difficult to combine different instruction datasets to optimize overall performance. |
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024.acl-long)
Copied to clipboard
Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, Ji-Rong Wen, Zhicheng Dou
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks, but their application to information retrieval tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. |
| Approach: | They propose to use instruction tuning to enhance LLMs' proficiency in IR tasks by combining a dataset with manually written templates to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions. |
| Outcome: | The proposed model can be used to perform query understanding, document understanding, and query-document relationship understanding tasks. |
An Empirical Study of Instruction-tuning Large Language Models in Chinese (2023.findings-emnlp)
Copied to clipboard
| Challenge: | emergence of ChatGPT validates the potential of large language models (LLMs) in artificial general intelligence (AGI) however, the closed source of LLMs coupled with the requirement for massive computing resources has deterred researchers from reaching the LLM training stage. |
| Approach: | They propose to use Chinese instruction-tuning LLMs as a cookbook for customizing LLM models that can better respond to Chinese instructions. |
| Outcome: | The proposed LLM can be used to customize Chinese LLMs that can better respond to Chinese instructions. |
X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models respond well in high-resource languages but struggle in low-resourced languages. |
| Approach: | They propose a method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. |
| Outcome: | The proposed method builds a large-scale cross-lingual instruction tuning dataset on 10 languages. |