Deep Exploration of Cross-Lingual Zero-Shot Generalization in Instruction Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on instruction tuning to show cross-lingual generalization . a novel non-English meta-dataset is used to study instruction tuning . |
| Approach: | They perform instruction tuning individually for two distinct language meta-datasets and assess the performance on unseen tasks in a non-English language. |
| Outcome: | The proposed model outperforms baseline training in English and Korean by 20.7% and 13.6%. |
Similar Papers
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show zero-shot performance on unseen multimodal tasks . instruction tuning has yet to be explored for vision and multimodal task. |
| Approach: | They propose a multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. |
| Outcome: | The proposed model performs well on unseen multimodal tasks and is highly scalable. |
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Instruction tuning is emerging in NLP, but has not been explored for dialogue-related tasks. |
| Approach: | They propose an instruction tuning framework for dialogue that leverages natural language instructions with language models to induce zero-shot generalization on unseen tasks. |
| Outcome: | The proposed framework enables good zero-shot performance on unseen datasets and tasks such as dialogue evaluation and intent detection. |
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that instruction tuning is effective in instruction learning for unseen tasks, but it relies on a large amount of human-annotated samples, which restricts its generalization. |
| Approach: | They propose an instruction tuning technique which fine-tunes a pre-trained language model on a massive collection of tasks described via human-craft instructions and then tests its generalization ability on unseen tasks. |
| Outcome: | The proposed method improves IT performance versus labeled data and training tasks by constructing pseudo-labeled data from unlabele . data is used to build a model that can learn from human instructions for zero-shot generalization on unseen tasks. |
The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction Tuning (2025.findings-acl)
Copied to clipboard
Bingxiang He, Ning Ding, Cheng Qian, Jia Deng, Ganqu Cui, Lifan Yuan, Haiwen Hong, Huan-ang Gao, Longtao Huang, Hui Xue, Huimin Chen, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing work on instruction tuning has focused on task level, without considering that tasks are artificially defined and, to LLMs, merely consist of tokens and representations. |
| Approach: | They propose a training data arrangement framework that allows for continual learning and loss reduction. |
| Outcome: | The proposed framework promotes continual learning and loss reduction on unseen tasks. |
Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models (2025.findings-acl)
Copied to clipboard
Ahmed Elshabrawy, Thanh-Nhi Nguyen, Yeeun Kang, Lihan Feng, Annant Jain, Faadil Abdullah Shaikh, Jonibek Mansurov, Mohamed Fazli Mohamed Imam, Jesus-German Ortiz-Barajas, Rendi Chevi, Alham Fikri Aji
| Challenge: | Large Language Models excel in zero-shot and few-shot tasks, but their architecture makes them difficult to use. |
| Approach: | They adapt Large Language Models (LLMs) for zero-shot generalization using Statement Tuning . they find encoders can achieve zero- shot cross-lingual generalization . |
| Outcome: | The proposed model generalizes well across languages while being more efficient. |
X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models respond well in high-resource languages but struggle in low-resourced languages. |
| Approach: | They propose a method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. |
| Outcome: | The proposed method builds a large-scale cross-lingual instruction tuning dataset on 10 languages. |
Multilingual Instruction Tuning With Just a Pinch of Multilinguality (2024.findings-acl)
Copied to clipboard
| Challenge: | Using multilingual instruction tuning, large language models can be used to follow instructions in multiple languages . a multilingual model can be tuned on a wide range of languages, yet most datasets are limited to English . |
| Approach: | They investigate how multilinguality during instruction tuning affects instruction-following across languages . they find that only 40 multilingual examples improve multilingual instruction- follow . |
| Outcome: | The results show that multilingual models perform better on multilingual mixtures compared to monolingual models . the results suggest that building multilingual instruction-tuned models can be done with only 2-4 languages . |
Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions? (2024.emnlp-main)
Copied to clipboard
Alexander Weber, Klaudia Thellmann, Jan Ebert, Nicolas Flores-Herr, Jens Lehmann, Michael Fromm, Mehdi Ali
| Challenge: | a study of multilingual pre-trained LLMs on parallel instruction-tuning benchmarks shows that instruction-following models can be used across languages by up to 9.9%. |
| Approach: | They conduct an extensive study of the performance of multilingual pre-trained LLMs instruction-tuned on parallel instruction-uning datasets. |
| Outcome: | The proposed model improves cross-lingual instruction following capabilities by 9.9% on a large and mid-sized LLM on parallel instruction-tuning datasets. |
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | generative multilingual models fine-tuned on English forget to generate non-English data when labeled data is only available in English . generative models fine tuned on English fail to generate multilingual summarization tasks when labeling data is available in other languages . |
| Approach: | They propose to use prompt tuning to overcome catastrophic forgetting in a generative task in . they assume a strict setting with no parallel data or machine translation . |
| Outcome: | The proposed method can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. |
Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on zero-shot cross-lingual transfer . mBERT, mBART and mT5 provide high-quality representations for texts in various languages . |
| Approach: | They propose to use mBART and NLLB-200 to finetune a multilingual pretrained language model on input-output pairs in one language and use it to make task predictions for inputs in other languages. |
| Outcome: | The proposed approach significantly reduces generation in the wrong language with full finetuning and can be competitive in some cases. |