BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2023.acl-long)
Copied to clipboard
Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina
| Challenge: | Existing language adaptation strategies for multilingual models are limited to 46 languages . a new language is added to the model to improve zero-shot prompting performance . |
| Approach: | They apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. |
| Outcome: | The proposed model can be extended to other languages without incurring prohibitively large costs. |
Similar Papers
Crosslingual Generalization through Multitask Finetuning (2023.acl-long)
Copied to clipboard
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel
| Challenge: | Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. |
| Approach: | They apply multitask prompted finetuning to pretrained multilingual models and generate variants called BLOOMZ and mT0. |
| Outcome: | The proposed models can generalize to non-English languages that have never been seen before. |
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models enable zero-shot learning even for unseen languages . current multilingual model covers only a small subset of the world's languages - due to data sparsity, they are not likely to obtain good results for many lowresource languages. |
| Approach: | They ask: how does the number of pretraining languages influence zero-shot learning for unseen languages? do the findings change if the languages used for pretraining are all related? |
| Outcome: | The results show that pretrained models can zero-shot learn for unseen languages even for limited amounts even for low-resource languages. |
How Many Languages Make Good Multilingual Instruction Tuning? A Case Study on BLOOM (2025.coling-main)
Copied to clipboard
| Challenge: | Many large language models (LLMs) support many languages, while others only support a few, e.g. the Llama series. |
| Approach: | They present a case study on BLOOM to understand three pertinent factors affecting performance: the number of languages, language exposure, and similarity between training and test languages. |
| Outcome: | The proposed model can be used to perform multilingual tasks on 1 to 52 languages. |
Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks. |
| Approach: | They propose a method that uses soft-prompt tuning to tune for language adaptation . prompt tuning outperforms continuously trained baselines on two benchmarks . |
| Outcome: | The proposed approach outperforms baselines on two text classification benchmarks while utilizing 0.28% of tuned parameters. |
Pre-trained Language Models Can be Fully Zero-Shot Learners (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to pre-trained language models require fine-tuning on labeled datasets or manually constructing proper prompts. |
| Approach: | They propose a nonparametric prompting PLM for fully zero-shot language understanding . they compare it to previous methods for text classification and text entailment . |
| Outcome: | The proposed method outperforms previous methods on diverse tasks. |
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models (LMs) have a surprising ability to perform zero-shot learning. |
| Approach: | They propose to fine-tune pre-trained language models to optimize the zero-shot learning objective by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering format. |
| Outcome: | The proposed model outperforms a same-sized QA model and the previous SOTA zero-shot learning system on unseen tasks. |
Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on monolingual prompts, but we study multilingual prompt for multilingual models. |
| Approach: | They propose a model that uses a unified prompt for all languages, called UniPrompt, to alleviate the effort of designing different prompts for multiple languages. |
| Outcome: | The proposed model outperforms baseline models in the zero-shot cross-lingual setting. |
Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)
Copied to clipboard
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li
| Challenge: | Large-scale generative language models such as GPT-3 are competitive few-shot learners. |
| Approach: | They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities. |
| Outcome: | The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions. |
Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has obtained strong zero-shot results by prompting language models. |
| Approach: | They propose a mining-based approach that uses regular expressions to mine labeled examples from unlabeled corpora and fine tune a pretrained model. |
| Outcome: | The proposed method outperforms prompting on a wide range of tasks when using comparable templates. |
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | generative multilingual models fine-tuned on English forget to generate non-English data when labeled data is only available in English . generative models fine tuned on English fail to generate multilingual summarization tasks when labeling data is available in other languages . |
| Approach: | They propose to use prompt tuning to overcome catastrophic forgetting in a generative task in . they assume a strict setting with no parallel data or machine translation . |
| Outcome: | The proposed method can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. |