Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks. |
| Approach: | They propose a method that uses soft-prompt tuning to tune for language adaptation . prompt tuning outperforms continuously trained baselines on two benchmarks . |
| Outcome: | The proposed approach outperforms baselines on two text classification benchmarks while utilizing 0.28% of tuned parameters. |
Similar Papers
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models enable zero-shot learning even for unseen languages . current multilingual model covers only a small subset of the world's languages - due to data sparsity, they are not likely to obtain good results for many lowresource languages. |
| Approach: | They ask: how does the number of pretraining languages influence zero-shot learning for unseen languages? do the findings change if the languages used for pretraining are all related? |
| Outcome: | The results show that pretrained models can zero-shot learn for unseen languages even for limited amounts even for low-resource languages. |
Cross-lingual Few-Shot Learning on Unseen Languages (2022.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated the ability to obtain good performance on downstream tasks with limited examples in resource-rich languages. |
| Approach: | They propose to use a downstream sentiment analysis task to analyze the effectiveness of several few-shot learning strategies across 12 languages, including 8 unseen languages, to compare results. |
| Outcome: | The proposed model, XLM-R, gives the best performance on a task with few examples in resource-rich languages. |
Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. |
| Approach: | They do cross-lingual evaluation using prompt tuning and compare it with fine-tuning . prompt tuning achieves much better cross-linguistic transfer than fine- tuning . |
| Outcome: | The results show that prompt tuning achieves better cross-lingual transfer than fine-tuning across datasets, with only 0.1% to 0.3% tuned parameters. |
Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages (2023.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that the pre-training in English does not transfer well to other languages in a zero-shot setting. |
| Approach: | They propose a simple yet efficient approach to adapt VLP to unseen languages using MPLM. |
| Outcome: | The proposed approach outperforms state-of-the-art models without large parallel corpora across three tasks. |
Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on monolingual prompts, but we study multilingual prompt for multilingual models. |
| Approach: | They propose a model that uses a unified prompt for all languages, called UniPrompt, to alleviate the effort of designing different prompts for multiple languages. |
| Outcome: | The proposed model outperforms baseline models in the zero-shot cross-lingual setting. |
Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning (2022.coling-1)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models have shown impressive performance on several downstream tasks for both high-resourced and low-resource languages. |
| Approach: | They propose to apply multilingual adaptive fine-tuning to 17 most-resourced African languages and three other high-resource languages to encourage cross-lingual transfer learning. |
| Outcome: | The proposed approach is competitive to LAFT on individual languages while requiring significantly less disk space. |
Modular Monolingual Adaptation using Pretrained Language Models (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing approaches to building monolingual models for low-resource languages require a full model tuning process. |
| Approach: | They propose a modular approach to build monolingual models for low-resource languages by finetuning the whole model on the target language. |
| Outcome: | The proposed model improves on natural language understanding tasks on Scottish Gaelic, Irish, and Quechua with Quechuan being a very low-resource language. |
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models (LMs) have a surprising ability to perform zero-shot learning. |
| Approach: | They propose to fine-tune pre-trained language models to optimize the zero-shot learning objective by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering format. |
| Outcome: | The proposed model outperforms a same-sized QA model and the previous SOTA zero-shot learning system on unseen tasks. |
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | generative multilingual models fine-tuned on English forget to generate non-English data when labeled data is only available in English . generative models fine tuned on English fail to generate multilingual summarization tasks when labeling data is available in other languages . |
| Approach: | They propose to use prompt tuning to overcome catastrophic forgetting in a generative task in . they assume a strict setting with no parallel data or machine translation . |
| Outcome: | The proposed method can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. |