When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Language models are a new standard to build state-of-the-art NLP systems. |
| Approach: | They compare multilingual and monolingual models on unseen languages . they show that some languages benefit from transfer learning whereas others don't . |
| Outcome: | The proposed model behaves in multiple ways on unseen languages, while others fail to transfer . the results provide a promising direction towards making multilingual models useful for a new set of unseense languages. |
Similar Papers
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2021.emnlp-main)
Copied to clipboard
| Challenge: | Massively multilingual language models offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks, but there is a profound performance gap between resource-rich and resource-poor target languages. |
| Approach: | They propose a series of data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to low-resource languages and unseen scripts. |
| Outcome: | The proposed methods improve learning of the new dedicated embedding matrix in the target language and for low-resource languages written in unseen scripts. |
Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | a large number of pretraining corpora are not publicly available, and it is unclear how much foreign language data exists in monolingual models. |
| Approach: | They propose to use English pretraining corpora to analyze their language composition . they find that even when less than 1% of data is not English, it facilitates cross-lingual transfer . |
| Outcome: | The proposed model is not truly monolingual when pretrained at scale, the authors show . they show that even when less than 1% of data is not English, it facilitates cross-lingual transfer . |
The Less the Merrier? Investigating Language Representation in Multilingual Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual models can be used to integrate multiple languages into one model and use cross-language transfer learning to improve performance for different NLP tasks. |
| Approach: | They propose to include languages in popular multilingual models and to use cross-language transfer learning to improve performance for different NLP tasks. |
| Outcome: | The proposed models perform better on downstream tasks for seen and unseen languages than community-centered models for low-resource languages. |
Cross-lingual Few-Shot Learning on Unseen Languages (2022.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated the ability to obtain good performance on downstream tasks with limited examples in resource-rich languages. |
| Approach: | They propose to use a downstream sentiment analysis task to analyze the effectiveness of several few-shot learning strategies across 12 languages, including 8 unseen languages, to compare results. |
| Outcome: | The proposed model, XLM-R, gives the best performance on a task with few examples in resource-rich languages. |
Massively Multilingual Transfer for NER (P19-1)
Copied to clipboard
| Challenge: | Existing approaches for cross-lingual transfer use a single source language, but there are exceptions. |
| Approach: | They propose two techniques for modulating the transfer, suitable for zero-shot or few-shot learning, respectively. |
| Outcome: | The proposed methods are much more effective than baseline models and rival oracle selection of the single best individual model. |
Few-Shot Learning Translation from New Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows strong transfer learning capability to unseen languages in sequence-to-sequence neural networks . current transfer learning methods require much less downstream task data than would otherwise be required. |
| Approach: | They first train word embeddings models on varying amounts of data and plug them into a machine translation model. |
| Outcome: | The proposed model can learn Flores with only 500 parallel sentences and 31,250 sentences of monolingual data, and it can exceed 15 BLEU on unseen languages. |
Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that results from high-resource languages cannot be easily transferred to realistic, low-resourced scenarios. |
| Approach: | They analyse performance of multilingual transformer models using available resources for Hausa, isiXhosa and NER and topic classification. |
| Outcome: | The proposed models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. |
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models enable zero-shot learning even for unseen languages . current multilingual model covers only a small subset of the world's languages - due to data sparsity, they are not likely to obtain good results for many lowresource languages. |
| Approach: | They ask: how does the number of pretraining languages influence zero-shot learning for unseen languages? do the findings change if the languages used for pretraining are all related? |
| Outcome: | The results show that pretrained models can zero-shot learn for unseen languages even for limited amounts even for low-resource languages. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
Cross-lingual Transfer of Monolingual Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual learning using multilingual models cast doubt on shared vocabulary and joint pre-training . et al. (2005) show that model knowledge learned in the source language enhances the learning of the target language independently of language proximity. |
| Approach: | They propose a method for transferring monolingual models to other languages through continuous pre-training and investigate their results in English. |
| Outcome: | The proposed method outperforms a model trained from scratch in the GLUE benchmark for English . it shows that model knowledge from the source language enhances the learning of syntactic and semantic knowledge in english. |