The Less the Merrier? Investigating Language Representation in Multilingual Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual models can be used to integrate multiple languages into one model and use cross-language transfer learning to improve performance for different NLP tasks. |
| Approach: | They propose to include languages in popular multilingual models and to use cross-language transfer learning to improve performance for different NLP tasks. |
| Outcome: | The proposed models perform better on downstream tasks for seen and unseen languages than community-centered models for low-resource languages. |
Similar Papers
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual language models are widely used to extend NLP systems to low-resource languages. |
| Approach: | They pre-train over 10,000 monolingual and multilingual language models for over 250 languages including multiple language families that are under-studied in NLP. |
| Outcome: | The results show that adding multilingual data improves low-resource language modeling performance, similar to increasing low-source dataset sizes by up to 33%. |
A Survey of Multilingual Models for Automatic Speech Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems have achieved human-like performance for a few languages, but the majority of the world’s languages do not have usable systems due to the lack of large speech datasets to train these models. |
| Approach: | They propose to use unlabeled speech data to build multilingual ASR models that can be used for improved performance on low-resource languages. |
| Outcome: | The proposed models can be used to improve performance on low-resource languages by using unlabeled speech data. |
The Linguistic Connectivities Within Large Language Models (2025.findings-acl)
Copied to clipboard
Dan Wang, Boxi Cao, Ning Bian, Xuanang Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Le Sun, Shanshan Jiang, Bin Dong, Xianpei Han
| Challenge: | Recent studies have discovered notable disparities in their performance across different languages. |
| Approach: | They conduct a systematic investigation into the behaviors of large language models across 27 different languages on 3 different scenarios and reveals a Linguistic Map correlates with the richness of available resources and linguistic family relations. |
| Outcome: | The proposed model demonstrates that there are significant disparities in performance across languages across 27 different languages on 3 different scenarios. |
Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multilingual pretrained models have proven effective at zero-shot transfer to a wide variety of languages, but this transfer is not universal, with many languages not currently understood by multilingual approaches. |
| Approach: | They propose a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model. |
| Outcome: | The proposed model can detect which languages are not well understood by a multilingual model on 350 low-resource languages. |
Targeted Multilingual Adaptation for Low-resource Language Families (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Massively multilingual models are known to have limited utility in any one language, and to perform poorly on low-resource languages. |
| Approach: | They propose to adapt a pre-trained multilingual model to a language family and evaluate its performance on two downstream tasks and 11 evaluation languages. |
| Outcome: | The proposed model outperforms mono- and multilingual models on two downstream tasks and 11 evaluation languages. |
Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan (2021.findings-acl)
Copied to clipboard
Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, Marta Villegas
| Challenge: | Multilingual language models have been a crucial breakthrough for under-resourced languages . however, the superiority of language-specific models has already been proven for underresourced ones . |
| Approach: | They propose to build a monolingual monolingual model that is comparable to state-of-the-art large multilingual models. |
| Outcome: | The proposed model consistently outperforms state-of-the-art models across tasks and settings. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
The Geometry of Multilingual Language Model Representations (2022.emnlp-main)
Copied to clipboard
| Challenge: | XLM-R models encode language-sensitive information in each language, allowing them to extract features for downstream tasks and cross-lingual transfer learning. |
| Approach: | They evaluate how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. |
| Outcome: | The proposed model can extract features for downstream tasks and cross-lingual transfer learning. |
Discovering Language-neutral Sub-networks in Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that multilingual pre-trained language models transfer well on cross-lingual downstream tasks. |
| Approach: | They conceptualize language neutrality as a function of overlap between language-encoding sub-networks of multilingual models. |
| Outcome: | The proposed model performs well on cross-lingual tasks despite being pre-trained on multiple languages . |
When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Language models are a new standard to build state-of-the-art NLP systems. |
| Approach: | They compare multilingual and monolingual models on unseen languages . they show that some languages benefit from transfer learning whereas others don't . |
| Outcome: | The proposed model behaves in multiple ways on unseen languages, while others fail to transfer . the results provide a promising direction towards making multilingual models useful for a new set of unseense languages. |