Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure (2022.acl-long)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models have shown impressive cross-lingual ability. |
| Approach: | They argue that cross-language ability comes from commonality between languages . they create an artificial language by modifying property in source language . |
| Outcome: | The proposed model can be implemented in multilingual and low-resource language scenarios without cross-lingual supervision or aligned data. |
Similar Papers
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Multilingual Language Models have shown a strong ability to transfer knowledge across languages. |
| Approach: | They examine factors contributing to the ability of MLLMs to perform zero-shot cross-lingual transfer . they identify consensuses among studies with consistent findings and resolve conflicts . |
| Outcome: | The authors outline and discuss factors that contribute to the ability of MLLMs to perform zero-shot cross-lingual transfer. |
Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on multilingual models have focused on their cross-lingual transfer behavior . a recent study examined multilingual model learning from the multilingual pretraining signal . |
| Approach: | They analyze checkpoints during multilingual pretraining to identify when models acquire in-language and cross-lingual abilities. |
| Outcome: | The proposed model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Subword Evenness (SuE) as a Predictor of Cross-lingual Transfer to Low-resource Languages (2022.emnlp-main)
Copied to clipboard
| Challenge: | English is the most natural choice for cross-lingual transfer, but it is often not the best choice for low-resource languages. |
| Approach: | They propose to use pre-trained multilingual models to improve performance in low-resource languages via cross-lingual transfer. |
| Outcome: | The results show that languages written in non-Latin and non-alphabetic scripts are the best choices for improving performance on Masked Language Modelling tasks in a diverse set of 30 low-resource languages. |
Analysis of Multi-Source Language Training in Cross-Lingual Transfer (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual transfer (XLT) methods address data scarcity problem . cross-linguistic transfer (xLT) techniques are effective at fine-tuning multilingual LMs . |
| Approach: | They propose to use multiple source languages to improve XLT by fine-tuning multilingual models . they propose to employ arbitrary combinations of source languages for XL to improve performance . |
| Outcome: | The proposed technique improves performance on language-agnostic or task-specific features by using multiple source languages. |
Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets (2024.lrec-main)
Copied to clipboard
| Challenge: | Multilingual Language Models (MLLMs) exhibit robust cross-lingual transfer capabilities for downstream tasks such as Named Entity Recognition (NER) challenges persist in MLLM implementations that are not cross-linguistically robust. |
| Approach: | They evaluate two well-known MLLMs on 13 pairs of languages with a geographic, genetic, or borrowing relationship. |
| Outcome: | The proposed models show that they can leverage information acquired in a source language and apply it to a target language. |
Finding Universal Grammatical Relations in Multilingual BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has found that multilingual masked language models learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. |
| Approach: | They propose an unsupervised method to find syntactic tree distances in languages other than English and that these subspaces are approximately shared across languages. |
| Outcome: | The proposed method shows that mBERT learns representations of syntactic dependency labels, in the form of clusters, which largely agree with the Universal Dependencies taxonomy. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |