| Challenge: | Pretrained contextual representation models have pushed forward the state-of-the-art on many NLP tasks. |
| Approach: | They propose to use a model that is pretrained on 104 languages for cross-lingual transfer. |
| Outcome: | The proposed model performs well on 5 NLP tasks covering 39 languages from various language families. |
Similar Papers
Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages. |
| Approach: | They propose to use multilingual BERT to enable zero-shot cross-lingual transfer of syntactic knowledge between different languages by generating grammatical relations in 24 different languages. |
| Outcome: | The results show that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms. |
Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. |
| Approach: | They propose to use parallel corpora and rotational alignment methods to improve transfer performance in a zero-shot setting. |
| Outcome: | The proposed method improves rotation-based alignment on Name Entity Recognition and Semantic Slot Filling tasks. |
Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) does not use any crosslingual signal during training. |
| Approach: | They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation. |
| Outcome: | The proposed setup with pretrained models with three languages shows that it works well. |
Finding Universal Grammatical Relations in Multilingual BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has found that multilingual masked language models learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. |
| Approach: | They propose an unsupervised method to find syntactic tree distances in languages other than English and that these subspaces are approximately shared across languages. |
| Outcome: | The proposed method shows that mBERT learns representations of syntactic dependency labels, in the form of clusters, which largely agree with the Universal Dependencies taxonomy. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
How Multilingual is Multilingual BERT? (P19-1)
Copied to clipboard
| Challenge: | Existing studies have shown that deep, contextualized language models can encode syntactic and named entity information, but they have focused on what models trained on English capture about English. |
| Approach: | They propose a multilingual model pre-trained from monolingual Wikipedia corpora . they show that multilingual BERT is surprisingly good at zero-shot cross-lingual model transfer . |
| Outcome: | The proposed model can find translation pairs, but it exhibits systematic deficiencies affecting certain language pairs. |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. |
| Approach: | They propose to use a layer ablation technique to create a multilingual model that is viewed as a stacking of two sub-networks: a language-agnostic encoder and a task-specific predictor. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer for many languages. |
Syntax-augmented Multilingual BERT for Cross-lingual Transfer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained multilingual text encoders capture language syntax, helping cross-lingual transfer. |
| Approach: | They provide language syntax and train mBERT to encode universal dependency tree structure. |
| Outcome: | The proposed model improves cross-lingual transfer on PAWS-X and MLQA benchmarks by 1.4 and 1.6 points on average across all languages. |
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual transformers are less effective in resource-lean scenarios and for distant languages. |
| Approach: | They propose to use massively multilingual transformers to pretrain languages . they show that MMTs are less effective in resource-lean scenarios and distant languages if they are pre-trained via language modeling . |
| Outcome: | The proposed model is less effective in resource-lean scenarios and for distant languages than cross-lingual word embeddings. |
Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations (D19-61)
Copied to clipboard
| Challenge: | Pretrained sentence representations have set the new state of the art in many language understanding tasks. |
| Approach: | They propose to use a multilingual corpus to train deep bidirectional sentence representations that are fully lexicalized to allow for the development of an unsupervised universal dependency parser. |
| Outcome: | The proposed approach outperforms the best CoNLL 2018 systems in all of the shared task’s six truly low-resource languages while using a single system. |