Probing Multilingual BERT for Genetic and Typological Signals (2020.coling-main)
Copied to clipboard
| Challenge: | Recent cross-lingual models provide representations for about 100 languages and vary in their training objectives. |
| Approach: | They probe the layers in multilingual BERT for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations. |
| Outcome: | The proposed model is best explained by phylogenetic and worst by structural factors and correlates with published ranked lists based on linguistic approaches. |
Similar Papers
Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages. |
| Approach: | They propose to use multilingual BERT to enable zero-shot cross-lingual transfer of syntactic knowledge between different languages by generating grammatical relations in 24 different languages. |
| Outcome: | The results show that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms. |
Finding Universal Grammatical Relations in Multilingual BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has found that multilingual masked language models learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. |
| Approach: | They propose an unsupervised method to find syntactic tree distances in languages other than English and that these subspaces are approximately shared across languages. |
| Outcome: | The proposed method shows that mBERT learns representations of syntactic dependency labels, in the form of clusters, which largely agree with the Universal Dependencies taxonomy. |
Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) does not use any crosslingual signal during training. |
| Approach: | They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation. |
| Outcome: | The proposed setup with pretrained models with three languages shows that it works well. |
Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages (2023.findings-emnlp)
Copied to clipboard
Elena Ruzzetti, Federico Ranaldi, Felicia Logozzo, Michele Mastromattei, Leonardo Ranaldi, Fabio Zanzotto
| Challenge: | a recent study examined how models for typologically similar languages encode structural information. |
| Approach: | They propose to layer-wise compare transformers for typologically similar languages to observe similarities . they use a domain adaptation on semantically equivalent texts to measure similarity . |
| Outcome: | The proposed model outperforms all other models on unseen sentences . the proposed model is based on a typologically similar language . |
BERT for Monolingual and Cross-Lingual Reverse Dictionary (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to find the proper word for a word are based on the surface form of words, but they cannot extract the semantic meaning. |
| Approach: | They propose a method to make BERT generate the target word for this task . cross-lingual reverse dictionary is the task to find the proper target word . |
| Outcome: | The proposed method can generate the target word for cross-lingual reverse dictionary task even without the parallel corpus. |
From BERT‘s Point of View: Revealing the Prevailing Contextual Differences (2022.findings-acl)
Copied to clipboard
| Challenge: | BERTology is a new approach to understanding the inner workings of large pretraining language models. |
| Approach: | They propose to invert the probing design to analyze the prevailing differences and clusters in BERT’s high dimensional space by extracting coarse features from masked token representations and predicting them by probing models with access to only partial information. |
| Outcome: | The proposed method extracts coarse features from masked token representations and predicts them by probing models with access to only partial information. |
Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. |
| Approach: | They propose to use parallel corpora and rotational alignment methods to improve transfer performance in a zero-shot setting. |
| Outcome: | The proposed method improves rotation-based alignment on Name Entity Recognition and Semantic Slot Filling tasks. |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. |
| Approach: | They propose to use a layer ablation technique to create a multilingual model that is viewed as a stacking of two sub-networks: a language-agnostic encoder and a task-specific predictor. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer for many languages. |
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (D19-1)
Copied to clipboard
| Challenge: | Pretrained contextual representation models have pushed forward the state-of-the-art on many NLP tasks. |
| Approach: | They propose to use a model that is pretrained on 104 languages for cross-lingual transfer. |
| Outcome: | The proposed model performs well on 5 NLP tasks covering 39 languages from various language families. |
What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In addition, information on part-of-speech tagging is spread over different parts of the network and the pipeline might not be as neat as it seems. |
| Approach: | They propose to probe Dutch BERT-based model and multilingual BERT model for Dutch NLP tasks to see if this holds true for other languages. |
| Outcome: | The proposed model is based on a Dutch model and a multilingual model for Dutch NLP tasks. |