Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has shown that multilingual BERT encodes sentences in structurally meaningful ways. |
| Approach: | They analyze how morphosyntactic alignment manifests across embedding spaces of languages . they train classifiers to recover subjecthood of mBERT embedds in transitive sentences . |
| Outcome: | The proposed model encodes a high-order grammatical feature of morphosyntactic alignment across languages . the results show that the classifier distributions reflect the morphological alignment of their training languages based on the results . |
Similar Papers
Finding Universal Grammatical Relations in Multilingual BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has found that multilingual masked language models learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. |
| Approach: | They propose an unsupervised method to find syntactic tree distances in languages other than English and that these subspaces are approximately shared across languages. |
| Outcome: | The proposed method shows that mBERT learns representations of syntactic dependency labels, in the form of clusters, which largely agree with the Universal Dependencies taxonomy. |
Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) does not use any crosslingual signal during training. |
| Approach: | They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation. |
| Outcome: | The proposed setup with pretrained models with three languages shows that it works well. |
Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages. |
| Approach: | They propose to use multilingual BERT to enable zero-shot cross-lingual transfer of syntactic knowledge between different languages by generating grammatical relations in 24 different languages. |
| Outcome: | The results show that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms. |
Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. |
| Approach: | They propose to use parallel corpora and rotational alignment methods to improve transfer performance in a zero-shot setting. |
| Outcome: | The proposed method improves rotation-based alignment on Name Entity Recognition and Semantic Slot Filling tasks. |
Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation (P19-1)
Copied to clipboard
| Challenge: | Pretrained contextual and non-contextual subword embeddings are available in over 250 languages, allowing massively multilingual NLP. |
| Approach: | They compare pretrained contextual and non-contextual subword embeddings with a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. |
| Outcome: | The proposed method outperforms non-contextual embeddings on multilingual named entity recognition and part-of-speech tagging. |
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (D19-1)
Copied to clipboard
| Challenge: | Pretrained contextual representation models have pushed forward the state-of-the-art on many NLP tasks. |
| Approach: | They propose to use a model that is pretrained on 104 languages for cross-lingual transfer. |
| Outcome: | The proposed model performs well on 5 NLP tasks covering 39 languages from various language families. |
From BERT‘s Point of View: Revealing the Prevailing Contextual Differences (2022.findings-acl)
Copied to clipboard
| Challenge: | BERTology is a new approach to understanding the inner workings of large pretraining language models. |
| Approach: | They propose to invert the probing design to analyze the prevailing differences and clusters in BERT’s high dimensional space by extracting coarse features from masked token representations and predicting them by probing models with access to only partial information. |
| Outcome: | The proposed method extracts coarse features from masked token representations and predicts them by probing models with access to only partial information. |
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (D19-60)
Copied to clipboard
| Challenge: | Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of their capabilities. |
| Approach: | They investigate BERT's ability to encode various commonsense features in its embedding space, but are still deficient in many areas. |
| Outcome: | The proposed model improves performance on a downstream commonsense reasoning task while using minimal data. |
Using Paraphrases to Study Properties of Contextual Embeddings (2022.naacl-main)
Copied to clipboard
| Challenge: | Previously, paraphrases have been used to probe whether compositionality is accurately captured by BERT, but we believe they can be used to explore many other questions. |
| Approach: | They propose to use paraphrases as a unique source of data to analyze contextualized embeddings, with a particular focus on BERT. |
| Outcome: | The proposed analysis of paraphrases and paraphrase representations using the Paraphrase Database shows that BERT handles polysemous words, but different representations in many cases. |
Syntax-augmented Multilingual BERT for Cross-lingual Transfer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained multilingual text encoders capture language syntax, helping cross-lingual transfer. |
| Approach: | They provide language syntax and train mBERT to encode universal dependency tree structure. |
| Outcome: | The proposed model improves cross-lingual transfer on PAWS-X and MLQA benchmarks by 1.4 and 1.6 points on average across all languages. |