An Isotropy Analysis in the Multilingual BERT Embedding Space (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored the advantages of multilingual pre-trained models in capturing shared linguistic knowledge. |
| Approach: | They investigate the anisotropic embedding space and outlier dimensions of the multilingual BERT model for two known issues of the monolingual models. |
| Outcome: | The proposed model has no outlier dimension and has highly anisotropic space . the results show that increasing the isotropy of multilingual space can improve its representation power and performance, similar to what had been observed for monolingual CWRs on semantic similarity tasks. |
Similar Papers
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that contextual language models display outlier dimensions . this is true for monolingual and multilingual models, but little work has been done on multilingual contexts . |
| Approach: | They investigate outlier dimensions and their relationship to anisotropy in multilingual contexts . they focus on cross-lingual semantic similarity tasks . |
| Outcome: | The proposed model improves on cross-lingual semantic similarity tasks. |
How Multilingual is Multilingual BERT? (P19-1)
Copied to clipboard
| Challenge: | Existing studies have shown that deep, contextualized language models can encode syntactic and named entity information, but they have focused on what models trained on English capture about English. |
| Approach: | They propose a multilingual model pre-trained from monolingual Wikipedia corpora . they show that multilingual BERT is surprisingly good at zero-shot cross-lingual model transfer . |
| Outcome: | The proposed model can find translation pairs, but it exhibits systematic deficiencies affecting certain language pairs. |
BERT is Not an Interlingua and the Bias of Tokenization (D19-61)
Copied to clipboard
| Challenge: | Cananical Correlation Analysis (CCA) of the internal representations of a pre- trained, multilingual BERT model reveals that the model partitions representations for each language rather than using a common, shared, interlingual space. |
| Approach: | They propose to use a multilingual BERT model to partition representations for each language rather than using a common, shared, interlingual space. |
| Outcome: | The results show that the model partitions representations for each language rather than using a common, shared, interlingual space. |
Is anisotropy really the cause of BERT embeddings not being semantic? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to train contextual language models for NLP use a lightweight approach called bi-encoder, which takes two sentences as input, but does not perform well with vanilla pre-trained Transformers. |
| Approach: | They conduct a set of experiments to improve our understanding of the lack of semantic isometry in contextualized word representations in BERT. |
| Outcome: | The proposed approach does not perform well with vanilla pre-trained Transformers. |
Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) does not use any crosslingual signal during training. |
| Approach: | They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation. |
| Outcome: | The proposed setup with pretrained models with three languages shows that it works well. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In addition, information on part-of-speech tagging is spread over different parts of the network and the pipeline might not be as neat as it seems. |
| Approach: | They propose to probe Dutch BERT-based model and multilingual BERT model for Dutch NLP tasks to see if this holds true for other languages. |
| Outcome: | The proposed model is based on a Dutch model and a multilingual model for Dutch NLP tasks. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses (2021.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. |
| Approach: | They propose a setup for analyzing lexical polysemy knowledge in pre-trained language models and multilingual BERT models by analyzing different sense distributions and controlling for parameters that are highly correlated with polysyntax. |
| Outcome: | The proposed model can be used to analyze lexical polysemy in English, French, Spanish, and Greek and in multilingual BERT. |
Extending Multilingual BERT to Low-Resource Languages (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual BERT (M-BERT) has been a huge success in both supervised and zero-shot cross-lingual transfer learning. |
| Approach: | They propose a simple but effective approach to extend multilingual BERT to any new language and show an increase in F1 on M-BERT and new languages. |
| Outcome: | The proposed approach improves on languages already in M-BERT and out of it on other languages. |