Do Explicit Alignments Robustly Improve Multilingual Encoders? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Explicit alignment objectives based on bitexts like Europarl and MultiUN have been shown to improve cross-lingual representations. |
| Approach: | They propose a new contrastive alignment objective that can better utilize bitexts . they propose to use a random sample of 1 million pair subset of OPUS data . |
| Outcome: | The proposed objective outperforms existing alignment objectives on a random 1 million pair subset of the OPUS dataset. |
Similar Papers
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained cross-lingual encoders have proven impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resourced languages. |
| Approach: | They propose a method to align multilingual encoders using two explicit alignment objectives that align the multilingual representations at different granularities. |
| Outcome: | The proposed method achieves gains of up to 1.1 average F1 score on sequence tagging and 27.3 average accuracy on retrieval over the XLM-R-large model. |
When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation? (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-training for many-to-many neural machine translation use manual cleaning of bilingual dictionaries, which are unavailable for most language pairs. |
| Approach: | They propose a word-level contrastive objective to leverage word alignments for many-to-many neural machine translation (NMT) Empirical results show that this leads to 0.8 BLEU gains for several language pairs. |
| Outcome: | Empirical results show that the proposed objective leads to 0.8 BLEU gains for several language pairs. |
Multilingual BERT Post-Pretraining Alignment (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work improves on the success of monolingual pretrained language models by adding cross-lingual tasks that always involve English. |
| Approach: | They propose a method to align multilingual contextual embeddings as a post-pretraining step for improved cross-lingual transferability of pretrained language models. |
| Outcome: | The proposed model outperforms XLM-R_Base on translation-train tasks while using less parallel data and fewer parameters. |
Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning (2025.acl-short)
Copied to clipboard
| Challenge: | Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. |
| Approach: | They propose to use image captions to implicitly align text representations between languages to make them usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
| Outcome: | The proposed approach is usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space (2021.emnlp-main)
Copied to clipboard
| Challenge: | Cross-lingual language models house representations for many different languages in the same space. |
| Approach: | They investigate linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. |
| Outcome: | The results show that word order agreement and agreement in morphological complexity are strongest predictors of cross-linguality. |
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that multilingual generative models exhibit a strong language bias toward high-resource languages. |
| Approach: | They propose a cross-lingual alignment framework exploiting pairs of translation sentences to improve cross-linguistic abilities. |
| Outcome: | The proposed framework improves cross-lingual abilities and mitigates performance gap. |
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning (2023.acl-long)
Copied to clipboard
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
| Challenge: | XY-LENT: X-Y bitext enhanced Language ENcodings achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands. |
| Approach: | They propose a method for building multilingual representation models that are competitive with existing models and more parameter efficient. |
| Outcome: | The proposed model outperforms XLM-R XXL and is 5x and 6x smaller respectively. |
Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. |
| Approach: | They propose to use parallel corpora and rotational alignment methods to improve transfer performance in a zero-shot setting. |
| Outcome: | The proposed method improves rotation-based alignment on Name Entity Recognition and Semantic Slot Filling tasks. |
Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) does not use any crosslingual signal during training. |
| Approach: | They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation. |
| Outcome: | The proposed setup with pretrained models with three languages shows that it works well. |