Papers with mPLMs
FAD-X: Fusing Adapters for Cross-lingual Transfer to Low-Resource Languages (2022.aacl-short)
Copied to clipboard
| Challenge: | Adapter-based tuning is a technique that selectively updates language-specific parameters to adapt to a new language, rather than fine-tuning all shared weights. |
| Approach: | They propose to add light-weight adapters to multilingual pretrained language models (mPLMs) and add language-specific parameters to adapt to a new language. |
| Outcome: | The proposed adapter can enhance cross-lingual transfer from pretrained adapters for well-known named entity recognition and classification benchmarks. |
ContrastiveMix: Overcoming Code-Mixing Dilemma in Cross-Lingual Transfer for Information Retrieval (2024.naacl-short)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have been widely adopted in cross-lingual transfer . however, training mPLMs on code-mixed data is counterproductive . |
| Approach: | They propose a method that disentangles contrastive loss between conflicting objectives . they propose mPLMs that train on code-mixed data for information retrieval . |
| Outcome: | The proposed model outperforms mDPR in cross-lingual transfer tasks . it is effective in sentence-level and token-level zero-shot tasks despite lack of target language data . |
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models encode strong language-specific signals, which are not explicitly provided during pretraining. |
| Approach: | They propose a language similarity measure that induces similarities across languages from mPLMs using multi-parallel corpora. |
| Outcome: | The proposed measure exhibits moderately high correlations with linguistic similarity measures, and more accurate similarity results on low correlation languages. |
TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data (2025.coling-main)
Copied to clipboard
| Challenge: | Existing mPLMs that handle non-transliterated data are not sufficient to train crosslingual models. |
| Approach: | They propose a framework to transliterate related languages into a common script by exploiting existing mPLMs and their tokenizer without any training. |
| Outcome: | The proposed framework can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. |
LangSAMP: Language-Script Aware Multilingual Pretraining (2025.acl-long)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models often avoid using language embeddings, which places a significant burden on token representations to encode all language-specific information. |
| Approach: | They propose a method that incorporates both language and script embeddings into the output of Transformer blocks before passing the final representations to the language modeling head for prediction. |
| Outcome: | The proposed method outperforms the baseline model in zero-shot crosslingual transfer across diverse downstream tasks. |
SERENGETI: Massively Multilingual Language Models for Africa (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretrained models acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning. |
| Approach: | They develop a set of massively multilingual language models that covers 517 African languages and language varieties. |
| Outcome: | The proposed models outperform 4 models that cover 4-23 African languages on eight natural language understanding tasks, achieving 82.27 average F_1. |
mPMR: A Multilingual Pre-trained Machine Reader at Scale (2023.acl-short)
Copied to clipboard
| Challenge: | Existing mPLMs only transfer NLU capability from source to target languages . mPMR allows direct inheritance of multilingual NLU capabilities to downstream tasks . |
| Approach: | They propose a method to guide multilingual pre-trained language models to perform natural language understanding in multiple languages. |
| Outcome: | mPMR enables multilingual pre-trained language models to perform natural language understanding (NLU) in multiple languages. |
TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | The world’s more than 7000 languages are written in at least 293 scripts, which poses a difficulty for multilingual pretrained language models in learning crosslingual knowledge through lexical overlap. |
| Approach: | They propose a framework that optimizes the Transliteration Contrastive Modeling objective to fine-tune an mPLM by contrasting sentences in its training data and transliterations in a unified script. |
| Outcome: | The proposed model outperforms Glot500-m on zero-shot crosslingual transfer tasks while retaining uniformity across scripts. |
Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale multilingual pretrained language models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLm. |
| Approach: | They propose to leverage the learned knowledge from well-performing languages to guide under-performing ones within the same mPLM. |
| Outcome: | The proposed model shows that it can guide under-performing languages while minimizing language-level performance disparities across different mPLMs. |
How Transliterations Improve Crosslingual Alignment (2025.coling-main)
Copied to clipboard
Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob ImaniGooghari, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, Hinrich Schütze
| Challenge: | Recent studies show that post-aligning multilingual pretrained language models improve crosslingual alignment, but it is unclear how and why this is achieved. |
| Approach: | They propose to explicitly evaluate crosslingual alignment by adding transliterations to models using original and transliterated data. |
| Outcome: | The proposed approach improves crosslingual alignment even for random sentences. |
Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have demonstrated remarkable cross-lingual capability of pre-trained language models . however, semantic alignments may be the reason behind such capability but remain under-explored. |
| Approach: | They propose token-level and semantic-level code-switched masked language modeling to improve cross-lingual interactions over mono-mPLMs without parallel sentences. |
| Outcome: | The proposed method outperforms mono-mPLMs on natural language understanding and unsupervised machine translation tasks. |
Multilingual Sentence Transformer as A Multilingual Word Aligner (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual pretrained language models (mPLMs) have shown their effectiveness in multilingual word alignment induction, but these methods usually start from mBERT or XLM-R. |
| Approach: | They propose to fine tune multilingual sentence Transformer LaBSE for alignment induction using parallel corpus and a parallel corpora model. |
| Outcome: | The proposed model outperforms existing models on seven language pairs and achieves new state-of-the-art on zero-shot language pairs. |
Improving Bilingual Lexicon Induction with Cross-Encoder Reranking (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Current methods for bilingual lexicon induction rely on the induction of cross-lingual word embeddings (CLWEs) such as VecMap or mPLMs are not available for multilingual NLP. |
| Approach: | They propose a semi-supervised post-hoc reranking method which combines cross-lingual lexical knowledge from multilingual pretrained language models with original CLWEs. |
| Outcome: | The proposed method outperforms existing methods on two standard benchmarks spanning a wide spectrum of languages and is robust to different CLWEs. |
ScriptMix: Mixing Scripts for Low-resource Language Parsing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work has considered transliteration and vocabulary augmentation, but the consideration of combining the two has been lacking. |
| Approach: | They propose a multilingual pretrained language model that combines two strengths and overcomes the hurdle of combining them. |
| Outcome: | The proposed model improves POS accuracy by 14% and improves DEP LAS score by 5.6%. |
Romanization-based Large-scale Adaptation of Multilingual Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large multilingual pretrained language models are limited by their vocabulary size and parameter budget. |
| Approach: | They explore the potential of leveraging transliteration on a massive scale to improve performance for multilingual pretrained language models. |
| Outcome: | The proposed transliteration tool outperforms other methods on low-resource languages. |
Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing mPLMs can align representations well for myriads of cross-lingual transfer tasks. |
| Approach: | They propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by anisotropic representations. |
| Outcome: | The proposed method improves on three zero-shot cross-lingual transfer tasks and over existing methods. |
Multilingual Lottery Tickets to Pretrain Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual pretrained language models (mPLMs) have become the de-facto standard for multilingual tasks. |
| Approach: | They propose to search for a per-language subnetwork with comparable performance to the full model by scaling the model to reduce interference and then redistributing parameters to keep the parameters reduced. |
| Outcome: | The proposed model reduces the inference cost of models for each language while keeping the capacity per language more or less the same. |
Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent mPLMs have shown impressive performance on crosslingual transfer tasks . however, the performance is often hindered when a lowresource target language is written in a different script than the high-resource source language. |
| Approach: | They propose a transliteration-based method to improve cross-lingual alignment between languages using diverse scripts. |
| Outcome: | The proposed method outperforms the original model on Englishcentric transfer tasks up to 50%. |
Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extend semantic parsing (SP) beyond English are challenging due to the complex slot alignment step after translation. |
| Approach: | They propose a method to enhance cross-lingual transfer for SP by utilizing mPLMs. |
| Outcome: | The proposed method synthesizes target language utterances from source meaning representations while maintaining high slot value alignment rates. |
Knowledge Enhanced Pre-training for Cross-lingual Dense Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing mPLMs neglect the importance of knowledge in cross-lingual dense retrieval. |
| Approach: | They propose a novel mPLM that leverages knowledge to learn language-agnostic semantic representations from a multilingual knowledge base and an annotation of Wiki. |
| Outcome: | The proposed model achieves strong multilingual and cross-lingual retrieval performance with significant improvements over existing mPLMs. |
Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks. |
| Approach: | They propose a method that uses soft-prompt tuning to tune for language adaptation . prompt tuning outperforms continuously trained baselines on two benchmarks . |
| Outcome: | The proposed approach outperforms baselines on two text classification benchmarks while utilizing 0.28% of tuned parameters. |