Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale multilingual pretrained language models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLm. |
| Approach: | They propose to leverage the learned knowledge from well-performing languages to guide under-performing ones within the same mPLM. |
| Outcome: | The proposed model shows that it can guide under-performing languages while minimizing language-level performance disparities across different mPLMs. |
Similar Papers
Pruning Multilingual Large Language Models for Multilingual Inference (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual large language models (MLLMs) demonstrate better zeroshot learning performance in non-English languages compared to large language model trained on English-dominant data. |
| Approach: | They propose a pruning approach to prune large language models using bilingual sentence pairs from English and other languages to enhance their performance in non-English language. |
| Outcome: | The proposed pruning strategy enhances the MLLMs’ performance in non-English language. |
Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent mPLMs have shown impressive performance on crosslingual transfer tasks . however, the performance is often hindered when a lowresource target language is written in a different script than the high-resource source language. |
| Approach: | They propose a transliteration-based method to improve cross-lingual alignment between languages using diverse scripts. |
| Outcome: | The proposed method outperforms the original model on Englishcentric transfer tasks up to 50%. |
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models encode strong language-specific signals, which are not explicitly provided during pretraining. |
| Approach: | They propose a language similarity measure that induces similarities across languages from mPLMs using multi-parallel corpora. |
| Outcome: | The proposed measure exhibits moderately high correlations with linguistic similarity measures, and more accurate similarity results on low correlation languages. |
FAD-X: Fusing Adapters for Cross-lingual Transfer to Low-Resource Languages (2022.aacl-short)
Copied to clipboard
| Challenge: | Adapter-based tuning is a technique that selectively updates language-specific parameters to adapt to a new language, rather than fine-tuning all shared weights. |
| Approach: | They propose to add light-weight adapters to multilingual pretrained language models (mPLMs) and add language-specific parameters to adapt to a new language. |
| Outcome: | The proposed adapter can enhance cross-lingual transfer from pretrained adapters for well-known named entity recognition and classification benchmarks. |
Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance (2025.naacl-industry)
Copied to clipboard
Somnath Banerjee, Avik Halder, Rajarshi Mandal, Sayan Layek, Ian Soboroff, Rima Hazra, Animesh Mukherjee
| Challenge: | Pretrained language models (PLMs) have revolutionized NLP but amplify linguistic inequities in multilingual applications. |
| Approach: | They evaluate pretrained language models including Mistral, TowerInstruct, OpenHathi, Tamil-Llama, and Kan-Lama across eight languages spanning high-resource and low-resourced settings. |
| Outcome: | The proposed models fail to bridge linguistic divides and are inefficient when compared to other models. |
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
Assessing the Role of Data Quality in Training Bilingual Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that adding more languages can degrade performance for some languages while improving others. |
| Approach: | They propose a data filtering strategy to select high-quality bilingual training data with only high quality English data. |
| Outcome: | The proposed approach improves bilingual model performance by 2–4% and reduces bilingual models performance gaps to 1%. |
TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | The world’s more than 7000 languages are written in at least 293 scripts, which poses a difficulty for multilingual pretrained language models in learning crosslingual knowledge through lexical overlap. |
| Approach: | They propose a framework that optimizes the Transliteration Contrastive Modeling objective to fine-tune an mPLM by contrasting sentences in its training data and transliterations in a unified script. |
| Outcome: | The proposed model outperforms Glot500-m on zero-shot crosslingual transfer tasks while retaining uniformity across scripts. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Multilingual Arbitration: Optimizing Data Pools to Accelerate Multilingual Progress (2025.acl-long)
Copied to clipboard
| Challenge: | Synthetic data generation relies on a single oracle teacher model, which can lead to model collapse and bias propagation. |
| Approach: | They propose a multilingual arbitration approach that exploits performance variations among multiple models for each language. |
| Outcome: | The proposed approach surpasses single-teacher distillation with 80% win rates over proprietary and open-weight models with the largest improvements in low-resource languages. |