Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | A study of multilingual fine-tuning yields better performance on downstream NLP applications . low resource languages such as Oriya and Punjabi are found to be the largest beneficiaries of multi-lingual fine tuning. |
| Approach: | They propose to leverage the relatedness of languages that belong to the same family in NLP models by multilingual fine-tuning. |
| Outcome: | The proposed approach improves performance on downstream NLP tasks by 15% compared to monolingual fine-tuning. |
Similar Papers
How to Improve LLMs’ Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit uneven performance across languages. |
| Approach: | They propose to use a framework to quantify the similarity within each language pair through both the lenses of language-specific performance patterns and cross-lingual transferability. |
| Outcome: | The proposed approach outperforms traditional linguistic typology and cross-lingual transferability measures on multilingual LLMs. |
Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research shows that a multilingual pre-trained language model fine-tuned with one (source) language performs well on downstream tasks for non-source languages . However, there is a clear performance gap between the source and non-sourced languages - this gap can be reduced by reducing forgetting. |
| Approach: | They propose a method to fine-tune a multilingual pre-trained language model fine- tuned with one (source) language and four training policies to address the performance gap. |
| Outcome: | The proposed method outperforms baselines on the XNLI dataset by a clear margin. |
The Effect of Language Diversity When Fine-Tuning Large Language Models for Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research on language diversity in LLM fine-tuning has reported benefits while others find no benefits. |
| Approach: | They find that expanding language diversity during fine-tuning improves translation quality . they also show that increased language diversity creates more language-agnostic representations . |
| Outcome: | The proposed model improves translation quality for unsupervised and supervised pairs . the results plateau or decrease beyond a certain diversity threshold. |
Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced natural language processing, yet their benefits remain concentrated in English and a small number of high-resource languages. |
| Approach: | They fine-tuned large language models (4B–671B parameters) on Arabic and evaluated zero-shot reading comprehension on Semitic languages and non-Semitic controls. |
| Outcome: | The results show that models with weak baselines improve across all languages, whereas strong-baseline models show only marginal gains regardless of language family. |
Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023.findings-eacl)
Copied to clipboard
| Challenge: | AfriBERTa shows that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. |
| Approach: | They propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. |
| Outcome: | The proposed model outperforms models trained on large, unrelated datasets on downstream NLP tasks. |
How Many Languages Make Good Multilingual Instruction Tuning? A Case Study on BLOOM (2025.coling-main)
Copied to clipboard
| Challenge: | Many large language models (LLMs) support many languages, while others only support a few, e.g. the Llama series. |
| Approach: | They present a case study on BLOOM to understand three pertinent factors affecting performance: the number of languages, language exposure, and similarity between training and test languages. |
| Outcome: | The proposed model can be used to perform multilingual tasks on 1 to 52 languages. |
Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study (2021.acl-long)
Copied to clipboard
Yash Khemchandani, Sarvesh Mehtani, Vaidehi Patil, Abhijeet Awasthi, Partha Talukdar, Sunita Sarawagi
| Challenge: | Recent research in multilingual language models (LMs) has demonstrated their ability to effectively handle multiple languages in a single model. |
| Approach: | They propose to exploit relatedness among languages in a language family to overcome corpora limitations of LRLs. |
| Outcome: | The proposed model exploits relatedness among languages in a language family to overcome corpora limitations for low web-resource languages. |
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence (2022.coling-1)
Copied to clipboard
| Challenge: | Several studies on the interpretation of Neural Language Models (NLMs) focus on the linguistic generalization abilities of pre-trained models, but little attention is paid to how the linguistic knowledge of the models changes during fine-tuning. |
| Approach: | They propose to examine whether a wide range of linguistic phenomena are forgotten during fine-tuning and whether it is possible to predict the fine- tuned accuracy solely relying on the assessed linguistic competence. |
| Outcome: | The proposed model can predict the evolution of written language competence of native language learners based on the assessed linguistic competence. |
Understanding the effects of language-specific class imbalance in multilingual fine-tuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models have been developed to reduce the amount of resources needed to perform classification tasks. |
| Approach: | They modify traditional class weighing approach to reduce imbalance by calculating class weights separately for each language. |
| Outcome: | The proposed model improves performance and reduces the promotion of uninformative features. |
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |