Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research shows that a multilingual pre-trained language model fine-tuned with one (source) language performs well on downstream tasks for non-source languages . However, there is a clear performance gap between the source and non-sourced languages - this gap can be reduced by reducing forgetting. |
| Approach: | They propose a method to fine-tune a multilingual pre-trained language model fine- tuned with one (source) language and four training policies to address the performance gap. |
| Outcome: | The proposed method outperforms baselines on the XNLI dataset by a clear margin. |
Similar Papers
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | A study of multilingual fine-tuning yields better performance on downstream NLP applications . low resource languages such as Oriya and Punjabi are found to be the largest beneficiaries of multi-lingual fine tuning. |
| Approach: | They propose to leverage the relatedness of languages that belong to the same family in NLP models by multilingual fine-tuning. |
| Outcome: | The proposed approach improves performance on downstream NLP tasks by 15% compared to monolingual fine-tuning. |
S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method (2022.acl-short)
Copied to clipboard
| Challenge: | Existing multilingual pre-trained language models allow to adapt to target languages with only few labeled examples. |
| Approach: | They propose a simple cross-lingual sub-network tuning method that detects the most essential sub-netzwork for each target language and updates it during fine-tuning. |
| Outcome: | The proposed method improves on three multi-lingual tasks involving 37 different languages. |
Cross-Lingual Optimization for Language Transfer in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Adapting large language models to other languages often suffers from an overemphasis on English performance. |
| Approach: | They propose a cross-lingual optimization technique that efficiently transfers an English-centric LLM to a target language while preserving its English capabilities. |
| Outcome: | The proposed model outperforms SFT in acquiring target language proficiency and maintaining English performance in low-resource languages. |
Consistency Regularization for Cross-Lingual Fine-Tuning (2021.acl-long)
Copied to clipboard
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
| Challenge: | Experimental results show that consistency regularization improves cross-lingual fine-tuning . pre-trained cross-linguistic models can transfer task-specific supervision from one language to the other . |
| Approach: | They propose to improve cross-lingual fine-tuning with consistency regularization . they use example consistency regularized to penalize prediction sensitivity to four types of data augmentations . |
| Outcome: | The proposed method improves cross-lingual fine-tuning across tasks . it can be generalized to other target languages without additional training . |
CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning (2025.acl-long)
Copied to clipboard
Yangfan Ye, Xiaocheng Feng, Zekun Yuan, Xiachong Feng, Libo Qin, Lei Huang, Weitao Ma, Yichong Huang, Zhirui Zhang, Yunfei Lu, Xiaohui Yan, Duyu Tang, Dandan Tu, Bing Qin
| Challenge: | Existing fine-tuning approaches that focus on English-centric training corpora often introduce implicit cross-lingual alignment, overlooking the potential for more profound, latent-level cross-linguistic interactions. |
| Approach: | They propose a multilingual fine-tuning paradigm that explicitly establishes a cross-lingual connection mechanism at the latent level. |
| Outcome: | The proposed model outperforms vanilla SFT and offers a strong latent-level alternative to data-level augmentation methods. |
Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work highlights the fallacies of zero-shot cross-lingual transfer with large multilingual models. |
| Approach: | They propose to replace sequential fine-tuning with joint fine-uning on source and target language instances. |
| Outcome: | The proposed techniques yield improved and more stable FS-XLT across the board. |
Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in training multilingual models on large datasets have shown promising results in knowledge transfer across languages. |
| Approach: | They challenge the assumption that high zero-shot performance reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. |
| Outcome: | The proposed model can achieve high performance on multilingual benchmarks and on low-resource languages. |
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work on multilingual evaluation has shown that there is a large gap between the performance of Large Language Models on English and other languages. |
| Approach: | They propose to finetune Llama-2 and Mistral models on two datasets to determine their effect on model performance on six downstream tasks covering forty one languages. |
| Outcome: | The proposed model can improve on six multilingual tasks while degrading on high-resource languages. |
Effective Fine-Tuning Methods for Cross-lingual Adaptation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large scale multilingual pre-trained language models have shown promising results in zero- and few-shot cross-lingual tasks. |
| Approach: | They propose a co-tuning method that aims to learn more generalized semantic equivalences when the languages are structurally dissimilar. |
| Outcome: | The proposed method improves on cross-lingual inference and review tasks by capturing the semantic relationship in the parallel data when a few translation pairs are available. |