| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
Similar Papers
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Effective Fine-Tuning Methods for Cross-lingual Adaptation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large scale multilingual pre-trained language models have shown promising results in zero- and few-shot cross-lingual tasks. |
| Approach: | They propose a co-tuning method that aims to learn more generalized semantic equivalences when the languages are structurally dissimilar. |
| Outcome: | The proposed method improves on cross-lingual inference and review tasks by capturing the semantic relationship in the parallel data when a few translation pairs are available. |
Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models enable zero-shot learning even for unseen languages . current multilingual model covers only a small subset of the world's languages - due to data sparsity, they are not likely to obtain good results for many lowresource languages. |
| Approach: | They ask: how does the number of pretraining languages influence zero-shot learning for unseen languages? do the findings change if the languages used for pretraining are all related? |
| Outcome: | The results show that pretrained models can zero-shot learn for unseen languages even for limited amounts even for low-resource languages. |
Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work highlights the fallacies of zero-shot cross-lingual transfer with large multilingual models. |
| Approach: | They propose to replace sequential fine-tuning with joint fine-uning on source and target language instances. |
| Outcome: | The proposed techniques yield improved and more stable FS-XLT across the board. |
CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning (2025.acl-long)
Copied to clipboard
Yangfan Ye, Xiaocheng Feng, Zekun Yuan, Xiachong Feng, Libo Qin, Lei Huang, Weitao Ma, Yichong Huang, Zhirui Zhang, Yunfei Lu, Xiaohui Yan, Duyu Tang, Dandan Tu, Bing Qin
| Challenge: | Existing fine-tuning approaches that focus on English-centric training corpora often introduce implicit cross-lingual alignment, overlooking the potential for more profound, latent-level cross-linguistic interactions. |
| Approach: | They propose a multilingual fine-tuning paradigm that explicitly establishes a cross-lingual connection mechanism at the latent level. |
| Outcome: | The proposed model outperforms vanilla SFT and offers a strong latent-level alternative to data-level augmentation methods. |
S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method (2022.acl-short)
Copied to clipboard
| Challenge: | Existing multilingual pre-trained language models allow to adapt to target languages with only few labeled examples. |
| Approach: | They propose a simple cross-lingual sub-network tuning method that detects the most essential sub-netzwork for each target language and updates it during fine-tuning. |
| Outcome: | The proposed method improves on three multi-lingual tasks involving 37 different languages. |
Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research shows that a multilingual pre-trained language model fine-tuned with one (source) language performs well on downstream tasks for non-source languages . However, there is a clear performance gap between the source and non-sourced languages - this gap can be reduced by reducing forgetting. |
| Approach: | They propose a method to fine-tune a multilingual pre-trained language model fine- tuned with one (source) language and four training policies to address the performance gap. |
| Outcome: | The proposed method outperforms baselines on the XNLI dataset by a clear margin. |
How Many Languages Make Good Multilingual Instruction Tuning? A Case Study on BLOOM (2025.coling-main)
Copied to clipboard
| Challenge: | Many large language models (LLMs) support many languages, while others only support a few, e.g. the Llama series. |
| Approach: | They present a case study on BLOOM to understand three pertinent factors affecting performance: the number of languages, language exposure, and similarity between training and test languages. |
| Outcome: | The proposed model can be used to perform multilingual tasks on 1 to 52 languages. |
VEEF-Multi-LLM: Effective Vocabulary Expansion and Parameter Efficient Finetuning Towards Multilingual Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a significant disadvantage for low-resource languages . VEEF-Multi-LLM-8B excels in multilingual instruction-following tasks . |
| Approach: | They propose a low-resource multilingual large language model that expands the vocabulary for multilingual support. |
| Outcome: | The proposed model outperforms existing models in multilingual instruction-following tasks, but lags behind English-centric models in some tasks. |
Cross-Lingual Optimization for Language Transfer in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Adapting large language models to other languages often suffers from an overemphasis on English performance. |
| Approach: | They propose a cross-lingual optimization technique that efficiently transfers an English-centric LLM to a target language while preserving its English capabilities. |
| Outcome: | The proposed model outperforms SFT in acquiring target language proficiency and maintaining English performance in low-resource languages. |