No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement (2025.coling-main)
Copied to clipboard
| Challenge: | Modular deep learning is the most effective way to lift the curse of multilinguality. |
| Approach: | They propose a method which enables training-free post-processing to address this limitation by adding learning to the language adapters and transitioning the framework from a multi-task to a multiple language setup. |
| Outcome: | The proposed method consistently improves baselines with significant gains, especially in the most challenging case of zero-shot application. |
Similar Papers
Investigating the Potential of Task Arithmetic for Cross-Lingual Transfer (2024.eacl-short)
Copied to clipboard
| Challenge: | Massively multilingual Transformer-based models (MMTs) can learn representations which have a degree of cross-lingual alignment despite being trained using purely unsupervised objectives. |
| Approach: | They propose a modular approach to cross-lingual transfer using task arithmetic . they show that modularity can be achieved even with full model fine-tuning . |
| Outcome: | The proposed approach shows strong performance on multilingual benchmarks encompassing both high-resource and low-resourced languages. |
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current deep pretrained models lack capacity to represent all languages . limited capacity is an issue even for high-resource languages where models are not included in training data at all. |
| Approach: | They propose an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. |
| Outcome: | The proposed framework outperforms state-of-the-art models on cross-lingual transfer across languages and typologically diverse models. |
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on multilingual pre-trained models has focused on pre-training transformers on concatenated corpora of a large number of languages. |
| Approach: | They propose a language-specific module approach that allows for more languages to be trained post-hoc. |
| Outcome: | The proposed model can be pre-trained on multiple languages with no drop in performance . |
Cross-Lingual Transfer with Target Language-Ready Task Adapters (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing frameworks for (zero-shot) cross-lingual transfer employ separate language and task adapters which can be arbitrarily combined to perform any task to any target language. |
| Approach: | They propose to fine-tune 'target language-ready' adapters to the target language to achieve better transfer performance without sacrificing the modularity of MAD-X. |
| Outcome: | The proposed adapters outperform MAD-X and BAD-X on most tasks and languages while maintaining the modularity of MAD. |
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021.findings-emnlp)
Copied to clipboard
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, Anna Korhonen
| Challenge: | Massively multilingual transformers (MMTs) have benefited from additional training of language-specific adapters, but this approach is not viable for the vast majority of languages due to limitations in their corpus size or compute budgets. |
| Approach: | They propose a multilingual ADapter generation approach which contextually generates language adapters from language representations based on typological features. |
| Outcome: | The proposed method improves cross-lingual transfer performance on part-of-speech tagging, dependency parsing, and named entity recognition tasks while remaining cost-effective. |
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
The Impact of Language on Arithmetic Proficiency: A Multilingual Investigation with Cross-Agent Checking Computation (2024.naacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) have garnered significant attention over the past year . previous studies have evaluated LLMs' performance in solving math word problems, but there is little discussion on whether they comprehend the operations they generate. |
| Approach: | They challenge the notion that arithmetic is language-independent and compare models with cross-agent collaborations to find significant limitations in their performance. |
| Outcome: | The proposed model outperforms collaborative approaches in basic arithmetic tasks. |
How to Adapt Your Pretrained Multilingual Model to 1600 Languages (2021.acl-long)
Copied to clipboard
| Challenge: | Pretrained multilingual models perform best for languages seen during pretraining . methods exist to improve performance for unseen languages, but have been evaluated using amounts of raw text only available for a small fraction of the world’s languages. |
| Approach: | They evaluate the performance of existing methods to adapt pretrained multilingual models to new languages using a resource available for close to 1600 languages: the New Testament. |
| Outcome: | The proposed models perform best for languages seen during pretraining . the results show that the most efficient approach is simplest and the most accurate . |
BAD-X: Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Massively multilingual Transformers (MMTs) have dominated research in multilingual NLP and cross-lingual transfer recently. |
| Approach: | They propose to learn bilingual language pair adapters (BAs) when the goal is to optimize performance for a particular source-target transfer direction. |
| Outcome: | The proposed framework improves performance in three standard downstream tasks and for the majority of low-resource languages. |
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE (2026.acl-long)
Copied to clipboard
Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She, Linjuan Wu, Hao-Ran Wei, Baosong Yang, Jiajun Chen, Shujian Huang
| Challenge: | Large Language Models (LLMs) are expensive and require extensive Continued Pre-Training and data-intensive alignment to expand. |
| Approach: | They propose a method which upcycles a dense model into a Mixture-of-Experts architecture, allocating different experts to different languages. |
| Outcome: | Experiments show that the proposed model upcycles a dense model into a Mixture-of-Experts(MoE) architecture, allocating different experts to different languages. |