Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work relies on training with multi-lingual ability-related data, which may not be available for low-resource languages. |
| Approach: | They propose a multi-lingual ability-enhanced LLM that extracts language-agnostic ability-related weights from LLMs and combine them across different languages by simple addition and subtraction operations without training. |
| Outcome: | The proposed approach extracts language-agnostic ability-related weights from LLMs and combine them across different languages without training. |
Similar Papers
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE (2026.acl-long)
Copied to clipboard
Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She, Linjuan Wu, Hao-Ran Wei, Baosong Yang, Jiajun Chen, Shujian Huang
| Challenge: | Large Language Models (LLMs) are expensive and require extensive Continued Pre-Training and data-intensive alignment to expand. |
| Approach: | They propose a method which upcycles a dense model into a Mixture-of-Experts architecture, allocating different experts to different languages. |
| Outcome: | Experiments show that the proposed model upcycles a dense model into a Mixture-of-Experts(MoE) architecture, allocating different experts to different languages. |
AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models excel in highresource languages but underperform in lowresource ones. |
| Approach: | They propose a cross-lingual transfer method that decouples "task ability" from "language ability" they propose to use adaptive adapter merging to obtain target adapters by combining other adapters. |
| Outcome: | The proposed method outperforms existing methods in highresource languages . it decouples "task ability" from "language ability" but fails to fully separate "task capability" from the "source language" |
Training-free LLM Merging for Multi-task Learning (2025.acl-long)
Copied to clipboard
Zichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang, Shanshan Ye, Hongzhi Yin, Yi Chang, Yefeng Zheng, Xiangyu Zhao
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing tasks. |
| Approach: | They propose a training-free method for unifying different specialized LLMs into a single model using model-wise and layer-wise pruning and scaling. |
| Outcome: | The proposed method outperforms existing merging techniques and surpasses models fine-tuned on combined datasets in most scenarios. |
Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive abilities in various domains such as text generation, instruction following, and reasoning. |
| Approach: | They propose a method to decompose the activations of Large Language Models into a sparse linear combination of SAE features. |
| Outcome: | The proposed method shows that some features are strongly related to specific languages, while others are unaffected by ablating them. |
VEEF-Multi-LLM: Effective Vocabulary Expansion and Parameter Efficient Finetuning Towards Multilingual Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a significant disadvantage for low-resource languages . VEEF-Multi-LLM-8B excels in multilingual instruction-following tasks . |
| Approach: | They propose a low-resource multilingual large language model that expands the vocabulary for multilingual support. |
| Outcome: | The proposed model outperforms existing models in multilingual instruction-following tasks, but lags behind English-centric models in some tasks. |
Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure (2022.acl-long)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models have shown impressive cross-lingual ability. |
| Approach: | They argue that cross-language ability comes from commonality between languages . they create an artificial language by modifying property in source language . |
| Outcome: | The proposed model can be implemented in multilingual and low-resource language scenarios without cross-lingual supervision or aligned data. |
MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MultiPL is a special case of multiple natural languages and requires limited computational resources to generate multilingual code. |
| Approach: | They propose to extend LLMs by combining two paired experts to optimize expert selection at token and segment levels. |
| Outcome: | The proposed extension improves the performance of the base LLMs while retaining the most popular ones using limited computational resources. |
Data and Model Centric Approaches for Expansion of Large Language Models to New languages (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | Existing LLMs mainly support English alongside a handful of high resource languages . this leaves a major gap for most low-resource languages despite increasing pace of research . |
| Approach: | This tutorial examines approaches to expand the language coverage of LLMs . they look at tokenizer training, pre-training, instruction tuning, alignment, evaluation, etc. |
| Outcome: | This tutorial examines approaches to expand the language coverage of LLMs . it provides an efficient and viable path to bring LLM technologies to low-resource languages . |
Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean (2024.lrec-main)
Copied to clipboard
ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim
| Challenge: | Large language models (LLMs) use pretraining to predict the subsequent word, but less-resourced languages are being overlooked. |
| Approach: | They propose to expand the MLLM vocabularies to enhance expressiveness and use bilingual data for pretraining to align the high- and less-resourced languages. |
| Outcome: | The proposed model outperforms existing models in qualitative analyses compared to Korean monolingual models. |
Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets (2024.findings-emnlp)
Copied to clipboard
Israel Azime, Atnafu Tonja, Tadesse Belay, Mitiku Yohannes Fuge, Aman Wassie, Eyasu Jada, Yonas Chanie, Walelign Sewunetie, Seid Yimam
| Challenge: | Low-resource languages are left behind due to the unavailability of resources. |
| Approach: | They propose to integrate task-specific and generative datasets to improve language model performance for Amharic by fine-tuning an Amharican instruction fine-to-tuned model. |
| Outcome: | The proposed model shows promising results in different NLP tasks and compares translated instruction datasets with the original model. |