ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs (2026.eacl-industry)
Copied to clipboard
Hangyeol Yoo, ChangSu Choi, Minjun Kim, Seohyun Song, SeungWoo Song, Inho Won, Jongyoul Park, Cheoneum Park, KyungTae Lim
| Challenge: | Recent studies have focused on enhancing multilingual large language models (MLLMs) for specific languages. |
| Approach: | They propose an efficient layer-specific optimization method to enhance continual pretraining (CP) for specific languages in multilingual large language models (MLLMs). |
| Outcome: | The proposed method achieves a training speedup of up to 6.46 times compared to existing methods while improving target language performance by up to 5.2% on qualitative benchmarks. |
Similar Papers
Efficient Continual Pre-training of LLMs for Low-resource Languages (2025.naacl-industry)
Copied to clipboard
| Challenge: | Open-source large language models (LLMs) are a promising tool for low-resource languages . however, there is still a substantial performance gap between high-resourced languages and LRLs . |
| Approach: | They develop an algorithm to select a subset of texts from a larger corpus and use it to select tokens for LLMs. |
| Outcome: | The proposed algorithm reduces the cost of continual pre-training (CPT) with large amounts of language-specific data. |
LaCo: Layer-wise Compensation for Pruned Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for predicting performance degradations of Large Language Models (LLMs) neglect the structural distortions caused by sparsity. |
| Approach: | They propose a framework that reorients the recovery paradigm from global adaptation to hierarchical representation alignment by sequentially optimizing each layer to reconstruct the model's hidden states. |
| Outcome: | The proposed framework surpasses parameter-efficient baselines in perplexity reduction and zero-shot reasoning. |
Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean (2024.lrec-main)
Copied to clipboard
ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim
| Challenge: | Large language models (LLMs) use pretraining to predict the subsequent word, but less-resourced languages are being overlooked. |
| Approach: | They propose to expand the MLLM vocabularies to enhance expressiveness and use bilingual data for pretraining to align the high- and less-resourced languages. |
| Outcome: | The proposed model outperforms existing models in qualitative analyses compared to Korean monolingual models. |
Exploring Design Choices for Building Language-Specific LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work focused on building multilingual models that cover a broad spectrum of languages. |
| Approach: | They conduct systematic experiments on how design choices impact the adapted LLM, both in terms of efficiency and end task performance. |
| Outcome: | The proposed model performs better on English-centric models than multilingual models despite poor performance on low-resource languages. |
ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining (2025.emnlp-industry)
Copied to clipboard
Seonwu Kim, Yohan Na, Kihun Kim, Hanhee Cho, Geun Lim, Mintae Kim, Seongik Park, Ki Hyun Kim, Youngsub Han, Byoung-Ki Jeon
| Challenge: | Domain Adaptive Continual Pretraining (DACP) is a method to mitigate performance degradation in small LLMs and enhance their effectiveness in target domains. |
| Approach: | They propose a continual pretraining methodology that optimizes sLLMs within service domains and enhances their effectiveness in target domains. |
| Outcome: | The proposed model achieves significant gains in target-domain performance while preserving general capabilities, offering a cost-efficient and scalable solution for enterprise-level deployment. |
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2022.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models are typically learned over a large, static corpus and fine-tuned for various downstream tasks. |
| Approach: | They propose to continuously update a pretrained language model to adapt to emerging data and to keep track of the model's performance. |
| Outcome: | The proposed model can adapt to new corpora while retaining knowledge in earlier domains. |
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive. |
| Approach: | They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them. |
| Outcome: | The proposed approach saves significant resources and accelerates convergence and performance. |
mDAPT: Multilingual Domain Adaptive Pretraining in a Single Model (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain-specific multilingual pretraining data is difficult to obtain due to regulations, legislation, or simply a lack of language- and domain- specific text. |
| Approach: | They propose to continue pretraining a language model on domain-specific unlabelled text . this allows for better modelling of text for downstream tasks within the domain . |
| Outcome: | The proposed approach outperforms the general multilingual model and performs close to its monolingual counterpart. |
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers (2026.acl-long)
Copied to clipboard
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
| Challenge: | Existing approaches to train multilingual large language models for many languages at once are limited due to limited model capacity, scarce high-quality data, and compute constraints. |
| Approach: | They propose to use a universal tokenizer to improve language plasticity and adaptability to new languages by up to 20%. |
| Outcome: | The proposed tokenizer improves language plasticity and improves plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. |
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |