Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive. |
| Approach: | They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them. |
| Outcome: | The proposed approach saves significant resources and accelerates convergence and performance. |
Similar Papers
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
Continual Learning of Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines . |
| Approach: | This tutorial offers a comprehensive exploration of continual learning in the context of large language models. |
| Outcome: | This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly . |
Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing large language models are notoriously English-centric, and their performance has been reported to drop significantly in lessresourced languages. |
| Approach: | They propose a language-agnostic benchmark for in-context learning that reveals catastrophic forgetting early on CPT when English is not included. |
| Outcome: | The proposed method does not impact validation perplexity but is critical for emergence of downstream capabilities in the target language. |
Efficient Continual Pre-training of LLMs for Low-resource Languages (2025.naacl-industry)
Copied to clipboard
| Challenge: | Open-source large language models (LLMs) are a promising tool for low-resource languages . however, there is still a substantial performance gap between high-resourced languages and LRLs . |
| Approach: | They develop an algorithm to select a subset of texts from a larger corpus and use it to select tokens for LLMs. |
| Outcome: | The proposed algorithm reduces the cost of continual pre-training (CPT) with large amounts of language-specific data. |
Towards Effective and Efficient Continual Pre-training of Large Language Models (2025.acl-long)
Copied to clipboard
Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, Ji-Rong Wen
| Challenge: | Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. |
| Approach: | They propose a Continual pre-training method that can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
| Outcome: | The proposed method can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2022.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models are typically learned over a large, static corpus and fine-tuned for various downstream tasks. |
| Approach: | They propose to continuously update a pretrained language model to adapt to emerging data and to keep track of the model's performance. |
| Outcome: | The proposed model can adapt to new corpora while retaining knowledge in earlier domains. |
Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | a lack of data in low-resource languages has limited the performance of a multilingual pre-trained model. |
| Approach: | They propose a continuous pre-training framework to adapt mBART to unseen languages . they construct noisy mixed-language text from the monolingual corpus of the target language . |
| Outcome: | The proposed framework improves finetuning performance on low-resource translation pairs . the proposed framework also improves on translation pairs where both languages are seen . |
Data-Centric Continual Pre-training for 500+ Languages: A New Bilingual Translation Corpus and Multilingual Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models pre-trained on massive data have promoted multilingual natural language processing (NLP). |
| Approach: | They construct a bilingual translation corpus with 2,500 language pairs and develop a suite of four models with parallel data. |
| Outcome: | The proposed model suites are evaluated across 7 tasks and 12 benchmarks. |
Continual Training of Language Models for Few-Shot Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. |
| Approach: | They propose to continuously post-train an LM with unlabeled domains to expand its knowledge without forgetting previous skills. |
| Outcome: | The proposed system improves few-shot end-task learning in these domains. |
Disentangling Continued Pre-Training: Attention-Driven Routing and Semantic Hub Preservation in Language Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Continued Pre-Training (CPT) enables Large Language Models (LLMs) to acquire second-language capabilities, yet the mechanisms underlying CPT remain poorly understood. |
| Approach: | They investigate how CPT adapts model representations across diverse language families and scripts, model sizes, and architectures. |
| Outcome: | The proposed model can be surgically transferred between base and CPT models with minimal loss. |