Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing large language models are notoriously English-centric, and their performance has been reported to drop significantly in lessresourced languages. |
| Approach: | They propose a language-agnostic benchmark for in-context learning that reveals catastrophic forgetting early on CPT when English is not included. |
| Outcome: | The proposed method does not impact validation perplexity but is critical for emergence of downstream capabilities in the target language. |
Similar Papers
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive. |
| Approach: | They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them. |
| Outcome: | The proposed approach saves significant resources and accelerates convergence and performance. |
Disentangling Continued Pre-Training: Attention-Driven Routing and Semantic Hub Preservation in Language Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Continued Pre-Training (CPT) enables Large Language Models (LLMs) to acquire second-language capabilities, yet the mechanisms underlying CPT remain poorly understood. |
| Approach: | They investigate how CPT adapts model representations across diverse language families and scripts, model sizes, and architectures. |
| Outcome: | The proposed model can be surgically transferred between base and CPT models with minimal loss. |
Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve? (2024.emnlp-main)
Copied to clipboard
| Challenge: | In the last decade, the generalization and adaptation abilities of deep learning models were evaluated on fixed training and test distributions. |
| Approach: | They propose to train large language models on unlabeled text corpora and train them online. |
| Outcome: | The proposed model training on a text domain could degrade its perplexity on the test portion of the same domain. |
Towards Effective and Efficient Continual Pre-training of Large Language Models (2025.acl-long)
Copied to clipboard
Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, Ji-Rong Wen
| Challenge: | Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. |
| Approach: | They propose a Continual pre-training method that can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
| Outcome: | The proposed method can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
Are Emergent Abilities in Large Language Models just In-Context Learning? (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models have been claimed to acquire certain capabilities without having been specifically trained on them. |
| Approach: | They propose a theory that explains emergent abilities by taking into account their potential confounding factors and rigorously substantiate this theory through over 1000 experiments. |
| Outcome: | The proposed theory proves that emergent abilities are not truly emergental, but result from a combination of in-context learning, model memory, and linguistic knowledge. |
Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to remarkable improvements in language understanding and text generation. |
| Approach: | They propose a framework to evaluate large language models for underrepresented languages . they examine CPT strategies for languages with limited representation in multilingual models . |
| Outcome: | The proposed evaluation framework is based on the case of Galician language . it assesses trade-offs between linguistic enrichment and task-solving capabilities . |
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages (2026.acl-long)
Copied to clipboard
Hao Yu, Tianyi Xu, Michael A. Hedderich, Wassim Hamidouche, Syed Waqas Zamir, David Ifeoluwa Adelani
| Challenge: | Continued pretraining (CPT) is a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning are limited. |
| Approach: | They propose to use CPT to adapt large language models to African languages . they use math, code, and synthetic translated data to analyze their models . |
| Outcome: | The proposed models improve on multilingual benchmarks and document-level translation. |
Emergent Abilities in Reduced-Scale Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models can solve new tasks without task-specific fine-tuning. |
| Approach: | They propose to use pre-training data to pre-train 36 language models with billions of parameters to investigate whether emergent properties are tied to model size or can be demonstrated by smaller models. |
| Outcome: | The proposed model performs comparable to models trained on unrestricted language. |
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models learn non-trivial abstractions during pretraining, but it is not well understood when and how these specific linguistic abilities emerge. |
| Approach: | They propose a method to track the evolution of linguistic features during pretraining by using sparse crosscoders to discover and align features across model checkpoints. |
| Outcome: | The proposed approach can detect features emergence, maintenance, and discontinuation during training stages. |