| Challenge: | Existing approaches to learning models (LMs) incorporate old task data or task-wise inductive bias into LMs, but old data and accurate task information are often unavailable or costly to collect. |
| Approach: | They propose a rehearsal-free method that updates model parameters with large magnitudes . they found that the L1-normalized magnitude distribution is different when different task data is used . |
| Outcome: | The proposed method improves accuracy and performance on four CL benchmarks. |
Similar Papers
Fine-tuned Language Models are Continual Learners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on large language models relies on intuition that most tasks can be described via natural language instructions. |
| Approach: | They propose that a model should be able to keep extending its knowledge without forgetting previous skills. |
| Outcome: | The proposed model can learn 8 new diverse language generation tasks while maintaining good performance on previous tasks, spanning in total of 70 datasets. |
Continual Learning of Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines . |
| Approach: | This tutorial offers a comprehensive exploration of continual learning in the context of large language models. |
| Outcome: | This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly . |
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive. |
| Approach: | They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them. |
| Outcome: | The proposed approach saves significant resources and accelerates convergence and performance. |
Gradient Localization Improves Lifelong Pretraining of Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for continual learning do not account for locality of knowledge . however, in practice language models are deployed in dynamic real-world settings and their learned knowledge becomes stale over time. |
| Approach: | They examine two types of knowledge relating to temporally sensitive entities . they hypothesize that lack of consideration of locality contributes to failed uptake of new information . |
| Outcome: | The proposed model can be improved by updating parameters to relevant layers . the proposed model is based on a large static web-scale dataset . |
Parameter-Efficient Finetuning for Robust Continual Multilingual Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to Continual Multilingual Learning (CML) are based on updating models using new data in stages. |
| Approach: | They propose a parameter-efficient finetuning strategy to increase the number of languages on which the model improves after an update while reducing the magnitude of loss for the remaining languages. |
| Outcome: | The proposed model improves on the languages included in the latest update while reducing the loss of performance on the remaining languages. |
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) offers efficiency but constrains the model’s ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. |
| Approach: | They propose a training strategy that synergistically combines full and low-rank parameters and jointly updating within a unified low-ranked gradient subspace. |
| Outcome: | Extensive experiments on continual learning benchmarks show that GORP improves performance compared to state-of-the-art approaches. |
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
Continual Training of Language Models for Few-Shot Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. |
| Approach: | They propose to continuously post-train an LM with unlabeled domains to expand its knowledge without forgetting previous skills. |
| Outcome: | The proposed system improves few-shot end-task learning in these domains. |
Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks. |
| Approach: | They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace . |
| Outcome: | The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting. |
Rethinking Long Context Generation from the Continual Learning Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with processing long contexts due to the limited context window. |
| Approach: | They propose to combine a limited context window with a continual learning perspective to improve LLMs' efficiency in processing long contexts. |
| Outcome: | The proposed models improve the performance of Large Language Models (LLMs) by integrating learning strategies with existing approaches. |