Harmonizing the Past, Present, and Future: A Null-Space Constrained Region-Specific Method for Continual Learning in LLMs (2026.acl-long)
Copied to clipboard
Jinhui Chen, Shizhu He, Xingchang Yang, Huanxuan Liao, Yequan Wang, Xiangwen Liao, Wenhao Teng, Kang Liu, Jun Zhao
| Challenge: | Existing continual learning paradigms prioritize instant performance through dense updates, leading to catastrophic forgetting and rapid exhaustion of model capacity. |
| Approach: | They propose a method that preserves previously acquired knowledge and acquires new task-specific skills while preserving sufficient parameter capacity for subsequent adaptation. |
| Outcome: | The proposed method is based on the brain's functional partitioning and can be used to map tasks between specialized and generalist neurons. |
Similar Papers
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
Unlocking Continual Learning Abilities in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learning models (LMs) incorporate old task data or task-wise inductive bias into LMs, but old data and accurate task information are often unavailable or costly to collect. |
| Approach: | They propose a rehearsal-free method that updates model parameters with large magnitudes . they found that the L1-normalized magnitude distribution is different when different task data is used . |
| Outcome: | The proposed method improves accuracy and performance on four CL benchmarks. |
ELLA: Efficient Lifelong Learning for Adapters in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to training Large Language Models (LLMs) suffer from catastrophic forgetting when adapted sequentially to new tasks in a continual learning (CL) setting. Existing methods are impractical and could potentially violate privacy. |
| Approach: | They propose a training framework built on the principle of selective subspace de-correlation that characterizes the structure of past updates and penalizes alignments along their high-energy, task-specific directions. |
| Outcome: | The proposed training framework achieves state-of-the-art CL performance on three popular benchmarks spanning both classification and generative tasks with relative accuracy gains of up to 9.6% and a 35 smaller memory footprint. |
Continual Learning of Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines . |
| Approach: | This tutorial offers a comprehensive exploration of continual learning in the context of large language models. |
| Outcome: | This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly . |
Spectral Disentanglement: Rank-Aware Task Adaptation for Rehearsal-free Continual Learning in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Continual Learning (CL) for Large Language Models faces a fundamental Stability-Plasticity Dilemma . Rank-Blindness enforces a single rank constraint across diverse tasks, leading to catastrophic forgetting of earlier tasks and underfitting on complex new ones. |
| Approach: | They propose a rank-spectrum-based rehearsal-free framework that explicitly disentangles knowledge into two orthogonal subspaces. |
| Outcome: | The proposed framework achieves a superior stability-plasticity balance compared to single-rank baselines. |
Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models underperform in languages absent or underrepresented in training data, creating barrier to equitable access for speakers worldwide. |
| Approach: | They propose a selective parameter update strategy that proactively preserves source knowledge by identifying critical parameters critical to maintaining source abilities. |
| Outcome: | Experiments in five typologically diverse languages show that SSU mitigates catastrophic forgetting. |
Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks. |
| Approach: | They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace . |
| Outcome: | The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting. |
Recyclable Tuning for Continual Pre-training (2023.findings-acl)
Copied to clipboard
Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou
| Challenge: | Continual pre-training is the paradigm where pre-trained language models acquire fresh knowledge and gradually get upgraded. |
| Approach: | They propose to use adapted weights to recycle old PLMs for continual pre-training . they propose to combine initialization and distillation methods to achieve better performance . |
| Outcome: | The proposed method improves the convergence and performance of the upgraded PLM. |
HFT: Half Fine-Tuning for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) with one or more fine-tuning phases can unlock various capabilities, but can be catastrophic forgetting during sequential training. |
| Approach: | They propose a method to regularly reset partial parameters to mitigate forgetting issues by using half fine-tuning instead of full fine-uning. |
| Outcome: | The proposed approach reduces the risk of catastrophic forgetting during training and the parametric knowledge lost during training may be overwhelmed by incoming training data. |
F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to address Catastrophic Forgetting (CF) have been developed to avoid forgetting and maintain system extensibility. |
| Approach: | They propose a method to reduce Catastrophic Forgetting (CF) by decomposing feed-forward layers into discrete memory cells and ensuring robust extendability. |
| Outcome: | The proposed method achieves higher BLEU scores and almost zero forgetting while maintaining robust extendability. |