LPC: A Logits and Parameter Calibration Framework for Continual Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to solve catastrophic forgetting problem are varied . current approaches to learn continuous learning are based on replay-based methods . |
| Approach: | They propose to calibrate parameters and logits so that preserving old parameters and generalized learning on new concepts can be solved simultaneously. |
| Outcome: | The proposed model achieves state-of-the-art performance in all scenarios. |
Similar Papers
Continual Prompt Tuning for Dialog State Tracking (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train a model on a sequence of tasks are not efficient enough to mitigate catastrophic forgetting. |
| Approach: | They propose a parameter-efficient framework that prevents forgetting and enables knowledge transfer between tasks by learning and freezing a pre-trained model. |
| Outcome: | The proposed framework avoids forgetting and enables knowledge transfer between tasks. |
Parameter-efficient Continual Learning Framework in Industrial Real-time Text Classification System (2022.naacl-industry)
Copied to clipboard
Tao Zhu, Zhe Zhao, Weijie Liu, Jiachi Liu, Yiren Chen, Weiquan Mao, Haoyan Liu, Kunbo Ding, Yudong Li, Xuefeng Yang
| Challenge: | Existing continual learning methods use data replay, parameter isolation and regularization to mitigate catastrophic forgetting. |
| Approach: | They propose a parameter-efficient continual learning framework that updates parameters offline and then trains using an online regularization method. |
| Outcome: | The proposed framework reduces catastrophic forgetting and saves the model with the changed parameters instead of all parameters. |
HFT: Half Fine-Tuning for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) with one or more fine-tuning phases can unlock various capabilities, but can be catastrophic forgetting during sequential training. |
| Approach: | They propose a method to regularly reset partial parameters to mitigate forgetting issues by using half fine-tuning instead of full fine-uning. |
| Outcome: | The proposed approach reduces the risk of catastrophic forgetting during training and the parametric knowledge lost during training may be overwhelmed by incoming training data. |
Rehearsal-free Continual Language Learning via Efficient Parameter Isolation (2023.acl-long)
Copied to clipboard
Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han, Ling Wang, Xu Shao, Wenqiu Zeng
| Challenge: | Existing methods for learning continual tasks do not cache history data, which makes the problem more challenging. |
| Approach: | They propose a method that allocates a small portion of private parameters and learns them with a shared pre-trained model. |
| Outcome: | The proposed method is comparable to existing methods and comparable to those using historical data. |
Rehearsal-Free Modular and Compositional Continual Learning for Language Models (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing methods to overcome catastrophic forgetting are rehearsal-based and parameter isolation-based. |
| Approach: | They propose a rehearsal-free framework which continuously adds new modules to language models and composes them with existing modules. |
| Outcome: | Experiments on benchmarks show that MoCL outperforms state-of-the-art and effectively facilitates knowledge transfer. |
Parameter-Efficient Finetuning for Robust Continual Multilingual Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to Continual Multilingual Learning (CML) are based on updating models using new data in stages. |
| Approach: | They propose a parameter-efficient finetuning strategy to increase the number of languages on which the model improves after an update while reducing the magnitude of loss for the remaining languages. |
| Outcome: | The proposed model improves on the languages included in the latest update while reducing the loss of performance on the remaining languages. |
FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning (2026.acl-long)
Copied to clipboard
Yujie Feng, Hao Wang, Jian Li, Xu Chu, Zhaolu Kang, Yiran Liu, Yasha Wang, Philip S. Yu, Xiao-Ming Wu
| Challenge: | Continual learning (CL) for large language models (LLMs) aims to enable sequential knowledge acquisition without catastrophic forgetting. |
| Approach: | They propose a framework that aligns replay schedules with a model-centric notion of time. |
| Outcome: | Experiments on three benchmarks show that FOREVER consistently mitigates catastrophic forgetting. |
SEE: Continual Fine-tuning with Sequential Ensemble of Experts (2025.findings-acl)
Copied to clipboard
| Challenge: | Continual fine-tuning of large language models suffers from catastrophic forgetting . some approaches use routers to assign tasks to experts, but continual learning often requires retraining . |
| Approach: | They propose a framework that integrates routing and response mechanisms within each expert . it eliminates the need for an additional router and allows each expert to decide whether a query should be handled . |
| Outcome: | The proposed framework outperforms previous approaches in continual fine-tuning . it can handle learning tasks and out-of-distribution instances, paving the way for distributed model ensembling. |
InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions (2024.naacl-long)
Copied to clipboard
| Challenge: | In order to perform downstream tasks, Large Language Models (LLMs) need continual adaptation without catastrophic forgetting. |
| Approach: | They propose a new paradigm that allows for continual adaptation without catastrophic forgetting . they propose to replay previous data based on task similarity with instructions . |
| Outcome: | The proposed method improves performance over 16 tasks with different training orders. |
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) offers efficiency but constrains the model’s ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. |
| Approach: | They propose a training strategy that synergistically combines full and low-rank parameters and jointly updating within a unified low-ranked gradient subspace. |
| Outcome: | Extensive experiments on continual learning benchmarks show that GORP improves performance compared to state-of-the-art approaches. |