FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning

Challenge:	Continual learning (CL) for large language models (LLMs) aims to enable sequential knowledge acquisition without catastrophic forgetting.
Approach:	They propose a framework that aligns replay schedules with a model-centric notion of time.
Outcome:	Experiments on three benchmarks show that FOREVER consistently mitigates catastrophic forgetting.

Similar Papers

Overview

Challenge

Approach

Outcome

InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions (2024.naacl-long)

Copied to clipboard

Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, Yujiu Yang

Language Models

Challenge:	In order to perform downstream tasks, Large Language Models (LLMs) need continual adaptation without catastrophic forgetting.
Approach:	They propose a new paradigm that allows for continual adaptation without catastrophic forgetting . they propose to replay previous data based on task similarity with instructions .
Outcome:	The proposed method improves performance over 16 tasks with different training orders.

Continual Learning of Large Language Models (2025.emnlp-tutorials)

Copied to clipboard

Tongtong Wu, Trang Vu, Linhao Luo, Gholamreza Haffari

Language Models

Challenge:	This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines .
Approach:	This tutorial offers a comprehensive exploration of continual learning in the context of large language models.
Outcome:	This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly .

Self-generated Replay Memories for Continual Neural Machine Translation (2024.naacl-long)

Copied to clipboard

Michele Resta, Davide Bacciu

Text Generation

Challenge:	Neural Machine Translation systems exhibit strong performance in several different languages, but their ability to learn continuously is limited by catastrophic forgetting.
Approach:	They propose a method that leverages a key property of encoder-decoder Transformers, i.e. their generative ability, to continuously learn Neural Machine Translation systems.
Outcome:	The proposed approach can counteract catastrophic forgetting without explicit memorization of training data.

Leitner-Guided Memory Replay for Cross-lingual Continual Learning (2024.naacl-long)

Copied to clipboard

Meryem M’hamdi, Jonathan May

Multilinguality

Challenge:	Various continual learning approaches have proposed to mitigate catastrophic forgetting by restricting the data buffer or limiting the data size of a model.
Approach:	They propose to use a human-inspired spaced-repetition technique to prioritize examples for cross-lingual continual learning.
Outcome:	The proposed approach significantly and consistently decreases forgetting while maintaining accuracy across natural language understanding tasks, language orders, and languages.

SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models (2024.emnlp-main)

Copied to clipboard

Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, Jinqiao Wang

Language Models

Challenge:	Existing methods fail to fully exploit the knowledge embedded in models from previous tasks . Existing techniques fail to exploit the information embedded in previous tasks, resulting in a large number of replay samples to achieve good results.
Approach:	They propose a method that uses attention weights to extract knowledge from previous tasks . they use a data replay strategy to extract the knowledge from the previous tasks.
Outcome:	The proposed method achieves comparable or even better performance with only 1/10 of replayed data used by other methods.

Don’t Half-listen: Capturing Key-part Information in Continual Instruction Tuning (2025.acl-long)

Copied to clipboard

Yongquan He, Wenyuan Zhang, Xuancheng Huang, Peng Zhang, Lingxun Meng, Xiang Zhou, Ke Zeng, Xunliang Cai

Language Models

Challenge:	Existing methods to improve instruction tuning for large language models may cause catastrophic forgetting (CF) CF is a problem where previously learned abilities are degraded .
Approach:	They propose a continual instruction tuning method that uses key-part information gain to replay data and refine training objective.
Outcome:	The proposed method achieves superior performance on both seen and held-out tasks.

History repeats: Overcoming catastrophic forgetting for event-centric temporal knowledge graph completion (2023.findings-acl)

Copied to clipboard

Mehrnoosh Mirtaheri, Mohammad Rostami, Aram Galstyan

Knowledge Representation

Challenge:	Existing methods for knowledge graph completion are incomplete and can lead to errors . retraining the model with the entire updated TKG can mitigate forgetting but is computationally burdensome.
Approach:	They propose a temporal regularization framework that allows repurposing of parameters . they propose 'clustering-based experience replay' that reinforces the past knowledge .
Outcome:	The proposed framework adapts to new events while reducing catastrophic forgetting.

Continual Learning Using Only Large Language Model Prompting (2025.coling-main)

Copied to clipboard

Jiabao Qiu, Zixuan Ke, Bing Liu

Semantic Text Processing

Challenge:	Existing continuous learning paradigms fine-tune language model parameters or use adapters or variants to adapt the LM.
Approach:	They propose a new continual learning paradigm wherein a large language model is regarded as a black box.
Outcome:	The proposed method outperforms baselines by a large margin in learning tasks incrementally.

LPC: A Logits and Parameter Calibration Framework for Continual Learning (2022.findings-emnlp)

Copied to clipboard

Xiaodi Li, Zhuoyi Wang, Dingcheng Li, Latifur Khan, Bhavani Thuraisingham

Responsible & Trustworthy NLP

Challenge:	Existing approaches to solve catastrophic forgetting problem are varied . current approaches to learn continuous learning are based on replay-based methods .
Approach:	They propose to calibrate parameters and logits so that preserving old parameters and generalized learning on new concepts can be solved simultaneously.
Outcome:	The proposed model achieves state-of-the-art performance in all scenarios.

TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining (2025.acl-long)

Copied to clipboard

Jeffrey Li, Mohammadreza Armandpour, Seyed Iman Mirzadeh, Sachin Mehta, Vaishaal Shankar, Raviteja Vemulapalli, Samy Bengio, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pouransari, Fartash Faghri

Language Models

Challenge:	Large language models (LLMs) trained on historical web data inevitably become outdated.
Approach:	They propose a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) they also design time-stratified evaluations to assess how well various continual learning methods adapt to new data while retaining past knowledge.
Outcome:	The proposed benchmarks show that autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x).