F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to address Catastrophic Forgetting (CF) have been developed to avoid forgetting and maintain system extensibility. |
| Approach: | They propose a method to reduce Catastrophic Forgetting (CF) by decomposing feed-forward layers into discrete memory cells and ensuring robust extendability. |
| Outcome: | The proposed method achieves higher BLEU scores and almost zero forgetting while maintaining robust extendability. |
Similar Papers
Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks. |
| Approach: | They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace . |
| Outcome: | The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting. |
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Neural machine translation models suffer from catastrophic forgetting during continual training . models tend to overfit to frequent observations in the in-domain data but forget previously learned knowledge. |
| Approach: | They investigated the causes of catastrophic forgetting in NMT models by examining their parameters and modules. |
| Outcome: | The proposed model forgets previously learned knowledge and swings to fit new data . the results show that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general- domain. |
Harmonizing the Past, Present, and Future: A Null-Space Constrained Region-Specific Method for Continual Learning in LLMs (2026.acl-long)
Copied to clipboard
Jinhui Chen, Shizhu He, Xingchang Yang, Huanxuan Liao, Yequan Wang, Xiangwen Liao, Wenhao Teng, Kang Liu, Jun Zhao
| Challenge: | Existing continual learning paradigms prioritize instant performance through dense updates, leading to catastrophic forgetting and rapid exhaustion of model capacity. |
| Approach: | They propose a method that preserves previously acquired knowledge and acquires new task-specific skills while preserving sufficient parameter capacity for subsequent adaptation. |
| Outcome: | The proposed method is based on the brain's functional partitioning and can be used to map tasks between specialized and generalist neurons. |
Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Currently, continuous learning methods suffer from catastrophic forgetting problem, causing model to forget previous knowledge while learning new knowledge. |
| Approach: | They propose a two-stage continuous learning method based on local features of the real loss to avoid catastrophic forgetting problem. |
| Outcome: | The proposed method achieves significant improvements on domain adaptation and more challenging language adaptation tasks. |
Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Semiparametric language models (LMs) use static storage, which lacks learning capability and is disconnected from the internal information flow of the parametric models. |
| Approach: | They reconceptualize the non-parametric memory represented by kNN-LM as a learnable Mixture-of-Neighbors Induction Memory (MoNIM) this synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks . |
| Outcome: | The proposed model is a learnable Mixture-of-neighbors induction memory (MoNIM) it synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks (FFNs). |
Self-generated Replay Memories for Continual Neural Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation systems exhibit strong performance in several different languages, but their ability to learn continuously is limited by catastrophic forgetting. |
| Approach: | They propose a method that leverages a key property of encoder-decoder Transformers, i.e. their generative ability, to continuously learn Neural Machine Translation systems. |
| Outcome: | The proposed approach can counteract catastrophic forgetting without explicit memorization of training data. |
Domain adapted machine translation: What does catastrophic forgetting forget and why? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) models can be specialized by domain adaptation, often fine-tuning on a dataset of interest. |
| Approach: | They propose a novel approach to understanding catastrophic forgetting during NMT adaptation by investigating the relationship between the data and the in-domain vocabulary coverage. |
| Outcome: | The proposed model can be specialized by fine-tuning on a domain of interest, but can fail to achieve the predicted quality of the target domain. |
SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models (2024.acl-long)
Copied to clipboard
Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che
| Challenge: | Existing methods to address catastrophic forgetting and knowledge transfer in large language models (LLMs) ignore potential of aligning the two modules to effectively address catastrophic forgetting and knowledge transfers simultaneously. |
| Approach: | They propose a Shared Attentive Learning & Selection module to align the PET learning and selection modules to address catastrophic forgetting and knowledge transfer simultaneously. |
| Outcome: | Experiments on two CL benchmarks show that the proposed framework is superior when scaled to different model sizes, different model architectures and unseen tasks. |
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
ELLA: Efficient Lifelong Learning for Adapters in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to training Large Language Models (LLMs) suffer from catastrophic forgetting when adapted sequentially to new tasks in a continual learning (CL) setting. Existing methods are impractical and could potentially violate privacy. |
| Approach: | They propose a training framework built on the principle of selective subspace de-correlation that characterizes the structure of past updates and penalizes alignments along their high-energy, task-specific directions. |
| Outcome: | The proposed training framework achieves state-of-the-art CL performance on three popular benchmarks spanning both classification and generative tasks with relative accuracy gains of up to 9.6% and a 35 smaller memory footprint. |