Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal (2024.acl-long)
Copied to clipboard
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, Jinsong Su
| Challenge: | Existing methods to train LLMs on previous training data are not feasible in real-world applications because of catastrophic forgetting. |
| Approach: | They propose a framework that uses the LLM to generate synthetic instances for rehearsal and refine the instance outputs based on the synthetic inputs. |
| Outcome: | The proposed framework achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. |
Similar Papers
Continual Learning of Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines . |
| Approach: | This tutorial offers a comprehensive exploration of continual learning in the context of large language models. |
| Outcome: | This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly . |
Self-training Large Language Models through Knowledge Detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) often require extensive labeled datasets and training compute to achieve impressive performance across downstream tasks. |
| Approach: | They propose a self-training paradigm where the LLM curates its own labels and selectively trains on unknown data samples identified through a reference-free consistency method. |
| Outcome: | The proposed model reduces the dependency on large labeled datasets and mitigates catastrophic forgetting in out-of-distribution benchmarks. |
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive performance across numerous NLP tasks, but fine-tuning them for Machine Translation (MT) often introduces catastrophic forgetting, compromising the broad general abilities of LLMs and introducing potential security risks. |
| Approach: | They propose a method that harnesses the strong generative capabilities of Large Language Models to create rationales for training data, which are then "replayed" to prevent forgetting. |
| Outcome: | The proposed approach harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting. |
Exploring Forgetting in Large Language Model Pre-Training (2025.acl-long)
Copied to clipboard
| Challenge: | Existing research on task-level forgetting in LLMs has focused on pretraining . but, there is limited attention to finer-grained forgetting during training . |
| Approach: | They investigated the existence and measurement of forgetting in pre-training . they examined low-cost, straightforward methods to mitigate forgetting during the pre- training phase . |
| Outcome: | The proposed methods could be used to mitigate forgetting during the pre-training phase and offer insights into the dynamics of forgetting. |
Revisiting Catastrophic Forgetting in Large Language Model Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Catastrophic Forgetting (CF) compromises the effectiveness of large language models during fine-tuning, yet the underlying causes of CF remain largely unexplored. |
| Approach: | They propose a method to flatten the model loss landscape to mitigate CF by flattening the loss landscape. |
| Outcome: | The proposed method complements existing anti-forgetting strategies, further enhancing the resistance of LLMs to CF. |
Unveiling and Addressing Pseudo Forgetting in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing efforts to mitigate catastrophic forgetting in continual learning have not been studied. |
| Approach: | They propose a rationale-guided replay framework that allows models to leverage their capabilities and provide partial external correct rationales to the original instructions. |
| Outcome: | The proposed framework mitigates pseudo forgetting while maintaining model plasticity. |
Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored using large language models to generate synthetic datasets . however, the effectiveness of the LLM-generated synthetic data is inconsistent across different classification tasks. |
| Approach: | They propose to use large language models to generate synthetic datasets to better understand factors that moderate the effectiveness of LLM-generated synthetic data. |
| Outcome: | The results show that subjectivity is negatively associated with the performance of the model trained on synthetic data. |
Temporal Sampling for Forgotten Reasoning in LLMs (2026.acl-long)
Copied to clipboard
Yuetai Li, Zhangchen Xu, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, Xiang Yue, Radha Poovendran
| Challenge: | a new metric measures the percentage of questions that were answered incorrectly during fine-tuning . |
| Approach: | They propose a decoding strategy that draws outputs from multiple checkpoints along the training trajectory. |
| Outcome: | The proposed method improves reasoning performance and consistency across benchmarks. |
Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown a powerful ability for text generation, but undesired behaviors such as toxicity and hallucinations can manifest. |
| Approach: | They propose to formalize text generation as a future-constrained generation problem to minimize undesirable behaviors and enforce faithfulness to instructions. |
| Outcome: | The proposed approach is effective across three tasks, including keyword-constrained generation, toxicity reduction, and factual correctness in question-answering. |