| Challenge: | Existing continuous learning paradigms fine-tune language model parameters or use adapters or variants to adapt the LM. |
| Approach: | They propose a new continual learning paradigm wherein a large language model is regarded as a black box. |
| Outcome: | The proposed method outperforms baselines by a large margin in learning tasks incrementally. |
Similar Papers
Continual Learning of Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines . |
| Approach: | This tutorial offers a comprehensive exploration of continual learning in the context of large language models. |
| Outcome: | This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly . |
Fine-tuned Language Models are Continual Learners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on large language models relies on intuition that most tasks can be described via natural language instructions. |
| Approach: | They propose that a model should be able to keep extending its knowledge without forgetting previous skills. |
| Outcome: | The proposed model can learn 8 new diverse language generation tasks while maintaining good performance on previous tasks, spanning in total of 70 datasets. |
Continual Lifelong Learning in Natural Language Processing: A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to continual learning (CL) are costly and time-consuming. |
| Approach: | They propose to examine the problem of continual learning in NLP through the lens of various NLP tasks and provide a critical review of existing methods. |
| Outcome: | The proposed methods are critical to the development of CL models and provide a critical review of existing methods and datasets. |
In-context Continual Learning Assisted by an External Continual Learner (2025.coling-main)
Copied to clipboard
| Challenge: | Existing continual learning methods suffer from catastrophic forgetting (CF) . Existing methods rely on fine-tuning or adapting large language models (LLMs) |
| Approach: | They propose an approach that integrates an external continual learner (ECL) with ICL to enable scalable CL without catastrophic forgetting (CF). |
| Outcome: | The proposed approach outperforms existing baselines while maintaining high performance. |
Empowering Large Language Model for Continual Video Question Answering with Collaborative Prompting (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing VideoQA models struggle to adapt to new questions or tasks posed by newly available content. |
| Approach: | They propose a continual learning framework that fine-tunes a large language model for a sequence of tasks and integrates specific question constraint prompting, knowledge acquisition prompting and visual temporal awareness prompting. |
| Outcome: | The proposed model achieves 55.14% accuracy on both NExT-QA and DramaQA datasets and 71.24% accuracy for DramaQA. |
Rethinking Long Context Generation from the Continual Learning Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with processing long contexts due to the limited context window. |
| Approach: | They propose to combine a limited context window with a continual learning perspective to improve LLMs' efficiency in processing long contexts. |
| Outcome: | The proposed models improve the performance of Large Language Models (LLMs) by integrating learning strategies with existing approaches. |
Soft Prompting for Unlearning in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing ethical and safety considerations for large language models are important for deployment . however, some ethical concerns have been raised due to the presence of private, sensitive, or harmful information in the training data. |
| Approach: | They propose a framework that learns prompt tokens that are prepended to a query to induce unlearning in LLMs. |
| Outcome: | The proposed method improves the trade-off between utility and forgetting for text classification and question-answering. |
Exploring Continual Learning for Code Generation Models (2023.acl-short)
Copied to clipboard
Prateek Yadav, Qing Sun, Hantian Ding, Xiaopeng Li, Dejiao Zhang, Ming Tan, Parminder Bhatia, Xiaofei Ma, Ramesh Nallapati, Murali Krishna Ramanathan, Mohit Bansal, Bing Xiang
| Challenge: | Large-scale code generation models such as Copilot and CodeT5 are expensive to train and re-train. |
| Approach: | They propose a benchmark for Continual Learning (CL) that covers a wide range of tasks with different input and output programming languages. |
| Outcome: | The proposed method improves on Prompt Pooling with Teacher Forcing, which suffers catastrophic forgetting due to stark distribution shifts in coding tasks. |
Continual Training of Language Models for Few-Shot Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. |
| Approach: | They propose to continuously post-train an LM with unlabeled domains to expand its knowledge without forgetting previous skills. |
| Outcome: | The proposed system improves few-shot end-task learning in these domains. |
Continual-learning for Modelling Low-Resource Languages from Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing models for low-resource languages with catastrophic forgetting pose several challenges, including learning to model multi-lingual scenarios. |
| Approach: | They propose to employ a continual learning strategy using parts-of-speech code-switching and replay adapter strategies to mitigate catastrophic forgetting gap while training LLM from LLM. |
| Outcome: | The proposed architecture is able to train LLMs from LLM and mitigate catastrophic forgetting gap on vision language tasks. |