ELLA: Efficient Lifelong Learning for Adapters in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to training Large Language Models (LLMs) suffer from catastrophic forgetting when adapted sequentially to new tasks in a continual learning (CL) setting. Existing methods are impractical and could potentially violate privacy. |
| Approach: | They propose a training framework built on the principle of selective subspace de-correlation that characterizes the structure of past updates and penalizes alignments along their high-energy, task-specific directions. |
| Outcome: | The proposed training framework achieves state-of-the-art CL performance on three popular benchmarks spanning both classification and generative tasks with relative accuracy gains of up to 9.6% and a 35 smaller memory footprint. |
Similar Papers
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks. |
| Approach: | They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace . |
| Outcome: | The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting. |
Continual Learning of Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines . |
| Approach: | This tutorial offers a comprehensive exploration of continual learning in the context of large language models. |
| Outcome: | This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly . |
Orthogonal Subspace Learning for Language Model Continual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks. |
| Approach: | They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks. |
Unlocking Continual Learning Abilities in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learning models (LMs) incorporate old task data or task-wise inductive bias into LMs, but old data and accurate task information are often unavailable or costly to collect. |
| Approach: | They propose a rehearsal-free method that updates model parameters with large magnitudes . they found that the L1-normalized magnitude distribution is different when different task data is used . |
| Outcome: | The proposed method improves accuracy and performance on four CL benchmarks. |
AGTAO: Robust and Stabilized LLM Unlearning via Adversarial Gating Training with Adaptive Orthogonality (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) unintentionally memorize sensitive data, posing privacy and security risks. |
| Approach: | They propose a framework that reconciles unlearning efficacy and utility preservation by using a latent-space gating mechanism to simulate internal recovery attempts. |
| Outcome: | The proposed framework achieves superior trade-off between unlearning efficacy and model utility. |
Decoupling Generalization and Adaptation in Meta-Learning for Large Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Adapting large language models to specific downstream tasks requires multi-step fine-tuning with substantial training data, incurring significant computational overhead. |
| Approach: | They propose a framework that separates learning generalizable initializations and adaptation through dedicated parameter spaces. |
| Outcome: | The proposed framework outperforms existing meta-learning and standard multi-task baselines on common-sense reasoning, mathematics, logic, medical and coding benchmarks. |
SLoRA: Balancing Plasticity and Forgetting in Large Language Models for Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved remarkable success across diverse tasks through large-scale pretraining. |
| Approach: | They propose a framework that filters noisy components from LoRA updates via subspace similarity with the base model. |
| Outcome: | The proposed framework improves accuracy by 12%, reduces forgetting by 29%, and filters out over 30% of LoRA parameters identified as noisy. |
Accelerating Dense LLMs via L0-regularized Mixture-of-Experts (2025.acl-short)
Copied to clipboard
| Challenge: | Existing methods for accelerating large language models (LLMs) suffer from slow and costly inference. |
| Approach: | They propose a lightweight MoE approach using cluster confusion matrix and dynamic batching to accelerate dense LLMs. |
| Outcome: | The proposed method achieves 2.5x speedup over dense models while maintaining competitive performance. |
DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong effectiveness and robustness when fine-tuned as dense retrievers. |
| Approach: | They propose a training framework that leverages pruned LLMs to train smaller generalizable dense retrievers. |
| Outcome: | The proposed training framework offers better multilingual and long-context capabilities than traditional encoder-based retrievers and achieves strong performance across multiple tasks and languages. |