| Challenge: | Low-Rank Adaptation (LoRA) offers efficiency but constrains the model’s ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. |
| Approach: | They propose a training strategy that synergistically combines full and low-rank parameters and jointly updating within a unified low-ranked gradient subspace. |
| Outcome: | Extensive experiments on continual learning benchmarks show that GORP improves performance compared to state-of-the-art approaches. |
Similar Papers
Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks. |
| Approach: | They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace . |
| Outcome: | The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting. |
Orthogonal Subspace Learning for Language Model Continual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks. |
| Approach: | They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks. |
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to mitigate catastrophic forgetting can be broadly categorized into data-based, architecture-based and learning-based methods. |
| Approach: | They propose a subspace regularization method on LoRA structure that imposes constraints on direction of updating matrix’s null space. |
| Outcome: | The proposed method reduces scale of output change while introducing minimal constraint on model capacity. |
HFT: Half Fine-Tuning for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) with one or more fine-tuning phases can unlock various capabilities, but can be catastrophic forgetting during sequential training. |
| Approach: | They propose a method to regularly reset partial parameters to mitigate forgetting issues by using half fine-tuning instead of full fine-uning. |
| Outcome: | The proposed approach reduces the risk of catastrophic forgetting during training and the parametric knowledge lost during training may be overwhelmed by incoming training data. |
FARSS: Fisher-Optimized Adaptive Low-Rank and Singular-Vector Selection for Knowledge-Preserving Fine-Tuning (2026.findings-acl)
Copied to clipboard
Renxing Chen, Ziwei Xiang, Peisong Wang, Hongjian Fang, Meng Li, Fanhu Zeng, Yanan Zhu, Peipei Yang, Xu-Yao Zhang, Jian Cheng
| Challenge: | Low-rank adaptation methods for large language models have limitations in preserving world knowledge and limiting updates to preserve world knowledge. |
| Approach: | They propose a Fisher-optimized adaptive low Rank and Singular-VectorSelection framework for knowledge-preserving fine-tuning that allows efficient and task-sensitive updates. |
| Outcome: | The proposed framework outperforms existing methods for knowledge-preserving fine-tuning. |
Unlocking Continual Learning Abilities in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learning models (LMs) incorporate old task data or task-wise inductive bias into LMs, but old data and accurate task information are often unavailable or costly to collect. |
| Approach: | They propose a rehearsal-free method that updates model parameters with large magnitudes . they found that the L1-normalized magnitude distribution is different when different task data is used . |
| Outcome: | The proposed method improves accuracy and performance on four CL benchmarks. |
Is Parameter Collision Hindering Continual Learning in LLMs? (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to learn multiple tasks in parallel often lead to catastrophic forgetting, resulting in overwriting knowledge. |
| Approach: | They propose a non-collision low-rank Adaptation approach that leverages low collision rates to enhance continual learning (CL) in large language models. |
| Outcome: | The proposed approach achieves better task orthogonality and higher task orthognality than existing SOTA methods. |
UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter Efficient Fine-Tuning of Large Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods such as LoRA and VeRA use a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Approach: | They propose a parameter-efficient fine-tuning approach that leverages a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Outcome: | The proposed approach outperforms existing methods on GLUE and E2E benchmarks and is effective in instruction-tuning large language models and image classification models. |
META-LORA: Memory-Efficient Sample Reweighting for Fine-Tuning Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Supervised fine-tuning (SFT) is widely adopted for tailoring large language models (LLMs) to specific downstream tasks. |
| Approach: | They propose a memory-efficient method for automatic sample reweighting that learns to re-weight fine-tuning samples by minimizing the loss on a small, high-quality validation set. |
| Outcome: | Meta-LoRA learns to reweight fine-tuning samples by minimizing the loss on a small, high-quality validation set through an end-to-end bi-level optimization framework based on meta-learning. |
LoRMA: Low-Rank Multiplicative Adaptation for LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive generalization capabilities, but can be expensive to fine-tune due to high computational costs. |
| Approach: | They propose a low-rank multiplicative Adaptation technique that shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. |
| Outcome: | The proposed approach overcomes computational complexity and rank bottlenecks in terms of matrix multiplication metrics. |