Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to mitigate catastrophic forgetting can be broadly categorized into data-based, architecture-based and learning-based methods. |
| Approach: | They propose a subspace regularization method on LoRA structure that imposes constraints on direction of updating matrix’s null space. |
| Outcome: | The proposed method reduces scale of output change while introducing minimal constraint on model capacity. |
Similar Papers
NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is essential for adapting large language models (LLMs). However, LoRA suffers from slow convergence and some recent LoRA variants, such as PiSSA, rely on Singular Value Decomposition (SVD) for initialization. |
| Approach: | They propose to introduce a small intermediate matrix between the low-rank matrices (A) and (B) and propose NyströmLoRA (NLoRA) which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. |
| Outcome: | The proposed approach improves on 5 natural language generation tasks and 8 natural language understanding tasks with minimal parameter overhead. |
Orthogonal Subspace Learning for Language Model Continual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks. |
| Approach: | They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks. |
Flexora: Flexible Low-Rank Adaptation for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized artificial intelligence, but performance on specific tasks is limited by knowledge boundaries. |
| Approach: | They propose a method that automatically selects the most critical layers for fine-tuning to optimize performance across diverse downstream tasks. |
| Outcome: | The proposed method outperforms baseline models and natural language tasks. |
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) fine-tunes large language models due to its significant reduction in trainable parameters, but its backward updates require storing high-dimensional intermediate activations and optimizer states, requiring high peak GPU memory. |
| Approach: | They propose a low-dimensional adaptation approach to fine-tune large language models which freezes a first projection matrix while introducing a lower-dimensional trainable square matrix. |
| Outcome: | The proposed approach reduces trainable parameters and peak GPU memory footprint while preserving low-dimensional trainable square matrix. |
Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks. |
| Approach: | They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace . |
| Outcome: | The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting. |
SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models (LLMs) introduce parameter interference, leading to a gap in generalization performance for specific tasks compared to full fine-uning. |
| Approach: | They propose a parameter-separated low-rank adapter to account for task differences by decomposing LoRA’s parameter matrix into multiple independent subspaces and assigning them differentially to distinct tasks. |
| Outcome: | The proposed method outperforms LoRA in trainable parameter efficiency and overall model performance on various NLP tasks. |
GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adaptive LoRA methods lack a theoretical foundation to guide this trade-off optimally. |
| Approach: | They propose a principled approach that estimates the intrinsic dimensionality of hidden data representations to adaptively select LoRA ranks. |
| Outcome: | Experiments show that GeLoRA outperforms adaptive LoRA methods by up to +1.0% . |
DenseLoRA: Dense Low-Rank Adaptation of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient approach for adapting large language models (LLMs) but many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization. |
| Approach: | They propose a low-rank adaptation approach that fine-tunes two low-ranked matrices and adapts them through a dense low-Rank matrix, improving parameter utilization and adaptation efficiency. |
| Outcome: | The proposed approach achieves 83.8% accuracy with only 0.01% of trainable parameters compared to LoRA's 80.8% with 0.70% of trainability parameters on LLaMA3-8B. |
ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular method . however, it is implemented with a fixed intrinsic rank that might not be ideal for downstream tasks. |
| Approach: | They propose a method that estimates the importance score of each LoRA rank and prunes abundant LoRA ranks to improve performance. |
| Outcome: | The proposed method outperforms baselines on a variety of tasks with comparable parameters. |
Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)
Copied to clipboard
Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable . |
| Approach: | They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition. |
| Outcome: | The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity . |