SDC-LoRA: Singular-Subspace Drift Controlled LoRA to Mitigate Knowledge Forgetting (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to adapt LLMs to new tasks focus on limiting knowledge forgetting . et al., 2023b) suggest a solution to this problem by limiting update energy in the principal singular subspace of W0 . |
| Approach: | They propose a low-rank Adaptation (LoRA) that steers early updates away from principal directions and mitigates forgetting by constraining update energy in the principal singular subspace of W0. |
| Outcome: | The proposed model mitigates forgetting on MMLU, TruthfulQA, and HellaSwag while keeping minor-subspace updates unchanged. |
Similar Papers
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to mitigate catastrophic forgetting can be broadly categorized into data-based, architecture-based and learning-based methods. |
| Approach: | They propose a subspace regularization method on LoRA structure that imposes constraints on direction of updating matrix’s null space. |
| Outcome: | The proposed method reduces scale of output change while introducing minimal constraint on model capacity. |
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (2025.naacl-long)
Copied to clipboard
| Challenge: | Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory costs. |
| Approach: | They propose a simple yet effective method that initializes low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrics frozen. |
| Outcome: | The proposed approach only updates the minor components of the weight matrix while keeping the principal singular components frozen. |
Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation? (2026.acl-long)
Copied to clipboard
| Challenge: | In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters. |
| Approach: | They propose a low-rank adaptation approach that injects parameterized singular components with spectral clipping into the pre-trained model. |
| Outcome: | The proposed method improves performance and retains pre-trained knowledge while preserving the weights of the model. |
G-LoRA: Global-Local Decoupled Low-Rank Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves the fine-tuning efficiency and performance of large language models. |
| Approach: | They propose a low-rank adaptive approach that decomposes update matrix into global and local adapters and assigns them to local and global adapters. |
| Outcome: | The proposed method achieves up to 2.7% accuracy improvement over LoRA and its variants on commonsense reasoning, mathematical reasoning, and code generation. |
Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks. |
| Approach: | They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace . |
| Outcome: | The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting. |
Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models, but is ineffective at removing backdoor behaviors from poisoned pretrained models when fine-timing on clean datasets. |
| Approach: | They propose a low-rank Adaptation method which increases spectral strength and corrects alignment through clean-strengthened regularization and trigger-insensitive constraints. |
| Outcome: | The proposed method significantly reduces attack success rates while maintaining clean accuracy. |
HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)
Copied to clipboard
Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers. |
| Approach: | They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters . |
| Outcome: | The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork. |
How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM? (2025.findings-naacl)
Copied to clipboard
Sergey Pletenev, Maria Marina, Daniil Moskovskiy, Vasily Konovalov, Pavel Braslavski, Alexander Panchenko, Mikhail Salnikov
| Challenge: | Low-rank adaptation (LoRA) is a popular training technique for updating or domain-specific adaptation of Large Language Models (LLMs). |
| Approach: | They propose to use low-rank adaptation to incorporate new facts into the LLM without compromising previously learned knowledge. |
| Outcome: | The proposed approach is harmful because the model's performance declines after such fine-tuning. |
LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models encode vast factual knowledge, yet their inability to selectively forget specific information hinders privacy protection, bias mitigation, and post-deployment correction. |
| Approach: | They propose a LoRA-based negative-only unlearning framework that updates only low-rank adapters while freezing the backbone. |
| Outcome: | The proposed framework reduces computational cost by about an order of magnitude compared to full fine-tuning and memory-editing methods. |
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture (2025.naacl-long)
Copied to clipboard
| Challenge: | balancing the training budget, downstream performance, and general capabilities of large language models remains a challenge in many applications. |
| Approach: | They propose a mixture of expert framework based on Soft LoRA and Identity Mixture . SLIM allows dynamic routing between LoRA adapters and identity layers . |
| Outcome: | The proposed framework reduces training cost while maintaining general capabilities . it can be open-sourced upon publication. |