| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
Similar Papers
SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models (LLMs) introduce parameter interference, leading to a gap in generalization performance for specific tasks compared to full fine-uning. |
| Approach: | They propose a parameter-separated low-rank adapter to account for task differences by decomposing LoRA’s parameter matrix into multiple independent subspaces and assigning them differentially to distinct tasks. |
| Outcome: | The proposed method outperforms LoRA in trainable parameter efficiency and overall model performance on various NLP tasks. |
ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular method . however, it is implemented with a fixed intrinsic rank that might not be ideal for downstream tasks. |
| Approach: | They propose a method that estimates the importance score of each LoRA rank and prunes abundant LoRA ranks to improve performance. |
| Outcome: | The proposed method outperforms baselines on a variety of tasks with comparable parameters. |
NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is essential for adapting large language models (LLMs). However, LoRA suffers from slow convergence and some recent LoRA variants, such as PiSSA, rely on Singular Value Decomposition (SVD) for initialization. |
| Approach: | They propose to introduce a small intermediate matrix between the low-rank matrices (A) and (B) and propose NyströmLoRA (NLoRA) which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. |
| Outcome: | The proposed approach improves on 5 natural language generation tasks and 8 natural language understanding tasks with minimal parameter overhead. |
MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)
Copied to clipboard
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei
| Challenge: | Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible. |
| Approach: | They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times . |
| Outcome: | The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. |
Flexora: Flexible Low-Rank Adaptation for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized artificial intelligence, but performance on specific tasks is limited by knowledge boundaries. |
| Approach: | They propose a method that automatically selects the most critical layers for fine-tuning to optimize performance across diverse downstream tasks. |
| Outcome: | The proposed method outperforms baseline models and natural language tasks. |
GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adaptive LoRA methods lack a theoretical foundation to guide this trade-off optimally. |
| Approach: | They propose a principled approach that estimates the intrinsic dimensionality of hidden data representations to adaptively select LoRA ranks. |
| Outcome: | Experiments show that GeLoRA outperforms adaptive LoRA methods by up to +1.0% . |
Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)
Copied to clipboard
Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable . |
| Approach: | They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition. |
| Outcome: | The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity . |
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-training/fine-tuning of pre-training models has become more expensive and resource-hungry. |
| Approach: | They propose a low-rank adaptation technique that trains LoRA blocks for a range of ranks instead of a single rank. |
| Outcome: | The proposed method trains LoRA blocks for a range of ranks instead of a single rank . it can train dynamic search-free models with DyLoRA at least 4 to 7 times faster than LoRA . |
Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity (2025.naacl-short)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient alternative to full-weight adaptation in federated fine-tuning of language models, significantly reducing computational costs. |
| Approach: | They propose a low-rank adaptation method that freezes original weights and trains only the update parametrized as a product of two low-ranked matrices. |
| Outcome: | The proposed method accelerates convergence and enhances the global model’s predictive performance. |
An Orthogonal High-Rank Adaptation for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) efficiently adapts LLMs to downstream tasks by decomposing LLM’s weight update into trainable low-rank matrices for fine-tuning. |
| Approach: | They propose an orthogonal high-rank adaptation for parameter-efficient fine-tuning that decomposes LLMs’ pre-trained weight matrices into orthogonals via QR decomposition and splits them into two low-redundancy high-ranked components. |
| Outcome: | Empirical results show that OHoRA outperforms LoRA and its variants and generates task-tailored representation spaces with 0.0371% trainable parameters. |