RA-LoRA: Rank-Adaptive Parameter-Efficient Fine-Tuning for Accurate 2-bit Quantized Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) with their extensive parameters and high memory demands are challenging to fine-tune for specific applications with limited resources. |
| Approach: | They propose a method that dynamically adjusts the adapter’s rank using rank-subspace analysis, optimizing performance with fewer parameters. |
| Outcome: | The proposed method improves model accuracy with minimal parameter changes and demonstrates the importance of rank dynamics in optimizing quantized LLMs. |
Similar Papers
Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)
Copied to clipboard
Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable . |
| Approach: | They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition. |
| Outcome: | The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity . |
Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large language models . standard LoRA lacks mechanisms for uncertainty quantification, leading to overconfident and poorly calibrated models. |
| Approach: | They propose a parameter-efficient Bayesian LoRA method that decomposes weight updates into low-rank matrices. |
| Outcome: | The proposed method achieves strong performance with improved calibration and generalization while maintaining computational efficiency. |
G-LoRA: Global-Local Decoupled Low-Rank Adaptation (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves the fine-tuning efficiency and performance of large language models. |
| Approach: | They propose a low-rank adaptive approach that decomposes update matrix into global and local adapters and assigns them to local and global adapters. |
| Outcome: | The proposed method achieves up to 2.7% accuracy improvement over LoRA and its variants on commonsense reasoning, mathematical reasoning, and code generation. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
NormAL LoRA: What is the perfect size? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are crucial for enabling intelligent experiences across applications. |
| Approach: | They propose a low-rank adaptive localization method that uses rank-norm regularization to determine the optimal rank for each weight matrix. |
| Outcome: | NormAL LoRA reduces adapter parameters by 37% while preserving full fine-tuning performance. |
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)
Copied to clipboard
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models. |
| Approach: | They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks. |
| Outcome: | The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank. |
L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware PEFT methods have been developed to reduce memory and computational costs associated with large language models. |
| Approach: | They propose a method that integrates Quantization-Aware Training (QAT) with LoRA to reduce memory overhead and improve model accuracy. |
| Outcome: | The proposed method significantly reduces QAT’s memory overhead while preserving the advantage of QAT in producing fully quantized LLMs with high accuracy. |
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) fine-tunes large language models due to its significant reduction in trainable parameters, but its backward updates require storing high-dimensional intermediate activations and optimizer states, requiring high peak GPU memory. |
| Approach: | They propose a low-dimensional adaptation approach to fine-tune large language models which freezes a first projection matrix while introducing a lower-dimensional trainable square matrix. |
| Outcome: | The proposed approach reduces trainable parameters and peak GPU memory footprint while preserving low-dimensional trainable square matrix. |
DenseLoRA: Dense Low-Rank Adaptation of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient approach for adapting large language models (LLMs) but many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization. |
| Approach: | They propose a low-rank adaptation approach that fine-tunes two low-ranked matrices and adapts them through a dense low-Rank matrix, improving parameter utilization and adaptation efficiency. |
| Outcome: | The proposed approach achieves 83.8% accuracy with only 0.01% of trainable parameters compared to LoRA's 80.8% with 0.70% of trainability parameters on LLaMA3-8B. |
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient way to fine-tune large language models (LLMs) but its memory overhead restricts it to scale up when the model size increases. |
| Approach: | They propose a low-rank adaptation method which decomposes model weight updates into a pair of low-ranked projection matrices and incorporates a regularizer to improve generalization. |
| Outcome: | The proposed method achieves better performance under few-shot settings and similar performance to the SOTA low-rank adaptation methods. |