Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying (2024.naacl-long)
Copied to clipboard
| Challenge: | a new paradigm for low-rank Adaptation (LoRA) uses weight tying and selective training to improve parameter efficiency. |
| Approach: | They propose a paradigm that uses weight tying and selective training to enhance parameter efficiency of Low-rank Adaptation. |
| Outcome: | The proposed paradigm achieves comparable performance to LoRA with reduced model complexity . the proposed paradigm can be used for a variety of tasks and languages . |
Similar Papers
LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is currently the most commonly used PEFT method for fine-tuning models with billions of parameters. |
| Approach: | They propose to use low-rank Adaptation to evaluate LoRA parameter features and then retain LoRA for important layers and the other layers share the same LoRA. |
| Outcome: | The proposed method achieves comparable performance to full fine-tuning and LoRA while retaining 50% of the LoRA parameters on average. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps (2025.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models. |
| Approach: | They propose a low-rank Adaptation technique that harnesses the expressiveness of spectral bases to re-parameterize LoRA from a sparse spectral subspace. |
| Outcome: | The proposed technique achieves greater efficiency with fewer parameters than baselines on various downstream tasks, including commonsense reasoning, math reasoning, and code generation. |
From Bottom to Top: Extending the Potential of Parameter Efficient Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models primarily focus on the interaction between different layers, ignoring the fact that different layers store different information. |
| Approach: | They propose a Parameter Efficient Fine-Tuning method which freeze pre-trained parameters and fine-tunes only a few task-specific parameters. |
| Outcome: | The proposed methods reduce parameter count to nearly half by omitting fine-tuning in the middle layers. |
Sparsity May Be All You Need: Sparse Random Parameter Adaptation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods aim at reducing computational and memory resources for fine-tuning large language models. |
| Approach: | They propose to train on a small number of parameters instead of all model parameters . they compare the method to LoRA and find it to be efficient . |
| Outcome: | The proposed method is competitive with LoRA when using a similar number of trainable parameters. |
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method for large language models. |
| Approach: | They propose a drop-in extension that reparameterizes a rank-rtot update as a sum of K *static* low-rank experts. |
| Outcome: | Experiments on reasoning and knowledge-intensive benchmarks show consistent gains over matched-budget LoRA. |
LoRAN: Improved Low-Rank Adaptation by a Non-Linear Transformation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods for fine-tuning large language models have shown great improvements on a wide range of NLP tasks. |
| Approach: | They propose to introduce a non-linear transformation to improve performance of adapters by introducing a low-rank adaptation to fit the accumulated weight updates. |
| Outcome: | The proposed method outperforms a baseline on SAMSum and 20 Newsgroups tasks and even improves the classification task by 1.95 points when a lower rank is applied. |
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Several approaches to parameter-efficient fine-tuning have been proposed . low-rank Adaptation (LoRA) does not consider the varying importance of each layer . |
| Approach: | They propose a method that allocates a different rank for each layer and performs pruning throughout the training process. |
| Outcome: | The proposed method is based on eight GLUE benchmarks and is currently the state of the art. |
BeamLoRA: Beam-Constraint Low-Rank Adaptation (2025.acl-long)
Copied to clipboard
Naibin Gu, Zhenyu Zhang, Xiyu Liu, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, Haifeng Wang
| Challenge: | Low-Rank Adaptation (LoRA) is one of the most efficient parameter-efficient fine-tuning methods. |
| Approach: | They propose to conceptualize each LoRA module as a beam where each rank corresponds to a potential sub-solution. |
| Outcome: | The proposed method improves performance on three base models and 12 datasets. |
CoLA: Collaborative Low-Rank Adaptation (2025.findings-acl)
Copied to clipboard
| Challenge: | The scaling law of Large Language Models (LLMs) reveals diminishing return on performance as model scale increases. |
| Approach: | They propose a more flexible LoRA architecture with an efficient initialization scheme . they propose combining three collaborative strategies to enhance performance . |
| Outcome: | The proposed model outperforms existing methods in low-sample scenarios. |