RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models use full finetunation, but this is impractical as language models continue to scale up. |
| Approach: | They propose a parameter-efficient fine-tuning method for large language models based on updating only a few rows and columns of the weight matrices in transformers. |
| Outcome: | The proposed method gives comparable or better accuracies than state-of-the-art methods while being more memory and computation-efficient. |
Similar Papers
MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work on multilingual evaluation has shown that there is a large gap between the performance of Large Language Models on English and other languages. |
| Approach: | They propose to finetune Llama-2 and Mistral models on two datasets to determine their effect on model performance on six downstream tasks covering forty one languages. |
| Outcome: | The proposed model can improve on six multilingual tasks while degrading on high-resource languages. |
UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter Efficient Fine-Tuning of Large Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods such as LoRA and VeRA use a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Approach: | They propose a parameter-efficient fine-tuning approach that leverages a low-rank approximation method that reduces the number of trainable parameters without compromising performance. |
| Outcome: | The proposed approach outperforms existing methods on GLUE and E2E benchmarks and is effective in instruction-tuning large language models and image classification models. |
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) fine-tunes large language models due to its significant reduction in trainable parameters, but its backward updates require storing high-dimensional intermediate activations and optimizer states, requiring high peak GPU memory. |
| Approach: | They propose a low-dimensional adaptation approach to fine-tune large language models which freezes a first projection matrix while introducing a lower-dimensional trainable square matrix. |
| Outcome: | The proposed approach reduces trainable parameters and peak GPU memory footprint while preserving low-dimensional trainable square matrix. |
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques. |
| Approach: | They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models. |
| Outcome: | The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies. |
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (2025.naacl-long)
Copied to clipboard
| Challenge: | Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory costs. |
| Approach: | They propose a simple yet effective method that initializes low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrics frozen. |
| Outcome: | The proposed approach only updates the minor components of the weight matrix while keeping the principal singular components frozen. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |
Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)
Copied to clipboard
Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable . |
| Approach: | They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition. |
| Outcome: | The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity . |
RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have strong generalizability, but fine-tuning involves updating all parameters, rendering full fine-uning resource-intensive. |
| Approach: | They propose a parameter-efficient fine-tuning method that updates all pre-trained parameters during inference. |
| Outcome: | The proposed method outperforms baseline methods on five benchmarks across 20 datasets. |
Full Parameter Fine-tuning for Large Language Models with Limited Resources (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require massive GPU resources for training. |
| Approach: | They propose a parameter-efficient optimization that fuses the gradient computation and parameter update in one step to reduce memory usage. |
| Outcome: | The proposed method reduces memory usage to 10.8% compared to the standard approach. |
Propulsion: Steering LLM with Tiny Fine-Tuning (2025.coling-main)
Copied to clipboard
| Challenge: | Propulsion is a parameter-efficient fine-tuning method that selectively re-scales specific dimensions of a pre-trained model without modifying the model’s parameters. |
| Approach: | They propose a parameter-efficient fine-tuning method that selectively re-scales specific dimensions of a pre-trained model without modifying the parameters. |
| Outcome: | The proposed method reduces parameter count from 355.3 million to 0.086 million while maintaining competitive performance across benchmarks. |