SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation (2025.naacl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) has been demonstrated effective in reducing the trainable parameter number when fine-tuning a large foundation model (LLM). |
| Approach: | They propose a low-rank adaptation approach that reduces the number of trainable parameters while enhancing model performance. |
| Outcome: | The proposed approach outperforms existing parameter-efficient fine-tuning methods while achieving substantial reductions in computational cost and memory requirements. |
Similar Papers
Sensitivity-LoRA : Low-Load Sensitivity-Based Fine-Tuning for Large Language Models (2025.findings-emnlp)
Copied to clipboard
Hao Zhang, Bo Huang, Zhenjia Li, Xi Xiao, Hui Yi Leong, Zumeng Zhang, Xinwei Long, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) is a promising approach to adapting LLMs to specialized tasks . existing rank allocation techniques remain computationally inefficient and unstable . |
| Approach: | They propose a low-rank adapted model that approximates model weight updates using low-ranked decomposition. |
| Outcome: | The proposed method is limited by its uniform rank allocation to each incremental matrix . it leverages the second-order derivatives of the loss function to capture weight sensitivity . |
Sparsity May Be All You Need: Sparse Random Parameter Adaptation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods aim at reducing computational and memory resources for fine-tuning large language models. |
| Approach: | They propose to train on a small number of parameters instead of all model parameters . they compare the method to LoRA and find it to be efficient . |
| Outcome: | The proposed method is competitive with LoRA when using a similar number of trainable parameters. |
Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps (2025.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models. |
| Approach: | They propose a low-rank Adaptation technique that harnesses the expressiveness of spectral bases to re-parameterize LoRA from a sparse spectral subspace. |
| Outcome: | The proposed technique achieves greater efficiency with fewer parameters than baselines on various downstream tasks, including commonsense reasoning, math reasoning, and code generation. |
GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods focus on layer-selective and data-selectory fine-tuning, but ignore the fact that different data points contribute varying degrees to distinct model layers. |
| Approach: | They propose a method that performs selective fine-tuning at both data and layer dimensions as integral components of a unified optimization strategy. |
| Outcome: | Experiments show that the proposed method outperforms baseline methods in terms of performance and performance. |
LoRAN: Improved Low-Rank Adaptation by a Non-Linear Transformation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods for fine-tuning large language models have shown great improvements on a wide range of NLP tasks. |
| Approach: | They propose to introduce a non-linear transformation to improve performance of adapters by introducing a low-rank adaptation to fit the accumulated weight updates. |
| Outcome: | The proposed method outperforms a baseline on SAMSum and 20 Newsgroups tasks and even improves the classification task by 1.95 points when a lower rank is applied. |
Sparse Low-rank Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained large language models in a parameter-efficient manner are gaining traction within the research community. |
| Approach: | They propose a method of low-rank adaptation that enables dynamic adjustments to the intrinsic rank during the adaptation process. |
| Outcome: | The proposed approach outperforms the current method with a fixed and unalterable intrinsic rank and a low-rank adaptation process. |
SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained models are often adapted to a desired domain or task through a fine-tuning stage. |
| Approach: | They propose an end-to-end solution for sparse parameter-efficient fine-tuning of large pre-trained models. |
| Outcome: | The proposed approach can be used to combine sparse weights with low-rank adapters without losing sparsity and accuracy. |
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
Beyond Full Fine-tuning: Harnessing the Power of LoRA for Multi-Task Instruction Tuning (2024.lrec-main)
Copied to clipboard
Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, Le Sun
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning algorithm for large-scale language models. |
| Approach: | They conduct a systematic study of Low-Rank Adaptation (LoRA) on diverse tasks and rich resources with different learning capacities. |
| Outcome: | The proposed algorithm can achieve remarkable performance in high-resource and multi-task scenarios, even comparable to full fine-tuning. |
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) fine-tunes large language models due to its significant reduction in trainable parameters, but its backward updates require storing high-dimensional intermediate activations and optimizer states, requiring high peak GPU memory. |
| Approach: | They propose a low-dimensional adaptation approach to fine-tune large language models which freezes a first projection matrix while introducing a lower-dimensional trainable square matrix. |
| Outcome: | The proposed approach reduces trainable parameters and peak GPU memory footprint while preserving low-dimensional trainable square matrix. |