Papers with A**daptation
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
SLoRA: Balancing Plasticity and Forgetting in Large Language Models for Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved remarkable success across diverse tasks through large-scale pretraining. |
| Approach: | They propose a framework that filters noisy components from LoRA updates via subspace similarity with the base model. |
| Outcome: | The proposed framework improves accuracy by 12%, reduces forgetting by 29%, and filters out over 30% of LoRA parameters identified as noisy. |
LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. |
| Approach: | They propose a method that localizes and optimizes critical parameters during training . they propose 'LoSiA-Pro' which reduces training latency by 27% . |
| Outcome: | The proposed method achieves minimal performance drop compared to full fine-tuning while requiring the least training time across domain specialization and common-sense reasoning tasks. |
DisLoRA: Task-specific Low-Rank Adaptation via Orthogonal Basis from Singular Value Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks demonstrate that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |
| Approach: | They propose a framework that leverages singular value decomposition to decompose pretrained weight matrices into orthogonal backbone and task-specific subspaces. |
| Outcome: | Extensive experiments on GLUE and Commonsense Reasoning benchmarks show that DisLoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, and SORSA. |