Papers by Ganglin Bao
DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution (2024.acl-long)
Copied to clipboard
| Challenge: | Existing parameter-efficient fine-tuning methods such as Low-Rank Adaptation ignore the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-uning outcomes. |
| Approach: | They propose a parameter-efficient low-rank Adaptation method that decomposes high-rank LoRA layers into structured single-rank components and allows dynamic pruning of parameter budget . |
| Outcome: | The proposed method outperforms LoRA and LoRA with the same parameter budget and performance. |