Papers by Zequan Liu
PARA: Parameter-Efficient Fine-tuning with Prompt-Aware Representation Adjustment (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning excel in the context of single-backbone multi-tenant applications. |
| Approach: | They propose to integrate a lightweight vector generator within each Transformer layer to improve prompt-aware representation adjustment. |
| Outcome: | The proposed method surpasses current benchmarks in terms of performance despite having a similar number of adjustable parameters. |
ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) has demonstrated commendable performance as a popular method . however, it is implemented with a fixed intrinsic rank that might not be ideal for downstream tasks. |
| Approach: | They propose a method that estimates the importance score of each LoRA rank and prunes abundant LoRA ranks to improve performance. |
| Outcome: | The proposed method outperforms baselines on a variety of tasks with comparable parameters. |