Papers by Hyock Kwon
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning (2024.emnlp-industry)
Copied to clipboard
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Kwon, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh
| Challenge: | Existing methods to fine tune large language models require huge memory, limiting the choice to acquire Larger models. |
| Approach: | They propose an efficient quantization approach for dynamic low-rank adaptation that can efficiently fine tune large language models on a set of pre-defined LoRA ranks. |
| Outcome: | The proposed method outperforms QLoRA and is competitive to QLouRA and outperformed when employing its optimal rank. |