Papers by Sihwa Lee
RA-LoRA: Rank-Adaptive Parameter-Efficient Fine-Tuning for Accurate 2-bit Quantized Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) with their extensive parameters and high memory demands are challenging to fine-tune for specific applications with limited resources. |
| Approach: | They propose a method that dynamically adjusts the adapter’s rank using rank-subspace analysis, optimizing performance with fewer parameters. |
| Outcome: | The proposed method improves model accuracy with minimal parameter changes and demonstrates the importance of rank dynamics in optimizing quantized LLMs. |
Understanding and Improving Knowledge Distillation for Quantization Aware Training of Large Transformer Encoders (2022.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) has been used for quantization-aware training to improve the ability of a lightweight model with the transferred knowledge from the teacher. |
| Approach: | They propose two methods to improve attention recovery of quantized large Transformers by combining attention-map and attention-output losses. |
| Outcome: | The proposed methods achieve state-of-the-art accuracy for quantized large Transformer encoder models with sub-2-bit weight quantization. |