Papers by Kyelim Lee
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing joint optimization methods prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component’s unique strengths. |
| Approach: | They introduce Outlier-Driven Low-Rank Initialization (ODLRI) which assigns low-rank components the specific role of capturing activation-sensitive weights. |
| Outcome: | Experiments on Llama2 (7B, 13B, 70B, and Mistral-7B) and Llma3-8B show that incorporating ODLRI into the joint optimization framework reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings. |