Papers by Utkarsh Saxena
Eigen Attention: Attention in Low-Rank Space for KV Cache Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been increasing context lengths to enhance their performance, but at long context length, the KV cache becomes the new bottleneck in memory usage during inference. |
| Approach: | They propose an approach which performs the attention operation in a low-rank space and reduces the KV cache memory overhead. |
| Outcome: | The proposed approach reduces the KV cache memory overhead and reduces memory usage with minimal drop in performance over OPT, MPT, and Llama model families. |