Papers by Neel Dani
MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art 2-bit KV cache quantization methods achieve excellent results in accelerating LLM inference while retaining accuracy on long context tasks. |
| Approach: | They propose a method based on 2-bit KV cache quantization with adaptive KV policies that retain LLM accuracy with only a subset of KV states. |
| Outcome: | The proposed method outperforms state-of-the-art methods on a wide range of long context tasks while retaining accuracy. |