Papers by Jianing Deng
FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference (2025.findings-emnlp)
Copied to clipboard
Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang
| Challenge: | Key-Value (KV) cache reading latency increases with context lengths hindering LLM inference . important tokens are sparsely distributed across the long context, making existing retrieval inaccurate . |
| Approach: | They propose a method to retain a small fraction of KV cache based on token importance . important tokens are often sparsely distributed across the long context . |
| Outcome: | The proposed method reduces decoding latency by 1.2 to 1.5. |