LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing KV cache compression methods mitigate memory bottlenecks but struggle in long reasoning tasks. |
| Approach: | They propose a lagged eviction framework that prioritizes evicts based on tokens’ recurrence patterns to reduce KV cache by 50% and maintain comparable accuracy. |
| Outcome: | The proposed framework reduces KV cache by 50% 70% while maintaining comparable accuracy, outperforming existing KV baselines. |
Similar Papers
NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time (2024.acl-long)
Copied to clipboard
Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, Hua Wu
| Challenge: | Large Language Models (LLMs) with extended context windows are expensive and infeasible on fixed memory hardware due to the surprisingly large memory consumption of KV Cache. |
| Approach: | They propose a general framework for long-context KV cache eviction that achieves more optimal and efficient evict in a single operation during the encoding phase. |
| Outcome: | The proposed framework improves performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 5 with over 95% performance maintenance. |
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference (2026.findings-eacl)
Copied to clipboard
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Anthony Kougkas, Xian Sun, Murali Emani, Venkatram Vishwanath, Bogdan Nicolae
| Challenge: | Large Language Models (LLMs) are exploding to large sizes, including GPT, LLaMA, and DeepSeek. |
| Approach: | They propose a fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention. |
| Outcome: | The proposed method integrates seamlessly with PagedAttention without any modifications to its CUDA attention kernels. |
Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression (2026.acl-long)
Copied to clipboard
Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu, Bing Qin
| Challenge: | Recent methods to reduce the KV cache size fail to identify crucial KVs for generation while excluding others accurately, resulting in severe information loss. |
| Approach: | They propose an intention-aware KV cache eviction method that identifies and retains crucial KVs according to the attention distribution of intention, which semantically reflects the user’s goal and determines which part of the context is relevant. |
| Outcome: | The proposed method can maintain the model performance while reducing the KV cache size from 128K to 2K, leading to a 6.3x increase in decoding speed and 7.8x enhancement in memory efficiency compared to the default setting. |
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries. |
| Approach: | They propose a lookahead q-cache framework that generates low-cost pseudo lookaheaded queries to better approximate the true decoding-stage queries. |
| Outcome: | The proposed framework outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks and can be flexibly combined to yield further improvements. |
MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference (2025.acl-long)
Copied to clipboard
Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, ZhaodeWang ZhaodeWang, Chengfei Lv, Shengyu Zhang, Fan Wu, Fei Wu
| Challenge: | Existing KV cache eviction methods fail to capture modality-specific information, resulting in suboptimal performance. |
| Approach: | They propose a modality-adaptive key-value (KV) cache eviction strategy to enhance the efficiency of multimodal large language models in long-context inference. |
| Outcome: | The proposed method reduces the KV cache memory footprint and model inference latency while maintaining high accuracy across multimodal long-context tasks. |
RefreshKV: Updating Small KV Cache During Long-form Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating long sequences of tokens are expensive and require memory and computation resources. |
| Approach: | They propose a method that alternates between full context attention and attention over a subset of input tokens during generation. |
| Outcome: | The proposed method achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks. |
REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evict keyvalue caches ignore diverse behavior in failure cases, such as bias and distraction. |
| Approach: | They propose a method to analyze attention head behaviors in success and failure scenarios by maximizing signal-to-noise ratio and minimizing noise from bias and distraction. |
| Outcome: | The proposed method achieves comparable accuracy to the strongest baseline, HeadKV-R2 on LongBench v2 while requiring 32x less space. |
Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention (2026.findings-acl)
Copied to clipboard
Mengqi Liao, Lu Wang, Chaoyun Zhang, Bo Qiao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Huaiyu Wan
| Challenge: | Existing methods to evict KV cache during inference phase are impractical for industrial-grade applications. |
| Approach: | They propose a method that combines token-wise KV cache eviction with PagedAttention and propose 'zipage' it achieves 95% of the performance of Full KV inference engines while delivering over 2.1 speedup . |
| Outcome: | The proposed method achieves 95% of the performance of Full KV inference engines while delivering over 2.1 speedup on large-scale mathematical reasoning tasks. |
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Rapid advances in Large Language Models have spurred demand for processing extended context sequences . however, performance degradation due to sequence lengths out-of-distribution and excessively long inference times are limiting LLMs in long-context scenarios. |
| Approach: | They propose a training-free method for efficient and accurate long-context inference . they selectively involves a few critical KV cache tokens in attention calculation . |
| Outcome: | The proposed method speeds up attention computation and accelerates inference time while reducing selection overhead. |
Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing cache eviction strategies for autoregressive language models fail to account for the role of mask tokens and specific characteristics in dLLMs. |
| Approach: | They propose a training-free cache eviction framework tailored to dLLMs that denies a fully masked sequence and allows parallel decoding at the expense of memory and computation. |
| Outcome: | The proposed framework reduces the cost of memory and cache eviction and improves efficiency by reducing allocation in intermediate layers and concentrating resources on prompt-preferring heads. |