NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time

Challenge:	Large Language Models (LLMs) with extended context windows are expensive and infeasible on fixed memory hardware due to the surprisingly large memory consumption of KV Cache.
Approach:	They propose a general framework for long-context KV cache eviction that achieves more optimal and efficient evict in a single operation during the encoding phase.
Outcome:	The proposed framework improves performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 5 with over 95% performance maintenance.

Similar Papers

Overview

Challenge

Approach

Outcome

PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference (2026.findings-eacl)

Copied to clipboard

Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Anthony Kougkas, Xian Sun, Murali Emani, Venkatram Vishwanath, Bogdan Nicolae

Semantic Text Processing

Challenge:	Large Language Models (LLMs) are exploding to large sizes, including GPT, LLaMA, and DeepSeek.
Approach:	They propose a fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention.
Outcome:	The proposed method integrates seamlessly with PagedAttention without any modifications to its CUDA attention kernels.

FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference (2025.findings-emnlp)

Copied to clipboard

Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang

Green & Sustainable NLP

Challenge:	Key-Value (KV) cache reading latency increases with context lengths hindering LLM inference . important tokens are sparsely distributed across the long context, making existing retrieval inaccurate .
Approach:	They propose a method to retain a small fraction of KV cache based on token importance . important tokens are often sparsely distributed across the long context .
Outcome:	The proposed method reduces decoding latency by 1.2 to 1.5.

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection (2025.emnlp-main)

Copied to clipboard

Wei Wu, Zhuoshi Pan, Kun Fu, Chao Wang, Liyi Chen, Yunchu Bai, Tianfu Wang, Zheng Wang, Hui Xiong

Responsible & Trustworthy NLP

Challenge:	Rapid advances in Large Language Models have spurred demand for processing extended context sequences . however, performance degradation due to sequence lengths out-of-distribution and excessively long inference times are limiting LLMs in long-context scenarios.
Approach:	They propose a training-free method for efficient and accurate long-context inference . they selectively involves a few critical KV cache tokens in attention calculation .
Outcome:	The proposed method speeds up attention computation and accelerates inference time while reducing selection overhead.

LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning (2026.acl-long)

Copied to clipboard

Haoyue Zhang, Hualei Zhang, Xiaosong Ma, Jie Zhang, Song Guo

Responsible & Trustworthy NLP

Challenge:	Existing KV cache compression methods mitigate memory bottlenecks but struggle in long reasoning tasks.
Approach:	They propose a lagged eviction framework that prioritizes evicts based on tokens’ recurrence patterns to reduce KV cache by 50% and maintain comparable accuracy.
Outcome:	The proposed framework reduces KV cache by 50% 70% while maintaining comparable accuracy, outperforming existing KV baselines.

MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference (2025.acl-long)

Copied to clipboard

Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, ZhaodeWang ZhaodeWang, Chengfei Lv, Shengyu Zhang, Fan Wu, Fei Wu

Responsible & Trustworthy NLP

Challenge:	Existing KV cache eviction methods fail to capture modality-specific information, resulting in suboptimal performance.
Approach:	They propose a modality-adaptive key-value (KV) cache eviction strategy to enhance the efficiency of multimodal large language models in long-context inference.
Outcome:	The proposed method reduces the KV cache memory footprint and model inference latency while maintaining high accuracy across multimodal long-context tasks.

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query (2025.emnlp-main)

Copied to clipboard

Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che

Challenge:	Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries.
Approach:	They propose a lookahead q-cache framework that generates low-cost pseudo lookaheaded queries to better approximate the true decoding-stage queries.
Outcome:	The proposed framework outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks and can be flexibly combined to yield further improvements.

Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression (2026.acl-long)

Copied to clipboard

Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu, Bing Qin

Challenge:	Recent methods to reduce the KV cache size fail to identify crucial KVs for generation while excluding others accurately, resulting in severe information loss.
Approach:	They propose an intention-aware KV cache eviction method that identifies and retains crucial KVs according to the attention distribution of intention, which semantically reflects the user’s goal and determines which part of the context is relevant.
Outcome:	The proposed method can maintain the model performance while reducing the KV cache size from 128K to 2K, leading to a 6.3x increase in decoding speed and 7.8x enhancement in memory efficiency compared to the default setting.

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression (2024.findings-emnlp)

Copied to clipboard

Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, Kaushik Roy

Challenge:	Large language models (LLMs) have been increasing context lengths to enhance their performance, but at long context length, the KV cache becomes the new bottleneck in memory usage during inference.
Approach:	They propose an approach which performs the attention operation in a low-rank space and reduces the KV cache memory overhead.
Outcome:	The proposed approach reduces the KV cache memory overhead and reduces memory usage with minimal drop in performance over OPT, MPT, and Llama model families.

GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction (2025.emnlp-main)

Copied to clipboard

Xuelin Li, Xiangqi Jin, Linfeng Zhang

Structured Data in NLP

Challenge:	Efficient Key-Value (KV) cache management is essential for processing long text sequences in large language models (LLMs).
Approach:	They propose a graph-based framework that redefines token selection for KV cache compression.
Outcome:	The proposed framework can be used in existing KV cache eviction methods such as SnapKV and PyramidKV in a plug-and-play manner.

Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference (2026.findings-acl)

Copied to clipboard

Jianuo Huang, Yaojie Zhang, Yicun Yang, Benhao Huang, Linfeng Zhang

Responsible & Trustworthy NLP

Challenge:	Existing cache eviction strategies for autoregressive language models fail to account for the role of mask tokens and specific characteristics in dLLMs.
Approach:	They propose a training-free cache eviction framework tailored to dLLMs that denies a fully masked sequence and allows parallel decoding at the expense of memory and computation.
Outcome:	The proposed framework reduces the cost of memory and cache eviction and improves efficiency by reducing allocation in intermediate layers and concentrating resources on prompt-preferring heads.