LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation

LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation (2025.findings-emnlp)

Copied to clipboard

Yiqun Shen, Song Yuan, Zhengze Zhang, Xiaoliang Wang, Daxin Jiang, Nguyen Cam-Tu

Challenge:	Existing methods for cache compression are heuristic and lack dynamic budget allocation . cnn's john mccartney and johnny mccain present a new approach for cache eviction and dynamic budgets .
Approach:	They propose a unified framework for cache compression that minimizes information loss in transformer residual streams.
Outcome:	The proposed method consistently maintains top performance across task types.

Similar Papers

Overview

Challenge

Approach

Outcome

EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025.findings-emnlp)

Copied to clipboard

Bohan Yu, Yekun Chai

Challenge:	Existing key-value (KV) cache compression methods ignore interplays between layer-specific feature patterns and task performance.
Approach:	They propose an adaptive framework for layer-wise, task-driven KV cache compression that optimizes memory efficiency and task performance.
Outcome:	EvolKV outperforms baseline methods on long-context tasks and surpasses heuristics by 7 percentage points on GSM8K.

ZigZagKV: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty (2025.coling-main)

Copied to clipboard

Meizhi Zhong, Xikai Liu, Chen Zhang, Yikun Lei, Yan Gao, Yao Hu, Kehai Chen, Min Zhang

Semantic Text Processing

Challenge:	Existing methods to accelerate inference of Large Language models (LLMs) are limited in their ability to retain key tokens as input length increases.
Approach:	They propose a method that leverages layer uncertainty to allocate budget size for each layer to reduce memory usage.
Outcome:	The proposed method reduces memory usage of the KV caches to only 20% when compared to full KV inference while achieving nearly lossless performance.

DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs (2025.findings-emnlp)

Copied to clipboard

Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, Liang Ding

Challenge:	Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics, which hampers the effective retention of essential information while discarding less important tokens.
Approach:	They propose a Task-Aware KV cache mechanism that dynamically adjusts the KV caching size across different layers based on the characteristics of the tasks.
Outcome:	The proposed method surpasses state-of-the-art methods by 11% on the LongBench dataset even under extreme compression (0.9%)

HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference (2026.acl-long)

Copied to clipboard

Zhiyuan Shi, Qibo Qiu, null Xuefeng, Zhonglin Jiang, Li Yu, Jian Jiang, Xiaofei He, Wenxiao Wang

Information Retrieval

Challenge:	Existing static compression methods suffer from coarse-grained caching and high I/O overhead.
Approach:	They propose a training-free dynamic compression framework that uses a sparse attention mechanism to categorize attention heads based on stability and similarity.
Outcome:	The proposed framework achieves state-of-the-art performance on long-context benchmarks and accelerates decoding by up to 3 compared to the original model with a 224K context.

HqeKV: Towards Hybrid Quantization and Eviction for KV Cache in Long-Context LLM Inference (2026.findings-acl)

Copied to clipboard

He Wang, Yu Gu, Fangfang Li, Zhigang Wang, Zhenghao Liu, Ning Wang, Xiaohua Li, Ge Yu

Challenge:	autoregressive inference requires repeated computation across transformer layers.
Approach:	They propose a hybrid compression framework built on both quantization and eviction . they propose varying importance metric and flexible conversion policies to reduce memory overhead .
Outcome:	The proposed framework outperforms state-of-the-art methods under memory constraints.

MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference (2025.naacl-long)

Copied to clipboard

Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, Mi Zhang

Multimodality

Challenge:	Long-context Multimodal Large Language Models (MLLMs) require substantial computational resources as their multimodal Key-Value (KV) cache grows with increasing input lengths, challenging memory and time efficiency.
Approach:	They propose a dynamic multimodal KV cache allocation strategy that dynamically allocating KV size based on attention entropy to better adapt to multimodal interactions.
Outcome:	The proposed model achieves up to 72% KV cache memory reduction and 2.82 faster decoding speeds while maintaining or enhancing performance on various multimodal tasks in a long context.

Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)

Copied to clipboard

Haoyi Wu, Kewei Tu

Language Models

Challenge:	Using a key-value cache, memory consumption is a bottleneck for high-throughput language models.
Approach:	They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput.
Outcome:	The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques.

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query (2025.emnlp-main)

Copied to clipboard

Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che

Challenge:	Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries.
Approach:	They propose a lookahead q-cache framework that generates low-cost pseudo lookaheaded queries to better approximate the true decoding-stage queries.
Outcome:	The proposed framework outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks and can be flexibly combined to yield further improvements.

GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction (2025.emnlp-main)

Copied to clipboard

Xuelin Li, Xiangqi Jin, Linfeng Zhang

Structured Data in NLP

Challenge:	Efficient Key-Value (KV) cache management is essential for processing long text sequences in large language models (LLMs).
Approach:	They propose a graph-based framework that redefines token selection for KV cache compression.
Outcome:	The proposed framework can be used in existing KV cache eviction methods such as SnapKV and PyramidKV in a plug-and-play manner.

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding (2025.findings-naacl)

Copied to clipboard

Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, Alham Fikri Aji

Language Models

Challenge:	Multi-Layer Key-Value (MLKV) sharing reduces memory usage by 6x compared to Multi-Query Attention and Grouped-Query Attributes.
Approach:	They propose a novel approach that extends KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention and Grouped-Query Attributes.
Outcome:	The proposed approach reduces KV cache size by 6x with minimal performance loss and scales linearly with model size, batch size, and sequence length.