KV

EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices (2025.acl-industry)

Challenge:	Retrieval-Augmented Generation (RAG) systems face efficiency bottlenecks in prefill due to attention mechanism, and traditional KV cache only accelerates decoding.
Approach:	They propose a multi-document KV cache reuse framework for multi-doc RAG workloads . they propose to resolve position and context misalignment while eliminating document-specific quadratic complexity in prefill.
Outcome:	The proposed framework solves position and context misalignment issues while eliminating document-specific quadratic complexity in prefill.

Copied to clipboard

Jiyu Chen, Shuang Peng, Daxiong Luo, Fan Yang, Renshou Wu, Fangyuan Li, Xiaoxin Chen

Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference (2026.findings-acl)

Challenge:	Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks . alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructures.
Approach:	They propose a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module.
Outcome:	The proposed solution achieves comparable performance to baseline Transformer-based LLMs while optimizing memory consumption and time to first token.

Copied to clipboard

Rei Taniguchi, Yuyang Dong, Makoto Onizuka, Chuan Xiao

Challenge:	Large language models (LLMs) have demonstrated remarkable capabilities in processing long contexts.
Approach:	They propose a training-free method that adaptively chooses the selection layer for KV cache reduction . they exploit the variance of token ranks ordered by attention score to optimize decoding .
Outcome:	The proposed method outperforms state-of-the-art token pruning methods on InfiniteBench, RULER, and NIAH benchmarks.

CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing (2026.acl-long)

Copied to clipboard

Rui Wang, Junda Wu, Yu Xia, Tong Yu, Ruiyi Zhang, Ryan A. Rossi, Subrata Mitra, Lina Yao, Julian McAuley

ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval (2026.acl-long)

Challenge:	Existing Large Language Models exhibit critical vulnerability to indirect prompt injection attacks, where instructions injected within in the prompt context can override the user's intent.
Approach:	They propose a neural pruning algorithm that prunes neurons associated with instruction-following during KV cache encoding of the prompt context.
Outcome:	The proposed approach significantly reduces the attack success rate while preserving the model's ability to follow user instructions.

Copied to clipboard

David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, Subhajit Chaudhury, Pin-Yu Chen

LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression (2025.findings-naacl)

Challenge:	Large language models generate long chain of thoughts but memory footprint grows with output length . prior work on KV cache optimization focused on compressing long input context .
Approach:	They propose a new approach that compresses verbose reasoning thoughts into summaries . they use a dynamic KV cache selection policy that leverages these summary keys .
Outcome:	The proposed approach reduces memory usage while avoiding full-cache attention at each step.

Copied to clipboard

Souvik Kundu, Anahita Bhiwandiwalla, Sungduk Yu, Phillip Howard, Tiep Le, Sharath Nittur Sridhar, David Cobbley, Hao Kang, Vasudev Lal

Multimodality

Challenge:	LVLMs have been shown to perform well on simple uni-modal benchmarks, but their detailed study on multi-modal models is still lacking.
Approach:	They propose a framework to analyze the impact of compression on LVLMs on multi-modal input driven tasks.
Outcome:	The proposed framework analyzes the impact of compression on generative performance of large vision language models on multi-modal input driven tasks.

EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025.findings-emnlp)

Copied to clipboard

Bohan Yu, Yekun Chai

Challenge:	Existing key-value (KV) cache compression methods ignore interplays between layer-specific feature patterns and task performance.
Approach:	They propose an adaptive framework for layer-wise, task-driven KV cache compression that optimizes memory efficiency and task performance.
Outcome:	EvolKV outperforms baseline methods on long-context tasks and surpasses heuristics by 7 percentage points on GSM8K.

TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering (2025.acl-industry)

Copied to clipboard

Vinay Joshi, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum

MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference (2025.naacl-long)

Challenge:	key-value caches in large language models consume memory, posing a major challenge for scalable deployment.
Approach:	They propose a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling.
Outcome:	The proposed technique reduces the KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy.

Copied to clipboard

Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, Mi Zhang

Multimodality

Challenge:	Long-context Multimodal Large Language Models (MLLMs) require substantial computational resources as their multimodal Key-Value (KV) cache grows with increasing input lengths, challenging memory and time efficiency.
Approach:	They propose a dynamic multimodal KV cache allocation strategy that dynamically allocating KV size based on attention entropy to better adapt to multimodal interactions.
Outcome:	The proposed model achieves up to 72% KV cache memory reduction and 2.82 faster decoding speeds while maintaining or enhancing performance on various multimodal tasks in a long context.

PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference (2026.findings-eacl)

Copied to clipboard

Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Anthony Kougkas, Xian Sun, Murali Emani, Venkatram Vishwanath, Bogdan Nicolae

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference (2024.findings-acl)

Challenge:	Large Language Models (LLMs) are exploding to large sizes, including GPT, LLaMA, and DeepSeek.
Approach:	They propose a fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention.
Outcome:	The proposed method integrates seamlessly with PagedAttention without any modifications to its CUDA attention kernels.

Copied to clipboard

Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, Hai Zhao

Challenge:	Existing methods to reduce memory usage for large language models neglect inter-layer dependency between layers and huge memory consumption in pre-computation.
Approach:	They propose a method that compresses the KV cache by layer-wise retaining crucial context.
Outcome:	The proposed method reduces memory usage by layer-wise retaining crucial context . it can improve 2.2x throughput compared to Accelerate with over 54% memory reduction .

KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches (2024.findings-emnlp)

Copied to clipboard

Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, Xia Hu

MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning (2026.acl-long)

Challenge:	Long context capability is a crucial competency for large language models as it mitigates the human struggle to digest long-form texts.
Approach:	They propose to evaluate 10+ state-of-the-art approaches for long context-capable LLMs.
Outcome:	The proposed methods are compared against 10+ state-of-the-art approaches across seven categories of long context tasks.

Copied to clipboard

Tao Zhang, Ziqian Zeng, Hao Peng, Huiping Zhuang, Cen Chen

Reasoning

Challenge:	Existing low-bit quantization methods often exhibit severe performance degradation on complex reasoning tasks.
Approach:	They propose a plug-and-play method that uses a key channel's intrinsic quantization difficulty and relevance to the query to identify and preserve critical key channels that need higher precision.
Outcome:	Experiments on complex reasoning datasets show that the proposed method outperforms low-bit methods at a substantially reduced memory footprint.

When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judges (2026.acl-long)

Copied to clipboard

Sichu Liang, Zhenglin Wang, null Chujiajia, Pengfei Xia, Hui Zang, Deyu Zhou

Challenge:	Multi-agent LLMs generate multiple candidate responses that are aggregated by an LLM judge.
Approach:	They propose to advocate KV cache reuse across partially shared contexts and report substantial speedups for generation agents.
Outcome:	The proposed reuse strategies weaken cross-candidate attention, especially for later candidate blocks, and highlight judge-centric inference as a distinct regime that requires dedicated, risk-aware system design.

Streamlining the Collaborative Chain of Models into A Single Forward Pass in Generation-Based Tasks (2025.findings-acl)

Copied to clipboard

Yuanjie Lyu, Chao Zhang, Yuhao Chen, Yong Chen, Tong Xu

Rectified Sparse Attention for Efficient Long-Sequence Generation (2026.findings-acl)

Challenge:	Recent advances in "Chain of Models" approach increase resource demands as each model must be deployed separately.
Approach:	They propose a prompt-tuning method that enables models to share hidden states . they modify input and attention masks during training to eliminate redundant forward passes .
Outcome:	Empirical results show that FTHSS matches the performance of traditional model chains while improving inference efficiency.

Copied to clipboard

Yutao Sun, Tianzhu Ye, Li Dong, Yuqing Xia, Jian Chen, Yizhao Gao, Shijie Cao, Jianyong Wang, Furu Wei

LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing (2026.findings-acl)

Challenge:	Recent sparse decoding methods improve efficiency but suffer from KV cache misalignment, resulting in performance degradation.
Approach:	They propose a method that combines block-sparse attention with periodic dense rectification to bound error accumulation and preserve alignment with the pretraining distribution.
Outcome:	Experiments on math reasoning, language modeling, and retrieval tasks show that ReSA achieves near-lossless generation quality with significantly improved efficiency.

Copied to clipboard

Dongfang Li, Zixuan Liu, Gang Lin, Baotian Hu, Min Zhang

Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention (2026.findings-acl)

Challenge:	Existing retrieval-based methods compromise semantic integrity through fixed-size chunking and suffer from inefficient linear scanning.
Approach:	They propose a method that preserves local semantic coherence through boundary-aware chunking and constructs a recursive hierarchical index rooted in the triangle inequality.
Outcome:	The proposed method achieves 3.6 end-to-end inference speedup with negligible degradation in model performance.

Copied to clipboard

Mengqi Liao, Lu Wang, Chaoyun Zhang, Bo Qiao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Huaiyu Wan

Reasoning

Challenge:	Existing methods to evict KV cache during inference phase are impractical for industrial-grade applications.
Approach:	They propose a method that combines token-wise KV cache eviction with PagedAttention and propose 'zipage' it achieves 95% of the performance of Full KV inference engines while delivering over 2.1 speedup .
Outcome:	The proposed method achieves 95% of the performance of Full KV inference engines while delivering over 2.1 speedup on large-scale mathematical reasoning tasks.

DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs (2025.findings-emnlp)

Copied to clipboard

Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, Liang Ding

Challenge:	Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics, which hampers the effective retention of essential information while discarding less important tokens.
Approach:	They propose a Task-Aware KV cache mechanism that dynamically adjusts the KV caching size across different layers based on the characteristics of the tasks.
Outcome:	The proposed method surpasses state-of-the-art methods by 11% on the LongBench dataset even under extreme compression (0.9%)

More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression (2025.findings-emnlp)

Copied to clipboard

Jiebin Zhang, Dawei Zhu, Yifan Song, Wenhao Wu, Chuqiao Kuang, Xiaoguang Li, Lifeng Shang, Qun Liu, Sujian Li

Challenge:	storing more tokens in the KV cache at lower precision can enhance the long-context performance of large language models.
Approach:	They propose a token-precision trade-off strategy to optimize KV cache compression . they also propose storing more tokens in the KV at lower precision .
Outcome:	The proposed method achieves an optimal point within the Information Bottleneck compared to standalone KV pruning or KV quantization.

SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation (2025.acl-long)

Copied to clipboard

Jialong Wu, Zhenglin Wang, Linhai Zhang, Yilong Lai, Yulan He, Deyu Zhou

Reasoning

Challenge:	Excessive compression during the prefill phase impairs comprehension of reasoning tasks . SCOPE is a framework that performs KV cache optimization during the decoding and prefill phases .
Approach:	They propose a framework that performs optimization during the prefill and decoding phases . they propose enabling a sliding strategy to select essential heavy hitters for the decoding phase .
Outcome:	Experiments show that SCOPE can optimize key-value cache for long-context generation tasks . the framework can preserve essential information while minimizing memory usage and transfer .

MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts (2025.acl-long)

Copied to clipboard

Wei Tao, Haocheng Lu, Xiaoyang Qu, Bin Zhang, Kai Lu, Jiguang Wan, Jianzong Wang

ZigZagKV: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty (2025.coling-main)

Challenge:	Existing approaches to optimize large language models for long-context inference are inefficient and consume memory.
Approach:	They propose a mixed-precision quantization method via mixture of experts that inputs tokens into router chunk by chunk to reduce inference overhead.
Outcome:	The proposed method outperforms state-of-the-art KV cache quantization methods on multiple benchmark datasets.

Copied to clipboard

Meizhi Zhong, Xikai Liu, Chen Zhang, Yikun Lei, Yan Gao, Yao Hu, Kehai Chen, Min Zhang

A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization (2025.findings-acl)

Challenge:	Existing methods to accelerate inference of Large Language models (LLMs) are limited in their ability to retain key tokens as input length increases.
Approach:	They propose a method that leverages layer uncertainty to allocate budget size for each layer to reduce memory usage.
Outcome:	The proposed method reduces memory usage of the KV caches to only 20% when compared to full KV inference while achieving nearly lossless performance.

Copied to clipboard

Junhui He, Junna Xing, Nan Wang, Rui Xu, Shangyu Wu, Peng Zhou, Qiang Liu, Chun Jason Xue, Qingan Li

MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference (2025.acl-long)

Challenge:	Long context large language models (LLMs) pose significant challenges for efficient serving due to the large memory footprint and high access overhead of KV cache.
Approach:	They propose a retrieval-based method to reduce the memory footprint of LLMs . they propose Windowed Rotary Position Embedding and query-aware vector quantization .
Outcome:	The proposed method can achieve lower performance degradation with lower overhead compared to existing methods . it can reduce the memory footprint and access overhead of long context large language models .

Copied to clipboard

Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, ZhaodeWang ZhaodeWang, Chengfei Lv, Shengyu Zhang, Fan Wu, Fei Wu

SpecCache: Speculative KV Cache Reuse for Efficient RAG Serving (2026.acl-long)

Challenge:	Existing KV cache eviction methods fail to capture modality-specific information, resulting in suboptimal performance.
Approach:	They propose a modality-adaptive key-value (KV) cache eviction strategy to enhance the efficiency of multimodal large language models in long-context inference.
Outcome:	The proposed method reduces the KV cache memory footprint and model inference latency while maintaining high accuracy across multimodal long-context tasks.

Copied to clipboard

Zijian Wen, Tao Zhang, Shuangwu Chen, Shenghao Ye, Yu Guo, Qirui Chen, Jingxian Shuai, Yunpeng Hou, Huasen He, null Jianyang

InfiniPot: Infinite Context Processing on Memory-Constrained LLMs (2024.emnlp-main)

Challenge:	Retrieval-Augmented Generation (RAG) improves LLMs but faces high prefill latency during long contexts.
Approach:	They propose a method that uses deep-layer hidden-state norms to guide token selection . they propose to use deep-layered hidden-status norms as a proxy to guide the token selection.
Outcome:	The proposed SpecCache outperforms state-of-the-art (SOTA) benchmarks.

Copied to clipboard

Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang

Challenge:	InfiniPot is a KV cache control framework that can handle long input contexts without additional training.
Approach:	They propose a KV cache control framework that can handle long input contexts efficiently without additional training.
Outcome:	The proposed framework outperforms models trained for long contexts in various NLP tasks and is highly efficient and versatile.

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression (2024.findings-emnlp)

Copied to clipboard

Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, Kaushik Roy

Challenge:	Large language models (LLMs) have been increasing context lengths to enhance their performance, but at long context length, the KV cache becomes the new bottleneck in memory usage during inference.
Approach:	They propose an approach which performs the attention operation in a low-rank space and reduces the KV cache memory overhead.
Outcome:	The proposed approach reduces the KV cache memory overhead and reduces memory usage with minimal drop in performance over OPT, MPT, and Llama model families.

RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction (2026.findings-acl)

Copied to clipboard

Sihao Liu, YuFan Xiong, Zhonghua Jiang, Zhaode Wang, Chengfei lv, Shengyu Zhang

Multimodality

Challenge:	Existing methods for evicting KV pairs rely on the "persistence of importance" hypothesis . visual tokens display "deferred importance" but become pivotal during later decoding, authors say .
Approach:	They propose an entropy-driven method that reformulates KV eviction from "discrete context truncation" to "continuous memory evolution" they propose to prune visual tokens with "deferred importance" visual token exhibiting low salience but becoming pivotal during later decoding .
Outcome:	The proposed method achieves 5.0 KV cache compression and 1.5 decoding acceleration.

FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping (2024.emnlp-main)

Copied to clipboard

Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Tianlong Chen, Shiwei Liu, Aditya Akella

Challenge:	Autoregressive Large Language Models (LLMs) are omnipresent but typically come with a substantial model size.
Approach:	They propose a novel fine-grained skip strategy for autoregressive large language models . they observe the saturation of computationally expensive feed-forward blocks of LLMs .
Outcome:	The proposed method can skip 25-30% of FFN blocks with marginal change in performance on knowledge-intensive generation tasks.

MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference (2025.findings-acl)

Copied to clipboard

Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, Minjia Zhang

IAM: Efficient Inference through Attention Mapping between Different-scale LLMs (2025.acl-long)

Challenge:	State-of-the-art 2-bit KV cache quantization methods achieve excellent results in accelerating LLM inference while retaining accuracy on long context tasks.
Approach:	They propose a method based on 2-bit KV cache quantization with adaptive KV policies that retain LLM accuracy with only a subset of KV states.
Outcome:	The proposed method outperforms state-of-the-art methods on a wide range of long context tasks while retaining accuracy.

Copied to clipboard

Yi Zhao, Zuchao Li, Hai Zhao

Deputy: Accelerating Large Language Model Inference with Dynamic Low-Rank Substitution (2026.findings-acl)

Challenge:	Large language models (LLMs) are a challenge due to their internal reasoning processes.
Approach:	They propose an algorithm that can optimize attention matrices by performing attention mapping between small and large LLMs.
Outcome:	The proposed framework can reduce KV cache usage by 22.1% and accelerate prefill by 15% without sacrificing performance.

Copied to clipboard

Yuhua Zhou, Shichao Weng, Changhai Zhou, Yuhan Wu, Qian Qiao, Jun Gao, Fei Yang, Aimin Pan

KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation (2025.findings-acl)

Challenge:	Existing dynamic schemes such as early-exit and layer-drop reduce FLOPs but break batch processing or introduce KV-cache inconsistency.
Approach:	They propose a dynamic low-rank substitution framework that employs a lightweight decision module at each layer to dynamically determine the execution branch for different tokens.
Outcome:	The proposed model reduces computation by approximately 40% compared to the original dense model while outperforming existing baseline methods.

Copied to clipboard

Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, Murali Annavaram

GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction (2025.emnlp-main)

Challenge:	Existing methods to inference large language models are limited by CPU capabilities and memory constraints.
Approach:	They propose an efficient I/O-aware LLM inference method that overlaps GPU computation with KV cache transfer to minimize idle GPU time.
Outcome:	The proposed method reduces the cost of auto-regressive decoding by 35.8% . it also achieves 46.2% higher throughput during decoding compared to state-of-the-art methods.

Copied to clipboard

Xuelin Li, Xiangqi Jin, Linfeng Zhang

Structured Data in NLP

Challenge:	Efficient Key-Value (KV) cache management is essential for processing long text sequences in large language models (LLMs).
Approach:	They propose a graph-based framework that redefines token selection for KV cache compression.
Outcome:	The proposed framework can be used in existing KV cache eviction methods such as SnapKV and PyramidKV in a plug-and-play manner.

When Efficiency Meets Safety: A Benchmark Security Analysis of KV Cache Compression in Large Language Models (2026.acl-long)

Copied to clipboard

Xiaoxiao Ma, Kuofeng Gao, Zeyi Lu, Wenxi Jiang, Hao Fang, Hao Wu, Bin Chen, Shu-Tao Xia

Latent-Condensed Transformer for Efficient Long Context Modeling (2026.acl-long)

Challenge:	Key-Value (KV) caching is widely used in large language models to enable long-context inference efficiently, yet its security implications remain underexplored.
Approach:	They propose a history-aware, per-head feedback merging strategy that prevents safety degradation while maintaining efficiency.
Outcome:	The proposed strategy prevents safety degradation while maintaining efficiency.

Copied to clipboard

Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan

Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters (2024.emnlp-main)

Challenge:	Existing approaches address these bottlenecks separately: Multi-head Latent Attention (MLA) reduces the KV cache by projecting tokens into a low-dimensional latent space, while sparse attention reduces computation.
Approach:	They propose a Latent-Condensed Attention mechanism that performs structured context condensation directly within MLA's latent space.
Outcome:	The proposed approach reduces KV cache size and attention cost without adding parameters.

Copied to clipboard

Zhiyu Guo, Hidetaka Kamigaito, Taro Watanabe

Challenge:	Recent studies have focused on scaling the context size of large language models (LLMs) however, the enormous inference costs of LLMs limit their applications.
Approach:	They propose a method which uses attention scores and the l 1 norm to evaluate token importance.
Outcome:	Extensive experiments on LLaMA2-7B-chat and Vicuna-v1.5-7B show that the proposed method outperforms attention-score-only baselines in over 12 tasks.

EQUIP: EQUivariant preserving In-Place updates for Efficient Token Pruning (2026.acl-long)

Copied to clipboard

Arun Ramachandran, R Govindarajan, Murali Annavaram, Prakash Raghavendra

Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression (2026.acl-long)

Challenge:	Token-pruning methods cause "holes" in KV tensors, posing major challenges . equip reduces recomputation of rotation operations through in-place update, caching and re-indexing .
Approach:	They propose an EQUIP-based in-place token update mechanism that preserves the equivariance property of the operations performed in the attention computation.
Outcome:	EQUIP reduces recomputation of rotation operations and reduces eviction overheads . it achieves geomean speedups of 1.62 (or 1.47) over StreamingLLM and 3.45 ( or 1.86)

Copied to clipboard

Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu, Bing Qin

Challenge:	Recent methods to reduce the KV cache size fail to identify crucial KVs for generation while excluding others accurately, resulting in severe information loss.
Approach:	They propose an intention-aware KV cache eviction method that identifies and retains crucial KVs according to the attention distribution of intention, which semantically reflects the user’s goal and determines which part of the context is relevant.
Outcome:	The proposed method can maintain the model performance while reducing the KV cache size from 128K to 2K, leading to a 6.3x increase in decoding speed and 7.8x enhancement in memory efficiency compared to the default setting.

Multi-matrix Factorization Attention (2025.findings-acl)

Copied to clipboard

Jingcheng Hu, Houyi Li, Yinmin Zhang, Zili Wang, Shuigeng Zhou, Xiangyu Zhang, Heung-Yeung Shum

Challenge:	Existing variants for Multi-Head Attention (MHA) fail to maintain strong performance under stringent Key-Value cache (KV cache) constraints.
Approach:	They propose to use multi-matrix factorization attention and MFA-Key-reuse attention architectures to increase model capacity under tight KV cache constraints.
Outcome:	The proposed architecture outperforms existing methods while reducing KV cache usage by 56% and 93.7% in large-scale experiments.

SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation (2025.emnlp-main)

Copied to clipboard

Aurick Qiao, Zhewei Yao, Samyam Rajbhandari, Yuxiong He

Hidden States as Early Signals: Step-level Trace Evaluation and Pruning for Efficient Test-Time Scaling (2026.findings-acl)

Challenge:	Large Language Models (LLMs) are an integral enabler of enterprise applications such as summarization, retrieval augmented generation, and agentic workflows.
Approach:	They propose a model transformation and distillation procedure that prefills later layers’ KV cache using an earlier layer’s output, allowing prompt tokens to skip those later layers.
Outcome:	The proposed procedure can reduce prefill computation by 25-50% across several LLM families while incurring minimum quality degradation.

Copied to clipboard

Zhixiang Liang, Beichen Huang, Zheng Wang, Minjia Zhang

SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers (2025.acl-long)

Challenge:	Existing approaches to speed up parallel scaling have relied on similarity-based or confidence-based pruning, but these signals do not reliably indicate trace quality.
Approach:	They propose a pruning framework that evaluates reasoning steps using hidden states and dynamically prunes unpromising traces during generation.
Outcome:	The proposed framework reduces end-to-end inference latency by 45%–70% on average compared to self-consistency while improving reasoning accuracy.

Copied to clipboard

Zicong Tang, Shi Luohe, Zuchao Li, Baoyuan Qi, Liu Guoming, Lefei Zhang, Ping Wang

Challenge:	Large language models (LLMs) have impressive capabilities across various fields, but their widespread use is facing a severe and realistic challenge, which is their high demand for GPU memory.
Approach:	They propose a KV cache reduction method which balances both shallow and deep layers by using an attention weight based eviction method and a codebook based replacement approach.
Outcome:	The proposed method reduces the KV cache for shallower layers while preserving similar or even better model performance.

VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization (2026.acl-long)

Copied to clipboard

Dingyu Yao, Chenxu Yang, Zhengyang Tong, Zheng Lin, Wei Liu, Jian Luan, Weiping Wang

ReFreeKV: Towards Threshold-Free KV Cache Compression (2026.findings-acl)

Challenge:	Existing quantization methods for large language models suffer performance degradation at ultra-low bit-widths due to key cache outliers.
Approach:	They propose a vector quantization method that suppresses outliers in the key cache and reduces memory access overhead.
Outcome:	The proposed method outperforms baseline quantization methods across long-context understanding and mathematical reasoning tasks while minimizing memory access overhead.

Copied to clipboard

Xuanfan Ni, Liyan Xu, Chenyang Lyu, Longyue Wang, Mo Yu, Lemao Liu, Fandong Meng, Jie Zhou, Piji Li

Anchoring the Cache: Mitigating Contextual Hallucination in KV-Compressed Long-Context Summarization (2026.acl-long)

Challenge:	Towards the KV cache efficiency, we propose a new objective that lifts the threshold constraints for robust KV compression.
Approach:	They propose a method that adjusts KV cache budgets while preserving full-cache performance.
Outcome:	The proposed method can reduce memory consumption while preserving full-cache performance.

Copied to clipboard

Yu Fu, Chen Luo, Josef Valvoda, Xin Zhang, Xuejing Lei, Xiao Pan, Hui Liu, Yue Dong

Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs (2025.acl-long)

Challenge:	Recent studies show that KV cache compression can increase hallucination scores in LLMs . modern LLM models support extremely long sequences, but their impact on model hallucinosity remains underexplored.
Approach:	They propose a decoding-phase strategy that selectively removes generated KV pairs from retrieval heads responsible for retrieving critical information from source context.
Outcome:	The proposed method reduces hallucination across multiple models and datasets while preserving computational efficiency.

Copied to clipboard

Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Shenlixing Shenlixing, Chenzhan Chenzhan, Xipeng Qiu, Qi Zhang, Tao Gui

FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration (2026.findings-acl)

Challenge:	Multi-head Latent Attention (MLA) is an innovative architecture designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector.
Approach:	They propose a data-efficient fine-tuning method for transitioning from MHA to MLA using a latent vector cache.
Outcome:	The proposed architecture reduces the KV cache size of Llama2-7B by 92.19%, with only 1% drop in LongBench performance.

Copied to clipboard

Dongwon Jo, Jiwon Song, Yulhwa Kim, Jae-Joon Kim

LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning (2026.acl-long)

Challenge:	Large language models (LLMs) excel at handling long-context sequences, but require substantial prefill computation and key-value (KV) cache.
Approach:	They propose a KV cache compression framework that decouples prefill computation from decoding KV budget.
Outcome:	The proposed framework reduces latency in prefill and decoding by leveraging the stabilization of token importance in later layers.

Copied to clipboard

Haoyue Zhang, Hualei Zhang, Xiaosong Ma, Jie Zhang, Song Guo

Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query (2025.emnlp-main)

Challenge:	Existing KV cache compression methods mitigate memory bottlenecks but struggle in long reasoning tasks.
Approach:	They propose a lagged eviction framework that prioritizes evicts based on tokens’ recurrence patterns to reduce KV cache by 50% and maintain comparable accuracy.
Outcome:	The proposed framework reduces KV cache by 50% 70% while maintaining comparable accuracy, outperforming existing KV baselines.

Copied to clipboard

Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, Wanxiang Che

Challenge:	Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries.
Approach:	They propose a lookahead q-cache framework that generates low-cost pseudo lookaheaded queries to better approximate the true decoding-stage queries.
Outcome:	The proposed framework outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks and can be flexibly combined to yield further improvements.

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization (2026.findings-acl)

Copied to clipboard

Jiantong Jiang, Peiyu Yang, Rui Zhang, Feng Liu