When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judges (2026.acl-long)
Copied to clipboard
| Challenge: | Multi-agent LLMs generate multiple candidate responses that are aggregated by an LLM judge. |
| Approach: | They propose to advocate KV cache reuse across partially shared contexts and report substantial speedups for generation agents. |
| Outcome: | The proposed reuse strategies weaken cross-candidate attention, especially for later candidate blocks, and highlight judge-centric inference as a distinct regime that requires dedicated, risk-aware system design. |
Similar Papers
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent studies have shown that sharing key-value (KV) cache across layers is effective in efficient inference of large language models. |
| Approach: | They propose a unified framework that covers several recent methods and their novel variants to investigate cross-layer KV sharing. |
| Outcome: | The proposed framework achieves higher throughput and better performance when reducing the size of the key-value cache by 2 while maintaining competitive performance. |
KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to inference large language models are limited by CPU capabilities and memory constraints. |
| Approach: | They propose an efficient I/O-aware LLM inference method that overlaps GPU computation with KV cache transfer to minimize idle GPU time. |
| Outcome: | The proposed method reduces the cost of auto-regressive decoding by 35.8% . it also achieves 46.2% higher throughput during decoding compared to state-of-the-art methods. |
When Efficiency Meets Safety: A Benchmark Security Analysis of KV Cache Compression in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Key-Value (KV) caching is widely used in large language models to enable long-context inference efficiently, yet its security implications remain underexplored. |
| Approach: | They propose a history-aware, per-head feedback merging strategy that prevents safety degradation while maintaining efficiency. |
| Outcome: | The proposed strategy prevents safety degradation while maintaining efficiency. |
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches (2024.findings-emnlp)
Copied to clipboard
Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, Xia Hu
| Challenge: | Long context capability is a crucial competency for large language models as it mitigates the human struggle to digest long-form texts. |
| Approach: | They propose to evaluate 10+ state-of-the-art approaches for long context-capable LLMs. |
| Outcome: | The proposed methods are compared against 10+ state-of-the-art approaches across seven categories of long context tasks. |
KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference (2026.eacl-industry)
Copied to clipboard
| Challenge: | Long-context Large Language Models (LLMs) face significant memory bottlenecks due to the linear growth of key-value (KV) cache with sequence length. |
| Approach: | They propose a framework that maps the trade-off frontier between total memory consumption and task accuracy across three complementary optimization techniques. |
| Outcome: | The proposed model-specific configurations achieve 68-78% total memory reduction with minimal (1-3%) accuracy degradation on long-context tasks. |
Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation (2026.eacl-srw)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) systems face efficiency bottlenecks in prefill due to attention mechanism, and traditional KV cache only accelerates decoding. |
| Approach: | They propose a multi-document KV cache reuse framework for multi-doc RAG workloads . they propose to resolve position and context misalignment while eliminating document-specific quadratic complexity in prefill. |
| Outcome: | The proposed framework solves position and context misalignment issues while eliminating document-specific quadratic complexity in prefill. |
SpecCache: Speculative KV Cache Reuse for Efficient RAG Serving (2026.acl-long)
Copied to clipboard
Zijian Wen, Tao Zhang, Shuangwu Chen, Shenghao Ye, Yu Guo, Qirui Chen, Jingxian Shuai, Yunpeng Hou, Huasen He, null Jianyang
| Challenge: | Retrieval-Augmented Generation (RAG) improves LLMs but faces high prefill latency during long contexts. |
| Approach: | They propose a method that uses deep-layer hidden-state norms to guide token selection . they propose to use deep-layered hidden-status norms as a proxy to guide the token selection. |
| Outcome: | The proposed SpecCache outperforms state-of-the-art (SOTA) benchmarks. |
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a powerful tool for high-performance inference serving. |
| Approach: | They focus on system-aware KV infrastructure for serving LLMs . they analyze cross-behavior co-design affinity and behavior-objective links . |
| Outcome: | The proposed key-value (KV) cache is crucial for low-latency, high-throughput LLM inference serving. |
FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference (2025.findings-emnlp)
Copied to clipboard
Dongwei Wang, Zijie Liu, Song Wang, Yuxin Ren, Jianing Deng, Jingtong Hu, Tianlong Chen, Huanrui Yang
| Challenge: | Key-Value (KV) cache reading latency increases with context lengths hindering LLM inference . important tokens are sparsely distributed across the long context, making existing retrieval inaccurate . |
| Approach: | They propose a method to retain a small fraction of KV cache based on token importance . important tokens are often sparsely distributed across the long context . |
| Outcome: | The proposed method reduces decoding latency by 1.2 to 1.5. |
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Existing static compression methods suffer from coarse-grained caching and high I/O overhead. |
| Approach: | They propose a training-free dynamic compression framework that uses a sparse attention mechanism to categorize attention heads based on stability and similarity. |
| Outcome: | The proposed framework achieves state-of-the-art performance on long-context benchmarks and accelerates decoding by up to 3 compared to the original model with a 224K context. |