HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models struggle to maintain stable understanding performance and low GPU memory overhead. |
| Approach: | They propose a training-free architecture for real-time and accurate understanding of video streams . HERMES reuses a compact KV cache, enabling efficient streaming understanding . |
| Outcome: | The proposed architecture achieves 10 faster TTFT compared to prior SOTA. |
Similar Papers
LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing retrieval-based methods compromise semantic integrity through fixed-size chunking and suffer from inefficient linear scanning. |
| Approach: | They propose a method that preserves local semantic coherence through boundary-aware chunking and constructs a recursive hierarchical index rooted in the triangle inequality. |
| Outcome: | The proposed method achieves 3.6 end-to-end inference speedup with negligible degradation in model performance. |
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Existing static compression methods suffer from coarse-grained caching and high I/O overhead. |
| Approach: | They propose a training-free dynamic compression framework that uses a sparse attention mechanism to categorize attention heads based on stability and similarity. |
| Outcome: | The proposed framework achieves state-of-the-art performance on long-context benchmarks and accelerates decoding by up to 3 compared to the original model with a 224K context. |
HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are hindered by the rapid growth of key–value (KV) caches. |
| Approach: | They propose a hybrid KV cache compression framework that reduces KV memory by 7.9 and speeds up decoding by 1.52. |
| Outcome: | Experiments on 11 multimodal benchmarks show that HYBRIDKV cuts KV cache memory by 7.9 and speeds up decoding by 1.52. |
StreamingEval: A Unified Evaluation Framework towards Realistic Streaming Video Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing research on streaming video understanding focuses on isolated aspects of visual understanding, but ignores practical deployability under realistic resource constraints. |
| Approach: | They propose a framework to evaluate streaming video understanding capabilities under realistic constraints. |
| Outcome: | StreamingEval benchmarks offline and online video models under a standardized protocol . it evaluates visual encoding efficiency, text decoding latency and task performance . |
Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using a key-value cache, memory consumption is a bottleneck for high-throughput language models. |
| Approach: | They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput. |
| Outcome: | The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques. |
Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs (2025.acl-short)
Copied to clipboard
| Challenge: | Recent advances in Video Large Language Models (Video-LLMs) have achieved exceptional performance on tasks like video question answering and captioning. |
| Approach: | They propose a decoding strategy that leverages sparse top-K attention and dense full attention to accelerate Video-LLMs without loss. |
| Outcome: | The proposed approach achieves a 1.94 walltime speedup in video processing. |
PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce memory usage for large language models neglect inter-layer dependency between layers and huge memory consumption in pre-computation. |
| Approach: | They propose a method that compresses the KV cache by layer-wise retaining crucial context. |
| Outcome: | The proposed method reduces memory usage by layer-wise retaining crucial context . it can improve 2.2x throughput compared to Accelerate with over 54% memory reduction . |
MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference (2025.naacl-long)
Copied to clipboard
| Challenge: | Long-context Multimodal Large Language Models (MLLMs) require substantial computational resources as their multimodal Key-Value (KV) cache grows with increasing input lengths, challenging memory and time efficiency. |
| Approach: | They propose a dynamic multimodal KV cache allocation strategy that dynamically allocating KV size based on attention entropy to better adapt to multimodal interactions. |
| Outcome: | The proposed model achieves up to 72% KV cache memory reduction and 2.82 faster decoding speeds while maintaining or enhancing performance on various multimodal tasks in a long context. |
ZigZagKV: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to accelerate inference of Large Language models (LLMs) are limited in their ability to retain key tokens as input length increases. |
| Approach: | They propose a method that leverages layer uncertainty to allocate budget size for each layer to reduce memory usage. |
| Outcome: | The proposed method reduces memory usage of the KV caches to only 20% when compared to full KV inference while achieving nearly lossless performance. |
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a powerful tool for high-performance inference serving. |
| Approach: | They focus on system-aware KV infrastructure for serving LLMs . they analyze cross-behavior co-design affinity and behavior-objective links . |
| Outcome: | The proposed key-value (KV) cache is crucial for low-latency, high-throughput LLM inference serving. |