SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are an integral enabler of enterprise applications such as summarization, retrieval augmented generation, and agentic workflows. |
| Approach: | They propose a model transformation and distillation procedure that prefills later layers’ KV cache using an earlier layer’s output, allowing prompt tokens to skip those later layers. |
| Outcome: | The proposed procedure can reduce prefill computation by 25-50% across several LLM families while incurring minimum quality degradation. |
Similar Papers
FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel at handling long-context sequences, but require substantial prefill computation and key-value (KV) cache. |
| Approach: | They propose a KV cache compression framework that decouples prefill computation from decoding KV budget. |
| Outcome: | The proposed framework reduces latency in prefill and decoding by leveraging the stabilization of token importance in later layers. |
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work mitigates memory overhead by offloading or compressing the Key-Value cache. |
| Approach: | They propose a method that integrates quantization and offloading into a generative large language model by using a hybrid compression method. |
| Outcome: | The proposed method outperforms the state-of-the-art in long-context evaluations. |
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. |
| Approach: | They propose an algorithm that uses early LLM layers as filters to select and compress input tokens, reducing the context length for subsequent processing. |
| Outcome: | The proposed method outperforms existing techniques on the Needle in a Haystack task while demonstrating comparable performance on the LongBench challenge. |
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly lengthy and require longer prompts . this paper presents a coarse-to-fine prompt compression method to reduce cost and increase performance. |
| Approach: | They propose a coarse-to-fine prompt compression method that maintains semantic integrity under high compression ratios and a token-level iterative compression algorithm to better model the interdependence between compressed contents. |
| Outcome: | The proposed method yields state-of-the-art performance and allows for up to 20x compression with little performance loss over four datasets from different scenarios. |
VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing quantization methods for large language models suffer performance degradation at ultra-low bit-widths due to key cache outliers. |
| Approach: | They propose a vector quantization method that suppresses outliers in the key cache and reduces memory access overhead. |
| Outcome: | The proposed method outperforms baseline quantization methods across long-context understanding and mathematical reasoning tasks while minimizing memory access overhead. |
KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to inference large language models are limited by CPU capabilities and memory constraints. |
| Approach: | They propose an efficient I/O-aware LLM inference method that overlaps GPU computation with KV cache transfer to minimize idle GPU time. |
| Outcome: | The proposed method reduces the cost of auto-regressive decoding by 35.8% . it also achieves 46.2% higher throughput during decoding compared to state-of-the-art methods. |
PLD+: Accelerating LLM Inference by Leveraging Language Model Artifacts (2025.findings-naacl)
Copied to clipboard
| Challenge: | speculative decoding is a novel decoding paradigm for large language models . however, its use is limited by its computational resources and fine-tuning requirements . |
| Approach: | They propose a tuning-free approach that accelerates inference of large language models . they use draft and verify principle to accelerate inference process . |
| Outcome: | The proposed approach outperforms tuning-free approaches on input-guided tasks and outperformed state-of-the-art EAGLE on four of the tasks. |
Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to generate draft tokens in large language models are expensive and resource-intensive. |
| Approach: | They propose an approach to generate draft tokens using a segment of the LLM and a self-distillation method to enhance the quality of draft token. |
| Outcome: | The proposed approach generates draft tokens using a segment of the LLM and a self-distillation method to improve quality and speed up generation. |
See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for video understanding suffer from autoregressive generation of tokens. |
| Approach: | They propose a training-free loosely SD framework for Video-LLMs that uses visual-relevant tokens to accurately pinpoint the latter. |
| Outcome: | The proposed framework boosts the accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs. |
RefreshKV: Updating Small KV Cache During Long-form Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating long sequences of tokens are expensive and require memory and computation resources. |
| Approach: | They propose a method that alternates between full context attention and attention over a subset of input tokens during generation. |
| Outcome: | The proposed method achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks. |