Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs (2025.acl-short)
Copied to clipboard
| Challenge: | Recent advances in Video Large Language Models (Video-LLMs) have achieved exceptional performance on tasks like video question answering and captioning. |
| Approach: | They propose a decoding strategy that leverages sparse top-K attention and dense full attention to accelerate Video-LLMs without loss. |
| Outcome: | The proposed approach achieves a 1.94 walltime speedup in video processing. |
Similar Papers
See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for video understanding suffer from autoregressive generation of tokens. |
| Approach: | They propose a training-free loosely SD framework for Video-LLMs that uses visual-relevant tokens to accurately pinpoint the latter. |
| Outcome: | The proposed framework boosts the accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs. |
Efficient Sparse Attention needs Adaptive Token Release (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks, however, their ‘large’ scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. |
| Approach: | They propose to release resources from caches and rebuild key-value states by a lightweight controller module to approximate an ideal top-K sparse attention. |
| Outcome: | The proposed method achieves a significant throughput improvement of 221.8% over full attention and a model with 7 billion tokens. |
Evolving Sparsity: Leveraging Token Importance Dynamics for Efficient LLM Decoding with Sparse Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Efficient long-context inference remains a major challenge for large language models (LLMs), as the cost of attention computation during auto-regressive decoding grows linearly with the context length. |
| Approach: | They propose to model token importance as a dynamic process that evolves over decoding steps and propagates through model layers. |
| Outcome: | The proposed method outperforms baseline sparse attention methods and achieves speedups of up to 5.36 for attention latency and 2.33 for end-to-end decoding. |
Accelerating Dense LLMs via L0-regularized Mixture-of-Experts (2025.acl-short)
Copied to clipboard
| Challenge: | Existing methods for accelerating large language models (LLMs) suffer from slow and costly inference. |
| Approach: | They propose a lightweight MoE approach using cluster confusion matrix and dynamic batching to accelerate dense LLMs. |
| Outcome: | The proposed method achieves 2.5x speedup over dense models while maintaining competitive performance. |
Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating attention importance for tokens are ineffective . dLLMs require bidirectional attention, which limits inference efficiency . |
| Approach: | They propose a training-free attention sparsification framework for efficient long-context inference . they propose 'sink-aware pruning strategy' to accurately estimate and remove redundant computation . |
| Outcome: | The proposed approach offers 29 lossless speedup under 32K context length. |
ViPE: Visual Perception in Parameter Space for Efficient Video-Language Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing video-language models rely on concatenating visual tokens with textual inputs for joint modeling, but this method suffers from significant inefficiency when scaling to long videos with dense visual inputs. |
| Approach: | They propose a video-to-parameter efficiency paradigm called ViPE that transforms video content into visual perceptual weights, which are directly injected into the LLM’s parameters. |
| Outcome: | The proposed model reduces FLOPs by 85% and inference time by up to 65% while reducing FLOP and FLOP inference times by up-to-65%. |
PLD+: Accelerating LLM Inference by Leveraging Language Model Artifacts (2025.findings-naacl)
Copied to clipboard
| Challenge: | speculative decoding is a novel decoding paradigm for large language models . however, its use is limited by its computational resources and fine-tuning requirements . |
| Approach: | They propose a tuning-free approach that accelerates inference of large language models . they use draft and verify principle to accelerate inference process . |
| Outcome: | The proposed approach outperforms tuning-free approaches on input-guided tasks and outperformed state-of-the-art EAGLE on four of the tasks. |
Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that Video Large Language Models (Vide-oLLMs) are efficient at video understanding but lack the quadratic complexity of visual tokens. |
| Approach: | They propose a plug-and-play inference acceleration framework for VideoLLM token compression that quantifies each frame’s uniqueness and adaptively adjusts compression intensity across frames. |
| Outcome: | Extensive experiments on video large language models and benchmarks show that the proposed framework can preserve essential information while reducing redundancy in video sequences. |
D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing training-free adaptation frameworks for video large language models are limited by the scarcity of high-quality video-text data. |
| Approach: | They propose a training-free adaptation framework that incorporates dynamic compression and question decomposition to address perception bottleneck and token overload. |
| Outcome: | The proposed framework improves video understanding across various benchmarks and on long-video benchmarks. |
Vista-LLM: Decoupled Query-Guided Visual Token Pruning for Efficient Long-Video Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Long-video understanding is bottlenecked by the high cost of processing massive visual tokens. |
| Approach: | They propose a decoupled framework for query-guided visual token pruning . their method reduces visual tokens by 90% and accelerates inference by 98% . |
| Outcome: | The proposed framework reduces visual tokens by 90% and accelerates inference while retaining over 98% of baseline performance on average. |