See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for video understanding suffer from autoregressive generation of tokens. |
| Approach: | They propose a training-free loosely SD framework for Video-LLMs that uses visual-relevant tokens to accurately pinpoint the latter. |
| Outcome: | The proposed framework boosts the accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs. |
Similar Papers
UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality. |
| Approach: | They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages. |
| Outcome: | The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments. |
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Video large language models (Vid-LLMs) rely on dense video token representations and require substantial memory and computational overhead in both prefilling and decoding. |
| Approach: | They propose a training-free speculative decoding framework that prunes up to 90% of video tokens to enable efficient speculation without sacrificing accuracy. |
| Outcome: | The proposed framework achieves 2.68 speedup on LLaVA-OneVision-72B and 2.11 speed up on Qwen2.5-VL-32B. |
Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs (2025.acl-short)
Copied to clipboard
| Challenge: | Recent advances in Video Large Language Models (Video-LLMs) have achieved exceptional performance on tasks like video question answering and captioning. |
| Approach: | They propose a decoding strategy that leverages sparse top-K attention and dense full attention to accelerate Video-LLMs without loss. |
| Outcome: | The proposed approach achieves a 1.94 walltime speedup in video processing. |
Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings. |
| Approach: | They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM. |
| Outcome: | The proposed model significantly speeds up inference time and out-of-domain speedup across various languages. |
MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification (2026.findings-acl)
Copied to clipboard
Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Tianyu Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai
| Challenge: | Autoregressive large language models suffer from high inference latency due to memorybandwidth constraints. |
| Approach: | They propose a method that decouples generation and verification by decoupling tokens and a lightweight draft model. |
| Outcome: | The proposed method delivers consistent and significant speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks. |
LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can process extremely long contexts, requiring efficient inference over extended inputs. |
| Approach: | They propose a model that uses a constant-sized key-value cache to train long-context models. |
| Outcome: | Experimental results show that LongSpec achieves 3.26x speedup over strong Flash Attention baselines and 2.34x wall clock time on four math reasoning tasks. |
Vista-LLM: Decoupled Query-Guided Visual Token Pruning for Efficient Long-Video Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Long-video understanding is bottlenecked by the high cost of processing massive visual tokens. |
| Approach: | They propose a decoupled framework for query-guided visual token pruning . their method reduces visual tokens by 90% and accelerates inference by 98% . |
| Outcome: | The proposed framework reduces visual tokens by 90% and accelerates inference while retaining over 98% of baseline performance on average. |
Draft
& Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding. |
| Approach: | They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model. |
| Outcome: | The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint. |
Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing speculative decoding models face performance collapse due to key-value cache explosion and context window mismatches. |
| Approach: | They propose a framework that offloads visual computation to the target model by using hidden state reuse. |
| Outcome: | The proposed framework achieves an average speedup of 2.82x even with 25k visual tokens . |
Speculative Verification: Exploiting Information Gain for Speculative Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used for many applications but their size and computational cost make inference serving a significant challenge. |
| Approach: | They propose an efficient augmentation to Speculative Decoding (SD) that predicts speculation accuracy and dynamically adapts the verification length to maximize throughput. |
| Outcome: | The proposed model reduces wasted verification on rejected tokens and improves decoding efficiency. |