HCSpec: Two-Tier Horizontal Cascade Speculative Decoding for High-Efficiency Large Language Model Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to decode large language models adopt a homogeneous architecture . autoregressive decoding is a bottleneck because tokens must be generated sequentially . |
| Approach: | They propose a framework that organizes heterogeneous position-specialized draft modules into a horizontal cascade. |
| Outcome: | The proposed framework outperforms the current state-of-the-art (EAGLE3) and achieves 3.72x acceleration over vanilla decoding. |
Similar Papers
Draft
& Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding. |
| Approach: | They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model. |
| Outcome: | The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint. |
Hierarchical Speculative Decoding with Dynamic Window (2025.findings-naacl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) uses an efficient draft model to generate multiple tokens . previous methods depend on simple heuristics to select K or dynamically adjust the window size . |
| Approach: | They propose a framework that allows a draft model to generate multiple tokens . they propose HSDDW, which allows the draft model autonomously decide when to stop generating tokens. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods on four datasets. |
QSpec: Speculative Decoding with Complementary Quantization Schemes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Quantization is widely adopted to accelerate inference and reduce memory consumption in large language models. |
| Approach: | They propose a quantization paradigm that decouples efficiency from quality by integrating two complementary schemes via speculative decoding. |
| Outcome: | The proposed approach achieves 1.64x speedup without quality degradation and outperforms state-of-the-art speculative decoding methods by 1.55x in batched settings. |
LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can process extremely long contexts, requiring efficient inference over extended inputs. |
| Approach: | They propose a model that uses a constant-sized key-value cache to train long-context models. |
| Outcome: | Experimental results show that LongSpec achieves 3.26x speedup over strong Flash Attention baselines and 2.34x wall clock time on four math reasoning tasks. |
HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Autoregressive decoding limits the inference throughput of Large Language Models due to its sequential dependency. |
| Approach: | They propose a framework that allocates verification effort in proportion to candidate uncertainty. |
| Outcome: | Speculative decoding achieves an average speedup over state-of-the-art methods . a small subset of high-confidence predictions accounts for most successful verifications . |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality. |
| Approach: | They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages. |
| Outcome: | The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments. |
PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding is limited by sequential stage dependencies that prevent full hardware utilization. |
| Approach: | They propose a framework that generalizes speculative decoding to use multiple models arranged in a hierarchical pipeline and enables asynchronous execution with lightweight coordination for prediction verification and rollback. |
| Outcome: | The proposed framework achieves 2.25 tokens/unit through pipelined parallelism with multiple models arranged in a hierarchical pipeline. |
DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive (AR) decoding in large language models is latency-bounded by strictly sequential token generation. |
| Approach: | They propose a diffusion-based drafter that proposes multi-token candidates and then verifies them in parallel by the target model. |
| Outcome: | The proposed drafter generates multi-token proposals in a single forward pass while remaining compatible with standard AR verifiers. |
Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding (2025.findings-naacl)
Copied to clipboard
Sukmin Cho, Sangjin Choi, Taeho Hwang, Jeongyeon Seo, Soyeong Jeong, Huije Lee, Hoyun Song, Jong C. Park, Youngjin Kwon
| Challenge: | Existing methods for drafting and verifying tokens require significant fine-tuning or have inconsistent performance across tasks. |
| Approach: | They propose a lossless drafting approach that organizes various token sources into multiple databases in a hierarchical framework based on temporal locality. |
| Outcome: | The proposed method outperforms existing database drafting methods on Spec-Bench using 7B and 13B parameters. |