Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding is a novel method to expedite inference in autoregressive (large) language models. |
| Approach: | They propose to use a smaller model as a draft model to speculate a block of tokens, which the target model then evaluates for acceptance. |
| Outcome: | The proposed method can be used to accelerate inference in autoregressive (large) language models by using smaller models as draft models to speculate tokens for multiple inference steps. |
Similar Papers
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
How Speculative Can Speculative Decoding Be? (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a largely increased latency due to their ability to autoregressively model . speculative decoding is a technique that trades generation quality for speed . |
| Approach: | They propose to use a draft model to draft tokens autoregressively and then verify them in parallel. |
| Outcome: | The proposed model could draft tokens autoregressively and then verify them in parallel . the proposed model trades quality for speed and could fail in verification stage . |
Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) is a powerful and efficient way to accelerate autoregressive generation. |
| Approach: | They propose a training-free framework that recovers valid tokens discarded by standard verification . they use online correction memory and Semantic Consistency Gating to analyze rejections . |
| Outcome: | The proposed framework outperforms existing methods and achieves peak throughput speedup of 2.33x. |
Decoding Speculative Decoding (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference. |
| Approach: | They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens. |
| Outcome: | The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models. |
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models. |
| Approach: | They propose a self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. |
| Outcome: | The proposed framework suppresses spurious confidence and bounds speculation length based on token-wise decoding difficulty. |
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a key technique for enhancing the inference speed of Large Language Models. |
| Approach: | They propose a method that adds padding tokens to ensure that the number of new tokens remains consistent across samples. |
| Outcome: | The proposed method can handle the issue of inconsistent prediction tokens without adding padding tokens. |
Speculative Decoding for Multi-Sample Inference (2025.findings-emnlp)
Copied to clipboard
Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Ji Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Speculative decoding method exploits consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Approach: | They propose a speculative decoding method that exploits the consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Outcome: | The proposed method exploits the intrinsic consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or databases. |
SSSD: Simply-Scalable Speculative Decoding (2026.acl-long)
Copied to clipboard
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
| Challenge: | Existing methods for accelerating inference in Large Language Models require additional training and training, resulting in a higher deployment and maintenance cost. |
| Approach: | They propose a training-free method that combines lightweight n-gram matching with hardware-aware speculation. |
| Outcome: | SSSD reduces latency by up to 2.9 and is faster than autoregressive decoding methods. |
Benchmarking the Energy Savings with Speculative Decoding Strategies (2026.findings-eacl)
Copied to clipboard
Rohit Dutta, Paramita Koley, Soham Poddar, Janardan Misra, Sanjay Podder, Naveen Balani, Saptarshi Ghosh, Niloy Ganguly
| Challenge: | Existing studies on speculative decoding have focused on the energy requirements of these models, despite their utility and utility. |
| Approach: | They propose to analyze the energy requirements of speculative decoding strategies and analyze how various factors influence the energy optimizations. |
| Outcome: | The proposed approach reduces decoding time while offloading a substantial portion of the sequential generation to a smaller, more efficient model. |
HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Autoregressive decoding limits the inference throughput of Large Language Models due to its sequential dependency. |
| Approach: | They propose a framework that allocates verification effort in proportion to candidate uncertainty. |
| Outcome: | Speculative decoding achieves an average speedup over state-of-the-art methods . a small subset of high-confidence predictions accounts for most successful verifications . |