| Challenge: | Speculative decoding has emerged as an alternative to autoregressive decoding for expediting inference in large language models (LLMs). prevailing assumptions focus solely on latency reduction, neglecting the computational expenses. |
| Approach: | They propose a speculative decoding enhancement to reduce the speculation set while validating more effective tokens. |
| Outcome: | The proposed method reduces the speculation set while validating more effective tokens. |
Similar Papers
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are highly memory-intensive when performing real-time inference. |
| Approach: | They propose a technique that allows for speculative decoding to be run on the fly to maximize the efficiency of LLM inferences. |
| Outcome: | The proposed solution can lead to 3.55-16.48% speed improvement over the standard speculative decoding, and 1.2-3.4 over the default LLMs. |
Speculative Verification: Exploiting Information Gain for Speculative Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used for many applications but their size and computational cost make inference serving a significant challenge. |
| Approach: | They propose an efficient augmentation to Speculative Decoding (SD) that predicts speculation accuracy and dynamically adapts the verification length to maximize throughput. |
| Outcome: | The proposed model reduces wasted verification on rejected tokens and improves decoding efficiency. |
How Speculative Can Speculative Decoding Be? (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a largely increased latency due to their ability to autoregressively model . speculative decoding is a technique that trades generation quality for speed . |
| Approach: | They propose to use a draft model to draft tokens autoregressively and then verify them in parallel. |
| Outcome: | The proposed model could draft tokens autoregressively and then verify them in parallel . the proposed model trades quality for speed and could fail in verification stage . |
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a key technique for enhancing the inference speed of Large Language Models. |
| Approach: | They propose a method that adds padding tokens to ensure that the number of new tokens remains consistent across samples. |
| Outcome: | The proposed method can handle the issue of inconsistent prediction tokens without adding padding tokens. |
SSSD: Simply-Scalable Speculative Decoding (2026.acl-long)
Copied to clipboard
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
| Challenge: | Existing methods for accelerating inference in Large Language Models require additional training and training, resulting in a higher deployment and maintenance cost. |
| Approach: | They propose a training-free method that combines lightweight n-gram matching with hardware-aware speculation. |
| Outcome: | SSSD reduces latency by up to 2.9 and is faster than autoregressive decoding methods. |
Decoding Speculative Decoding (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference. |
| Approach: | They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens. |
| Outcome: | The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models. |
MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification (2026.findings-acl)
Copied to clipboard
Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Tianyu Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai
| Challenge: | Autoregressive large language models suffer from high inference latency due to memorybandwidth constraints. |
| Approach: | They propose a method that decouples generation and verification by decoupling tokens and a lightweight draft model. |
| Outcome: | The proposed method delivers consistent and significant speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks. |
Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate only one token at each decoding step, leading to high latency. |
| Approach: | They propose a speculative decoding paradigm that stores tokens in an adjacency matrix and employs a breadth-first-search algorithm to construct a draft tree. |
| Outcome: | The proposed method outperforms existing train-free methods by 30% and even a training method by 25%. |
Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) is a powerful and efficient way to accelerate autoregressive generation. |
| Approach: | They propose a training-free framework that recovers valid tokens discarded by standard verification . they use online correction memory and Semantic Consistency Gating to analyze rejections . |
| Outcome: | The proposed framework outperforms existing methods and achieves peak throughput speedup of 2.33x. |