Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show draft-then-verify paradigm can achieve around 5x speedup for the popular Transformer architectures with comparable generation quality to beam search decoding. |
| Approach: | They propose to use Spec-Drafter and Spec Verification to accelerate autoregressive (AR) decoding by combining a model optimized for efficient and accurate drafting and a reliable method for verifying the drafted tokens efficiently. |
| Outcome: | The proposed method achieves 5x speedup on seq2seq tasks with comparable generation quality to beam search decoding, refreshing the impression that draft-then-verify paradigm introduces only 1.4x2x speed up. |
Similar Papers
Speculative Streaming: Efficient and Scalable Speculative Decoding with Multi-Stream Attention (2025.emnlp-main)
Copied to clipboard
Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Antonie Lin, Mohammad Rastegari, Mahyar Najibi
| Challenge: | Speculative decoding is a prominent technique for accelerating LLM inference by leveraging an auxiliary draft model, but its effectiveness is limited by the autoregressive nature of draft generation. |
| Approach: | They propose a method that integrates speculative draft generation directly within the target model using multi-stream attention. |
| Outcome: | The proposed method improves acceptance but also latency and speculation latency, limiting overall speedup. |
Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (2025.naacl-long)
Copied to clipboard
Jacob K Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto
| Challenge: | Existing methods to accelerate large language model inference are limited by the reliance on incremental token generation in existing draft models. |
| Approach: | They propose an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences and allows parallelization of both the drafting and verification steps. |
| Outcome: | The proposed approach provides 7.2x speedups over standard generation processes and 1.75x speed ups over existing speculative decoding approaches. |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
RASD: Retrieval-Augmented Speculative Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating draft tokens rely on lightweight draft models or additional model structures to generate tokens and retrieve context from databases. |
| Approach: | They propose to use a pruning method to enhance model-based speculative decoding by combining the best-fit model with the best retrieval tree. |
| Outcome: | The proposed method achieves state-of-the-art inference acceleration across tasks such as DocQA, Summary, Code, and In-Domain QA. |
Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism (2026.acl-long)
Copied to clipboard
| Challenge: | Parallel Speculative Decoding (PSD) has limitations due to speedup limits and high computational waste . a novel synchronous mechanism solves the Retrieval Precision-Efficiency Dilemma . |
| Approach: | They propose a framework that combines a draft-verification-based approach with a synchronous mechanism to solve the Retrieval Precision-Efficiency Dilemma. |
| Outcome: | The proposed framework breaks speedup limits for Speculative Decoding by overlapping draft generation with verification. |
SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding performance degrades as input length increases, with significant drops even at moderate lengths. |
| Approach: | They propose a drop-in enhancement that improves speculative decoding on long sequences without additional training. |
| Outcome: | The proposed enhancement accelerates speculative decoding by up to 2.84 on 16K-token long document summarization and up to 3.86 on long-form reasoning while preserving the short-input performance of state-of-the-art frameworks. |
Draft
& Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding. |
| Approach: | They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model. |
| Outcome: | The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint. |
Decoding Speculative Decoding (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference. |
| Approach: | They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens. |
| Outcome: | The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models. |
LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) is a promising technique for LLM inference acceleration. |
| Approach: | They propose a method to generate draft tokens in a retrieval-based manner to reduce drafting overhead and improve inference speed. |
| Outcome: | Extensive tests show that *LogitSpec* can achieve 2.61 speedup and 3.28 mean accepted tokens per decoding step. |
Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding uses a small draft model to generate a single input token, instead of sequentially generating tokens until completion. |
| Approach: | They propose a method that generates varying draft models adapted to the input context using simple rules. |
| Outcome: | The proposed method is competitive with the current SOTA for self-speculative decoding while being a truly plug-and-play method. |