Graph-Structured Speculative Decoding

Challenge:	Speculative decoding is a promising technique to accelerate the inference of Large Language Models.
Approach:	They propose a method that uses a token graph to record multiple sequence hypotheses within a single draft stage.
Outcome:	The proposed method significantly accelerates the inference of Large Language Models (LLMs) it allows the LLM to choose from and select the longest sequence that meets its standards.

Similar Papers

Overview

Challenge

Approach

Outcome

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)

Copied to clipboard

Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui

Language Models

Challenge:	Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding.
Approach:	They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding .
Outcome:	The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models.

Decoding Speculative Decoding (2025.naacl-long)

Copied to clipboard

Minghao Yan, Saurabh Agarwal, Shivaram Venkataraman

Language Models

Challenge:	Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference.
Approach:	They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens.
Outcome:	The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models.

How Speculative Can Speculative Decoding Be? (2024.lrec-main)

Copied to clipboard

Zhuorui Liu, Chen Zhang, Dawei Song

Semantic Text Processing

Challenge:	Large language models (LLMs) have a largely increased latency due to their ability to autoregressively model . speculative decoding is a technique that trades generation quality for speed .
Approach:	They propose to use a draft model to draft tokens autoregressively and then verify them in parallel.
Outcome:	The proposed model could draft tokens autoregressively and then verify them in parallel . the proposed model trades quality for speed and could fail in verification stage .

Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)

Copied to clipboard

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra

Language Models

Challenge:	Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding.
Approach:	They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model.
Outcome:	The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint.

Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)

Copied to clipboard

Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun

Multilinguality

Challenge:	Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings.
Approach:	They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM.
Outcome:	The proposed model significantly speeds up inference time and out-of-domain speedup across various languages.

Hierarchical Speculative Decoding with Dynamic Window (2025.findings-naacl)

Copied to clipboard

Shensian Syu, Hung-yi Lee

Challenge:	Speculative decoding (SD) uses an efficient draft model to generate multiple tokens . previous methods depend on simple heuristics to select K or dynamically adjust the window size .
Approach:	They propose a framework that allows a draft model to generate multiple tokens . they propose HSDDW, which allows the draft model autonomously decide when to stop generating tokens.
Outcome:	The proposed framework outperforms existing state-of-the-art methods on four datasets.

Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (2025.naacl-long)

Copied to clipboard

Jacob K Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto

Text Generation

Challenge:	Existing methods to accelerate large language model inference are limited by the reliance on incremental token generation in existing draft models.
Approach:	They propose an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences and allows parallelization of both the drafting and verification steps.
Outcome:	The proposed approach provides 7.2x speedups over standard generation processes and 1.75x speed ups over existing speculative decoding approaches.

RASD: Retrieval-Augmented Speculative Decoding (2025.findings-acl)

Copied to clipboard

Guofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang, Yuewei Zhang, Hao Henry Wang

Information Retrieval

Challenge:	Existing methods for generating draft tokens rely on lightweight draft models or additional model structures to generate tokens and retrieve context from databases.
Approach:	They propose to use a pruning method to enhance model-based speculative decoding by combining the best-fit model with the best retrieval tree.
Outcome:	The proposed method achieves state-of-the-art inference acceleration across tasks such as DocQA, Summary, Code, and In-Domain QA.

A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models (2025.acl-long)

Copied to clipboard

Jiesong Liu, Brian Park, Xipeng Shen

Language Models

Challenge:	Large Language Models (LLMs) are highly memory-intensive when performing real-time inference.
Approach:	They propose a technique that allows for speculative decoding to be run on the fly to maximize the efficiency of LLM inferences.
Outcome:	The proposed solution can lead to 3.55-16.48% speed improvement over the standard speculative decoding, and 1.2-3.4 over the default LLMs.

Speculative Decoding for Multi-Sample Inference (2025.findings-emnlp)

Copied to clipboard

Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Ji Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li

Reasoning

Challenge:	Speculative decoding method exploits consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases.
Approach:	They propose a speculative decoding method that exploits the consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases.
Outcome:	The proposed method exploits the intrinsic consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or databases.