A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models

Challenge:	Large Language Models (LLMs) are highly memory-intensive when performing real-time inference.
Approach:	They propose a technique that allows for speculative decoding to be run on the fly to maximize the efficiency of LLM inferences.
Outcome:	The proposed solution can lead to 3.55-16.48% speed improvement over the standard speculative decoding, and 1.2-3.4 over the default LLMs.

Similar Papers

Overview

Challenge

Approach

Outcome

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)

Copied to clipboard

Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui

Language Models

Challenge:	Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding.
Approach:	They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding .
Outcome:	The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models.

Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)

Copied to clipboard

Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun

Multilinguality

Challenge:	Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings.
Approach:	They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM.
Outcome:	The proposed model significantly speeds up inference time and out-of-domain speedup across various languages.

Decoding Speculative Decoding (2025.naacl-long)

Copied to clipboard

Minghao Yan, Saurabh Agarwal, Shivaram Venkataraman

Language Models

Challenge:	Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference.
Approach:	They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens.
Outcome:	The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models.

Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity (2024.findings-emnlp)

Copied to clipboard

Michael Metel, Peng Lu, Boxing Chen, Mehdi Rezagholizadeh, Ivan Kobyzev

Challenge:	Speculative decoding uses a small draft model to generate a single input token, instead of sequentially generating tokens until completion.
Approach:	They propose a method that generates varying draft models adapted to the input context using simple rules.
Outcome:	The proposed method is competitive with the current SOTA for self-speculative decoding while being a truly plug-and-play method.

Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)

Copied to clipboard

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra

Language Models

Challenge:	Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding.
Approach:	They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model.
Outcome:	The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint.

How Speculative Can Speculative Decoding Be? (2024.lrec-main)

Copied to clipboard

Zhuorui Liu, Chen Zhang, Dawei Song

Semantic Text Processing

Challenge:	Large language models (LLMs) have a largely increased latency due to their ability to autoregressively model . speculative decoding is a technique that trades generation quality for speed .
Approach:	They propose to use a draft model to draft tokens autoregressively and then verify them in parallel.
Outcome:	The proposed model could draft tokens autoregressively and then verify them in parallel . the proposed model trades quality for speed and could fail in verification stage .

UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (2026.acl-long)

Copied to clipboard

Truong Dinh Do, Nguyen-Khang Le, Le-Minh Nguyen

Robustness in NLP

Challenge:	Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality.
Approach:	They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages.
Outcome:	The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments.

An Empirical Study of Speculative Decoding for Small Language Models (2026.eacl-long)

Copied to clipboard

Luca Mainardi, Selcuk Sandikci, Joaquin Vanschoren

Language Models

Challenge:	Existing studies focus on 7B-70B parameters models, leaving a knowledge gap for small language models.
Approach:	They propose a draft-then-verify paradigm that allows for a single forward pass through a model and transfer of all model parameters to the GPU cache.
Outcome:	The proposed method can be used to accelerate small language models with low computational overhead.

SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences (2026.findings-acl)

Copied to clipboard

Jungyoub Cha, Hyunjong Kim, Sungzoon Cho

Challenge:	Speculative decoding performance degrades as input length increases, with significant drops even at moderate lengths.
Approach:	They propose a drop-in enhancement that improves speculative decoding on long sequences without additional training.
Outcome:	The proposed enhancement accelerates speculative decoding by up to 2.84 on 16K-token long document summarization and up to 3.86 on long-form reasoning while preserving the short-input performance of state-of-the-art frameworks.

FastDraft: How to Train Your Draft (2025.findings-acl)

Copied to clipboard

Ofir Zafrir, Igor Margulis, Dorin Shteyman, Shira Guskin, Guy Boudoukh

Language Models

Challenge:	Speculative Decoding relies on the availability of efficient draft models, which are often lacking due to a stringent constraint of vocabulary compatibility.
Approach:	They propose a novel approach for pre-training and aligning a draft model to any large language model by incorporating efficient pre-train and fine-tuning over synthetic datasets generated by the target model.
Outcome:	The proposed model can be trained on a single server with 8 Intel Gaudi 2 accelerators in under 24 hours and achieves 3x acceptance rate, block efficiency and 2x memory bound speedup.