Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding

Challenge:	Existing methods for drafting and verifying tokens require significant fine-tuning or have inconsistent performance across tasks.
Approach:	They propose a lossless drafting approach that organizes various token sources into multiple databases in a hierarchical framework based on temporal locality.
Outcome:	The proposed method outperforms existing database drafting methods on Spec-Bench using 7B and 13B parameters.

Similar Papers

Overview

Challenge

Approach

Outcome

Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)

Copied to clipboard

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra

Language Models

Challenge:	Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding.
Approach:	They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model.
Outcome:	The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint.

LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification (2026.acl-long)

Copied to clipboard

Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, Bo An

Green & Sustainable NLP

Challenge:	Large Language Models (LLMs) can process extremely long contexts, requiring efficient inference over extended inputs.
Approach:	They propose a model that uses a constant-sized key-value cache to train long-context models.
Outcome:	Experimental results show that LongSpec achieves 3.26x speedup over strong Flash Attention baselines and 2.34x wall clock time on four math reasoning tasks.

Hierarchical Speculative Decoding with Dynamic Window (2025.findings-naacl)

Copied to clipboard

Shensian Syu, Hung-yi Lee

Challenge:	Speculative decoding (SD) uses an efficient draft model to generate multiple tokens . previous methods depend on simple heuristics to select K or dynamically adjust the window size .
Approach:	They propose a framework that allows a draft model to generate multiple tokens . they propose HSDDW, which allows the draft model autonomously decide when to stop generating tokens.
Outcome:	The proposed framework outperforms existing state-of-the-art methods on four datasets.

UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (2026.acl-long)

Copied to clipboard

Truong Dinh Do, Nguyen-Khang Le, Le-Minh Nguyen

Robustness in NLP

Challenge:	Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality.
Approach:	They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages.
Outcome:	The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments.

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)

Copied to clipboard

Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui

Language Models

Challenge:	Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding.
Approach:	They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding .
Outcome:	The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models.

An Empirical Study of Speculative Decoding for Small Language Models (2026.eacl-long)

Copied to clipboard

Luca Mainardi, Selcuk Sandikci, Joaquin Vanschoren

Language Models

Challenge:	Existing studies focus on 7B-70B parameters models, leaving a knowledge gap for small language models.
Approach:	They propose a draft-then-verify paradigm that allows for a single forward pass through a model and transfer of all model parameters to the GPU cache.
Outcome:	The proposed method can be used to accelerate small language models with low computational overhead.

Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)

Copied to clipboard

Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun

Multilinguality

Challenge:	Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings.
Approach:	They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM.
Outcome:	The proposed model significantly speeds up inference time and out-of-domain speedup across various languages.

Graph-Structured Speculative Decoding (2024.findings-acl)

Copied to clipboard

Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan

Green & Sustainable NLP

Challenge:	Speculative decoding is a promising technique to accelerate the inference of Large Language Models.
Approach:	They propose a method that uses a token graph to record multiple sequence hypotheses within a single draft stage.
Outcome:	The proposed method significantly accelerates the inference of Large Language Models (LLMs) it allows the LLM to choose from and select the longest sequence that meets its standards.

Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling (2025.acl-long)

Copied to clipboard

Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu

Semantic Text Processing

Challenge:	Large Language Models (LLMs) generate only one token at each decoding step, leading to high latency.
Approach:	They propose a speculative decoding paradigm that stores tokens in an adjacency matrix and employs a breadth-first-search algorithm to construct a draft tree.
Outcome:	The proposed method outperforms existing train-free methods by 30% and even a training method by 25%.

Decoding Speculative Decoding (2025.naacl-long)

Copied to clipboard

Minghao Yan, Saurabh Agarwal, Shivaram Venkataraman

Language Models

Challenge:	Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference.
Approach:	They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens.
Outcome:	The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models.