| Challenge: | Large foundational speech and language models require more memory and computational resources to generate long sequences. |
| Approach: | They propose to optimize speculative sampling for parallel hardware accelerators by combining multiple GPU threads to reduce profiling time. |
| Outcome: | The proposed approach improves profiling time from 6% to 13% without compromising accuracy. |
Similar Papers
BASS: Batched Attention-optimized Speculative Sampling (2024.findings-acl)
Copied to clipboard
Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, Anoop Deoras
| Challenge: | Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. |
| Approach: | They propose a batched speculative decoding system that generates sequences at an average speed of 5.8ms per token and a batch size of 8 at a 2.15 speed-up over optimized regular decoding. |
| Outcome: | The proposed system achieves state-of-the-art latency and speed-up over optimized regular decoding. |
Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are demanding more memory and computational resources . however, these devices typically feature weaker GPUs and stronger CPUs . |
| Approach: | They propose a lossless inference acceleration method that leverages the characteristics of heterogeneous devices and the advantages of speculative decoding. |
| Outcome: | The proposed method achieves speedups ranging from 1.79 to 10.1 across different devices . it uses a draft model on the GPU to perform preliminary predictions, while a target model on CPU validates these outputs . |
Speculative Decoding for Multi-Sample Inference (2025.findings-emnlp)
Copied to clipboard
Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Ji Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Speculative decoding method exploits consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Approach: | They propose a speculative decoding method that exploits the consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Outcome: | The proposed method exploits the intrinsic consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or databases. |
Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show draft-then-verify paradigm can achieve around 5x speedup for the popular Transformer architectures with comparable generation quality to beam search decoding. |
| Approach: | They propose to use Spec-Drafter and Spec Verification to accelerate autoregressive (AR) decoding by combining a model optimized for efficient and accurate drafting and a reliable method for verifying the drafted tokens efficiently. |
| Outcome: | The proposed method achieves 5x speedup on seq2seq tasks with comparable generation quality to beam search decoding, refreshing the impression that draft-then-verify paradigm introduces only 1.4x2x speed up. |
Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (2025.naacl-long)
Copied to clipboard
Jacob K Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto
| Challenge: | Existing methods to accelerate large language model inference are limited by the reliance on incremental token generation in existing draft models. |
| Approach: | They propose an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences and allows parallelization of both the drafting and verification steps. |
| Outcome: | The proposed approach provides 7.2x speedups over standard generation processes and 1.75x speed ups over existing speculative decoding approaches. |
Speculative Streaming: Efficient and Scalable Speculative Decoding with Multi-Stream Attention (2025.emnlp-main)
Copied to clipboard
Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Antonie Lin, Mohammad Rastegari, Mahyar Najibi
| Challenge: | Speculative decoding is a prominent technique for accelerating LLM inference by leveraging an auxiliary draft model, but its effectiveness is limited by the autoregressive nature of draft generation. |
| Approach: | They propose a method that integrates speculative draft generation directly within the target model using multi-stream attention. |
| Outcome: | The proposed method improves acceptance but also latency and speculation latency, limiting overall speedup. |
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a key technique for enhancing the inference speed of Large Language Models. |
| Approach: | They propose a method that adds padding tokens to ensure that the number of new tokens remains consistent across samples. |
| Outcome: | The proposed method can handle the issue of inconsistent prediction tokens without adding padding tokens. |
Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism (2026.acl-long)
Copied to clipboard
| Challenge: | Parallel Speculative Decoding (PSD) has limitations due to speedup limits and high computational waste . a novel synchronous mechanism solves the Retrieval Precision-Efficiency Dilemma . |
| Approach: | They propose a framework that combines a draft-verification-based approach with a synchronous mechanism to solve the Retrieval Precision-Efficiency Dilemma. |
| Outcome: | The proposed framework breaks speedup limits for Speculative Decoding by overlapping draft generation with verification. |
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference (2025.findings-emnlp)
Copied to clipboard
Hao Mark Chen, Wayne Luk, Yiu Ka Fai Cedric, Rui Li, Konstantin Mishchenko, Stylianos Venieris, Hongxiang Fan
| Challenge: | Auto-regressive decoding of Large Language Models results in significant overheads in hardware performance . a novel parallel prompt decoding approach is proposed to overcome these limitations . |
| Approach: | They propose a parallel prompt decoding that uses a single model for speculation and verification. |
| Outcome: | The proposed approach speeds up auto-regressive decoding of large language models 2.49 times . it can be used on mobileLlama to Vicuna-13B on a wide range of benchmarks . |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |