UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware

Challenge:	Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality.
Approach:	They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages.
Outcome:	The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments.

Similar Papers

Overview

Challenge

Approach

Outcome

Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)

Copied to clipboard

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra

Language Models

Challenge:	Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding.
Approach:	They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model.
Outcome:	The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint.

Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)

Copied to clipboard

Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun

Multilinguality

Challenge:	Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings.
Approach:	They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM.
Outcome:	The proposed model significantly speeds up inference time and out-of-domain speedup across various languages.

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs (2026.acl-long)

Copied to clipboard

Yicheng Ji, Jun Zhang, Jinpeng Chen, Cong Wang, Lidan Shou, Gang Chen, Huan Li

Multimodality

Challenge:	Existing methods for video understanding suffer from autoregressive generation of tokens.
Approach:	They propose a training-free loosely SD framework for Video-LLMs that uses visual-relevant tokens to accurately pinpoint the latter.
Outcome:	The proposed framework boosts the accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.

PLD+: Accelerating LLM Inference by Leveraging Language Model Artifacts (2025.findings-naacl)

Copied to clipboard

Shwetha Somasundaram, Anirudh Phukan, Apoorv Saxena

Language Models

Challenge:	speculative decoding is a novel decoding paradigm for large language models . however, its use is limited by its computational resources and fine-tuning requirements .
Approach:	They propose a tuning-free approach that accelerates inference of large language models . they use draft and verify principle to accelerate inference process .
Outcome:	The proposed approach outperforms tuning-free approaches on input-guided tasks and outperformed state-of-the-art EAGLE on four of the tasks.

Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference (2025.findings-emnlp)

Copied to clipboard

Hao Mark Chen, Wayne Luk, Yiu Ka Fai Cedric, Rui Li, Konstantin Mishchenko, Stylianos Venieris, Hongxiang Fan

Responsible & Trustworthy NLP

Challenge:	Auto-regressive decoding of Large Language Models results in significant overheads in hardware performance . a novel parallel prompt decoding approach is proposed to overcome these limitations .
Approach:	They propose a parallel prompt decoding that uses a single model for speculation and verification.
Outcome:	The proposed approach speeds up auto-regressive decoding of large language models 2.49 times . it can be used on mobileLlama to Vicuna-13B on a wide range of benchmarks .

DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding (2026.findings-acl)

Copied to clipboard

Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, Jun Wang

Language Models

Challenge:	Autoregressive (AR) decoding in large language models is latency-bounded by strictly sequential token generation.
Approach:	They propose a diffusion-based drafter that proposes multi-token candidates and then verifies them in parallel by the target model.
Outcome:	The proposed drafter generates multi-token proposals in a single forward pass while remaining compatible with standard AR verifiers.

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)

Copied to clipboard

Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui

Language Models

Challenge:	Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding.
Approach:	They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding .
Outcome:	The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models.

A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models (2025.acl-long)

Copied to clipboard

Jiesong Liu, Brian Park, Xipeng Shen

Language Models

Challenge:	Large Language Models (LLMs) are highly memory-intensive when performing real-time inference.
Approach:	They propose a technique that allows for speculative decoding to be run on the fly to maximize the efficiency of LLM inferences.
Outcome:	The proposed solution can lead to 3.55-16.48% speed improvement over the standard speculative decoding, and 1.2-3.4 over the default LLMs.

FastDraft: How to Train Your Draft (2025.findings-acl)

Copied to clipboard

Ofir Zafrir, Igor Margulis, Dorin Shteyman, Shira Guskin, Guy Boudoukh

Language Models

Challenge:	Speculative Decoding relies on the availability of efficient draft models, which are often lacking due to a stringent constraint of vocabulary compatibility.
Approach:	They propose a novel approach for pre-training and aligning a draft model to any large language model by incorporating efficient pre-train and fine-tuning over synthetic datasets generated by the target model.
Outcome:	The proposed model can be trained on a single server with 8 Intel Gaudi 2 accelerators in under 24 hours and achieves 3x acceptance rate, block efficiency and 2x memory bound speedup.

MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification (2026.findings-acl)

Copied to clipboard

Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Tianyu Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai

Challenge:	Autoregressive large language models suffer from high inference latency due to memorybandwidth constraints.
Approach:	They propose a method that decouples generation and verification by decoupling tokens and a lightweight draft model.
Outcome:	The proposed method delivers consistent and significant speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks.