TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs (2026.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) is a useful tool for accelerating large language models . but its utility is limited by a fundamental constraint: draft and target models must share the same vocabulary . |
| Approach: | They propose an algorithm that uses a draft token sequence to get a new target token sequence and then uses DTW to build a mapping to transfer probability distributions. |
| Outcome: | The proposed method shows 1.57x speedup on various tasks. |
Similar Papers
EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Speculative decoding is a key technique for enhancing the inference speed of Large Language Models. |
| Approach: | They propose a method that adds padding tokens to ensure that the number of new tokens remains consistent across samples. |
| Outcome: | The proposed method can handle the issue of inconsistent prediction tokens without adding padding tokens. |
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to accelerate autoregressive generation of large language models require training costs. |
| Approach: | They propose a training-free alignment-augmented speculative decoding algorithm . it leverages the output distribution obtained in the prefilling phase to provide more aligned draft candidates . |
| Outcome: | The proposed method increases the average generation score by 3.3 points for the LLaMA3 model. |
Hierarchical Speculative Decoding with Dynamic Window (2025.findings-naacl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) uses an efficient draft model to generate multiple tokens . previous methods depend on simple heuristics to select K or dynamically adjust the window size . |
| Approach: | They propose a framework that allows a draft model to generate multiple tokens . they propose HSDDW, which allows the draft model autonomously decide when to stop generating tokens. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods on four datasets. |
Multi-Drafter Speculative Decoding with Alignment Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to accelerate large language model (LLM) inference use a smaller model to draft future tokens, which are then verified by the target LLM. |
| Approach: | They propose a unified framework that integrates multiple drafters into the SD process. |
| Outcome: | Extensive experiments show that MetaSD outperforms single-drafter approaches. |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings. |
| Approach: | They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM. |
| Outcome: | The proposed model significantly speeds up inference time and out-of-domain speedup across various languages. |
SPIDE: Serial and Parallel Intertwined Speculative Decoding (2026.findings-acl)
Copied to clipboard
Wenru Xu, Peixuan Xu, Ziqi Yang, Ming Hu, Zihui Wang, Jianzhong Qi, Rongshan Yu, Xiaoliang Fan, Cheng Wang
| Challenge: | Speculative decoding (SD) is a training-free SD framework that orchestrates dynamic alternation combining serial dynamic drafting with parallel draft verification. |
| Approach: | They propose a serial and parallel intertwined speculative DEcoding framework that orchestrates dynamic alternation combining serial dynamic drafting and parallel draft verification. |
| Outcome: | The proposed framework accelerates inference while reducing the LLM usage costs. |
Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on test-time alignment, where additional compute is allocated during inference to enhance LLM safety and reasoning capabilities. |
| Approach: | They propose a reward-shifted speculative sampling algorithm that aligns a draft model with human preferences while the target model remains unchanged. |
| Outcome: | The proposed algorithm achieves superior gold reward scores at a significantly reduced inference cost in test-time weak-to-strong alignment experiments. |
Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate only one token at each decoding step, leading to high latency. |
| Approach: | They propose a speculative decoding paradigm that stores tokens in an adjacency matrix and employs a breadth-first-search algorithm to construct a draft tree. |
| Outcome: | The proposed method outperforms existing train-free methods by 30% and even a training method by 25%. |
Speculative Streaming: Efficient and Scalable Speculative Decoding with Multi-Stream Attention (2025.emnlp-main)
Copied to clipboard
Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Antonie Lin, Mohammad Rastegari, Mahyar Najibi
| Challenge: | Speculative decoding is a prominent technique for accelerating LLM inference by leveraging an auxiliary draft model, but its effectiveness is limited by the autoregressive nature of draft generation. |
| Approach: | They propose a method that integrates speculative draft generation directly within the target model using multi-stream attention. |
| Outcome: | The proposed method improves acceptance but also latency and speculation latency, limiting overall speedup. |