RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for decoding large language models generate one token per step, causing high inference latency. |
| Approach: | They propose a method that integrates retrieved exact patterns with logit-driven future cues. |
| Outcome: | Experiments on Spec-Bench, HumanEval, and MGSM-ZH show that RACER outperforms training-free methods and accelerates inference. |
Similar Papers
Speculative Streaming: Efficient and Scalable Speculative Decoding with Multi-Stream Attention (2025.emnlp-main)
Copied to clipboard
Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Antonie Lin, Mohammad Rastegari, Mahyar Najibi
| Challenge: | Speculative decoding is a prominent technique for accelerating LLM inference by leveraging an auxiliary draft model, but its effectiveness is limited by the autoregressive nature of draft generation. |
| Approach: | They propose a method that integrates speculative draft generation directly within the target model using multi-stream attention. |
| Outcome: | The proposed method improves acceptance but also latency and speculation latency, limiting overall speedup. |
RASD: Retrieval-Augmented Speculative Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating draft tokens rely on lightweight draft models or additional model structures to generate tokens and retrieve context from databases. |
| Approach: | They propose to use a pruning method to enhance model-based speculative decoding by combining the best-fit model with the best retrieval tree. |
| Outcome: | The proposed method achieves state-of-the-art inference acceleration across tasks such as DocQA, Summary, Code, and In-Domain QA. |
LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) is a promising technique for LLM inference acceleration. |
| Approach: | They propose a method to generate draft tokens in a retrieval-based manner to reduce drafting overhead and improve inference speed. |
| Outcome: | Extensive tests show that *LogitSpec* can achieve 2.61 speedup and 3.28 mean accepted tokens per decoding step. |
PLD+: Accelerating LLM Inference by Leveraging Language Model Artifacts (2025.findings-naacl)
Copied to clipboard
| Challenge: | speculative decoding is a novel decoding paradigm for large language models . however, its use is limited by its computational resources and fine-tuning requirements . |
| Approach: | They propose a tuning-free approach that accelerates inference of large language models . they use draft and verify principle to accelerate inference process . |
| Outcome: | The proposed approach outperforms tuning-free approaches on input-guided tasks and outperformed state-of-the-art EAGLE on four of the tasks. |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
SSSD: Simply-Scalable Speculative Decoding (2026.acl-long)
Copied to clipboard
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
| Challenge: | Existing methods for accelerating inference in Large Language Models require additional training and training, resulting in a higher deployment and maintenance cost. |
| Approach: | They propose a training-free method that combines lightweight n-gram matching with hardware-aware speculation. |
| Outcome: | SSSD reduces latency by up to 2.9 and is faster than autoregressive decoding methods. |
MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification (2026.findings-acl)
Copied to clipboard
Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Tianyu Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai
| Challenge: | Autoregressive large language models suffer from high inference latency due to memorybandwidth constraints. |
| Approach: | They propose a method that decouples generation and verification by decoupling tokens and a lightweight draft model. |
| Outcome: | The proposed method delivers consistent and significant speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks. |
SAM Decoding: Speculative Decoding via Suffix Automaton (2025.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) methods are inefficient and rely on single retrieval resources. |
| Approach: | They propose a retrieval-based speculative decoding method that adapts the suffix automaton for efficient draft generation by utilizing the generating text sequence and static text corpus. |
| Outcome: | The proposed method can find the longest suffix match and can be integrated with existing methods to generalize to broader domains. |
DReSD: Dense Retrieval for Speculative Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) uses an efficient draft model to propose the next few tokens, which are verified by the LLM in a single forward call, reducing latency while preserving its outputs. |
| Approach: | They propose a draft model that proposes the next few tokens from a non-parametric datastore and uses a framework that uses approximate nearest neighbour search with contextualised token embeddings to retrieve the most semantically relevant sequences for SD. |
| Outcome: | The proposed framework achieves (on average) 87% higher acceptance rates, 65% longer accepted tokens and 19% faster generation speeds compared to sparse retrieval (REST). |
Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings. |
| Approach: | They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM. |
| Outcome: | The proposed model significantly speeds up inference time and out-of-domain speedup across various languages. |