MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification (2026.findings-acl)
Copied to clipboard
Jingwei Song, Xinyu Wang, Hanbin Wang, Xiaoxuan Lei, Tianyu Shi, Shixin Han, Eric Yang, Xiao-Wen Chang, Lynn Ai
| Challenge: | Autoregressive large language models suffer from high inference latency due to memorybandwidth constraints. |
| Approach: | They propose a method that decouples generation and verification by decoupling tokens and a lightweight draft model. |
| Outcome: | The proposed method delivers consistent and significant speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks. |
Similar Papers
Speculative Verification: Exploiting Information Gain for Speculative Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used for many applications but their size and computational cost make inference serving a significant challenge. |
| Approach: | They propose an efficient augmentation to Speculative Decoding (SD) that predicts speculation accuracy and dynamically adapts the verification length to maximize throughput. |
| Outcome: | The proposed model reduces wasted verification on rejected tokens and improves decoding efficiency. |
Multi-Drafter Speculative Decoding with Alignment Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to accelerate large language model (LLM) inference use a smaller model to draft future tokens, which are then verified by the target LLM. |
| Approach: | They propose a unified framework that integrates multiple drafters into the SD process. |
| Outcome: | Extensive experiments show that MetaSD outperforms single-drafter approaches. |
Draft
& Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for accelerating Large Language Models have been criticized for their inference costs and inefficient decoding. |
| Approach: | They propose a self-speculative decoding approach for accelerating Large Language Models without an auxiliary model. |
| Outcome: | The proposed method achieves a speedup of up to 1.99 with no additional neural network training and no extra memory footprint. |
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to accelerate autoregressive generation of large language models require training costs. |
| Approach: | They propose a training-free alignment-augmented speculative decoding algorithm . it leverages the output distribution obtained in the prefilling phase to provide more aligned draft candidates . |
| Outcome: | The proposed method increases the average generation score by 3.3 points for the LLaMA3 model. |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Speculative decoding (SD) is a powerful and efficient way to accelerate autoregressive generation. |
| Approach: | They propose a training-free framework that recovers valid tokens discarded by standard verification . they use online correction memory and Semantic Consistency Gating to analyze rejections . |
| Outcome: | The proposed framework outperforms existing methods and achieves peak throughput speedup of 2.33x. |
From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) allows a lightweight draft model to propose outputs that a stronger target model verifies. |
| Approach: | They propose a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. |
| Outcome: | Experiments show that SpecGuard outperforms both SD and reward-guided SD in accuracy and reliability tests. |
UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality. |
| Approach: | They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages. |
| Outcome: | The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments. |
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models. |
| Approach: | They propose a self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. |
| Outcome: | The proposed framework suppresses spurious confidence and bounds speculation length based on token-wise decoding difficulty. |
Hierarchical Speculative Decoding with Dynamic Window (2025.findings-naacl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) uses an efficient draft model to generate multiple tokens . previous methods depend on simple heuristics to select K or dynamically adjust the window size . |
| Approach: | They propose a framework that allows a draft model to generate multiple tokens . they propose HSDDW, which allows the draft model autonomously decide when to stop generating tokens. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods on four datasets. |