Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs (2024.lrec-main)
Copied to clipboard
Chenxi Sun, Hongzhi Zhang, Zijia Lin, Jingyuan Zhang, Fuzheng Zhang, Zhongyuan Wang, Bin Chen, Chengru Song, Di Zhang, Kun Gai, Deyi Xiong
| Challenge: | Large language models have demonstrated exceptional capability in natural language understanding and generation, but their generation speed is limited by the inherently sequential nature of their decoding process. |
| Approach: | They propose a method that accelerates decoding process without sacrificing quality . they propose lexical unit decoding, which can be integrated with other methods . |
| Outcome: | The proposed method significantly reduces decoding time while maintaining quality while maintaining output quality. |
Similar Papers
Segmenting Natural Language Sentences via Lexical Unit Analysis (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on sequence segmentation models suffer from invalid predictions and a lack of consistency. |
| Approach: | They propose a unified span-based model that embeds every span and computes a score for each segmentation candidate. |
| Outcome: | The proposed model achieves state-of-the-art on 6 of the 3 tasks tested. |
Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models generate tokens sequentially and are memory-bound, resulting in a memory-based inference stage that is memory-limited. |
| Approach: | They propose an approach to accelerate the inference speed of large language models with billions of parameters by integrating semi-autoregressive inference and speculative decoding capabilities. |
| Outcome: | The proposed approach has demonstrated inference speedups of 2.7x-4.0x on humanEval-X while maintaining output quality. |
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference (2025.findings-emnlp)
Copied to clipboard
Hao Mark Chen, Wayne Luk, Yiu Ka Fai Cedric, Rui Li, Konstantin Mishchenko, Stylianos Venieris, Hongxiang Fan
| Challenge: | Auto-regressive decoding of Large Language Models results in significant overheads in hardware performance . a novel parallel prompt decoding approach is proposed to overcome these limitations . |
| Approach: | They propose a parallel prompt decoding that uses a single model for speculation and verification. |
| Outcome: | The proposed approach speeds up auto-regressive decoding of large language models 2.49 times . it can be used on mobileLlama to Vicuna-13B on a wide range of benchmarks . |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
DecoCal: Decoding with Calibration in Diffusion Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Diffusion Large Language Models (DLLMs) generate text via iterative token denoising . but decoding is challenging, with many tokens appearing predictable early . |
| Approach: | They propose a Decoding framework that performs Calibration of token-level confidence across diffusion steps and leverages the calibrated results to guide decoding decisions. |
| Outcome: | Experiments on multiple DLLMs and benchmarks show that DecoCal improves generation accuracy compared to existing strategies. |
PLD+: Accelerating LLM Inference by Leveraging Language Model Artifacts (2025.findings-naacl)
Copied to clipboard
| Challenge: | speculative decoding is a novel decoding paradigm for large language models . however, its use is limited by its computational resources and fine-tuning requirements . |
| Approach: | They propose a tuning-free approach that accelerates inference of large language models . they use draft and verify principle to accelerate inference process . |
| Outcome: | The proposed approach outperforms tuning-free approaches on input-guided tasks and outperformed state-of-the-art EAGLE on four of the tasks. |
Exploring the Hidden Capacity of LLMs for One-Step Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Approach: | They show that large language models can reconstruct surprisingly long texts via autoregressive generation from just one trained input embedding. |
| Outcome: | The proposed model can generate hundreds of accurate tokens in one token-parallel forward pass, when provided with only two learned embeddings. |
A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
SSSD: Simply-Scalable Speculative Decoding (2026.acl-long)
Copied to clipboard
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
| Challenge: | Existing methods for accelerating inference in Large Language Models require additional training and training, resulting in a higher deployment and maintenance cost. |
| Approach: | They propose a training-free method that combines lightweight n-gram matching with hardware-aware speculation. |
| Outcome: | SSSD reduces latency by up to 2.9 and is faster than autoregressive decoding methods. |
UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality. |
| Approach: | They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages. |
| Outcome: | The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments. |