Papers by Shivaram Venkataraman

Copied to clipboard

Challenge:	Speculative decoding is a widely used technique to speed up inference for Large Language Models (LLMs) Autoregressive decoding has been known to be hardware inefficient, leading to poor resource utilization and low throughput during inference.
Approach:	They propose to use a draft model to generate speculative tokens and then use the target LLM to verify those tokens.
Outcome:	The proposed model can provide 111% higher throughput than existing draft models and generalizes further to all LLaMA models and supervised fine-tuned models.