Papers by Zhuorui Liu
How Speculative Can Speculative Decoding Be? (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a largely increased latency due to their ability to autoregressively model . speculative decoding is a technique that trades generation quality for speed . |
| Approach: | They propose to use a draft model to draft tokens autoregressively and then verify them in parallel. |
| Outcome: | The proposed model could draft tokens autoregressively and then verify them in parallel . the proposed model trades quality for speed and could fail in verification stage . |