How It Works
Speculative decoding
A way to make a large language model answer faster without changing what it writes. A small, fast “draft” model guesses the next several words, and the large model checks all of those guesses in one pass, keeping the ones it agrees with and correcting the first one it doesn’t. Because checking is cheaper than generating word by word, responses arrive sooner at the same quality.
Origin
Introduced in 2022 by Yaniv Leviathan, Matan Kalman and Yossi Matias at Google in the paper “Fast Inference from Transformers via Speculative Decoding”; a DeepMind team led by Charlie Chen published a closely related method, speculative sampling, in early 2023.