I explored speculative decoding, a technique that adds a small, fast draft model to accelerate generation of a large autoregressive LLM. The draft model predicts a short sequence, then the large model verifies the whole draft in one forward pass, accepting tokens based on a probability ratio.
A smaller, faster model drafts four possible tokens.
For a proposed token x, the acceptance probability is: min(1, p_target(x) / p_draft(x))
I wouldn't evaluate this with a single "tokens per second" number. Instead I measure acceptance rate, inter‑token latency, concurrency performance.
If a token is rejected, the remaining draft is discarded and a correction is sampled from the residual probability distribution.