I explored speculative decoding, a technique that pairs a small draft model with a large target model to accelerate autoregressive LLM inference by generating several candidate tokens in one cheap pass and verifying them in a single expensive pass.
- Metrics to monitor:
- Accepted tokens per verification step (acceptance rate).
- Inter‑token latency.
- Performance under realistic concurrent request loads.
- Results (illustrative):
- When the draft correctly predicts four tokens, the large model performs a single verification pass, reducing work from four passes to one.
- Acceptance rates range from 70‑90 % with high‑quality drafts to below 30 % with poor drafts.
- Reported latency improvements span 2×‑4× versus baseline autoregressive decoding, depending on draft quality and concurrency.
- Open Questions:
- How does draft model size versus quality affect overall throughput under varying request loads?
- What is the optimal draft length for different hardware configurations?
- Can adaptive strategies dynamically adjust draft length based on observed acceptance rates?
- How does speculative decoding interact with beam search or other constrained decoding methods?
- What are the security or alignment implications of introducing a draft model that may bias the token distribution before verification?
- Decisions:
- Do not evaluate solely by "tokens per second" because efficiency depends on acceptance rate and verification latency.
- Methods:
- Draft Generation: run the small model to produce a short sequence (e.g., four tokens).
- Target Evaluation: feed the whole candidate sequence to the large model in one forward pass with causal masking.
- Acceptance Test: compute p_target and p_draft for each token and accept with probability min(1, p_target/p_draft).
- Rejection Handling: if a token is rejected, discard remaining draft tokens and sample a correction token from the residual distribution.
- Iterate using accepted tokens as context.