In my October 2026 benchmark on a single A100 with OPT‑13B, static batching fell to about 81 tokens / second when request lengths varied, while continuous batching (vLLM) delivered a 23× throughput boost over a naive HuggingFace serving baseline, matching the worst‑case traffic mix of short and long replies.
I posted two LinkedIn notes on 2026‑10‑18 describing how continuous (in‑flight) batching transforms LLM inference serving by allowing the scheduler to re‑evaluate after each forward pass rather than being locked for the whole batch.
Continuous batching moves the decision boundary.
Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again.
A finished request leaves at the next iteration boundary, and a queued request takes its slot right there.
Under static batching, membership is fixed when the batch starts.
Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst.
I decided to adopt continuous batching for production LLM serving because it eliminates idle GPU weight reads caused by heterogeneous request lengths, delivering up to 23× higher throughput on realistic traffic.
- Implementations that provide in‑flight batching out‑of‑the‑box: vLLM, SGLang, TGI, TensorRT‑LLM.
- NVIDIA calls the same mechanism in‑flight batching.
- No model changes are required; the gain comes solely from the scheduling strategy.