I observed that a single LLM forward pass is composed of many tiny GPU kernels—attention, matmul, normalization, sampling, data movement, and more—rather than one monolithic GPU operation.
Without CUDA Graphs, the CPU prepares and submits these operations individually, adding per‑kernel launch overhead.
Some kernels finish in only a few microseconds, allowing the GPU to sit idle while the CPU queues the next launch.
CUDA Graphs change how this work reaches the GPU by capturing the repeatable sequence of operations—including dependencies—once and replaying it through a single graph launch.
If a live batch size does not match any captured graph, the engine falls back to eager execution or pads the batch to a captured size.
Most of the gain comes from removing CPU launch work between kernels.
Engines such as vLLM and TensorRT-LLM therefore capture graphs for several common execution cases and batch sizes.
Later iterations replay the captured sequence through one graph launch instead of submitting every operation separately.
I decided to use CUDA Graphs for the decoding phase where many short kernels are launched repeatedly, because the launch overhead dominates latency. For the prefill phase I keep eager execution, since longer kernels hide the launch cost. I capture multiple graphs for batch sizes 1, 2, 4, and 8 to balance coverage against GPU memory usage, accepting that workloads outside these sizes will fall back to eager execution.