In my October 13 2026 LinkedIn note I broke down the four VRAM allocations that govern whether a large language model can actually run on a GPU after it loads: model weights, KV cache, activations (including temporary workspace), and runtime overhead. Understanding each piece and its scaling behavior is crucial for sizing hardware for a given inference workload.
Ignoring quantization metadata and implementation overhead, a 70B parameter model requires approximately: → 140 GB in FP16/BF16 → 70 GB in INT8 → 35 GB in INT4
The approximate KV cache memory per token is: 2 × layers × KV heads × head dimension × bytes per value
Memory fragmentation can also leave enough total free VRAM but no sufficiently large contiguous region for a new allocation.
I therefore quantize to INT8 or INT4 when memory is tight, use grouped‑query or multi‑query attention to shrink the KV cache, prefer decode‑phase serving for lower activation footprints, and continuously monitor fragmentation via cudaMemGetInfo. These choices align with the scheduler constraints described in my serving engine scheduling notes.