Your LLM fits on the GPU and then 10 people start using it, and you run out of memory. The missing piece is the KV cache. Start from first principles: when a model generates the next token, that token needs to attend to earlier tokens. Recalculating the earlier tokens' keys and values every time would waste compute, so the server keeps them in memory. The model weights stay roughly fixed. The KV cache grows with the number of tokens being processed across active requests. Take a hypothetical model with: 32 layers 8 KV heads 128 dimensions per head FP16 cache (2 bytes per value) Memory per token = 2 (key + value) x 32 x 8 x 128 x 2 = 128 KiB An 8,192-token request needs about 1 GiB of KV cache. Ten independent requests at that length? About 10 GiB, before counting model weights and other serving overhead. Now the optimizations make more sense: - Grouped-query attention reduces the number of KV heads, so each token needs less cache. - PagedAttention allocates cache in blocks to reduce wasted memory. It doesn't make the tokens disappear. - Prefix caching reuses blocks when requests share the same prefix. Different prompts won't get the same benefit.