I analyzed the memory pressure of the key‑value (KV) cache when serving large prompts on a GPU and identified three optimizations to reduce its footprint.
Memory per token = 2 (key + value) x 32 x 8 x 128 x 2 = 128 KiB.
An 8,192-token request needs about 1 GiB of KV cache.
Ten independent requests at that length? About 10 GiB, before counting model weights and other serving overhead.
- Keep KV cache in GPU memory for fast attention lookup.
- Adopt grouped‑query attention to reduce the number of KV heads per token.
- Use PagedAttention to allocate cache in pages/blocks and avoid waste.
- Enable prefix caching only when requests share common prompt prefixes.