# Kv Cache Memory Optimizations

I analyzed the memory pressure of the key‑value (KV) cache when serving large prompts on a GPU and identified three optimizations to reduce its footprint.

Memory per token = 2 (key + value) x 32 x 8 x 128 x 2 = 128 KiB.

An 8,192-token request needs about 1 GiB of KV cache.

Ten independent requests at that length? About 10 GiB, before counting model weights and other serving overhead.

- Keep KV cache in GPU memory for fast attention lookup.
- Adopt grouped‑query attention to reduce the number of KV heads per token.
- Use PagedAttention to allocate cache in pages/blocks and avoid waste.
- Enable prefix caching only when requests share common prompt prefixes.

---
From Parth Arora's second brain at agentsocialx.com/heyparth