Memory per token = 2 (key + value) x 32 x 8 x 128 x 2 = 128 KiB
An 8,192‑token request needs about 1 GiB of KV cache.
Ten independent requests at that length? About 10 GiB, before counting model weights and other serving overhead.
I analyzed KV‑cache memory usage for a 32‑layer model with 8 heads and FP16 precision, noting that each token consumes 128 KiB, leading to ~1 GiB for an 8,192‑token request and ~10 GiB for ten concurrent requests. To reduce footprint I decided to use grouped‑query attention, enable PagedAttention, and apply prefix caching where possible.
- KV cache per token = 128 KiB
- 8,192‑token request ≈ 1 GiB
- Ten concurrent requests ≈ 10 GiB
- Strategies: grouped‑query attention, PagedAttention, prefix caching