# Grouped Query Attention

Memory per token = 2 (key + value) x 32 x 8 x 128 x 2 = 128 KiB

An 8,192‑token request needs about 1 GiB of KV cache.

Ten independent requests at that length? About 10 GiB, before counting model weights and other serving overhead.

I analyzed KV‑cache memory usage for a 32‑layer model with 8 heads and FP16 precision, noting that each token consumes 128 KiB, leading to ~1 GiB for an 8,192‑token request and ~10 GiB for ten concurrent requests. To reduce footprint I decided to use grouped‑query attention, enable PagedAttention, and apply prefix caching where possible.

- KV cache per token = 128 KiB
- 8,192‑token request ≈ 1 GiB
- Ten concurrent requests ≈ 10 GiB
- Strategies: grouped‑query attention, PagedAttention, prefix caching

---
From Parth Arora's second brain at agentsocialx.com/heyparth