KV‑streams deletes discarded turns directly from the live KV cache and continues from the entries that survived.
In the researchers' vLLM setup, KV entries are stored in 16-token blocks.
On SWE-bench Verified, KV‑streams reached peak performance in about 32 hours.
RoPE handling: cached keys keep their original Rotated‑Position Encoding (RoPE) even after compaction, while logical sequence indices are tracked separately to preserve correct attention.
Eviction policy: the system evicts whole conversation turns rather than individual tokens to maintain coherent message boundaries.
In October 2026 I posted a deep dive on KV‑streams, a compaction method that modifies the live KV cache in‑place instead of rebuilding the prompt, and argued for its adoption in agentic RL training.
- KV‑streams delete discarded turns directly from the KV cache, preserving RoPE‑rotated positions and re‑linking 16‑token blocks to keep the attention kernel dense.
These results motivated me to adopt KV‑streams for my RL pipelines, as they cut training time roughly in half and keep KV consistency between generation and training, which is crucial for low‑bias policy updates.
For more background on KV‑cache management, see my september 2026 x posts part 2 series.
- Whole‑turn eviction is required; evicting individual tokens leads to degenerate text.
- Training‑inference alignment logs each compaction event and replays it during training, yielding near‑zero KL mismatch (≈ 0.001).
- Performance: KV‑streams reach peak performance on SWE‑bench Verified in ~32 hours, whereas re‑prefill needs ~60 hours.
- Recall test shows 100 % recall of a removed assignment when sufficient cache remains.