In my September 2026 X posts I outlined how to scale voice‑based LLM inference on a single GPU for many concurrent callers.
I decided to front‑load first tokens of new callers.
I deployed separate machines for reading and writing.
I used sticky sessions or fingerprint routing to preserve KV cache.
I pre‑fetch data because lookups exceed 500 ms.
In late September 2026 I shared a series of X updates announcing new OpenAI products, model performance claims, and the goals of our Voice AI “Research on Tap” workshop. I also reached out to GPU‑stack providers and highlighted a partnership with Convex for Build Sprint Season 04.
Pro 500 delivers 8X the speed at 300 tokens/second.
Sol 6.1 is 80% cheaper on input tokens.
Sol 6.1 is 90% cheaper on cache input.
Sol 6.1 is 50% cheaper on output.
OpenAI launches Dots.
TTS latency is 200–300 ms.
The workshop set a TTFA goal of 500 ms and a cost ceiling of ≤ ₹2 per minute, using sound‑based turn‑end detection (80‑100 ms) and stable‑word streaming to cut redundant GPU work to 14 %.