I noted Together AI’s agent-driven inference optimization work as an alternative to hand-tuning for every model, hardware setup, and traffic shape. Its reported performance figures are company claims, and access remains limited.
Together AI announced Together Compiler on 4 November, the first step in its Polymorphic Inference programme.
Agent teams rewrite GPU code in a continuous loop, testing candidates on persistent engines that hot-reload without restarting.
Together reports up to 2.6x higher tokens per second on frontier open models in under 24 hours.
Together reports 1.7x over open-source megakernels on a Llama 1B test.
Together reports 1,111 and 1,205 tokens per second on MiniMax M3.1-Flash and DeepSeek v4.1-Flash at batch size 1.
The results are company-reported; dedicated-endpoint use is in limited beta, with self-serve access still to come.
I asked: If inference keeps retuning itself, what happens to the job of optimising it by hand?