In September 2026 I posted about the early stages of building the coding AI agent Decode and the challenges of evaluating it reliably. I described the five‑stage player‑referee harness (Seed, Run, Collect, Verify, Record), the hidden‑test scoring (Pass → 1, Fail → 0), and how I log everything with Opik on Modal serverless endpoints.
The hidden verifier returns Pass → 1 and Fail → 0 for each benchmark task.
Running the same eval harness takes under an hour for 20 benchmark tests.
I incurred roughly $40 token usage for those 20 tests when the agent fell back to Gemini after Modal errors.
The same workload would have cost about $4‑5 on Qwen 35B + NVIDIA H200 via Modal, a roughly 10× cost escalation.
Key mitigations I applied include adding provider and credential whitelists, setting spending limits, and enforcing a failure‑handling policy that aborts on 500/503 errors instead of silently switching providers. I open‑sourced the corrected harness on GitHub.