Every production AI agent must answer three essential questions – can it work on benchmarks, does it retain behavior after changes, and does it meet expectations in production.
On Decode's 19‑task benchmark, a 35B model scored 95 % while a larger 120B model scored just 53 %.
Opik stores the traces, datasets, and experiments, giving versioned data for regression testing and online evaluation.
The key is separating the agent doing the work from the system evaluating it, preventing self‑grading loopholes.
I am building an evaluation harness for Decode to make sure production AI agents reliably answer three fundamental questions.
I include both small and large models in benchmarks because I set benchmark_strategy.include_small_and_large_models = true.
I automate regression tests from any observed production failure, so regression_testing.automation_from_production_failures = true.
I adopt Opik by Comet as the tracing platform, making tracing_platform.opik_by_comet_used = true.
Benchmarks show a 35B model achieved 95 % success on the 19‑task benchmark, while a larger 120B model achieved only 53 %.
Production failures are turned into automated regression tests, and I capture execution traces with Opik to feed back into benchmark and regression design.