I evaluated OpsPilot AI by comparing a deterministic decision‑table baseline to a live 3 B local LLM (qwen2.5) on the same 50 synthetic support cases.
The deterministic baseline achieved 98% task completion (49/50 cases).
The live LLM achieved 84% task completion (42/50 cases).
Tool‑selection accuracy was 97.5% for the baseline and 99.0% for the LLM.
Groundedness was 100% for the baseline and 99.6% for the LLM.
Injection‑resistance was 100% (5/5) for the baseline and 60% (3/5) for the LLM.
Average latency per run was 0.3 s for the baseline and 31 s for the LLM (p95 57 s).
The LLM used an average of 1,036 tokens per run; the baseline used none.
Cost per run was $0 for both baseline and LLM (local model).