Your AI agent doesn't need to be perfect.
It needs to be good enough for the job, cheap enough to run, and predictable enough to trust.
Alexey built a RAG system for @DataTalksClub that manages FAQs for 100,000+ learners.
But his generation eval suite passes only 42 out of 61 cases.
That's roughly 69%.
His automation classifies FAQ submissions into four categories: NEW UPDATE DUPLICATE WRONG_COURSE
When he discovers a failure, he adds it to the eval dataset before fixing it.
Good evals help you understand which failures are worth fixing.
Track classification accuracy, placement accuracy, and cost in Opik by @Cometml.
Keep eval cases in Git as the source of truth.
Chasing a 100% eval pass rate can lead to a giant system prompt full of edge-case instructions.