In late October 2026 I posted two LinkedIn entries that dig into two intertwined themes: how to design evaluation suites for LLMs and why a reactive database is the missing ingredient for proactive AI agents.
I decided against a fixed target count of evals; instead I adopt a "failure‑mode clarity" test: each eval must answer the question "which failure mode does this eval catch?"
I propose building a common contract for events – publishing, subscribing, filtering, and reliable resumption after disconnects – as the foundation for proactive AI behaviours.
- Flight delay → draft re‑booking options.
My tentative blueprint for the protocol layer follows five steps: define event schema, publish API, subscription mechanics, resume semantics, and prototype extensions to MCP.
Evals are tricky to get right, mostly because you can get them wrong in both directions. Too few, and you risk missing important failures. Too many, and you risk over-constraining the model. My mental model is simple - I ask one simple question: which failure mode does this eval catch?
The central thesis was simple: "For AI to be proactive, the database has to be reactive."
- Cloud bill exceeds budget → list resources to shut down.
- Competitor price change → short diff + response suggestion.
- PR touches owned code → summary + review checklist.
- p99 latency alert → identify culprit deploy + fix draft.
- Calendar conflict → propose alternative slot.
- Unanswered support ticket → draft reply with linked docs.