Raising the floor: what actually works when evaluating production agents

Key Takeaways

• Ceiling versus floor is the framing that matters. The ceiling is the most impressive emergent thing your agent can do. The floor is the worst — recommending a competitor, deleting a pile of data, sending an AI-slop email to a customer because it technically had access to your mail. The floor is what breaks user trust, and raising it is a different job from maxing a benchmark.
• Keep evals as code, in your test runner. The prompt is no longer a string you tune in a playground; it is the whole harness — tools, skills, hooks, middleware, everything you wire up. So evals should look like tests: unit tests and end-to-end tests, run locally against your agent. Sentry ships a vitest-evals package that is essentially Vitest with syntactic sugar; OpenAI calls the same idea macro evals. Hylak notes he sees very few companies still using hosted prompt-playground tooling.
• Big offline eval sets break the moment anything moves. An eval suite that asserts the agent calls a particular tool for a particular question can lose most of its value when you switch models or switch harnesses. Be careful about sinking months into a suite that then slows down your model upgrades — in practice almost nobody actually delays shipping a new model for two weeks to rewrite evals. The goal is safety, not theater.
• Every issue needs two facts: when it started, and what percent of users it affects. This sounds obvious and is the thing teams skip. "Is this new?" is what converts a vague user report into a tractable investigation — if an issue started three days ago your mind immediately goes to what changed, which model, which downstream dependency. And three affected users versus a hundred thousand is the difference between ignoring it and stopping everything.
• Clusters are not issues. The naive approach is to take all the traces and cluster them. That is useful for one-off error analysis, but it does not scale: cluster boundaries drift so you can't reliably track one over time, and what counts as "the same issue" is specific to each product. A cluster like "pricing issues" merges a wrong quoted price and a wrong refund calculation, which can have entirely unrelated root causes. The telemetry analogy holds — nobody finds production bugs by clustering their logs.
• Code mode scales to trace analysis. The pattern people apply to MCP works on traces too: write classifiers as code and run them in a sandbox at production scale, rather than passing every trace through a model.
• Agents are bad at anomaly detection — so don't ask them to find anomalies. Ask them to investigate anomalies you have already found. Pull out as many deterministic signals as you can first, such as keyword frequency; a spike doesn't prove there's an issue, but it is a tractable thing to hand an agent and say "go look at this."
• Your user volume decides the method. At tens of millions of messages a day, experiments become extremely valuable — you can run them on a small sample of your free tier. At five or ten users, A/B tests are meaningless. But a five-user internal enterprise app serving critical information can still be very high stakes, so low volume is not the same as low risk.
• Consequence of error depends on who your users are. If a tab-complete or a coding-agent suggestion is wrong you delete it, and much of the responsibility for catching it sits with a technical user. Products where the agent replaces rather than augments the expert — the AI-doctor shape — carry a fundamentally different responsibility, and should be designed accordingly.