AI Agents in Production: Guardrails and Evals

The demo took two weeks and everyone was delighted. Then it met real users, and the question nobody could answer was the only one that mattered: is it getting better or worse?

Without evaluation you cannot answer that, which means every change after launch is a guess. This is what separates AI features that survive from the ones quietly switched off after a quarter.

Guardrails and evals are not the same thing

	Guardrails	Evals
When they run	Every request, in production	Before you ship a change
What they do	Stop a bad output reaching a user	Tell you whether quality moved
Failure mode they prevent	One harmful answer	A silent regression across thousands
Cost of skipping	An incident	Months of changes you cannot justify

You need both, and teams usually build the first and skip the second, because guardrails feel urgent and evals feel like homework.

Guardrails that earn their place

Constrain the output shape

If the answer must be JSON matching a schema, enforce the schema rather than asking politely in the prompt. Anything that fails validation is retried once and then fails loudly. A malformed response that silently becomes a null in your database is worse than an error.

Put permissions in the tool layer, not the prompt

This is the one that matters most and the one most often got wrong. "Do not delete anything" in a system prompt is a suggestion to a probabilistic system. If the agent must not delete, the credential it holds must not have delete rights. Prompts express intent; infrastructure enforces it, and only one of the two is auditable.

Gate the irreversible

Anything that moves money, contacts a customer or destroys data goes behind a human approval, at least until the evaluation record justifies otherwise. "At least until" is doing real work in that sentence: the gate is not permanent, it is the thing you remove once you have evidence.

Bound the loop

Cap tool calls per turn and total spend per conversation. An agent in a retry loop is the most common way to turn a small bug into a large invoice, and it happens at 3am on a weekend.

Make every decision replayable

Log the inputs, the retrieved context, the tool calls and the output, for every run. When someone asks why the agent did something three weeks ago, this is the difference between an answer and a shrug. It is also the raw material for your evaluation set.

Evals, which is the part people skip

An eval is a fixed set of inputs with known good outputs, plus a way to score what the system actually produced. That is the whole idea, and the reason it feels like homework is that building the set is unglamorous work with no visible output on the day you do it.

Start with twenty cases, not two hundred

Twenty real examples beat two hundred invented ones, and you can assemble twenty in an afternoon from your own logs. Include the obvious cases, the ones that embarrassed you last month, and three that should be refused outright. That last group catches more regressions than the rest combined.

Score what you can score cheaply

Exact match for anything structured. String or numeric checks for extracted fields. For open text, a second model scoring against a short rubric works well enough to catch regressions, which is the point: you are not grading an essay, you are detecting movement.

Run it on every change

Prompt edits, model upgrades, retrieval tuning, a new tool. All of them. The entire value of an eval set is that it runs automatically and tells you a change made things worse before a user does. An eval suite you run manually when you remember is an eval suite you do not have.

Grow it from production

Every incident becomes a case. Every complaint becomes a case. After six months the set encodes what your organisation has actually learned about the task, which is an asset nobody can copy from you.

The order we build in

1. Logging first. Before guardrails, before evals. You cannot improve what you cannot see, and you cannot build an eval set without examples.
2. Permissions in the tool layer. The agent gets credentials scoped to exactly what it may do.
3. Twenty case eval set. From real logs, with the refusals included.
4. Human approval on anything irreversible. Removed later on evidence, not on confidence.
5. Loop and spend limits. Cheap, and they prevent the expensive surprise.

Notice that none of this is about the model. Changing models is a one line change when this scaffolding exists and a leap of faith when it does not, which is exactly why teams with evals upgrade models casually and teams without them stay on an old one for a year.

What this costs

Two to four weeks on top of the agent itself, and it is the difference between a feature you can keep changing and one you are afraid to touch. Our breakdown of what an AI agent costs puts this work in context, and it is the line item most often cut and most often regretted.

Frequently asked questions

Where to start

Open your logs and pull the twenty most recent real interactions. Mark each one good or bad. You now have an eval set, and probably a surprise about the current quality level.

If the logs do not exist yet, that is the first thing to fix, and we can tell you what to capture so the eval set builds itself while you keep shipping.