A release gate for AI agents: what we test before every update

Traditional software has a simple safety net: if the tests pass, you ship. AI agents don’t get that for free. The same code can behave differently tomorrow because the model provider shipped an update, someone tweaked a prompt, a tool returns a new field, or customers started asking a new kind of question.

Most teams handle this with a handful of manual checks before a release. Someone chats with the agent for twenty minutes, it looks fine, it ships. That is how a hallucinated refund policy or a successful prompt injection reaches a customer.

The fix is a release gate: an automated suite of tests that every update to the model, prompt, tools or data has to pass before it goes live. Here is what we put in one, and how to make its verdict mean something.

What changes, and why it matters

An agent is a model plus instructions plus tools plus data. Any of the four can change behaviour:

• The model. Hosted models are updated, deprecated or swapped for a cheaper version. Even a minor version can shift tone, refusal behaviour or tool-calling accuracy.
• The prompt. A one-line fix for one complaint often breaks something unrelated.
• The tools. A changed API response, a new permission or a slower endpoint changes what the agent does next.
• The data. New documents in the knowledge base, new products, new policies, new kinds of customer.

A release gate treats every one of these as a release, not just code changes.

What goes in the suite

We build the suite from five kinds of test. Each answers a different question.

1. Scenario tests: does it do the job?

Real tasks with known good outcomes, ideally taken from logs or from the people who do the work today. For a claims assistant: “customer asks about excess on a windscreen claim”, “customer uploads a photo of the wrong car”. Aim for coverage of the common cases and the expensive ones. A few hundred is a good start; the right number depends on how varied the work is.

2. Adversarial tests: can it be pushed off course?

Prompt injection, jailbreaks and social engineering, including the indirect kind: instructions hidden inside a document, an email or a web page the agent reads. If your agent reads anything a stranger can write, this is the test that matters most. We keep a library of attack patterns and adapt them to each agent’s tools and data.

3. Policy and tone tests: does it stay inside the lines?

Things it must never say (promises about refunds, legal or medical advice, competitors), things it must always do (disclose it’s an AI, offer a human), and the brand voice. These catch policy drift: the slow slide that happens after a few prompt edits and a model update.

4. Tool-use tests: does it act safely?

Did it call the right tool with the right arguments? Did it stop when it should have asked? Could it be talked into an action outside its permissions? For agents that move money, change records or send messages, these are the tests that prevent real damage.

5. Regression tests: does it still pass what it failed before?

Every failure found in production, in an audit or in a complaint becomes a permanent test. Over time this becomes the most valuable part of the suite, because it encodes exactly how your agent breaks.

How to score it without fooling yourself

Use the right checker for each test. Exact or structured checks wherever possible (the right tool was called, the number matches). An LLM judge for open-ended answers is fine, but treat the judge as a model too: label a sample by hand and measure how often the judge agrees with people before you trust its scores.

Set thresholds from business risk, not round numbers. Ask what one failure costs. A wrong answer about opening hours is cheap; a wrong answer about cover is not. Typical gates look like: task success at least 92%, groundedness at least 95%, zero critical prompt-injection failures, policy drift under 1%, and cost and latency inside budget.

Read “zero failures” honestly. Passing 380 out of 380 injection tests does not mean the failure rate is zero. A handy rule (the “rule of three”) says that with zero failures in n independent tests, the true failure rate is below about 3/n with 95% confidence. So 0 of 380 means “probably under 0.8%”. If your risk tolerance is 0.1%, you need around 3,000 tests, not 380. The same logic applies to every pass rate: report it with an interval, and the gate makes decisions on the interval, not the point estimate.

Compare against the current version, not just the threshold. An update that passes the gate but is clearly worse than what’s live on one important slice should still be questioned.

What the gate decides

A gate produces one of three outcomes:

• Pass. Ship, with monitoring on.
• Block. At least one critical check failed. The report lists every failure with examples, ranked by risk, so the fix is obvious.
• Hold for review. Nothing critical, but a meaningful drop on a slice. A person decides.

The point is that the decision is made by evidence the whole team agreed on beforehand, not by whoever happens to chat with the agent before the deadline.

When the agent isn’t sure

The strongest agents also know when not to answer. Calibrated confidence and conformal prediction let an agent answer when it’s sure and hand the case to a person when it isn’t, with an error rate you choose. We wrote about how that works, including for closed models behind an API, in Accuracy is not enough. A good gate tests that behaviour too: does the agent abstain on the cases it should?

After release

The gate doesn’t end at launch. Run a slice of it on live traffic every day, alert on drift in the metrics, and turn every escalation or complaint into a new regression test. Under the EU AI Act, high-risk systems must show appropriate accuracy, robustness and cybersecurity throughout their lifecycle (Article 15). A release gate with its history of reports is the most practical evidence there is.

A checklist to start this week

1. Write down the five worst things your agent could do, and what each would cost.
2. Collect 100 real tasks with known good outcomes.
3. Add 50 adversarial prompts, including at least 10 hidden inside documents or emails the agent reads.
4. Add policy checks for the five worst things from step 1.
5. Pick a threshold for each metric from what a failure costs, and write it down.
6. Run the suite on your current version to get a baseline.
7. Make the suite run automatically on every change to the model, prompt, tools or data. Block on critical failures.
8. Every new failure becomes a test.

That is a working gate in about a week. It will catch more in its first month than any amount of manual checking.

If you’d like a second pair of eyes, this is exactly what our AI System Audit does: we build the suite for your agent, run it, and hand you the gate.