AI Agent Testing: 10 Scenarios, CI Checks and Replay

AI agent testing checks three things: the code, the model's judgement, and a human view of the risky calls. A normal test can't catch a plan that goes wrong halfway through a task. So start small. Build 10 golden scenarios from real work and wire one deterministic check into your CI pipeline. Everything fancier can wait.

Why AI agent testing is different from normal testing

A normal test asks one question. Does output A match expected output B? That stops working the moment your system is an agent.

An agent doesn't just answer. It makes a plan, picks tools, calls them in some order, reads the results and decides what to do next. Any one of those steps can go wrong on its own.

Non-determinism is the first problem. Ask the same agent the same question twice and you can get two different answers that are both right. There's no single string to match, so an old assertEquals test is useless.

The path is the second problem. Picture a support agent that refunds the right customer, but only after checking the wrong account first. By the old rules it passed. By the rules that matter it failed. You have to check the whole path, not just where it landed. NVIDIA's guide calls this evaluating full trajectories: the plan, the tool calls and the result at each step.

The failure modes are the third problem. They're stranger than a normal bug. Watch for these:

• Invented tool parameters. The agent makes up a field that doesn't exist and calls the tool wrong.
• Loops. It retries the same failed step, burning tokens and time.
• Ignored results. A tool call fires, but the agent never reads what came back.
• Scope creep. It does something helpful that nobody asked for.
• Early mistakes that spread. One wrong assumption at step two corrupts every step after it.

None of these show up in a simple input and output check. They only show up when you watch the whole run.

Amazon learnt this at scale. Its teams have built thousands of agents since 2025, and its write-up on evaluating them tests at three levels: the final output, each part of the agent, and the model underneath. It also found that badly defined tool schemas led agents to call the wrong APIs, which added latency and cost.

The simplest way to think about it: treat the agent like an enthusiastic intern. You wouldn't just read their final report. You'd watch how they got there, have someone senior judge the tricky calls, and check anything that touches money twice.

This is a different job from scoring a model. You're testing a worker doing a task inside your systems. Treat it that way from day one. Plenty of "bugs" turn out to be the agent doing exactly what it was told, just not what you meant.

Define what "good" looks like for your agent

Decide what "pass" means before you write a single test. Skip this and you'll build tests nobody trusts.

Start with task success rate. It's your headline number. For each scenario, did the agent finish the job? Not "did it sound sure of itself". Did the customer get the refund? Did the ticket reach the right queue?

Track it per scenario, not as one blended average. A high average can hide the one task type that fails every time. NVIDIA makes the same point: agent evaluation puts task success per scenario ahead of model accuracy.

Then add the numbers that explain why it moved:

• Path efficiency. How many steps did the agent take against the fewest needed? Four steps instead of twelve costs less and breaks less.
• Tool-call accuracy. Right tool, right parameters, right order. This is where invented parameters get caught.
• Latency. How long the whole run took, end to end.
• Cost per task. Tokens and API calls for one finished job.

Path efficiency and tool-call accuracy both come from logging the full run, so set that up early.

Pro tip: Set your target pass rate before you run a single test, not after you see the results.

Here's the distinction most teams miss. Capability evals and regression evals are not the same test with different data. Anthropic's engineering team splits them like this:

Eval type	What it asks	Expected pass rate
Capability eval	What can this agent do well?	Starts low, and that's fine
Regression eval	Does it still handle everything it used to?	Near 100%

A capability eval that always passes isn't testing anything hard. A regression eval that fails now and then means your live agent isn't reliable. Once a capability eval passes reliably, it can graduate into the regression suite.

Set these numbers first. Your CI gates, release thresholds and human review triggers all point back to them.

Three-layer testing strategy for AI agents

Anthropic's guide describes agent evals as a mix of three kinds of grader: code-based, model-based and human. Each has its own job.

Layer 1: code checks

These are the cheap, boring checks. Schema validation: did the tool call have the right fields? Order checks: did step three happen before step four? Unit tests on the logic. Run them on every commit. They're the smoke alarm, not the fire brigade.

Layer 2: model-based graders

Some output has no single right answer. Tone, say, or whether an explanation makes sense to a customer. For those, a second model grades the first. It isn't perfect, and it has to be checked against human grades. But it's the only way to score thousands of transcripts without an army of reviewers.

Layer 3: human review

Keep people for the calls that matter most: safety, money, legal exposure, a client relationship. People are slow and expensive, so spend them where a wrong call costs real money. How much human time to budget is its own design question, and this guide to human-in-the-loop workflows covers it.

The rule for combining them is simple. Push as much as you can down to layer one. Use layer two for the volume layer one can't judge. Save layer three for the cases where being wrong is expensive.

Anthropic adds one habit worth copying: read the transcripts. You won't know if your graders work until you read the runs and the grades side by side. When a task fails, the transcript tells you whether the agent made a real mistake or the grader rejected a valid answer.

Pro tip: Track how often your human reviewers agree with each other, not just how often they agree with the model grader.

Teams now automate parts of layer two as well. Microsoft Research's Agent-Pex pulls behaviour rules out of an agent's prompts and traces, checks thousands of traces against them, and turns the rules into adversarial tests. That's far more coverage than anyone could write by hand.

Running several agents together changes the maths again. Read how multi-agent systems compare with single agents before you commit to an architecture.

Designing scenario-based tests and golden datasets

A golden dataset is your answer key. Get it wrong and every test built on it lies to you.

Use real scenarios, not made-up ones. Your own production traces are the best source. Pull the last 200 real sessions and look for patterns. The requests that took five extra steps. The ones where a customer had to rephrase twice. The ones that failed quietly with no error.

Databricks makes the same call from the production side: custom benchmarks built on your own workload consistently beat generic test sets. They hold your real documents, your real tools and your real failures.

Give each golden example three parts:

• The input. What the user said or did.
• The expected path. Which tools should fire, in roughly what order.
• The pass criteria. What success means for this case.

Then break things on purpose. A suite built only from happy paths tells you nothing about stress.

Add these:

• Prompts written to push the agent off task
• Tool failures, where a tool times out or returns rubbish
• Unclear requests, where the right move depends on context the agent doesn't have

Pro tip: Seed your golden set from support tickets and complaints, not just successful sessions. The cases people complained about are the ones most likely to break again after your next update.

Aim for about 10 scenarios to start. That covers your main paths and your worst edge cases without making test upkeep a full-time job. Add more as production throws up new failures. Don't write hundreds of near-copies that test the same thing.

Tracing, logging and replay for repeatable runs

You can't fix what you can't see. Good logs turn a mystery failure into a two-minute fix.

Log every step, not just the final answer:

• The plan the agent made before acting
• Every tool call, with full arguments
• Every tool response, exactly as it came back
• Tokens used per step
• Time taken per step

That list looks basic. Most teams skip half of it, then wonder why a bug takes three days to reproduce. Our guide to AI agent observability goes deeper on what to capture.

Cassette replay is what makes this cheap. Record one real run, with its tool calls, responses and timing, and save it as a fixture. Now you can replay that run in CI as often as you like. No live API calls. No token bill. Open-source tools like agentverify do exactly this: record the model calls once, commit the recording to git, and replay it in CI at zero cost. It removes the flakiness you get when a live model returns slightly different text on every run.

A trace viewer turns logs into something you can read. Build your own or use a standard like OpenTelemetry. The goal is the same: click into one failed run and see step by step where it went wrong. Was it the plan, or a tool that returned rubbish? Without a viewer you're reading raw JSON at midnight and guessing.

Tooling patterns for agent evaluation

You don't need to build all of this from scratch. Here's how the pieces fit.

• Test runners fake a user and mock the tools your agent calls. They slot into CI like any other test suite, with no live API calls and no surprise costs.
• Eval platforms sit one level up. They score whole transcripts and work out the numbers from earlier: task success, path efficiency, cost per task. Run them nightly or in staging, where real model calls are affordable.
• Monitoring covers you once the agent is live. Feed production failures back into your golden set and the loop closes.

Open source or managed is a real trade-off. Open-source tools cost nothing up front and you can read every line, but you maintain them. Managed platforms cost money, save engineering time and usually ship better dashboards.

A rough rule. Use open-source runners in CI, where speed and control matter. Consider a managed platform once you run dozens of scenarios a night and the dashboards become a job in themselves. Keep monitoring in whatever stack your team already trusts. Agents don't need a separate philosophy.

If you build with Claude Code, the first layer is less work than it sounds. The code checks are an ordinary test suite, and Claude Code can write them and run them. The broader evals can run on a schedule as a Claude Code routine. What Claude Code can't supply is the pass mark. That comes from the person whose decisions the agent is copying.

For where testing sits inside wider quality control, see this breakdown of AI QA frameworks. And if you want the traditional baseline these methods grew out of, DevWiz has a plain guide to what software testing is and why it matters.

Integrating tests into CI/CD and release gates

Testing that only happens before a big launch arrives too late. Build it into every change instead. Anthropic describes automated evals in CI/CD as the first line of defence, running on every agent change and every model upgrade. Without them, debugging is reactive: wait for complaints, reproduce by hand, fix it, and hope nothing else broke.

1. Run code checks on every commit. Schema validation, tool order, unit logic. They take seconds, cost nothing and catch the most common breakages before anyone reviews the pull request.
2. Run the model-graded evals nightly or on staging. These cost more in tokens and time, so don't let them block every commit. Run the full scenario suite once a day or on each staging deploy. Treat a falling trend line as seriously as a failed test.
3. Gate releases on the regression suite. Databricks describes deployment gates where a new version must clear threshold checks before rollout. When a check fails, raise a ticket automatically with the failing trace attached, so someone owns it.

Pro tip: Tag each failure with the layer that caught it: code check, model grader or human. Six months in, those tags show where your suite is weak and where people are reviewing things a code check should have caught.

Cost, sampling and scaling: keeping evaluation affordable

Every eval run costs tokens. Run enough of them and finance will start asking about your test suite.

Measure cost per task first. Track tokens and API calls per finished scenario, like cost per transaction anywhere else in the business. Anthropic notes that once evals exist, you can track latency, token use, cost per task and error rates on a fixed bank of tasks for free.

Set a token budget per scenario type. NVIDIA recommends tracking tokens, tool calls and latency per successful task against explicit budgets. Treat a blown budget as a failure, the same as a timeout.

Test the unstable scenarios more often. Stable scenarios that always pass don't need ten runs a day. Spend the budget on the tasks where the agent sometimes nails it and sometimes doesn't.

• Run stable, reliable scenarios weekly, not daily
• Run unstable scenarios on every commit
• Rebalance the split every month as pass rates shift

Use replay and synthetic data wherever you can. A replayed test costs nothing in API calls. A synthetic scenario generated once and reused across ten CI runs costs a fraction of ten live calls. Keep paid model calls for the nightly runs and for live monitoring, where real data matters.

Ten tests to run before you let users near the agent

Run through this before launch. Code checks first, then model graders, then a human.

1. Schema validation on every tool call. Expect near 100%. Anything less means fix the code, not the prompt.
2. Tool-order test. Expect near 100%. A failure here usually means a planning bug.
3. Ten golden happy-path scenarios. Expect 90% or better. Below that, don't ship.
4. Five adversarial prompts. Lower pass rates are fine here. This is a capability eval, not a regression one.
5. Tool failure test. Confirm the agent handles a timeout or a bad response without looping.
6. Model-graded review of open-ended answers. Send anything under your quality bar to a human.
7. Latency check per scenario. A run far over target usually means an inefficient path.
8. Cost-per-task check. Anything over budget gets flagged before launch, not after the invoice.
9. Human review of every safety-sensitive scenario. No shortcuts, however well the model grader scored it.
10. Full regression suite against last week's golden set. Expect near 100%. If it drops, a change broke something you didn't test directly.

Every failed check gets one action: fix the code, change the prompt, or escalate to a human. Never just rerun the test and hope.

My take: set the pass mark by the stakes

My LinkedIn runs on Claude routines now. It accepts requests, posts, comments and answers my DMs. I haven't looked at it in days.

It didn't start there. It started at about 70% right. Working alongside it, I got it to about 95%. Then I stopped, on purpose. Nobody needs the last 5% on LinkedIn.

A process that touches money or a client is a different world. There I want 99%.

That's why you set the pass rate before you test. The number comes from what a wrong answer costs, not from what the agent happens to hit.

I give clients the same rule as a count. Five out of ten right is a starting score, not a failure. Work on the five it got wrong. When it gets nine out of ten, it earns the automation. That count is a golden set, whether you call it one or not.

Then there are the failures nobody writes a test for. One of our automations told about a hundred people we never got their webinar scorecard. Every one of them had sent it. Some got the message twice. "This person has already submitted" is exactly the kind of case a golden set should hold.

The quiet failures are the worst. A member of my mastermind found a daily automation that wasn't written down anywhere, and a backup that had been failing silently for weeks. I'd done the same thing. I once rebuilt something three months later because I'd forgotten I owned it. Now a sweep checks my stack and pings me when anything fails. Most of it runs on plain cron jobs, because a check that needs no judgement needs no model.

James Killick

Build the testing into the agent, not after it

Most teams don't have three months to work out evaluation from scratch while they ship features.

The founders we work with have a sharper version of the problem. The agents that matter most are the ones making the calls the founder used to make. Those need golden scenarios built from the founder's own past decisions, because that judgement is the IP.

Our 90-day program runs on Explore, Map, Transform. It turns the founder's decisions into AI employees built with Claude Code, delivered as a working prototype your team can run without the founder in every loop. Testing those agents against the founder's real calls is part of doing that properly.

If terms like "trajectory" or "grader" are new, the AI orchestration glossary is a good place to check your footing. When you're ready to look at your own agents, book an assessment.

Sources

• Evaluating AI agents: Real-world lessons from building agentic systems at Amazon
• Mastering agentic techniques: AI agent evaluation (NVIDIA)
• Agent-Pex: Automated evaluation and testing of AI agents (Microsoft Research)
• Demystifying evals for AI agents (Anthropic)
• What is agent evaluation? (Databricks)
• agentverify: deterministic testing for AI agents (GitHub)