The evaluation model

Benchmarks like SWE-bench measure average model capability across a standard task set. They're useful for comparing models against each other, but they can't tell you whether an agent uses your technology correctly. That requires your domain knowledge. You know the right SDK version and the authentication pattern the product team recommends. A benchmark doesn't have that information, and can't check for it.

AX evaluation answers a more useful question. Given your technology, does the agent use it the way it should? That question is only worth asking if you can trust the answer, and an evaluation can lie to you convincingly. We've seen evaluations report perfect scores for code that never compiled, because nobody actually built it. We've seen a "used Platform X" check pass whether or not the agent used Platform X, because it matched the word and never checked the meaning. A result that looks confident and turns out to be wrong is worse than no result, because you'll act on it.

The model in this chapter is the set of commitments that make a result trustworthy. Whatever system you run it in, a serious AX evaluation holds to the same discipline.

Every evaluation is built from four things. Scenarios are the prompts developers actually type into coding agents, run against a workspace: blank or pre-populated. Criteria are the checks that decide whether the agent made the right choices. Profiles are the environments you run in: operating system, harness, model and its configuration, and whatever extensions are loaded. Gates are the mechanical checks that confirm the generated code actually works. Each of these has a standard to meet before you can trust what it tells you.

Criteria judge meaning, and judge it the same way twice

Criteria are where your domain expertise becomes measurable. You know the correct SDK version and the partition key strategy the product team would sign off on. A criterion captures one of those judgments so an LLM judge can rule on it: pass, fail, or skip.

Two things make a criterion trustworthy. The first is that it checks meaning. "The response mentions Platform X" passes whether the agent embraced Platform X or argued against it, so matching words hands you a confident wrong answer. A real criterion reads the generated artifacts in context and judges what the agent actually did. The second is that it reaches the same verdict every time. If the same output scores pass on one run and fail on the next, the criterion is ambiguous, and the problem is in your wording. Chapter 5 is entirely about writing good criteria.

To further understand the agent's strengths and weaknesses, you can group criteria into areas, say whether dependencies are current or how idiomatic the code is. It's helpful, but you're not required to do it. Each criterion just needs to test something that matters to you and rule on it consistently.

Run in the environment your developers use

A result only means something if it comes from the environment your developers actually work in. The operating system, harness, model, and extensions in context all change how the agent behaves, and the harness matters more than people expect. Two surfaces from the same vendor can run different harnesses, so a strong result in one doesn't carry over to another, even on the same model. Running a CLI on Linux because it's easy to automate and cheap to run, when your developers work in an IDE on Windows, gives you a precise number about a situation none of them are in. You'll fix problems they don't have and miss the ones they do. Pick the profile that represents most of your audience and make that your reference point. Chapter 6 goes deeper on choosing profiles, and Chapter 7 on reproducing them faithfully.

Prove the code runs, don't assume it

Gates are mechanical, binary checks, and they stack, so each one runs only if the previous one passed. A prerequisite gate comes first. Did the agent even pick the right technology? If you asked for Cosmos DB and it built on MongoDB, nothing else is worth measuring. After that, the gates follow the code's own lifecycle: does it build, do its tests pass (if any), does the app run, and does it deploy when deployment is part of the job. Deploy comes last because there's no sense deploying something you haven't confirmed runs.

Skip these and you get the most common lie in agent evaluation: a perfect criteria score for a project that never compiled. Gates are exactly what stops that from reaching your readout.

Compare against a baseline, and count the cost

A score on its own tells you very little. What you want to know is whether your extension made things better or worse, so every result is measured against a bare baseline: the same scenario and profile, with no extensions. Where the extension helps, you have lift, and where it hurts, drag.

Keep the score easy to understand. Take the pass rate with your extension, subtract the pass rate of the baseline, and you have an answer a product owner can read and act on without a data science degree. If you grouped criteria into areas, do this per area and compare across them.

Be mindful of two traps. The first is treating usage as success. The agent calling your extension doesn't mean it helped. If the numbers don't move, the extension is cost without benefit. The second is ignoring what the task costs to run. Most developers aren't on unlimited plans. A profile that lifts quality at three times the price forces a real decision, and you can't have that conversation if you only ever looked at the score. Measure cost alongside quality, every time.

Run it more than once

Language models are non-deterministic. Send the same prompt five times and you can get five different results. A single run tells you almost nothing, because it might be the outlier. Run each scenario and profile combination several times, at least five, so you can tell a real pattern from noise. A criterion that fails once in five is variance. But a criterion that fails five times in five is a problem you can fix. While five runs isn’t statistically significant, in our experience, it allows you to strike balance between evaluations not being overly expensive and starting to see results group and can discern obvious failures.

Keep the evidence

A score tells you what happened. It can't however tell you why. For that you need the full record of the run: every tool the agent called, every file it read, every decision it made along the way. Keep that trajectory for every run, along with the generated output and the verdicts. Without it, you can see that a criterion failed but you're guessing at the cause, and guessing is how you end up fixing the wrong thing. Chapters 8 and 9 are about reading this evidence.

What this gives you

Hold to these commitments and you get a result you can defend: a clear picture of what the agent does with your technology, an honest comparison of whether your extensions help or hurt and at what cost, and specific failures that point to specific fixes. You can point past a headline number like "the agent scored 72%" to the cause behind it: the agent installs the wrong SDK in 4 out of 5 runs because a docs page hardcodes an outdated version.

Further reading

• What AI benchmarks are not telling you. Why leaderboard scores don't transfer to your technology, and how to start a small, representative eval of your own.
• How to measure AI agent extension effectiveness. Why you can't judge an extension by looking at it, and what measuring lift, drag, and cost actually involves.