Writing evaluation criteria: AX Practitioner Playbook | Developer Agent Experience (DevAX)

Writing criteria is where your domain expertise becomes measurable. This is the hardest part of the process. Get it right and your eval produces actionable signal. Get it wrong and you'll get scores that look confident, feel consistent, and mean nothing.

What criteria are

Criteria are the rules your evaluation system uses to judge whether the agent made the right choices: the right SDK version, authentication approach, or product pattern. An LLM judge reads the generated artifacts and decides whether each criterion passes, fails, or should be skipped. Build and run behavior belongs in mechanical gates instead.

ASIDE: What about deterministic judges? There are pros and cons to using deterministic judges or LLM judges when doing coding agent evals. Deterministic judges can run faster and return more predictable results, but they are often more brittle and prone to evaluation errors caused by incorrect pattern matching. LLM judges, meanwhile, are slower and require careful configuration to produce repeatable, usable results, but they are better at evaluating the code in context and avoiding simple pattern match errors. A mix is often useful, but it is important to be aware of the limits and advantages of both.

What good criteria look like

Good criteria test things that matter to the product and its users, and express each check precisely enough that the judge reaches the same verdict each time. "The code uses Cosmos DB" works as a prerequisite gate, but it doesn't tell you whether the agent used Cosmos DB well.

The judge is an LLM too, with its own knowledge gaps. "Uses the current SDK" forces it to decide what current means. Name the package, accepted version range, APIs, and specific wrong patterns instead.

For Cosmos DB, meaningful criteria might check:

Does it use hierarchical partition keys where the scenario calls for multi-tenant isolation? (Specific: "Uses PartitionKeyBuilder or equivalent hierarchical partition key API. Fails: uses a single flat partition key like /customerId")
Does it use identity-based authentication? (Specific: "Passes: code uses DefaultAzureCredential, ManagedIdentityCredential, or equivalent. Fails: connection string contains AccountKey=")

For an SPFx upgrade, name the exact target version and enumerate the required files and properties, such as dependency versions in package.json and the generator version in .yo-rc.json. Put each required change in a separate criterion so you can see exactly what the agent missed.

Writing criteria that work

Each criterion needs four things:

What you're checking. One specific aspect of the output.
What passes. The concrete conditions that indicate correct usage. Name packages, versions, APIs.
What fails. The concrete conditions that indicate incorrect usage. Name the specific wrong patterns.
When to skip. When the criterion doesn't apply to the output.

Criteria rarely land right on the first attempt. Here's how one evolves:

Draft 1: "Uses current SDK version." Current means nothing to the judge, so it has to supply its own version.
Draft 2: "Uses Azure Cosmos DB SDK v4 for JavaScript (@azure/cosmos)." Better, but still ambiguous. Does any v4 release count? What about a prerelease?
Draft 3: "Uses @azure/cosmos v4.x. Passes: package.json specifies a stable version from 4.0.0 up to, but not including, 5.0.0. Fails: it specifies an earlier major version, a prerelease version, or the deprecated documentdb package. Skipped: the application doesn't use Cosmos DB." Now the judge doesn't need to define current or decide whether a prerelease counts.

You can also use a model to help with this. Ask it whether your criteria are ambiguous and how to clarify them. Models are good at catching ambiguity, and it'll save you time running and re-running evaluations to discover the issue.

Calibrate before you lock

A criterion can read clearly and still produce the wrong verdict or score inconsistently. Before using a criteria set for a production evaluation, run a calibration cycle:

Test each criterion against an artifact you know should pass and one you know should fail.
Score the same artifacts repeatedly to see whether the judge reaches the same verdict.
Have someone who knows the product confirm technical accuracy.
Confirm that the wording produces consistent verdicts, using the repeated scores from step 2.
Revise and repeat until the criteria are both accurate and consistent.

Calibration runs are for improving the criteria, not measuring AX. Keep them separate from production results. Once the criteria set is locked, don't rewrite it in response to a surprising score. Investigate the output and trajectory first. The result might be telling you something real.

Maintaining criteria

Once a criteria set is locked, maintenance keeps the definition of success accurate without moving the goalposts after you've seen the results. Products release new versions, recommended patterns change, and scenarios evolve. Each can change what a valid result looks like.

Version the criteria set

When a product or SDK release changes correct usage, or when the scenario or starting workspace changes materially, review the criteria. If the changes affect what passes, fails, or skips, create a new version of the criteria set. Keep previous results tied to the version that produced them.

Put every revised criteria set through the same calibration process before locking the new version. Calibration is part of writing the revision; versioning and preserving its relationship to previous results are the maintenance work around it.

Results produced by different criteria-set versions don't form a clean trend. You can compare them for investigation, but don't present the difference as AX lift or regression without accounting for the changed yardstick.

Keep criteria for older product versions while those versions remain supported. A scenario targeting an older supported release should be evaluated against the guidance for that release, not today's latest recommendation. Retire the criteria when the product version is no longer supported.

If you discover a defect in a locked criterion after a production evaluation, document it with the affected results and correct it in a new criteria-set version. Keep the original results with a clear warning rather than silently changing their meaning.

Watch the system around the criteria

The judge can evolve too. A changed judge model or prompt shouldn't silently change verdicts for a locked criteria set. The evaluation system needs to preserve backward compatibility, validate the change against existing evaluations, or restore affected results. If your evaluation system can't make that guarantee, treat a judge change as a new measurement boundary and don't compare its scores directly with earlier results.

Maintaining criteria takes two kinds of ownership, whether they sit with one person or two teams. Someone needs to own whether the criterion describes correct product usage. Someone needs to own whether the wording can be judged consistently. Drop either and the criteria set stops being trustworthy.

Common traps

Letting an LLM write your criteria. Hand the model your scenarios to generate the criteria and it fails three ways: its knowledge cutoff bakes stale patterns into what you're measuring, it can't know what you actually care about, and what it writes is either biased (when it also judges) or too vague for a different judge to score consistently. Instead, draft them yourself, then use a model to tighten the wording and catch ambiguity.

Testing for style instead of correctness. "Code is well-structured and follows best practices" is not an AX criterion. Focus on whether the agent used your technology correctly.

Testing for things the agent can't control. If the harness decides which tools to load, don't write a criterion that penalizes the agent for not loading a tool.

Conflating multiple concerns in one criterion. "Uses correct SDK with proper authentication and appropriate error handling" is three criteria pretending to be one. Split them so you get clear signal on each.

Using vague language the judge can't verify. "Correct," "proper," "current," "appropriate," "best practices." If you wouldn't accept these words from a junior dev in a code review, don't use them in criteria.

Your turn

Write 3 criteria for your scenario. Before you move on, ask yourself:

Would a perfect implementation pass?
Would a clearly broken implementation fail?
Would the judge produce the same verdict if you ran it five times on the same output?

If you're not confident about that last one, your criteria are ambiguous. Tighten the language until there's nothing left to interpret. The quality checklist in Appendix A has the full self-review process.

Then calibrate the criteria against known artifacts, get accuracy and consistency confirmed, and lock the criteria-set version before running the production evaluation.