Running evals and generating the readout

This is the mechanical part. The hard thinking is behind you. Now you execute and let the data come back.

Choosing an evaluation system

The methodology in this playbook doesn't depend on a specific evaluation system. You can use one your team built, an off-the-shelf evaluation product, or Scope, the open source evaluation system we built in DevRel at Microsoft. What matters is whether the system can run a representative evaluation and produce evidence you can trust.

That comes down to a handful of capabilities. The system needs to reproduce the environment your developers actually use, because a convenient setup like a CLI-based agent on Linux-based cloud runners tells you little when most of your audience works in an IDE on Windows. It needs to reset every run to the same starting workspace and run each scenario-profile combination repeatedly.

It needs deterministic checks for what's mechanically verifiable, such as build gates or structured checks against package.json, and an LLM judge with explicit pass, fail, and skip conditions for decisions that depend on meaning. Text matching can't tell "used Platform X" from "didn't use Platform X." Finally, it needs to keep the evidence behind every verdict (artifacts, execution results, costs, and full trajectories) even after results are aggregated. The full list is in Appendix A under "Evaluation system readiness."

If your current system covers these, keep using it. If the gaps are small, extending it might be the most pragmatic choice. If reproducing representative environments or producing trustworthy verdicts would take substantial custom work, adopting an existing system might cost less than building the missing pieces yourself.

Where Scope fits

This playbook isn't just a method you're left to build tooling for. Scope implements it: you configure scenarios, criteria, and agent profiles, run them repeatedly, and get a readout backed by the evidence from every run. It's open source, so you can use it as-is, extend it, or use it as a reference for your own system. You don't need Scope to use this playbook, but you don't have to start from scratch either.

Running the evaluation

Use the evaluation system available to you to configure the scenario, criteria, starting workspace, and agent profiles. Then run each scenario-profile combination multiple times (at least 5) because agent behavior varies between runs. A single run can be an outlier. Multiple runs tell you whether a result is signal or noise.

For every run, preserve the generated output, criteria verdicts, execution gate results, token or cost data, and the agent trajectory. Without the trajectory, you can see what failed but not reliably diagnose why.

Generating the readout

When runs are done, aggregate the raw results into a readout. Whether your evaluation system generates it automatically or you build it separately, the readout should combine criteria verdicts, gate outcomes, task costs, and profile comparisons into a report you can act on.

The readout translates "I ran some evals" into "here's what agents do with this technology, here's what's broken, and here's what to fix."

Further reading

• The hidden variables in your agent eval. How the operating system, file paths, user identity, and tooling quietly change results between machines, and how to control for them.
• Your AI coding agent evaluation is only as good as its sandbox. Why a correct answer can still invalidate the measurement, and how to define the sandbox around what the agent is allowed to know.
• How to test agent skills without hitting real APIs. Evaluate skills that call APIs against lightweight emulation instead of production endpoints.