Developer

Playbook, chapter 8

Interpreting the readout

The readout translates agent behavior into decisions: what works, what doesn't, and what to fix.

A useful readout combines the results with interpretation and recommended next steps. Whether it is generated by your evaluation system or assembled separately, you need to understand what you're reading so you can explain it, defend it, and act on it.

What a readout should contain

A readout covers each scenario-profile combination. Whatever system produces it, here's what it should include:

First, the prerequisite gate: did the agent pick the right technology? If you asked for Cosmos DB and the agent built on MongoDB, everything else is irrelevant. The prerequisite gate must pass before the other gates and criteria are evaluated.

The output evaluation tells you whether the agent's code builds, runs, and deploys. These are mechanical gates, pass or fail. If the code doesn't build, nothing else matters for that run.

The criteria results show how the agent performed on each criterion across all runs. A criterion that passes 5/5 is strong. One that passes 2/5 is inconsistent. One that passes 0/5 is a clear gap.

The profile comparison is where you see lift or drag. If the bare profile scores 4/5 on a criterion and your skill profile scores 2/5, your skill is making things worse.

Task cost shows what each profile costs to run. Tokens come at different prices (input, output, cache tokens each have different price tiers), and what matters is the total cost per task, not just the token count. Drag at a higher price is easy to spot. Lift at a 3x price leads to an interesting discussion: is it worth it? Is there a way to optimize?

Finally, recommended next steps suggest what to investigate or fix. Whether your system generates them or you write them yourself, treat them as starting points, not conclusions. Your domain expertise determines which ones to pursue.

To see what this looks like in practice, browse the readouts from the SPFx upgrade evaluation that Appendix B walks through.

What to look for first

  • Is your extension even being used? Just because you added an extension doesn't mean the agent calls it. Check the readout, and if needed the trajectories, to see if the extension was invoked. If not, it's dead weight costing users money.
  • Criteria where the bare profile outperforms an extension profile. This is active drag. Investigate why.
  • Criteria that fail consistently (0/5) across all profiles. A gap no extension is filling.
  • Criteria that are inconsistent (2/5, 3/5) across runs. The agent is on the edge of a decision. Small content changes can tip inconsistent results toward consistent passes.
  • Large cost differences between profiles for similar outcomes. Your extensions are adding expense without adding value.

What the readout can't tell you

A readout summarizes what happened, and it might include some explanation of why, but it's aggregated across multiple runs and profiles. When you need to understand exactly what path the agent followed in a specific run, you'll look at the collected trajectories. That's what the next chapter is for.

Further reading