Developer

Playbook, chapter 4

Designing scenarios

A scenario isn't a test case. It's the question you're asking about your technology. The quality of what you learn depends entirely on the quality of the question you ask.

A scenario is the prompt a developer would type into a coding agent. Nothing more. No evaluation instructions mixed in, no scoring rubric, no hints about what the "right" answer is. The agent should see exactly what a real developer would type.

What makes a good scenario

The only true requirement: your scenario should be as close to what a developer would actually put into a coding agent as possible. If developers use vague or ambiguous prompts, that's exactly what you should be testing. The moment you start crafting artificially clear prompts, you're losing touch with reality.

A scenario can be anything from "build me a todo app with Cosmos DB" to "upgrade this project to the latest version" (yes, explicitly leaving out the tech stack to see if the agent properly discovers it!). Both are valid because both are things a developer would actually ask.

Propensity vs. efficacy

You can test two different things, and both matter equally. Which you start with depends on your business context.

Propensity (discovery): does the agent find your technology and choose it? "Build an API that stores customer data in a document database" doesn't name Cosmos DB. If the agent picks MongoDB instead, that's a propensity finding. If the product is new, propensity is probably where you start. You need to know whether agents even pick it up, and if they don't, what other technologies are competing for attention.

Efficacy (quality): given that the agent uses your technology, does it use it correctly? "Create a Cosmos DB container with hierarchical partition keys for multi-tenant isolation" assumes Cosmos DB. Now you're testing whether the agent knows the right patterns. If you're working on technology that's been out there for years and models already know about it, efficacy is where you'll find the most value, unless you're looking at competitive scenarios where propensity matters again.

Neither is better than the other. Start with the one most relevant to your situation.

Things to be aware of

Complex prompts increase variance. A prompt that asks the agent to do five things will produce more variable results than a simple one. That's fine, but be aware that you'll need more precise evaluation criteria to interpret what happened. The more complex the prompt, the more careful you need to be about what you're checking for.

Prompt-crafting is a trap. If your eval only works with perfectly worded prompts, it won't tell you what happens in the real world. Developers don't use your team's internal vocabulary. They don't always know the right product names. Be mindful of that gap.

Your turn

Pick a scenario for your technology. Before you run it, check:

  • Is it something a real developer would actually type into a coding agent?
  • Are you aware of what you're testing (propensity or efficacy)?
  • If the agent gets it wrong, will your evaluation criteria be able to surface what went wrong?

The quality checklist in Appendix A covers this in more detail.

Further reading