The eval lifecycle

You already know what good looks like for your technology. The eval lifecycle is how you make that knowledge visible in data that can drive decisions and improvements.

The process has five steps:

1. Design a scenario. Write the prompt a developer would give a coding agent.
2. Write evaluation criteria. Define what "correct" looks like for the output. Writing criteria isn't trivial and we discuss it in more detail in chapter 5.
3. Choose agent profiles. Pick which combinations of harness, model, and extensions to run against.
4. Run the evaluation. Execute each scenario-profile combination multiple times and collect the outputs and trajectories.
5. Generate and interpret the readout. Turn the raw results into an actionable report.

After step 5, you have data. What you do with that data (investigating trajectories and driving improvements) is Part 3.

The chapters that follow cover each step in detail. If a step feels mechanical, it's short. If it requires judgment, it's longer. Chapter 7 also covers what to look for in an evaluation system.