Playbook, chapter 6
Choosing agent profiles
Every combination of harness, model, tools, and a myriad of other configuration levers shapes and impacts the outputs of an AX evaluation. It is easy to get overwhelmed by the full matrix of possible combinations, so instead define profiles for testing.
An agent profile is a combination of: the operating system, the harness (eg. GitHub Copilot in Visual Studio Code, Cursor, Claude Code CLI), the model and its settings (reasoning level and context window size), and the extensions (skills, MCP servers, instruction files).
Each profile is an environment the agent runs in, chosen to test a hypothesis, such as "adding skill X improves results at a lower cost."
Start with the baseline
Always run a bare profile first: harness + model, no extensions. This tells you what the agent does with your technology out of the box: its training data plus the harness's built-in tools, such as web search.
The baseline is your reference point. If the baseline is already strong, your extensions need to add value without creating drag. If the baseline is weak, you know where the gaps are and your extensions need to fill them. And if there's no extension yet, there's nothing to compare against, so running just the baseline is perfectly fine. It tells you whether there are gaps and helps you decide how to address them.
Add extensions one at a time
After the baseline, add agent-facing extensions incrementally. Test each skill, MCP server, plugin, or instruction set in isolation before testing combinations. Each combination tests a different question:
- Bare: What does the model already know about your technology?
- Skill only: Does your skill improve outcomes, or create drag?
- MCP only: Does your MCP server help, or does it overwhelm the context?
- Full stack: Do your extensions compose well together, or do they interfere?
We've seen cases where a skill alone improves outcomes, an MCP server alone improves outcomes, and both together make everything worse. It turns out, that composition isn't additive. You can't assume that two good extensions produce a better result than either one alone.
Common profiles and what they test
| Profile | What it tests |
|---|---|
| Bare (no extensions) | Training data coverage for your technology |
| Skill only | Whether the skill adds lift or drag |
| Skill + docs MCP server | Whether general docs help or hurt alongside the specific skill |
| Full extension stack | Real-world developer experience (most realistic, hardest to diagnose) |
| Different harness, same model | Whether your extensions work across harnesses |
For most teams starting out, bare + one extension is enough. That gives you a baseline and a delta.
When custom profiles matter
If you count the possible permutations of operating systems, harnesses, models, reasoning levels, and extensions, not to mention scenarios, you end up with an endless list of options. Enough to paralyze you before you even start.
Keep it pragmatic: for the baseline, use the profile that represents the majority of your audience. Which OS do they use? Which agent and model do they typically work with? Start there, then look at other combinations. With limited time, focus first on where you can improve the experience for the most developers.
If developers using your technology commonly have specific MCP servers installed (like a cloud provider's MCP server), include those in your profiles. Your extensions don't exist in isolation - they compete with whatever else is in the developer's context window.
Don't add custom profiles for the sake of coverage. Each profile multiplies your run count and makes interpretation harder. Add a profile when you have a specific question it answers.
Further reading
- When the model has never seen your code. What the baseline tells you when your technology is proprietary or internal and the model has nothing to start from.
- Your work might not need the smartest model. Set the bar before you run the eval, then find the least expensive model that clears it.