Every coding agent best practice is a guessuntil you measure it.
Developer Agent Experience (DevAX) helps you understand, measure, and improve how AI coding agents work with your technology when developers ask them to build with it. Find what works, fix what doesn't, and measure the impact on outcome quality and task cost.
Or ask your coding agent. Install the AX Practitioner skill and ask the playbook questions while you work.
npx skills add microsoft/scope/docs/ax-playbook/ax-practitionerRead it as text
- Every coding agent best practice is a guess until you measure it.
- Your audience now includes AI coding agents.
- What the developer saw: they asked an agent to create a SharePoint Framework web part. The build passed, and the agent reported "Done. Your web part is ready." What they got: version 1.11.0, from July 2020. Six years out of date.
- Developers judge your technology by what their agent builds with it.
- Does it hurt or help? Letting agents pass your CLI arguments as one JSON payload hurts: 4–11× the cost per task, and correctness never went up. Using the smartest model available helps at a price: +1 point at 5.6× the cost per run.
- Developer Agent Experience (DevAX). Find what works. Fix what doesn't. Measure the impact.
- Worked example: one SPFx upgrade, no new extension. We fixed the docs the agent already reads. Configuration checks passed went from 30/80 before to 83/85 after.
- What do you want to improve? Your technology: docs, skills, MCP servers, SDKs, and APIs. Your agent setup: skills, MCP servers, models, and instructions. One method.
- Start here: the AX Practitioner Playbook. Read it, download it, or ask your coding agent. aka.ms/ax-playbook
- Measure, don't guess. aka.ms/agent-experience
Test your instincts
Call it.
8 changes teams make so developers get better results from their coding agents. We measured each one against a bare baseline. Did it help (lift), not help (drag), or help at a price too high to keep (expensive lift)?
Let agents pass your CLI arguments as one JSON payload.
You called . The eval found: Drag.
4–11×cost per task. Correctness never went up.
With plain arguments, every model got every run right. With JSON, the strong models got there after retrying shell-escaping failures. Haiku 4.5 dropped from 5 of 5 to 2 of 5.
- Harness
- GitHub Copilot Chat
- Models
- Sonnet 5, Sonnet 4.6, GPT-5.3-Codex, Haiku 4.5, MAI-Code-1-Flash
- OS
- Windows, PowerShell
- Runs
- 5 per model per mode
Add structured JSON output to your CLI.
You called . The eval found: Lift.
72 → 85of 85 configuration checks.
The agent read structured output instead of guessing at text. Similar is not the same, though: switching that output to JSONL cost 2.6× more tokens, with worse outcomes on every dimension.
- Task
- SPFx project upgrade, changes emulated with Dev Proxy
Read the research: How to test agent experience changes without shipping them
Add a tip to your docs that points agents to the right tool.
You called . The eval found: Drag.
1/5runs used the tool.
The agent planned to bump npm packages before it read the page. A tip offers an option, it doesn’t say the plan is wrong. A warning that named the approach that would fail moved it to 5 of 5.
- Task
- SPFx 1.21.1 to 1.22.2 upgrade
- Runs
- 5
Use the smartest model available.
You called . The eval found: Expensive lift.
+1 pointat 5.6× the cost per run.
Upgrading SPFx projects with a skill, Claude Sonnet 4.6 scored 98% at $0.92 per run. GPT-6 Astra scored 99% at $5.14. On a JDK migration with a skill, Astra scored about 4 points lower at 5.3× the cost.
- Harness
- GitHub Copilot Chat in VS Code
- Models
- Claude Sonnet 4.6, GPT-6 Astra
- OS
- Windows
- Runs
- 5 per profile
Read the research: Your work might not need the smartest model
Give the agent another documentation source through an MCP server.
You called . The eval found: Drag.
0/5runs called its tools.
The context7 MCP server added no meaningful lift. Its tools didn’t load in 3 of 5 runs and weren’t called in the other 2.
- Task
- SPFx project upgrade
- Runs
- 5
Tell the agent which CLI to use for upgrades.
You called . The eval found: Lift.
30 → 75of 80 configuration checks.
It only helps developers who know about the CLI or install the skill that carries the instruction. Fixing the release notes the agent already read made every run use the CLI, for every developer.
- Harness
- GitHub Copilot Chat
- Models
- Claude Sonnet 4.6
- OS
- Windows
- Runs
- 5
Switch to the newer model. Its tokens are 33% cheaper.
You called . The eval found: Expensive lift.
3.7×cost per run.
On SPFx code upgrades, Claude Sonnet 5 completed 100% of tasks to Sonnet 4.6’s 60%, at $2.01 per run to $0.55. On Azure architecture work, it completed the same share at lower quality, 78% against 90%.
- Harness
- GitHub Copilot Chat in VS Code
- Models
- Claude Sonnet 4.6, Claude Sonnet 5
- OS
- Windows
- Runs
- 5 per model per scenario
Ship a ready-to-run upgrade script with your docs.
You called . The eval found: Drag.
0/10runs ran the script.
In zero runs out of ten did the agent pipe the script to a shell. A script nobody runs changes nothing.
- Task
- SPFx project upgrade
- Runs
- 10
Read the research: How to test agent experience changes without shipping them
Your intuition might be right. But without measuring, you don't know. That's what DevAX is for.
Two ways in
What do you want to improve?
Your technology
Make your SDKs, APIs, and frameworks work better with AI coding agents.
Discover where agents struggle, identify opportunities for improvement, and measure whether your changes help.
Your agent setup
Help your developers get better results from AI coding agents without spending more than you need to.
Compare models, skills, and configurations on representative work to find the best balance of outcome quality and task cost.
Worked example
See what measurement reveals.
30/80 → 83/85configuration checks passed when GitHub Copilot upgraded a SharePoint Framework project, before and after.
- What we evaluated
- Whether GitHub Copilot with Claude Sonnet 4.6 can upgrade a SharePoint Framework (SPFx) project from v1.21.1 to v1.22.2 when a developer just types "Upgrade the project to 1.22.2". Every run compiled and ran, but only 30 of 80 configuration checks passed. A skill the SPFx team had already built raised that to 37.
- What changed
- The docs the agent already reads. The agent knew SPFx, but planned a manual upgrade and read the 1.22 release notes instead of 1.22.2. Release notes pages now use full version numbers, and the migration guide warns that editing package.json by hand breaks the build, pointing to CLI for Microsoft 365. We tested changes with Dev Proxy before shipping them, and several hypotheses didn't survive.
- What the numbers mean
- Configuration checks passed, added up across five runs: 30 of 80 in April, and 83 of 85 in September after the fixes were live, when every run found and used the CLI without being told. One check was added in between, so read it as a before-and-after pattern, not an exact delta. No new extension, so every developer benefits without installing anything.
Why it matters
Developers judge your technology by what their agent builds with it.
When an agent picks a competing service, calls an API you deprecated, or reports success on a broken build, your technology looks broken. Even when your docs are excellent for people.
Agent Experience (AX) is the experience AI agents have when discovering, choosing, and using your technology. Developer Agent Experience (DevAX) narrows that to one kind of agent and one kind of user: the experience a coding agent has when a developer asks it to build with your SDK, API, CLI, or framework. Practicing DevAX means improving it systematically. Measurement is how you know whether you did.
What the developer saw
A new project. A passing build. An agent reporting success.
What they got
SharePoint Framework v1.11.0, from July 2020, instead of v1.23.0. Twelve versions and six years behind, from an unpinned npx.
The status quo
How teams handle it today.
Skills, MCP servers, and documentation can all help agents work better with your technology. The question is where they make a difference—and whether that difference is worth the cost.
- Follow a list of AX best practices
- A good start. But a change that sounds agent-friendly can make agents worse, or more expensive. Passing CLI arguments as JSON did both.
- Ship a skill or an MCP server
- It helps developers who install it, at a token cost on every run. Fixing the docs agents already read helps every developer.
- Wait for smarter models
- Newer isn't always better at your tasks. On architecture work, a newer model completed the same share of tasks at lower quality, with 12× the tokens at the median.
- Add an llms.txt
- We haven't found evidence that agents read it, so this site doesn't have one. The playbook ships as an agent skill instead.
The AX stack
Where you can win.
- The model
- Fixed. You can't change it, and a model that learned your docs in training won't see your update until the next one ships.
- The harness
- The agent app that runs the model. You can only influence it through its vendor.
- Everything the agent reads and calls
- Yours. Your docs, which an agent reading them at work sees on its very next run. Your agent extensions: skills, MCP servers, instruction files, and custom agents. And your interfaces: APIs, SDKs, CLIs, error messages, and scaffolders.
The practice
Measure, don't guess.
Start with the work developers actually need to accomplish. Measure how agents perform today, then improve what matters.
- Define the work. Choose representative development scenarios and success criteria.
- Establish a baseline. Compare relevant agents and models without added interventions.
- Find the gaps. Identify failures, unnecessary task costs, and opportunities to improve.
- Make targeted changes. Improve documentation, SDKs, skills, MCP servers, or configurations.
- Measure again. Compare outcome quality and total task cost, and reevaluate as conditions change.
The AX Practitioner Playbook.
From your first baseline to measuring improvements, the AX Practitioner Playbook shows you how to evaluate AI coding agents on the work that matters to your developers.
Or install the skill and ask the playbook from your agent. The AX Practitioner skill answers questions, walks you through the method, and reviews your eval against the checklist.
npx skills add microsoft/scope/docs/ax-playbook/ax-practitioner- The game has changed1
- The evaluation model2
- The eval lifecycle3
- Designing scenarios4
- Writing evaluation criteria5
- Choosing agent profiles6
- Running evals and generating the readout7
- Interpreting the readout8
- Investigating agent trajectories9
- Common failure patterns10
- Turning findings into improvements11
- Quality checklistApp. A
- Worked example: SPFx end-to-endApp. B
Open source
Tools for each step.
- LearnHow do we practice DevAX?
- ObserveWhat did the agent actually do?
- MeasureHow often does it get it right, and at what cost?
- CompareLift, drag, or expensive lift?
- Change without shippingWill this fix work before we publish it?
Research
Read what we measured.
Is our skill or MCP server actually working?
- How to measure AI agent extension effectivenessStatic checks passed 90%+. The projects didn’t compile.
- How to diagnose conflicts between AI agent extensions
- Stop overloading your skills
Why does the agent use our old version?
- Your agent just scaffolded a project from 2020SPFx v1.11.0 instead of v1.23.0. Twelve versions, six years.
- Why AI coding agents keep using deprecated CLIs and SDKs
- Your agent already has a planA tip: 1 of 5 runs. A warning: 5 of 5.
Which model should our developers use?
- Not all model upgrades are upgradesSame prompt, same model: 16,000 tokens in one run, 6.6 million in another.
- Your work might not need the smartest modelOne more point of quality at 5.6× the cost.
- What AI benchmarks are not telling you
Objections
Questions we get.
- Isn't this just developer experience?
- DevAX is where DX and AX meet: your developers' experience now runs through their coding agent. It keeps one of DX's oldest lessons: conventions help, but you still watch real users to know whether your product works. Agents are users now too, and they're easier to watch. You can run the same task five times with and without a change, which you'd never ask of a room full of developers.
- Does this apply to agents that browse my site or use my app?
- Some of it transfers, but that's not what we measured. Everything here comes from coding agents doing development work for developers. For Agent Experience across every kind of agent, see agentexperience.ax.
- Isn't this generative engine optimization?
- GEO is about getting AI search to mention your product. That overlaps with propensity. DevAX goes further and asks whether the coding agent uses your technology correctly once it has picked it.
- We'll deal with agents later.
- Your developers are already asking coding agents to build with your technology. Without measurement, the only evidence you have is a demo that looked fine.
Stop guessing.
Find out how AI coding agents work with your technology when developers build with it—and what you can do to improve the outcome.
Or ask your coding agent. Install the AX Practitioner skill and ask the playbook questions while you work.
npx skills add microsoft/scope/docs/ax-playbook/ax-practitioner