Developer

Playbook, chapter 1

The game has changed

Whether you build a technology or help developers adopt it, your audience now includes AI coding agents. Agents pick technology stacks and choose services. Left alone, they make these choices based on stale training data and whatever extensions happen to be loaded. You need to influence agents now, not just developers.

Ask a coding agent to build something with the technology you work on: a Cosmos DB app, a SharePoint Framework (SPFx) web part, a payments integration, whatever is in your space, and watch what happens.

The agent will generate code. It will look reasonable, and might even compile. But look closer and you'll find the wrong SDK version, a deprecated authentication pattern, a partition key strategy that no one on the product team would recommend. The agent didn't make a random mistake. It did exactly what its training data told it to do, and its training data is months or years behind what the product team shipped last quarter.

How close does the agent get to correct usage? That's the agent experience of your technology. Measuring it with evals is gaining popularity, but most teams are still figuring out how to do it well. Many haven't started yet and judge how their extensions perform on anecdotal observations.

Agents are another audience

Product teams, maintainers, and developer advocates have always helped developers succeed with technology. You build products and extensions. You write docs and samples. You make the technology approachable, and you make sure developers can find what they need when they need it.

All of that still matters. But it was built for a world where the developer reads your docs, studies your samples, and writes their own code. Developers increasingly work through AI coding agents instead. They describe what they want, and the agent figures out how to build it. The agent picks which SDK to use, which version to install, which authentication pattern to follow, which configuration to apply. The developer reviews the result, but the agent made the decisions.

When the agent makes those decisions, it draws on its training data, its tool descriptions, whatever extensions are loaded, and whatever docs it can reach. If your technology isn't represented well in those sources, the agent will get it wrong. Not sometimes. Reliably. Across every developer who uses that agent with your technology. Albeit to different degrees and in different ways.

That changes what product and advocacy work means. It isn't enough that a developer can find the right information. An agent needs to find and correctly apply it through the surfaces you control: docs, MCP tools, skills, plugins, instructions, CLIs, and APIs.

Newer models don't necessarily help either. A knowledge cutoff tells you the latest date a model's training data could cover, not how well the model knows your product. Its product knowledge might end months earlier or be patchy across the product's history, and we've seen a newer model know less about a technology than its predecessor. Knowledge cutoff is a poor proxy for model capability, so test the baseline to understand what you're up against.

Waiting for models to get better isn't a strategy. Model improvements are unpredictable, and developers are using your technology today. Fixing the sources agents rely on improves the current experience and gives future models better information to work with.

What this looks like in practice

In fall 2025, we in DevRel at Microsoft started measuring how agents actually perform with the technologies we support. We've since run hundreds of agent sessions across Azure, Cosmos DB, SPFx, and Microsoft 365 Copilot extensions, using the same kinds of prompts real developers use. What we found wasn't subtle.

Agents ignored available CLI tools and manually edited files instead, introducing version errors the CLI would have prevented. They picked up documentation but anchored on the wrong section, collapsing three distinct architectural patterns into one because a single docs page overemphasized it. They loaded extension tools but never called them, choosing to rely on training data that was two SDK versions behind.

These aren't edge cases. They're patterns that repeat across technologies and agent harnesses. And they repeat because few teams are looking.

What happened when we looked

Looking was the easy part. What mattered was what came after: turning findings into specific changes the owning teams could ship. Across the technologies we evaluated, that turned into dozens of shipped fixes to docs and agent extensions. For the Azure Cosmos DB Agent Kit alone, you can browse the 46 improvements that came out of our evaluations. The SPFx dev skill followed the same path.

One example: SPFx release notes used filenames like release-1.22.md. That seems fine to a human, but an agent looking for version 1.22.2 sees release-1.22 and assumes it has what it needs. It never looks for the version-specific page. Renaming the file to release-1.22.0.md made it obvious that the page covered 1.22.0 and nothing later, so the agent realized there was a gap and went looking for the right version. Before the change, none of the 4 runs that read the release notes went on to the 1.22.2 page. After it shipped, 4 of 5 runs did.

Another: SPFx docs included a tip pointing developers to the CLI for Microsoft 365 with the exact upgrade command. No run used the CLI, because the agent had already formed a plan from training data. Tips are suggestions, and agents treat them that way. After we replaced the tip with a warning that said "manually updating package.json alone will result in build failures" and rewrote the migration guide, every run used the CLI instead. The agent didn't need encouragement. It needed its existing plan invalidated.

These are the kinds of improvements that compound. Each one is small. Together they transform how reliably agents work with a technology.

Who this is for

This playbook is for anyone who builds technology or extensions that AI coding agents use. If you build an SDK, API, service, CLI, MCP server, skill, plugin, or instruction set, or write the docs behind them, you control what agents discover and apply. That includes product teams and open source maintainers, docs writers, and platform teams building internal tooling for their own developers. Evaluation shows you whether those surfaces work as intended.

If you advocate for a technology rather than own it, you bring the scenario knowledge and sustained attention needed to evaluate the developer experience across those surfaces. You can run the work with the owning team, or bring evidence that helps them decide what to change. Chapter 11 covers what to do when you don't have a direct line to the owner.

Either way, the work depends on domain expertise. Without it, criteria end up miscalibrated: too easy and the base model passes already, telling you nothing. Too hard and you optimize for something theoretical that no developer actually needs. The strongest evaluations combine product knowledge with realistic developer scenarios, whether that's one person or two teams working together. If you're looking to tune your own coding agent setup, some of the ideas here transfer, but that's not what this playbook is about.

What you'll be able to do

By the end of this playbook, you'll be able to:

  1. Evaluate your technology. Run an AX evaluation against the products, extensions, SDKs, and APIs you build or support, and understand what the results mean.
  2. Diagnose failures. Identify whether agents fail to discover your technology or apply it correctly, and trace root causes to specific docs, tools, or extension surfaces.
  3. Drive improvements. Translate findings into specific changes, test them, and work with the people who own each surface to ship fixes.

Further reading