Common failure patterns: AX Practitioner Playbook | Developer Agent Experience (DevAX)

These patterns repeat across every technology we've evaluated. Learning them means you won't start from scratch. You'll recognize what you're seeing and know where to look next.

Each pattern describes what you'll see, gives a real example, and points to what to fix.

The agent ignores your extension and uses training data instead

Your extension is loaded: you can see it in the trajectory. But the agent never calls it. Instead it generates code from training data, which is outdated or wrong.

The model feels confident enough from what it learned in training and doesn't see a reason to invoke your tool. Often the issue is vocabulary: your tool says "manage database operations" but the developer asked to "build an API." The agent doesn't connect the two.

Rewrite tool descriptions to match how developers and models think about the task. Include vocabulary for all task types: "build," "create," "diagnose," "compare." And measure baseline first. If the model already does well without your tool, the tool might not be needed for this scenario.

The agent follows docs too literally

This one's subtle. The agent finds your docs, reads them, and follows them. That sounds like success, right? Except the docs include npm install @azure/cosmos@4.1.0 as an example, and the agent installs 4.1.0 even though the developer asked for the latest version. It anchored on the version number in the example.

Agents treat retrieved content as authoritative. Whatever version, pattern, or approach your docs foreground becomes what the agent does. If your docs cover multiple approaches but emphasize one, the agent picks the emphasized one and ignores the rest.

Use version variables or @latest in doc examples instead of hardcoded versions. Ensure balanced coverage when a page describes multiple approaches. If one approach is recommended, say so explicitly rather than relying on emphasis.

Also: keep your docs up to date. When agents retrieve docs while working on a task (as opposed to relying on crawled training data), up-to-date docs are the easiest and cheapest way to fix agent behavior.

The agent has a plan before it reads your content

The agent already knows how to do the task. It reads your docs but treats them as confirmation, not instruction. You added a tip pointing to the correct CLI command. The agent ignored it in every run. Why? Because tips are suggestions, and the agent had already decided what to do.

LLMs form plans from training data before consuming extension content. Suggestions sit alongside the existing plan as one more input. Only explicit contradiction forces plan abandonment.

Don't describe the right path. Invalidate the wrong one. "Manually updating package.json alone will result in build failures" works. "Consider using the CLI instead" doesn't. Use this sparingly, though. If you put plan-breaking language everywhere, it becomes noise.

The agent picks the wrong tool when multiple overlap

Multiple extensions could handle the same task. The agent picks the wrong one, calls both and mixes patterns from each, or calls one and assumes it covered everything.

The issue is vocabulary overlap. "Manage database operations" vs. "database management and queries." The agent can't distinguish which matches the developer's intent, and its choice depends on factors you don't control: context position, training data associations, or just which tool appears first.

Use distinctive, product-specific vocabulary. "Configure Cosmos DB hierarchical partition keys" instead of "manage database settings." Test alongside the most common extensions developers in your space use.

Too many extensions make everything worse

The full extension stack produces worse results than the bare baseline. You add more tools and the scores go down.

Every extension consumes context tokens whether it's used or not. Past a threshold, the cumulative cost exceeds the benefit. We measured this directly: loading several overlapping extensions for the same platform at once used 4x the tokens while producing the worst results across 2 out of 3 scenarios. The agent read too many tool descriptions, got conflicting guidance, and made worse decisions than it would have with training data alone. None of those extensions was the problem on its own. The combination was.

Track net lift across the full stack. If adding an extension doesn't measurably improve outcomes, remove it. "Install everything and let the agent figure it out" is a losing strategy.

Communicate this in your documentation too, so that developers working with your extensions are aware of it. If your extension works best as a standalone, say so.

The agent does manual work instead of running your CLI

Your CLI provides a ready-to-execute command. But the agent ignores it and implements each step manually by editing files. The manual edits introduce version errors the CLI would have prevented.

Why would it do that? Because the agent's default behavior is to use file editing tools. That's its primary modality. CLI commands in docs or tool responses are treated as specifications to implement, not commands to run.

Foreground the script as the primary action. Add explicit instructions: "Run the provided commands rather than manually editing files." Consider output formats that make scripts unambiguous actions rather than readable documents.

The agent anchors on retrieved content over correct knowledge

This is the counter-intuitive one. The agent's training data has the right answer. But after calling a tool or reading a page, it overwrites its correct knowledge with whatever the retrieved content emphasizes. The result is worse than baseline.

Models give disproportionate weight to retrieved content. If your docs page emphasizes one architectural pattern, the agent anchors on that pattern and forgets the other two it actually knew about. We saw this in one of our tests: the bare baseline correctly distinguished three multi-tenancy models. Adding a docs skill that routed to a page emphasizing one model caused the agent to collapse all three into one.

Ensure retrieved content provides balanced coverage. If your tool returns docs, make sure those docs don't inadvertently steer the agent away from patterns it already knows.

The agent scaffolds from 2020

The agent uses an outdated version of your technology. An old scaffolding tool, a deprecated SDK, a legacy API pattern. And it does this confidently, as if the old version is still current.

Training data volume is the culprit. Your old tool has years of blog posts, tutorials, and Stack Overflow answers. Your new tool has months of content. The model's confidence is proportional to how much it has seen, not how recent it is. Runtime resolution can compound this too: npx without a pinned version can silently resolve to a package published in 2020.

Put explicit deprecation signals in the sources you control: package deprecation metadata (like npm deprecate), a warning at the top of the old tool's README and docs, runtime warnings in the CLI, and redirects from outdated docs pages. "DEPRECATED: Do not use X. Use Y instead." works because it invalidates the plan the agent already has. Reach for an agent extension only to bridge the gap while those changes ship. Choose distinctive names for new tools rather than generic category descriptions.

The agent can't follow your tool's routing instructions

Your MCP server uses subrouting: a parent tool returns instructions telling the agent to call a subtool for the actual content. The agent treats the routing response as a final answer and never calls the subtool.

This happens when a competing tool in the same context returns actual content in one turn. The model decides it has enough information and stops. It optimizes for efficiency, and a direct answer beats a routing chain.

Test tools in realistic multi-extension compositions, not in isolation. If your subrouting pattern breaks when other tools are present, consider returning content directly instead of routing.

From failure to surface

The pattern tells you where to investigate first. It doesn't prove the fix, but it narrows the search.

What you observe	First surface to inspect
The extension never loads	Packaging, registration, startup, and harness limits
The extension loads but is never invoked	Tool or skill name, description, and vocabulary
The extension is invoked but the output is wrong	Returned content, response shape, docs, and instructions
The agent edits files instead of running the CLI	Docs examples, tool response, and CLI output format
The full extension stack performs worse than baseline	Overlapping descriptions, conflicting guidance, and context cost
The agent confidently uses an old version	Docs versions, deprecation signals, and runtime version resolution

Change one surface at a time and rerun the same evaluation. Otherwise, you won't know which change caused the result.