Research and opinions
Read what we measured.
Find out what coding agents actually do with your technology, why they get it wrong, and which fixes make a difference. The research is based on measured agent runs. The opinions are what we make of it.
Start with the question you're facing, or browse all 48 articles. Get new ones with RSS.
Where do we start?
Is our skill or MCP server actually working?
- How to measure AI agent extension effectiveness AX series, part 3Static checks passed 90%+. The projects didn’t compile.
- How to diagnose conflicts between AI agent extensions AX series, part 4
- Stop overloading your skills
- Stop skillmaxxing, save your tokens50 skills: about 10,000 tokens before the first prompt.
- Test whether your knowledge skill actually works Opinion
- Does every agent skill need an eval? Opinion
Why does the agent use our old version?
- Your agent just scaffolded a project from 2020SPFx v1.11.0 instead of v1.23.0. Twelve versions, six years.
- Why AI coding agents keep using deprecated CLIs and SDKs
- Your agent already has a planA tip: 1 of 5 runs. A warning: 5 of 5.
- When the model has never seen your code AX series, part 5
- Knowledge cutoff is a poor proxy for model capability18% of Dev Proxy tasks passed, on both sides of the cutoff.
- A knowledge cutoff is a ceiling, not a snapshot Opinion
- What's volatile about your technology? Opinion
Which model should our developers use?
- Not all model upgrades are upgradesSame prompt, same model: 16,000 tokens in one run, 6.6 million in another.
- Your work might not need the smartest modelOne more point of quality at 5.6× the cost.
- What AI benchmarks are not telling you AX series, part 6
- Models don’t have preferences, they have context
- You can't judge an AI agent until you know what the task is worth Opinion
Can we trust our eval?
- Building AX evals that actually work AX series, part 8
- The hidden variables in your agent eval AX series, part 7An “azureuser” home folder nudged the agent toward Azure.
- Your AI coding agent evaluation is only as good as its sandbox
- Your eval should describe the user's success, not your product's behavior Opinion
- An eval score is only half the result Opinion
- Your agent eval is more than its result Opinion
How do we test a fix before we ship it?
- How to test agent experience changes without shipping themA ready-to-run script: used in 0 of 10 runs.
- How to test agent skills without hitting real APIs
- Don’t rewrite your CLI for agentsJSON input: 4–11× the cost, no gain in correctness.
All articles
- If you can't automate your workflow, neither can your users' agentsYou want to test whether an agent can deploy an app to your platform. But before you can check the agent's work, your eval harness has to click through a portal and an interactive sign-in. Whose problem is that?
- Be honest about what your evals proveYour coding agent scored 100% on its eval, and every criterion passed. Then someone tried to build the code it generated.
- Your eval judge doesn't know what you meanDoes the solution follow best practices? Your LLM judge answers without hesitation and explains its verdict convincingly. So which best practices did it check?
- As models get better, your agent tools should get smallerYou wrote your agent skill to fix what the agent kept getting wrong. What happens to those instructions when a new model stops getting it wrong?
- Introducing the Agent Experience (AX) Practitioner PlaybookAI coding agents now pick the SDK, the version, and the pattern for your developers. The AX Practitioner Playbook is our method for finding out whether they get your technology right, and for fixing the sources they rely on when they don't.
- What is Agent Experience (AX)?Your agent says it's done, and the code compiles, but did it pick your technology? Did it use it correctly? Learn what Agent Experience (AX) is, how to measure it, and why the obvious fix isn't always the right one.
- Your eval should describe the user's success, not your product's behaviorYour eval checks whether the agent called the tool you built, and every run passes. What does that tell you about the developer who asked for help?
- Knowledge cutoff is a poor proxy for model capabilityA model can fail on features released before its knowledge cutoff, then succeed on ones released after it. We tested hundreds of product changes and found that the date tells you far less than the work does.
- An eval score is only half the resultYour agent's score improved. Its inference bill increased 15 times. Before you celebrate, decide what the improvement is actually worth.
- Your agent eval is more than its resultYour agent completed the task and every grader passed. What happened before the score reached 100%?
- Your AI coding agent evaluation is only as good as its sandboxYour AI coding agent passed the eval. But did the model know the answer, or did it find it somewhere on your machine? A correct answer can still invalidate your measurement.
- A knowledge cutoff is a ceiling, not a snapshotA model's knowledge cutoff looks precise. But how much does that date actually tell you about what it knows?
- Your work might not need the smartest modelThe smartest model can cost five times more and deliver the same result, or even a worse one. See how evaluating your own work helps you get more value from your agent budget.
- Test whether your knowledge skill actually worksYour knowledge skill gives reasonable answers to the questions you've tried. What would you find in the parts of the source you didn't think to test?
- What if coding agents had a learning mode?Coding agents can take us from intent to production in minutes. What happens to the understanding we used to build along the way?
- Your product didn't get worseCustomers still like your product and keep renewing. Meanwhile, their expectations may be changing in ways your usual signals won't reveal.
- Your coding agent extension needs continuous evaluationYour extension passed its evals. Days later, its code is identical, but can you still trust the result?
- The red penAI can keep an entire working session in context. Yet changing one sentence still requires directions that sound like a confused treasure map.
- Your app is losing its grip on your customersYour customers still use your app and renew their contracts. Meanwhile, the work they once did in it might already be moving somewhere else.
- Does every agent skill need an eval?A skill can make an agent better and still be a bad trade. How much evidence should you expect before trusting someone else's skill?
- What's volatile about your technology?A model's knowledge cutoff doesn't tell you how old its knowledge of your SDK is. Technology providers know something more useful.
- Your agent should start with a hypothesis, not a planAgents can turn a plausible first answer into a plan before current documentation gets a say. By then, correcting an outdated assumption is already expensive.
- DRY might be wrong for agent documentationDRY documentation saves maintainers from repeating themselves. What if it makes every agent pay for that convenience?
- Your agents should learn for the organizationYour agents discover stale guidance and hidden constraints every day. When the conversation ends, who gets to keep what they learned?
- Every team needs a knowledge interfaceOrganizations have spent years improving how people find knowledge. Agents don't just find it. They make decisions with it.
- Inference efficiency is about protecting attentionEvery extra tool call looks like a small tax. The larger cost is what it takes away from the agent while it works on your problem.
- We're measuring AI productivity the wrong wayWhen we talk about AI and productivity, we ask how much time it saved us. That's the wrong question. The one that matters is how much time it saved everyone. AI didn't remove the cost of communication. It shifted it from the author to the audience, and we're celebrating the wrong side of the ledger.
- Every configuration option is a question we couldn't answerOpen your favorite AI coding agent and you're greeted with a surprising number of decisions. Which model? How much reasoning? Which MCP servers? Most of us don't know the right answers. We treat that as the cost of cutting-edge AI. It's a sign the technology hasn't matured yet.
- The next literacy is externalizing judgmentWhen ChatGPT took off, everyone wanted to learn prompt engineering. We've been optimizing the wrong thing. The biggest productivity gains from LLMs come from learning to externalize judgment.
- You can't judge an AI agent until you know what the task is worthWe keep asking whether AI coding agents are good or bad, fast or slow, cheap or expensive. But good compared to what? Until you know what a task is actually worth to you, you have no way to tell whether the agent's output was a bargain or a rip-off.
- How to test agent experience changes without shipping themMost changes you think will improve AI agent behavior won't. We tested a dozen hypotheses on a real project upgrade scenario and the majority failed. Learn how to emulate documentation, API, and MCP server changes locally so you can validate what works before shipping anything to production.
- How to test agent skills without hitting real APIsYour agent skill calls an API. The moment you start evaluating it, every run either costs money or mutates production data. Learn how to mock APIs transparently so you can run evals without changing your skill or hitting real endpoints.
- Building AX evals that actually work
- The hidden variables in your agent eval
- Don’t rewrite your CLI for agentsThere’s advice making the rounds: replace your CLI args with a single --json payload so agents can use your tool more effectively. The thinking being, that agents already think in structured formats, and nested data maps cleanly to JSON.
- Not all model upgrades are upgradesA new model drops with lower per-token pricing and better benchmarks. You switch. A week later someone asks why the agent is burning 12x more tokens on the same task while producing worse output.
- What AI benchmarks are not telling you
- Your agent already has a planIf an agent isn’t doing the right thing, the obvious move is to make the docs clearer. Add a tip, spell out the correct command, describe the right approach more prominently. You do all of that, and the agent still ignores it.
- When the model has never seen your code
- Models don’t have preferences, they have contextYou open a fresh chat, type “What framework should I use for a web app?”, and the model says “React.” You screenshot it, share it, and write “Claude prefers React.” It gets engagement. People nod along. A few reply with their own results.
- Stop overloading your skillsYou built a skill for your technology. API references, authentication flows, SDK patterns, error handling, version info, all packed into one skill. The agent calls it, gets all that context, and generates code.
- How to diagnose conflicts between AI agent extensionsYour extension works in isolation. You measured it, confirmed it creates lift. Then a developer installs it alongside 14 other extensions, and outcomes get worse. The symptoms look like a bug in your code, but the problem is elsewhere: extensions fighting each other for the same context window and model attention.
- Why AI coding agents keep using deprecated CLIs and SDKsYou deprecated the old CLI and shipped something better. Developers are migrating, but AI coding agents aren’t. They keep reaching for the deprecated tool, confidently scaffolding projects with something you sunset months ago.
- Your agent just scaffolded a project from 2020Your agent ran a scaffold command. Project generated, dependencies resolved, no errors. Everything looks fine. Except it’s based on the project structure from 2020, and neither you nor the agent noticed.
- How to measure AI agent extension effectivenessYou shipped your skill. Everything looks like it's working. But is the generated code actually better because of your extension? You can't tell without measuring, and measuring agent extension impact is harder than it looks.
- Stop skillmaxxing, save your tokensYou built a dozen skills for your technology: authentication, CRUD, error handling, deployment, testing, monitoring. Then you installed a cloud platform bundle with 15 more covering diagnostics, storage, compliance, and cost optimization.
- How AI coding agents actually use your technologyYou ship an SDK, a CLI, an API, and developers use it. Now AI coding agents use it too, except they use it differently than humans do.
- How to improve AI coding agent support for your SDK, API, or CLIAI coding agents use your SDK, API, or CLI, but often get it wrong. The stack has layers you can't change and one you can: agent extensions. Learn how to measure whether yours are helping or hurting.