Turning findings into improvements: AX Practitioner Playbook

An evaluation has no impact until someone changes the source of the behavior. The owning team is whoever controls the affected surface: the documentation, SDK, API, CLI, extension, or other source the agent relies on.

How you turn evidence into a change depends on your relationship to that owner. You might own the surface yourself, collaborate directly with the people who do, or contribute through public channels. In every case, the goal is the same: fix the source, verify the result, and avoid adding another layer unless the source can't be changed.

If you own the affected surface

Start with failures that prevent the agent from selecting or using the technology at all. Then look for active drag, where an extension performs worse than the bare baseline. Inconsistent results deserve attention too. A 2 out of 5 or 3 out of 5 result often means the agent is close to making the right decision, and a small source change might tip it.

Trace each result to a surface your team controls. Fix unclear guidance in the docs. Change the name and description of a tool that isn't invoked. Clarify ambiguous output in the response shape or CLI format. Building another extension on top of a source problem adds another dependency without repairing the experience for everyone else.

Treat the proposed change as a hypothesis. Change one thing, rerun the same scenarios and profiles, and compare the result with the original baseline. If it doesn't improve the outcome, don't ship it just because the new wording sounds better.

If you can collaborate directly with the owner

Approach the work as a partnership, not a verdict on the quality of the team's product. Before you meet, understand what outcomes matter to them, what they already know about AX, and which agent-facing surfaces they maintain.

Lead with baseline results from a scenario relevant to their technology. No extensions, just: "Here's what happens when a developer asks a coding agent to build something with your SDK."

People who work on a technology every day know its vocabulary and intended patterns so well that those assumptions become invisible. Showing how an agent actually approaches the technology reveals where that internal perspective differs from the developer experience.

The owner might respond in several ways:

They want to test a fix. If the change is small, test it together. If the problem is broader, agree on additional scenarios before recommending a solution.

They expect models to improve. Push back on waiting: it leaves today's experience unchanged, with no guarantee the next model will behave differently. Simulate the smallest meaningful change the team could ship, for example by intercepting the agent's requests and serving the modified content, and show whether it makes results better, more consistent, or cheaper. A result the team can see before committing anything is far easier to say yes to.

They already measure AX. Compare scenarios, criteria, environments, and results. Where findings disagree, inspect the trajectories together. Different evaluation approaches can reveal different parts of the same experience.

They can't prioritize the change. Record the evidence and the constraint. Revisit it when priorities, contribution options, or the underlying technology change.

Match your involvement to what helps the owner act. They may ask you to collaborate on the evaluation, hand over the evidence, contribute a fix, or step back.

If you need to contribute through public channels

You won't always have a direct relationship with the owning team. Use the contribution path the project provides:

Open an issue with the scenario, profile, criteria, run count, results, and a minimal reproduction.
Link to the relevant outputs and trajectories, removing credentials, personal data, and other sensitive information first.
Propose the smallest source-level change supported by the evidence.
Submit a focused pull request when the project accepts external contributions.
Use the project's private reporting or support channel when the evidence shouldn't be disclosed publicly.

Make it easy for maintainers to reproduce the result. Separate observations from conclusions, state the evaluation's limits, and include the baseline so they can see whether the affected surface helps or hurts.

The owner may disagree, lack capacity, or have constraints you can't see. That doesn't invalidate the finding, and it doesn't make the owner an obstacle. Document the evidence, keep the evaluation reproducible, and revisit it when something changes.

When the source can't change

Sometimes the source can't be fixed quickly because of priorities, release constraints, or a closed contribution process. A temporary skill or instruction set can bridge the gap until the owner ships a source-level fix.

Coordinate with the owning team when possible. If you publish an independent bridge:

Label it clearly as independent and don't imply endorsement.
State which gap it addresses and link to the supporting evidence.
Name who maintains it and how updates will be handled.
Define the condition for retiring it when the source is fixed.

A bridge creates another surface that users must find, install, trust, and update. Treat it as temporary infrastructure, not a substitute for repairing the source.

Presenting the evidence

Frame findings constructively. If a skill improves implementation tasks but not advisory scenarios, say exactly that. A bounded observation is more useful than a broad judgment about the quality of the product or the team's work.

Lead with what agents do, not with the evaluation machinery. Keep the scenarios, criteria, outputs, trajectories, and environment details available so others can inspect how you reached the conclusion.

Be precise about scope. Results belong to the tested scenario, profiles, versions, and point in time. They don't establish that a product, extension, or model is universally good or bad.

Keep measuring

AX improvement isn't one-off work. Coding agents, models, harnesses, extensions, and products change independently. What works today can regress even when the owning team hasn't changed anything.

Rerun the evaluation regularly using the same locked definition so results stay comparable. Set the cadence to match how fast the agents your audience uses are changing and how much capacity there is to investigate and act on what you find. A system like Scope can automate that repetitive work and keep the signal current. When the product, scenario, or criteria change, create a new version instead of presenting the results as a continuation of the old trend.

Following up

If a fix shipped, rerun the evaluation and verify the result.
If the owner requested more scenarios, run them and share the evidence.
If no change was possible, record why and revisit in 2-3 months or when the product, model, or contribution path changes.
If you published a bridge, check whether its retirement condition has been met.

Whether you own the surface, collaborate with its owner, or contribute through public channels, the loop is the same: measure, diagnose, change, and verify. The improvements compound the same way the problems did.