Investigating agent trajectories: AX Practitioner Playbook

This is where you develop intuition. Reading trajectories is like watching an agent think out loud about your technology, and discovering what it actually believes vs. what you intended. The readout tells you what failed. The trajectories tell you why.

A trajectory is the complete record of what the agent did during a single run: every tool call, every file read, every decision. ATIF (Agent Trajectory Interchange Format) is one format for storing this data and comparing it across the different agents. When a criterion fails, the trajectory shows you the moment things went sideways.

Use whichever trajectory viewer your evaluation system provides. For ATIF files, we like to use the ATIF Preview VS Code extension, which renders tool calls, file operations, and decision points in a readable format, allowing you to quickly find the relevant information.

How to read a trajectory

A trajectory is a sequence of events. Each event records an action: reading a file, calling a tool, generating code, receiving tool output. You're looking for decision points, the moments where the agent chose one path over another.

Some models also expose their reasoning (chain-of-thought). If present, this is great additional information to understand why the agent chose to do something. Look at the reasoning to see what the agent considered and what it dismissed.

Scan for:

Tool calls. Which tools did the agent call? Which were available but never called? If your extension was loaded but never invoked, the agent decided it didn't need your help.
File reads. Which docs did the agent read? Did it find the right page? Did it read the whole page or stop partway?
Decision sequences. Follow the chain of events leading to a wrong output. Often the agent made the wrong call early, and everything after was downstream of that first mistake.

Discovery, invocation, and application

When something goes wrong, the first question is: at what stage did the agent fail?

Discovery failure: The agent failed to load your extension entirely. The MCP server didn't start, or there was an error loading its tools, or the skill exceeded the agent's size limits. Your extension never made it into the context. This is an infrastructure problem.

Lack of invocation: The extension loaded successfully, you can see it in the trajectory, but the agent decided not to use it. This typically means either the tool's description doesn't match the intent the model inferred from the user's instruction, or the model was confident enough in its own knowledge and didn't see a need for additional information.

Application failure: The agent found your technology and tried to use it, but applied it incorrectly. It chose the wrong SDK version, wrong pattern, or ignored critical configuration. This is all a content problem. The fix is usually in your docs, skill content, or tool responses.

Why do these distinctions matter? Because the fixes are completely different. A discovery failure is an infrastructure fix. Lack of invocation needs better tool descriptions or vocabulary alignment. An application failure needs better content or plan-invalidation language in your docs.

Contradictory behaviors

One of the most common findings: the agent does X in one run and the opposite in another, given the exact same scenario and profile. Run 1 uses your CLI tool. Run 2 ignores it and edits files manually. Same prompt, same extensions, but different behavior.

This is exactly how LLMs work. The model's token sampling introduces variability. When the agent is on the boundary of a decision, small sampling differences tip it one way or the other.

Contradictory behavior tells you the agent's decision is weakly motivated. Something in the context is almost-but-not-quite convincing enough to consistently steer behavior. And that's actually good news: these are the highest-value improvements. A small content change can tip a 2/5 result to 5/5 because the agent was already close to the right decision.

Attribution: whose problem is it

When you find a failure, trace it to its source:

The documentation's problem. Ambiguous content, misleading emphasis, or missing information.
The extension's problem. Your skill or MCP tool provided content that confused the agent or conflicted with other sources.
The training data's problem. The model's training data is stale and no extension is correcting it. The most common root cause.
The agent's problem. The agent is the harness and the model together. A reasoning error, or harness behavior like a fetch tool that pulls in irrelevant page content, that no content change would fix. Rare. When it's the harness, report it to the harness vendor.

The source determines the fix. A doc problem needs a doc fix. A training data problem needs an extension that provides current information. Don't band-aid a doc problem with a skill when you could fix the doc. We can't emphasize this enough.

If you own the source, fix it directly. If you don't, bring the evidence to the team that does. A new skill should be the last resort because the improvement then depends on developers finding, installing, and updating it. Everyone without the skill still has the broken experience.

Investigating: a worked example

In an SPFx upgrade evaluation targeting v1.22.0, the bare profile installed the wrong version in 4 out of 5 runs. The readout showed only the symptom: a failed version criterion. The trajectory showed the cause. The agent fetched the migration guide, read npm commands hardcoded to v1.22.1, and followed them instead of the version the user asked for.

That moved the investigation away from an abstract model weakness and toward a specific documentation fix. Appendix B follows the complete evaluation, from scenario design through the shipped fixes.