Developer

Playbook, appendix b

Worked example: SPFx end-to-end

A complete AX evaluation cycle for SharePoint Framework (SPFx) project upgrades, from scenario design through evaluation, investigation, and shipped improvements. This is a condensed version of the full engagement. The complete readouts are linked at the end of this appendix.

We wanted to understand how well GitHub Copilot handles SPFx project upgrades out of the box. We chose upgrades because they're one of the most impactful scenarios for SPFx developers: upgrading is how developers get the latest capabilities. The evaluation was a collaboration: the SPFx product team knew what a correct upgrade should look like, and we, as developer advocates, knew how to measure whether an agent gets there. That split is common, but it isn't required. The steps are the same whether you own the technology, advocate for it, or contribute from the outside, as Chapter 11 describes. What changes is who ships the fixes.

The scenario

Prompt: "Upgrade the project to 1.22.2"

No additional context, no guidance. Just the outcome a developer would type. The starting workspace contained an SPFx v1.21.1 web part project. We deliberately kept the prompt minimal to test how well the agent reasons about the project on its own.

The environment

We matched a typical SPFx developer setup:

  • VS Code with GitHub Copilot Chat
  • Windows 11
  • Claude Sonnet 4.6 (a popular model at the time)

The criteria

We grouped criteria into four classifications:

  • Prerequisite gate. Was the project upgraded to SPFx 1.22.2? We check the version of @microsoft/sp-webpart-base in package.json. If this is wrong, everything else is noise.
  • Dependency currency. Did the agent move the project to the correct dependency state? Exact versions for runtime/build dependencies, required pins/overrides, removal of legacy Gulp-era packages.
  • Configuration correctness. Did the agent perform the migration mechanics correctly? Config files, scripts, .yo-rc.json, .gitignore, SASS config, Heft migration details.
  • Idiomatic use. Did the agent align with current SPFx conventions? Expected modern lint rules and patterns.

Plus execution gates: Prerequisite (passes), Build (compiles), Test (tests pass), Run (starts successfully).

The profiles

We started with three profiles:

Baseline results

Every run passed all execution gates (the project compiled and ran). But the criteria told a different story:

Classification Baseline + Skill + Skill + context7
Prerequisite 5/5 5/5 5/5
Idiomatic use 6/10 10/10 9/10
Dependency currency 34/50 38/50 39/50
Configuration correctness 30/80 37/80 41/80

The skill provided some lift, most clearly in idiomatic use, but still left the project only partially upgraded. Adding context7 on top made little difference, and context7 itself contributed nothing. In 3 of 5 runs, its tools never loaded into the agent's context. In the other 2, they were available but never invoked. Available tools are not always used.

What investigation revealed

The agent already knows SPFx. Before invoking any tools, the agent identified the workspace as an SPFx project from the file structure alone, formulated an upgrade plan, and knew the URL of the release notes page. It was going to Microsoft Learn, Microsoft's documentation site, to confirm versions, not to discover what to do.

The model knows docs URLs but struggles with SemVer. The agent fetched release-1.22 when it needed release-1.22.2. After reading the page, it realized "v1.22.0 is not v1.22.2" but never attempted to fetch the version-specific page. It treated release-1.22 as "close enough."

Documentation hardcodes led the agent astray. The migration guide hardcoded @1.22.1 in all npm install examples. In 4 of 5 bare runs, the agent faithfully followed these examples, installing v1.22.1 instead of the requested v1.22.0 (in the v1.22.0 scenario variant).

Hypothesis-driven fixes

We used Dev Proxy, an API simulator, to intercept the agent's requests to Microsoft Learn and return modified pages. That let us test documentation changes before committing them to live docs. Several hypotheses were wrong. Had we shipped those changes without testing, we would have degraded the experience.

Fix 1: Rename release notes pages to use full version. Replacing release-1.22.md with release-1.22.0.md, and updating the version number in the contents, made the agent recognize that v1.22.0 and v1.22.2 are different pages. In the April baseline, none of the 4 runs that read the release notes followed up with the 1.22.2 page. When we reran the scenario in September against the live docs, 4 of 5 runs did. Shipped as PR #10855.

Fix 2: Add a warning to invalidate the agent's manual plan. A tip saying "you can also use CLI for Microsoft 365" didn't change the agent's plan. In the April baseline, no run used the CLI. In our simulations, a warning saying "manually updating package.json alone will result in build failures" got the agent to discover and use the CLI on its own, with no prompt engineering from the developer. Shipped as PR #10921.

Fix 3: Rewrite the migration guide. The migration guide contained step-by-step npm commands and JSON snippets. In our simulations, those actionable code blocks pulled the agent back to manual steps, even with the warning in place. We rewrote the migration guide as a conceptual page (describing what changes, not how to apply them manually), directing developers to the CLI. Shipped in the same PR.

Once both changes were live, the September rerun showed the combined effect. Every bare run discovered and used the CLI without being told, and configuration correctness reached 83/85, up from 30/80 in April. The criteria set gained one configuration criterion between the two runs, so read this as a before-and-after pattern, not an exact delta.

CLI for Microsoft 365: the highest-value tool

CLI for Microsoft 365 is an open source CLI whose spfx project upgrade command generates the version-specific changes an incremental upgrade needs. When we explicitly told the agent to use it, dependency currency and configuration correctness improved dramatically:

Classification Baseline + CLI for Microsoft 365
Prerequisite 5/5 5/5
Idiomatic use 6/10 6/10
Dependency currency 34/50 49/50
Configuration correctness 30/80 75/80

Notice, that idiomatic use didn't move. The CLI encodes the upgrade mechanics, not the lint rules and patterns that make the code look current.

We then tested whether the CLI report format affected the outcome, using the v1.22.0 variant of the scenario:

Format Idiomatic Deps Config Avg tokens
Markdown (default) 8/10 42/50 72/85 888K
Markdown, summary first 9/10 50/50 85/85 1,380K
JSON 9/10 50/50 85/85 1,133K
JSONL 7/10 43/50 84/85 2,927K

JSON was the clear winner. It tied the summary-first Markdown report for the best quality while using about 18% fewer tokens. Of the two structured formats, it beat JSONL on every dimension at less than half the token cost. The structured format removes parsing ambiguity, and the resolutionType field tells the agent exactly how to apply each change.

Harness findings

Not every finding belongs to the technology you're evaluating. While investigating, we noticed an opportunity in how the harness retrieves docs. Microsoft Learn can return pages directly as Markdown, which gives agents cleaner content than converting the full HTML page. We shared this with the GitHub Copilot teams, who added Accept: text/markdown support to GitHub Copilot Chat and GitHub Copilot CLI. When harness behavior affects how agents use your technology, share it with the harness vendor. It's part of your agent experience too.

What shipped

  • Release notes pages updated to full patch versions (PR #10855)
  • Warning callout and rewritten migration guide (PR #10921)
  • Accept: text/markdown support in GitHub Copilot Chat and GitHub Copilot CLI

Two doc changes and one harness fix, and none of them needed a new extension. They fixed the sources the agent already relied on, so every developer benefits without installing anything.

The readouts behind this example

The tables and results in this appendix come from three readouts. Each folder contains the readout, the per-run results, and the trajectories:

Each readout includes readout.json, which holds the complete data and is easiest to explore in VS Code with the Scope extension. The April readouts also include a readout.md that renders directly on GitHub.