TL;DR
A benchmark for reference-guided web development. Working software serves as both the specification and verifier. Coding agents inspect a reference app, infer its behavior, restore it in an editable app, and validate the result.
Apps become verifiable SWE tasks. ProgramDistill turns replayable behaviors into repair tasks while preserving their prerequisite structure.
Difficulty scales with dependencies. Restoration depth controls how many behaviors must be restored together.
Across 26 applications, the pipeline discovers 1,975 replay-verified behaviors and constructs 4,063 tasks.
From a working reference to working code
In practice, web development does not always begin with a complete specification. The intended behavior may instead be demonstrated by an earlier product version, an interactive prototype, another application, or a working reference.
A developer can interact with that reference to fill in the missing details. They can observe what changes after an action, which behaviors depend on earlier state, what persists, and how a workflow unfolds. The goal is then to reproduce that behavior in the application being developed.
ProgramDistill turns this workflow into a benchmark. The agent can interact with the working reference but cannot inspect its source code. It edits a current application with missing functionality and is evaluated on whether the resulting application reproduces the reference behavior when the same workflow is executed. The implementation itself does not need to match the original source code.
A synthetic task generation method built for stateful web applications
To evaluate this workflow at scale, each task needs both a reproducible behavior and a reliable verifier. Constructing such tasks requires exploring a working application, establishing prerequisite state, identifying successful outcomes, and removing the corresponding implementation without breaking earlier behaviors.
ProgramDistill automates this process with the mine-craft-patch pipeline, orchestrated by multiple LLM agents. Starting from a working application and its source code, the pipeline discovers behaviors, records them as replayable traces, and removes the source implementations responsible for those behaviors to create repair tasks.
Each stores browser actions, expected success signals, and an optional prerequisite link. Because the actions can be replayed and the expected signals checked automatically, the trace itself serves as an executable behavioral verifier.
-
Mine
Mining agents explore the application and record replayable traces. Verified traces can seed the discovery of dependent behaviors, and only reproducible traces are retained.
-
Craft
Crafting agents remove the implementation of selected behaviors. Build and replay checks ensure that prerequisites still work, the target behavior fails, and the gold patch restores it.
-
Patch
Coding agents repair the current application by interacting with the working reference, without access to its source code or the gold patch. Replay then evaluates the repair.
The same replay mechanism checks the original app, the masked app, and the repair. It executes the recorded actions and checks the expected signals without LLM intervention.
Walk through the pipeline!
Choose one of the three examples below and follow the mine-craft-patch pipeline from mined behaviors to masks and saved repair runs. The explorer shows recorded examples rather than launching new agent runs. Each application is repository-based, with its GitHub source linked below the example list.
Loading interactive examples...
Choose an application
Each application opens one recorded repair example with a different agent and task.
Explore the workflow
Move through how its behavior is mined, turned into a task, and repaired.
Product workflows become the curriculum
Product features rarely stand alone. A card must exist before it can be moved or edited, and creating that card may first require a board and a list. Reproducing a later behavior therefore means recreating the state established by earlier ones.
Mining preserves these relationships as prerequisite lineages. Each verified trace can build on the state produced by an earlier trace, forming branching dependency trees across applications.
Crafting turns this structure into two kinds of task. An atomic task removes one behavior while leaving its prerequisites intact. A cumulative task combines accepted masks along one lineage, requiring the agent to restore several connected behaviors together.
| Task type | What the agent restores | Tasks |
|---|---|---|
| Atomic | One missing behavior | 2,862 |
| Cumulative | Several behaviors along one prerequisite lineage | 1,201 |
Partial-Application Reconstruction
Nine coding agents tackle ProgramDistill-300, a suite of 300 tasks across 26 applications at restoration depths 1 through 8. Each starts from a mostly working app with missing functionality. Under binary scoring, a cumulative task succeeds only if every repair target in the workflow passes.
Astra solves every depth-1 task, establishing a strong baseline for individual repair solvability. Yet success drops to 64.0% at depth 8, when multiple dependent behaviors must be restored together. Every other agent retains less than half of its depth-1 performance at depth 8.
The strongest agent observes more and edits less
Astra is distinctive for an observation-intensive, edit-light workflow. It performs substantially more reference and current-app observations than the other models, with the difference particularly pronounced for current-app observation, while making the fewest edit/write steps. It averages 96.3 current-app observations per trajectory, roughly twice Sol’s 45.8, alongside only 9.9 edit/write steps. Its trajectories are thus characterized by extensive behavioral checking, especially of its own implementation, followed by comparatively selective code changes. More broadly, the strongest-performing model differs not only in repair accuracy but also in how it allocates effort across observation, validation, and editing.
Deeper tasks receive less checking per behavior
As restoration depth increases, reconstruction burden grows roughly linearly as agents must recover more interdependent behaviors along a prerequisite lineage. At depth 8, for example, eight behaviors must be restored in sequence, with later behaviors relying on state established by earlier ones.
From depth 1 to depth 8, the amount of code to restore grows by more than 9×, while the total number of actions across target behavior traces grows by more than 10×. Agent effort, however, does not keep pace. Although agents do more work overall, observation and editing effort per repair target decline as tasks deepen, with observation shrinking particularly sharply. Final patches also leave more of the masked implementation unrestored.
Example: How an agent completes a depth-8 repair
In vdevired/trello-clone, Claude Opus 5 successfully completes a depth-8 task spanning eight dependent workflow stages, from login and project setup through board and card operations, movement, and comment editing.
The repair takes 317 agent steps. Rather than solving the task in a single pass, the agent repeatedly inspects the reference, edits the current implementation, checks the repaired application, and returns to remaining mismatches. The trajectory looks much more like iterative development against a live reference than one-shot code generation.
Full-Application Reconstruction
Full reconstruction removes the existing implementation as a starting point. The agent receives a minimal executable scaffold, a product-level capability description, and browser access to the working reference. It must rebuild the application, which is then evaluated using the replay-verified behaviors discovered during mining.
Across 12 applications, each agent is evaluated on 590 individual behaviors and 413 cumulative workflows. Individual tests measure behavior recovery, while a cumulative workflow passes only when all of its targets pass.
| Agent | Individual behaviors | Cumulative workflows |
|---|---|---|
| GPT-6 Astra (max) | 58.98% | 49.15% |
| Claude Opus 5 (max) | 42.03% | 28.81% |
| GPT-5.6 Sol (max) | 33.39% | 21.07% |
Even the strongest agent recovers only 49.2% of cumulative workflows. All three agents recover a larger share of individual behaviors than cumulative workflows, showing that implementing useful pieces does not guarantee that those pieces work together. These runs average roughly 700 agent steps and reach as high as 1,921 steps.
Failure analysis: What agents miss
Reference exploration remains a major bottleneck. The failure analysis covers 977 failed atomic behaviors across 36 reconstruction runs. All percentages below are calculated over these 977 failures.
- Reference coverage
- 59.2% involve behaviors that were never observed in the reference.
- Faithful implementation
- 27.9% produce the wrong state, route, or result, 11.1% have the wrong observable form, and 1.8% were observed but not implemented.
These failures point to gaps in both reference exploration and faithful implementation. The largest category consists of behaviors the agent never observed, but the remaining failures show that observing a behavior is not enough to reproduce it correctly.
An application that runs is not necessarily an application that matches the reference. Syntax checks, successful backend requests, or selected text matches can miss an incorrect state transition or a workflow that was never rechecked after the final relevant edit. This echoes partial repair, where current-app observation per target falls as tasks deepen.
Example: A working feature can still be wrong
In one reconstruction of knowankit/trello-clone, Astra implements card dragging, and the card visibly moves. However, after the same center drop, the resulting card order differs from the working reference.
The interaction works, but the resulting state is wrong. The error remains because after its final relevant edit, the agent validates other board interactions instead of replaying the exact drag workflow that would expose the mismatch.
Full reconstruction also shows that more browser feedback alone is not enough. Claude Opus 5 receives substantially more browser-state feedback than Astra, yet recovers fewer behaviors. What matters is not only how much of the reference the agent inspects, but whether it identifies the right details, implements them faithfully, and rechecks the relevant workflow after making changes.
What’s next?
Train reference-guided agents with a curriculum. Use restoration depth to organize replay-verifiable tasks from simpler repairs to deeper workflows, with agent trajectories for distillation and replay-based rewards for reinforcement learning.
Extend ProgramDistill to multimodal agents. Add screenshot-based reference interaction and evaluate visual fidelity alongside behavioral correctness.