Grader: diff-empty

diff-empty passes when the stimulus produced no changes to the workspace — the working tree at the end of the run is identical to its starting state. This is the canonical check for no-op evals: did the agent correctly decide that nothing needed changing?

Like the other diff graders, it works for both environment.files and environment.git starting states (as well as --workspace), and diffs the final workspace against a baseline commit recorded in a private, throwaway git index just before the agent ran. Because the diff is git-tracked, the workspace’s own .git directory is excluded and .gitignore rules are honored, so ignored build output never makes the diff look non-empty.

“No changes” means no tracked changes

diff-empty only sees git-tracked changes. If the stimulus workspace ships a .gitignore (e.g. *.log, dist/, secrets/), an agent that writes only to ignored paths still produces an empty diff — so diff-empty passes even though files changed. Don’t rely on it to prove the agent made no changes to ignored paths; see Limitations.

Taxonomy

Property	Value
Determinism	complex-static
Cost	low
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: diff-empty

diff-empty takes no config.

Behavior

1. Computes the unified diff between the pre-run baseline commit and the final workspace.
2. Passes when the diff is empty; fails when any file was added, modified, or deleted.

Limitations

• Requires the workspace diff to be available. vally eval runs the agent against a local workspace (--workspace, environment.files, environment.git) and captures the diff during the run, so it is always available there. Relocating backends (e.g. Docker/remote) own their own workspace — they must grade where that workspace lives, or persist the diff for a later re-grade — otherwise the grader fails with a clear message.
• Re-grading needs the recorded diff. The baseline is a snapshot of the workspace taken before the agent runs, so it only exists during the vally eval pass. That pass persists the resulting diff to the JSONL output — inline (trajectory.diff) or as a sidecar file (trajectory.diffPath) — so vally grade can re-grade a saved run; point it at the run directory with --run-dir so sidecar paths resolve. Only inputs that never recorded a diff (e.g. ATIF or session-log inputs not produced by a diff-grader eval) have no diff to grade, and the grader fails with a clear message.
• The diff is trajectory-level (start of run → end of run).
• Ignored paths are invisible to the diff. The snapshot honors the workspace’s own .gitignore, and a nested .git directory is recorded as a gitlink rather than its contents. A change to an ignored path (e.g. dist/, .env, logs) or to a file inside a nested repo will not register, so diff-empty can pass even though such a file changed.
• Requires git on the PATH. The baseline snapshot shells out to git. If git is missing or the snapshot can’t be captured, the run still proceeds and the grader fails with a clear message rather than aborting the eval.

Evidence examples

✔ Workspace diff is empty
✘ Workspace diff is not empty — the stimulus changed the workspace
✘ No workspace diff is available for this run. …