Grader: diff-contains

The diff graders assert on the working-tree diff a stimulus produced — the cumulative change from the workspace’s starting state (after environment setup, before the agent ran) to its ending state. This is ideal for code-fix and PR-style evals where you want to validate the patch directly rather than re-reading individual files.

The diff works for both environment.files (inline) and environment.git (worktree or clone) starting states, as well as --workspace. When a diff grader is configured, the run records a baseline commit of the workspace in a private, throwaway git index just before the agent runs, then diffs the final workspace against it. Because the diff is git-tracked, the workspace’s own .git directory is excluded and .gitignore rules are honored (so build output such as node_modules/ never pollutes the diff).

The diff is computed once, right after the agent finishes, and stored on the trajectory as an artifact — so it appears in the run’s JSONL output and every diff grader reuses the same diff instead of recomputing it. The baseline snapshot lives in a private, owner-only temp directory outside the agent’s workspace and is removed after grading, while the diff artifact is kept in the output.

Taxonomy

Property	Value
Determinism	complex-static
Cost	low
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: diff-contains
    config:
      pattern: "def divide"

Field	Type	Required	Description
pattern	string	one of these	JavaScript regex matched against the unified diff text
value	string	one of these	Literal substring matched verbatim against the diff text
negate	boolean	No	When true, passes when the pattern is NOT present in the diff

Provide exactly one of pattern (regex, the default) or value (literal-string convenience).

Negated variant

Use diff-not-contains as a shorthand for negate: true — handy for restraint checks (“the agent must not have touched this”):

graders:
  - type: diff-not-contains
    config:
      pattern: "secrets\\."

This is equivalent to type: diff-contains with negate: true.

Restraint checks don’t see ignored paths

diff-not-contains only sees git-tracked changes. If the stimulus workspace ships a .gitignore (e.g. *.log, dist/, secrets/), an agent that writes only to ignored paths produces an empty diff — so a restraint or security check (diff-not-contains "secrets\\.") passes even though the file changed. Don’t rely on diff graders to prove the agent avoided an ignored path; see Limitations.

Behavior

1. Validates the regex pattern up front (fails immediately if invalid).
2. Computes the unified diff between the pre-run baseline commit and the final workspace.
3. Tests the diff text against the regex (or literal value).

Passes when the pattern matches the diff (or does NOT match, when negated). The match is against the standard unified-diff text, so added lines are prefixed with +, removed lines with -, and file headers read a/<path> / b/<path>.

Regex flags

Use the (?flags) prefix syntax for inline regex flags:

• (?i)pattern — case-insensitive
• (?m)pattern — multiline (^/$ match line start/end)
• (?s)pattern — dotAll (. matches newlines)

Limitations

• Requires the workspace diff to be available. vally eval runs the agent against a local workspace (--workspace, environment.files, environment.git) and captures the diff during the run, so it is always available there. Relocating backends (e.g. Docker/remote) own their own workspace — they must grade where that workspace lives, or persist the diff for a later re-grade — otherwise the grader fails with a clear message.
• Re-grading needs the recorded diff. The baseline is a snapshot of the workspace taken before the agent runs, so it only exists during the vally eval pass. That pass persists the resulting diff to the JSONL output — inline (trajectory.diff) or as a sidecar file (trajectory.diffPath) — so vally grade can re-grade a saved run; point it at the run directory with --run-dir so sidecar paths resolve. Only inputs that never recorded a diff (e.g. ATIF or session-log inputs not produced by a diff-grader eval) have no diff to grade, and the grader fails with a clear message.
• Per-turn diff scoping is not supported and is out of scope for this release. The diff is always trajectory-level (full run start → end). Supporting per-turn diffs would require the pipeline to capture a baseline snapshot at the start of each turn and expose a turn-scoped grading surface — infrastructure that doesn’t exist yet. This is tracked in #560.
• Ignored paths are invisible to the diff. Snapshots honor the workspace’s own .gitignore, and a nested .git directory is recorded as a gitlink rather than its contents. Changes to ignored paths (e.g. dist/, .env, logs) or files inside a nested repo therefore never appear in the diff. Assert only on tracked files; to match an otherwise-ignored path, ensure it is not ignored in the workspace.
• Requires git on the PATH. The baseline snapshot shells out to git. If git is missing or the snapshot can’t be captured, the run still proceeds and the grader fails with a clear message rather than aborting the eval.

Evidence examples

✔ Pattern /def divide/ found in workspace diff
✘ Pattern /def divide/ not found in workspace diff
✘ Invalid regex pattern: [bad. Error: Unterminated character class
✘ No workspace diff is available for this run. …