Grader: diff-contains

The diff graders assert on the working-tree diff a stimulus produced — the cumulative change from the workspace’s starting state (after environment setup, before the agent ran) to its ending state. This is ideal for code-fix and PR-style evals where you want to validate the patch directly rather than re-reading individual files.

The diff works for both environment.files (inline) and environment.git (worktree or clone) starting states, as well as --workspace. When a diff grader is configured, the run records a baseline commit of the workspace in a private, throwaway git index just before the agent runs, then diffs the final workspace against it. Because the diff is git-tracked, the workspace’s own .git directory is excluded and .gitignore rules are honored (so build output such as node_modules/ never pollutes the diff).

Without turn, the diff is computed once after the agent finishes and covers the full run. For a multi-turn stimulus, the three static diff graders also support the existing 0-based top-level turn field. turn: N selects only changes made after prompt N was sent and before that turn completed; it does not select the cumulative patch through turn N. Vally records a private baseline after each completed turn and persists each turn diff alongside the cumulative artifact.

Taxonomy

Property	Value
Determinism	complex-static
Cost	low
Reference	reference-free
Temporal scope	trajectory-level
Score kind	code

Config

graders:
  - type: diff-contains
    config:
      pattern: "def divide"

Field	Type	Required	Description
pattern	string	one of these	JavaScript regex matched against the unified diff text
value	string	one of these	Literal substring matched verbatim against the diff text
negate	boolean	No	When true, passes when the pattern is NOT present in the diff

Provide exactly one of pattern (regex, the default) or value (literal-string convenience).

Per-turn diff

turns:
  - "Inspect the project but do not edit files."
  - "Implement the requested fix."
graders:
  - type: diff-empty
    turn: 0
  - type: diff-contains
    turn: 1
    config:
      value: "requested fix"

turn is only supported by diff-contains, diff-not-contains, and diff-empty; it remains unsupported for file-based graders, agent scope:, and prompt/panel LLM evidence selecting diff. An executor must advertise turn-completion support for a multi-turn scoped diff. A turn: 0 check uses the same diff as the full run only when both the stimulus and recorded trajectory are single-turn; otherwise Vally requires a dedicated turn diff.

Negated variant

Use diff-not-contains as a shorthand for negate: true — handy for restraint checks (“the agent must not have touched this”):

graders:
  - type: diff-not-contains
    config:
      pattern: "secrets\\."

This is equivalent to type: diff-contains with negate: true.

Restraint checks don’t see ignored paths

diff-not-contains only sees git-tracked changes. If the stimulus workspace ships a .gitignore (e.g. *.log, dist/, secrets/), an agent that writes only to ignored paths produces an empty diff — so a restraint or security check (diff-not-contains "secrets\\.") passes even though the file changed. Don’t rely on diff graders to prove the agent avoided an ignored path; see Limitations.

Behavior

1. Validates the regex pattern up front (fails immediately if invalid).
2. Computes the unified diff between the pre-run baseline commit and the final workspace.
3. Tests the diff text against the regex (or literal value).

Passes when the pattern matches the diff (or does NOT match, when negated). The match is against the standard unified-diff text, so added lines are prefixed with +, removed lines with -, and file headers read a/<path> / b/<path>.

Regex flags

Use the (?flags) prefix syntax for inline regex flags:

• (?i)pattern — case-insensitive
• (?m)pattern — multiline (^/$ match line start/end)
• (?s)pattern — dotAll (. matches newlines)

Limitations

• Requires the workspace diff to be available. vally eval runs the agent against a local workspace (--workspace, environment.files, environment.git) and captures the diff during the run, so it is always available there. Relocating backends (e.g. Docker/remote) own their own workspace — they must grade where that workspace lives, or persist the diff for a later re-grade — otherwise the grader fails with a clear message.
• Re-grading needs the recorded diff. The baseline is a snapshot of the workspace taken before the agent runs, so it only exists during the vally eval pass. That pass persists the resulting diff to the JSONL output — inline (trajectory.diff) or as a sidecar file (trajectory.diffPath) — so vally grade can re-grade a saved run; point it at the run directory with --run-dir so sidecar paths resolve. Only inputs that never recorded a diff (e.g. ATIF or session-log inputs not produced by a diff-grader eval) have no diff to grade, and the grader fails with a clear message.
• Turn evidence must be recorded. Re-grading a multi-turn scoped diff requires the corresponding recorded turn diff (inline or a sidecar resolved with --run-dir). Vally never falls back to the cumulative diff: a missing, malformed, or capture-failed turn artifact fails the grader rather than producing a potentially misleading pass.
• Ignored paths are invisible to the diff. Snapshots honor the workspace’s own .gitignore, and a nested .git directory is recorded as a gitlink rather than its contents. Changes to ignored paths (e.g. dist/, .env, logs) or files inside a nested repo therefore never appear in the diff. Assert only on tracked files; to match an otherwise-ignored path, ensure it is not ignored in the workspace.
• Requires git on the PATH. The baseline snapshot shells out to git. If git is missing or the snapshot can’t be captured, the run still proceeds and the grader fails with a clear message rather than aborting the eval.

Evidence examples

✔ Pattern /def divide/ found in workspace diff
✘ Pattern /def divide/ not found in workspace diff
✘ Invalid regex pattern: [bad. Error: Unterminated character class
✘ No workspace diff is available for this run. …