CLI: oracle

Usage

vally oracle --eval-spec eval.yaml [options]
vally oracle --eval-spec eval.yaml --stimulus <name>

Description

Re-grade a stimulus against its golden patch (a reference-solution diff) without running an agent. oracle materializes the stimulus’s starting environment into a fresh workspace, applies the golden patch, and runs the eval spec’s graders against the result.

Use it to:

• Verify the harness. If a grader doesn’t pass on the correct answer, the grader is wrong. Run oracle to catch that before shipping.
• Gate CI before publishing. Require every task to pass an oracle run so you never ship an unscorable benchmark.

To re-grade an existing agent trajectory (e.g. after tweaking grader configs) without re-running the agent, use vally grade instead.

Re-grading the same inputs is deterministic for static graders, so an oracle run is safe to gate CI on.

A stimulus declares its golden patch in eval.yaml:

stimuli:
  - name: fix-the-bug
    prompt: "Fix the off-by-one error in pagination"
    golden_patch:
      path: solutions/fix-the-bug.patch # or use `inline:` for an embedded diff
    graders:
      - type: run-command
        config:
          command: "npm test"

Golden trajectory (validating behavior graders)

A golden patch only captures the final file state, so oracle synthesizes an empty trajectory (no events, zeroed metrics). Graders that inspect the agent’s behavior — events, metrics, or the conversation transcript — therefore have nothing to check and are rejected (see below).

To validate those graders offline, add a golden_trajectory: a reference ATIF trajectory (fake or real, inline or a path to a JSON file). Oracle then grades against its real events and metrics instead of the empty synthesized one. You can hand-author the events you expect a passing run to produce and confirm the behavior graders agree — no agent run required.

stimuli:
  - name: runs-the-tests
    prompt: "Run the test suite"
    golden_trajectory:
      path: solutions/runs-the-tests.trajectory.json
    graders:
      - type: tool-calls
        config:
          required: ["bash"]

When hand-authoring the ATIF, keep each observation in the same agent step as the tool_call it answers: tool results are matched to calls per step, so a source_call_id referencing a call in a different step resolves to tool "unknown" and a tool-calls grader won’t count it. See the complete inline example in the eval-spec reference.

A stimulus may declare a golden_patch, a golden_trajectory, or both: the patch drives file-state graders (against the patched workspace) while the trajectory drives behavior/metric graders. With a golden trajectory that has user-delimited turns (at least one user step), event graders may also be scoped to a specific turn. The diff graders (diff-contains, diff-not-contains, diff-empty) remain rejected with turn: oracle has one golden patch, not one workspace diff per agent turn. A user-less (autonomous-agent) trajectory has no turn boundaries, so all turn-scoped graders are rejected against it. The --no-golden-input baseline withholds every golden input (patch, trajectory, and custom metrics), so behavior graders correctly fail on the empty baseline too.

Golden custom metrics (validating custom-metrics graders)

The custom-metrics grader reads a JSON file from the workspace (default custom_metrics.json). Since oracle runs no agent, that file only exists if the golden patch writes it. Add a golden_custom_metrics (fake or real, inline object or a path to a JSON file) and oracle materializes it into the workspace at every path a custom-metrics grader reads:

stimuli:
  - name: tests-pass
    prompt: "Make the tests pass"
    golden_custom_metrics:
      inline:
        tests_failed: 0
    graders:
      - type: custom-metrics
        config:
          assertions:
            - metric: tests_failed
              equals: 0

Like the patch and trajectory, it is withheld by --no-golden-input.

Options

Flag	Type	Required	Description
--eval-spec, -e <path>	string	Yes	Path to eval spec file (grader + golden-input source of truth)
--stimulus <name>	string	No	Grade a single stimulus. Default: every stimulus that declares a golden_patch, golden_trajectory, or golden_custom_metrics
--no-golden-input	boolean	No	Grade the baseline with all golden inputs withheld — a sanity check that should fail
--output jsonl	string	No	Emit graded results as JSONL (one outcome per line)
--judge-model <model>	string	No	Model for LLM judge graders
--judge-reasoning-effort <level>	string	No	Reasoning effort for the judge model: low, medium, high, or xhigh
--grader-plugin <specifier>	string	No	Grader plugin to load (npm package name or local path). Repeatable.
--keep-workspace	boolean	No	Keep the materialized workspace (a temp dir) and print its path (for debugging)
--workspace <path>	string	No	Materialize into this directory instead of a temp dir; contents are preserved. Requires a single stimulus.
--verbose	boolean	No	Show detailed grader evidence

Golden patch reference for graders

When the golden patch is resolved, its diff text is passed to every grader. The prompt LLM judge includes it as a Reference Solution section so the judge can compare the agent’s work against the known-good answer — except in oracle mode, where the applied patch is also reported as the agent output. Handing the judge identical reference and output is circular, so the Reference Solution section is omitted whenever the two are the same.

Oracle works best with static / file-based graders

Oracle applies the golden patch to files, so file-state graders (file-exists, file-contains, run-command, …) verify the harness directly. An LLM judge under oracle instead sees the patch diff as the agent output and scores it against the rubric — there is no separate reference solution to compare against, so it’s a weaker harness-verification signal. To have an LLM judge compare a real agent run against the golden patch, use vally grade instead, where the reference and the agent output differ.

Grader support under oracle

Oracle grades a single materialized workspace. Which graders are supported depends on whether the stimulus declares a golden_trajectory:

Grader category	Grades	Requires golden_trajectory?	Turn-scopable?
File-state (file-exists, file-contains, run-command)	The patched workspace (final file state)	No	No
Diff (diff-contains, diff-not-contains, diff-empty)	The golden patch text itself	No	No
Output-scoped (output-contains, output-matches, and their negations)	The trajectory output	Yes	Yes
Trajectory / behavior / metric / transcript (completed, exit-success, tool-calls, skill-invocation, max-repeat, token-budget, tool-call-count, turn-count, error-count, wall-time, transcript-contains, transcript-not-contains, transcript-matches, transcript-not-matches)	The agent’s events, metrics, and transcript	Yes	Yes

Without a golden_trajectory, oracle’s synthesized trajectory has no events, zeroed metrics, and no assistant messages, so output-scoped and trajectory-scoped graders would grade an empty run — they are rejected, along with any grader scoped to a specific turn. With a golden_trajectory, those graders grade its real events and metrics, and may be scoped to a turn when the trajectory has user-delimited turns (at least one user step); turn-scoped graders are rejected against a user-less trajectory, which has no turn boundaries. The three diff graders (diff-contains, diff-not-contains, diff-empty) also cannot be turn-scoped under oracle: a golden trajectory supplies events per turn, but oracle has only one cumulative golden patch. (A workspace-reading grader such as file-exists still cannot be turn-scoped, since it grades final state.) Copying a typical eval spec into oracle surfaces a clear error naming the offending graders and pointing at golden_trajectory.

Exit codes

The meaning of a “good” run is inverted for the baseline sanity check, which is expected to fail:

Code	Apply mode (default)	Baseline (--no-golden-input)
0	All graders passed	Graders correctly failed (the sanity check holds)
1	A grader failed, or an error occurred	A stimulus unexpectedly passed, or an error occurred

Examples

# Verify the harness: apply each golden patch and assert the graders pass
vally oracle --eval-spec eval.yaml

# Oracle-grade a single stimulus, verbose
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --verbose

# Baseline sanity check: graders should FAIL without the patch
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --no-golden-input

# CI gate, machine-readable
vally oracle --eval-spec eval.yaml --output jsonl > oracle.jsonl

# Inspect the materialized + patched workspace
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --keep-workspace

# Materialize into a directory you control (preserved after the run)
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --workspace ./oracle-ws

Re-grading a real agent run

oracle verifies graders against the correct answer (the golden patch). To re-grade an existing agent trajectory without re-running the agent — e.g. after tweaking grader configs — use vally grade instead.

Output format

✅ fix-the-bug [oracle] (1/1 graders passed)

Score: 100.0% | PASSED

Baseline (--no-golden-input) — expected to fail, so a correct failure is reported as OK:

✅ fix-the-bug [baseline (no golden inputs)] (0/1 graders passed, 1 failed)
  ✗ [run-command] Command "npm test" exited with code 1

Score: 0.0% | baseline OK (graders correctly FAILED on the baseline (no golden inputs))