CLI: oracle

Usage

vally oracle --eval-spec eval.yaml [options]
vally oracle --eval-spec eval.yaml --stimulus <name>

Description

oracle materializes a stimulus’s starting environment and golden inputs without running an agent. By default, it then runs the eval spec’s graders against that reference solution. With --skip-grade, it stops after materialization without running graders.

Use it to:

• Verify the harness. If a grader doesn’t pass on the correct answer, the grader is wrong. Run oracle to catch that before shipping.
• Gate CI before publishing. Require every task to pass an oracle run so you never ship an unscorable benchmark.
• Materialize a golden solution. Use --skip-grade to populate a workspace or emit its trajectory without running the configured graders. Use --output atif when another tool needs a standalone ATIF document.

To re-grade an existing agent trajectory (e.g. after tweaking grader configs) without re-running the agent, use vally grade instead.

Re-grading the same inputs is deterministic for static graders, so an oracle run is safe to gate CI on.

A stimulus declares its golden patch in eval.yaml:

stimuli:
  - name: fix-the-bug
    prompt: "Fix the off-by-one error in pagination"
    golden_patch:
      path: solutions/fix-the-bug.patch # or use `inline:` for an embedded diff
    graders:
      - type: run-command
        config:
          command: "npm test"

Golden command (running a reference script)

Use golden_command when the correct solution must execute a workflow rather than only change files. Oracle runs the string through the platform shell in the materialized workspace after applying golden_patch and before writing explicit golden custom metrics or running graders:

stimuli:
  - name: reproduces-ui-bug
    prompt: "Reproduce the bug and collect the interaction log"
    golden_command: node scripts/reproduce-ui-bug.mjs
    graders:
      - type: file-contains
        config:
          path: interaction.log
          value: opened-settings

The command must exit successfully within five minutes. A startup failure, nonzero exit, or timeout fails oracle. Relative paths are workspace-relative, so scripts and files may be supplied by agent_environment.files, an agent_environment.git checkout, or an earlier golden_patch; external tools and runtimes must be installed by agent_environment.commands or already be available on PATH. The command must remain in the foreground and must not intentionally detach or background child processes. Command output does not become trajectory output; add a golden_trajectory when graders need behavior, metric, output, or transcript evidence.

For diff graders and LLM judges using diff evidence, Oracle captures the cumulative workspace diff after the patch and command are materialized; golden custom metrics are not included. --diff-output <path> writes the captured unified diff to a file for integrations that consume ATIF separately.

When a command-only stimulus uses a prompt or panel grader whose selected evidence includes neither diff nor repo, Oracle emits an advisory warning: command-created workspace changes are not visible in trajectory-only evidence. Select diff or repo when the judge should inspect those changes, or provide a golden_patch or golden_trajectory. Oracle does not reject the configuration or change its evidence automatically because it cannot infer the command’s intent.

Golden trajectory (validating behavior graders)

A golden patch only captures the final file state, so oracle synthesizes an empty trajectory (no events, zeroed metrics). Graders that inspect the agent’s behavior — events, metrics, or the conversation transcript — therefore have nothing to check and are rejected (see below).

To validate those graders offline, add a golden_trajectory: a reference ATIF trajectory (fake or real, inline or a path to a JSON file). Oracle then grades against its real events and metrics instead of the empty synthesized one. You can hand-author the events you expect a passing run to produce and confirm the behavior graders agree — no agent run required.

stimuli:
  - name: runs-the-tests
    prompt: "Run the test suite"
    golden_trajectory:
      path: solutions/runs-the-tests.trajectory.json
    graders:
      - type: tool-calls
        config:
          required: ["bash"]

When hand-authoring the ATIF, keep each observation in the same agent step as the tool_call it answers: tool results are matched to calls per step, so a source_call_id referencing a call in a different step resolves to tool "unknown" and a tool-calls grader won’t count it. See the complete inline example in the eval-spec reference.

A stimulus may combine a golden_patch, golden_command, and golden_trajectory: the patch and command drive file-state graders (against the materialized workspace) while the trajectory drives behavior/metric graders. With a golden trajectory that has user-delimited turns (at least one user step), event graders may also be scoped to a specific turn. The diff graders (diff-contains, diff-not-contains, diff-empty) remain rejected with turn: oracle has one golden patch, not one workspace diff per agent turn. A user-less (autonomous-agent) trajectory has no turn boundaries, so all turn-scoped graders are rejected against it.

The --no-golden-input baseline withholds every golden input (patch, command, trajectory, and custom metrics) and grades the resulting empty workspace/trajectory as a negative control — see Baseline (negative control) below.

Golden custom metrics (validating custom-metrics graders)

The custom-metrics grader reads a JSON file from the workspace (default custom_metrics.json). Since oracle runs no agent, that file only exists if the golden patch or golden command writes it. Add a golden_custom_metrics (fake or real, inline object or a path to a JSON file) and oracle supplies it at every path a custom-metrics grader reads:

stimuli:
  - name: tests-pass
    prompt: "Make the tests pass"
    golden_custom_metrics:
      inline:
        tests_failed: 0
    graders:
      - type: custom-metrics
        config:
          assertions:
            - metric: tests_failed
              equals: 0

Like the patch and trajectory, it is withheld by --no-golden-input.

Options

Flag	Type	Required	Description
--eval-spec, -e <path>	string	Yes	Path to eval spec file
--stimulus <name>	string	No	Select a single stimulus. Default: every stimulus that declares a golden_patch, golden_command, golden_trajectory, or golden_custom_metrics
--no-golden-input	boolean	No	Materialize or grade the baseline with all golden inputs withheld; grading it is a sanity check that should fail
--output jsonl | atif	string	No	Emit JSONL outcomes, or one bare ATIF document. ATIF requires --skip-grade and exactly one selected stimulus.
--run-id <id>	string	No	Stable identifier placed on every JSONL outcome. Defaults to one generated UUID per invocation. Does not modify ATIF output.
--skip-grade	boolean	No	Materialize and exit without grading
--diff-output <path>	string	No	Write the cumulative materialized workspace diff; requires exactly one selected stimulus
--judge-model <model>	string	No	Model for LLM judge graders
--judge-reasoning-effort <level>	string	No	Non-empty, model-supported reasoning effort for the judge model. Common values include none, minimal, low, medium, high, xhigh, and max
--grader-plugin <specifier>	string	No	Grader plugin to load (npm package name or local path), additive with project plugins.grader. Repeatable.
--keep-workspace	boolean	No	Keep the materialized workspace (a temp dir) and print its path (for debugging)
--workspace <path>	string	No	Materialize into this directory instead of a temp dir; contents are preserved. Requires a single stimulus.
--verbose	boolean	No	Show detailed grader evidence

Golden patch reference for graders

When the golden patch is resolved, prompt and panel graders that select golden_patch read it from a temporary reference file (evidence/golden-patch.patch), not a section of the prompt.

When oracle builds the trajectory from the patch, that patch is also the agent output. Comparing it against an identical reference would be circular, so Vally omits the separate reference file whenever the golden patch equals the output. If no selected evidence remains, the grader fails rather than scoring the rubric alone. Pair golden_patch with trajectory, diff, or repo; here, trajectory supplies the patch as the output being graded.

Oracle works best with static / file-based graders

Oracle applies the golden patch to files, so file-state graders (file-exists, file-contains, run-command, …) verify the harness directly. An LLM judge under oracle instead sees the patch diff as the agent output and scores it against the rubric — there is no separate reference solution to compare against, so it’s a weaker harness-verification signal. To have an LLM judge compare a real agent run against the golden patch, use vally grade instead, where the reference and the agent output differ.

Grader support under oracle

Oracle grades a single materialized workspace. Which graders are supported depends on whether the stimulus declares a golden_trajectory:

Grader category	Grades	Requires golden_trajectory?	Turn-scopable?
File-state (file-exists, file-contains, run-command)	The materialized workspace (final file state)	No	No
Diff (diff-contains, diff-not-contains, diff-empty)	The cumulative materialized workspace diff	No	No
Output-scoped (output-contains, output-matches, and their negations)	The trajectory output	Yes	Yes
Trajectory / behavior / metric / transcript (completed, exit-success, tool-calls, skill-invocation, system-event, max-repeat, token-budget, tool-call-count, turn-count, step-count, error-count, wall-time, transcript-contains, transcript-not-contains, transcript-matches, transcript-not-matches)	The agent’s events, metrics, and transcript	Yes	Yes

Without a golden_trajectory, oracle’s synthesized trajectory has no events, zeroed metrics, and no assistant messages, so output-scoped and trajectory-scoped graders would grade an empty run — in apply mode they are rejected, along with any grader scoped to a specific turn. With a golden_trajectory, those graders grade its real events and metrics, and may be scoped to a turn when the trajectory has user-delimited turns (at least one user step); turn-scoped graders are rejected against a user-less trajectory, which has no turn boundaries. The three diff graders (diff-contains, diff-not-contains, diff-empty) also cannot be turn-scoped under oracle: a golden trajectory supplies events per turn, but oracle has only one cumulative workspace diff. (A workspace-reading grader such as file-exists still cannot be turn-scoped, since it grades final state.) Copying a typical eval spec into oracle surfaces a clear error naming the offending graders and pointing at golden_trajectory.

With --no-golden-input, output and trajectory graders—including those with turn, scope, or both—run against the empty baseline. Workspace/diff scoping restrictions still apply.

Baseline (negative control)

--no-golden-input is a negative control: it withholds every golden input and verifies that the graders actually depend on the solution. A healthy grader should now fail — if it still passes, it isn’t really checking anything.

For file-state and behavior graders this is direct: withholding the golden patch and command leaves an unmodified workspace and an empty trajectory, so solution-dependent checks — file-exists, run-command, tool-calls with required:, skill-invocation with required:, and the like — run and fail. Each such failure is healthy, and the run exits 0 when every grader either fails, is N/A (below), or has no outstanding problem.

Absence assertions are reported N/A. Some graders pass precisely because nothing happened — a tool-calls or skill-invocation grader with only disallowed: (and no required:), a system-event grader with only disallowed:, negated output and file graders, diff-not-contains, diff-empty, max-repeat (an empty run repeats nothing), loop-outcome with should_loop: false (the default — an empty run didn’t loop), a custom-metrics grader whose assertions are all absent:, or a max-only metric-threshold grader. On the empty baseline there is nothing to withhold, so passing is correct, not a broken grader. Oracle reports these as N/A and excludes them from the control. A stimulus whose graders are all absence assertions (a “no action needed” task) is therefore reported N/A and passes.

A grader that passes on the baseline but does not assert absence (for example, output-matches with a pattern that matches empty output) is flagged as trivially passing and fails the run (exit 1) — the negative control caught a grader that doesn’t depend on the solution.

custom-metrics on the baseline

When a stimulus defines golden_custom_metrics, the baseline synthesizes an empty metrics file in place of the withheld golden data, so a custom-metrics value/presence assertion fails cleanly on the missing key — a healthy expected-fail, not a file-not-found error. An absent:-only grader instead passes and is reported N/A. A custom-metrics grader with no golden_custom_metrics to withhold has no file to synthesize, so it genuinely errors (exit 1) — the negative control needs golden metrics to withhold to be meaningful.

Note

The metric-threshold family — token-budget, tool-call-count, step-count, turn-count, error-count, and wall-time — checks a numeric ceiling (max:). An empty baseline scores zero on every metric, so a max-only threshold is an absence assertion and is reported N/A. Pair it with a solution-dependent grader (e.g. tool-calls with required:) when the negative control must prove that the solution performs an expected action.

A grader that errors on the baseline (e.g. an invalid regex, or a thrown exception) is a harness fault, not the control holding — it is reported separately and also fails the run (exit 1), rather than being mistaken for a healthy failure.

Exit codes

The meaning of a “good” run is inverted for the baseline sanity check, which is expected to fail. Baseline graders that legitimately pass on empty input (absence assertions like disallowed-only tool-calls or output-not-contains) are reported N/A and do not fail the run:

Code	Apply mode (default)	Baseline (--no-golden-input)
0	All graders passed	Every grader failed as expected, or is N/A (an absence assertion)
1	A grader failed, or an error occurred	A grader trivially passed without depending on the solution, or error

Baseline JSONL carries an explicit verdict

With --output jsonl, baseline outcomes include an oracleBaseline: { mode: "baseline", ok } field. The negative-control verdict is a per-grader classification, not the aggregate score — a healthy baseline emits score: 0, so a threshold-aware re-reader (score >= threshold) would otherwise contradict the exit code. Consumers that re-read baseline outcomes (e.g. vally grade, the analytics server) MUST prefer oracleBaseline.ok over any score/threshold rule.

Examples

# Verify the harness: apply each stimulus's golden inputs and assert the graders pass
vally oracle --eval-spec eval.yaml

# Oracle-grade a single stimulus, verbose
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --verbose

# Baseline sanity check: graders should FAIL without the patch
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --no-golden-input

# CI gate, machine-readable
vally oracle --eval-spec eval.yaml --output jsonl > oracle.jsonl

# Materialize a golden solution without grading and preserve it at a known path
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug \
  --skip-grade --workspace ./oracle-ws

# Materialize one golden solution as a standalone ATIF document
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug \
  --skip-grade --output atif > trajectory.json

# Inspect the materialized + patched workspace
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --keep-workspace

# Materialize into a directory you control (preserved after the run)
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --workspace ./oracle-ws

Re-grading a real agent run

oracle verifies graders against the correct answer (the golden patch). To re-grade an existing agent trajectory without re-running the agent — e.g. after tweaking grader configs — use vally grade instead.

Output format

--output atif writes exactly one ATIF JSON document to stdout. It requires --skip-grade and exactly one selected stimulus. A supplied golden_trajectory is emitted unchanged after validation. Without one, oracle emits a single agent step containing the golden patch when present.

The default human-readable output is:

✅ fix-the-bug [oracle] (1/1 graders passed)

Score: 100.0% | PASSED

Baseline (--no-golden-input) — expected to fail, so a correct failure is reported as OK:

✅ fix-the-bug [baseline (no golden inputs)] (1 failed as expected)
  ✓ [run-command] failed as expected — Command "npm test" exited with code 1

Score: 0.0% | baseline OK (graders correctly FAILED on the baseline (no golden inputs))

A trajectory-scoped grader (e.g. tool-calls with required:) fails the same way on the empty baseline. Absence-asserting graders that legitimately pass are reported N/A:

✅ no-op-task [baseline (no golden inputs)] (0 failed as expected, 1 n/a)
  ⊘ [tool-calls] N/A — asserts absence, satisfied by the empty baseline

Score: 100.0% | baseline OK (all graders assert absence — nothing to withhold, reported N/A)

A grader that trivially passes without depending on the solution fails the run. Its evidence is always shown (even without --verbose), since a trivially-passing grader is the defect the negative control exists to surface:

❌ fix-the-bug [baseline (no golden inputs)] (0 failed as expected, 1 unexpectedly passed)
  ✗ [output-matches /^$/] passed without the golden solution (trivially passing) — Output matches pattern /^$/

Score: 100.0% | baseline PROBLEM (some graders PASSED on the baseline without depending on the golden solution)