CLI: oracle

Usage

vally oracle --eval-spec eval.yaml [options]
vally oracle --eval-spec eval.yaml --stimulus <name>

Description

oracle materializes a stimulus’s starting environment and golden inputs without running an agent. By default, it then runs the eval spec’s graders against that reference solution. With --skip-grade, it stops after materialization without running graders.

Use it to:

• Verify the harness. If a grader doesn’t pass on the correct answer, the grader is wrong. Run oracle to catch that before shipping.
• Gate CI before publishing. Require every task to pass an oracle run so you never ship an unscorable benchmark.
• Materialize a golden solution. Use --skip-grade to populate a workspace or emit its trajectory without running the configured graders. Use --output atif when another tool needs a standalone ATIF document.

To re-grade an existing agent trajectory (e.g. after tweaking grader configs) without re-running the agent, use vally grade instead.

Re-grading the same inputs is deterministic for static graders, so an oracle run is safe to gate CI on.

A stimulus declares its golden patch in eval.yaml:

stimuli:
  - name: fix-the-bug
    prompt: "Fix the off-by-one error in pagination"
    golden_patch:
      path: solutions/fix-the-bug.patch # or use `inline:` for an embedded diff
    graders:
      - type: run-command
        config:
          command: "npm test"

Golden trajectory (validating behavior graders)

A golden patch only captures the final file state, so oracle synthesizes an empty trajectory (no events, zeroed metrics). Graders that inspect the agent’s behavior — events, metrics, or the conversation transcript — therefore have nothing to check and are rejected (see below).

To validate those graders offline, add a golden_trajectory: a reference ATIF trajectory (fake or real, inline or a path to a JSON file). Oracle then grades against its real events and metrics instead of the empty synthesized one. You can hand-author the events you expect a passing run to produce and confirm the behavior graders agree — no agent run required.

stimuli:
  - name: runs-the-tests
    prompt: "Run the test suite"
    golden_trajectory:
      path: solutions/runs-the-tests.trajectory.json
    graders:
      - type: tool-calls
        config:
          required: ["bash"]

When hand-authoring the ATIF, keep each observation in the same agent step as the tool_call it answers: tool results are matched to calls per step, so a source_call_id referencing a call in a different step resolves to tool "unknown" and a tool-calls grader won’t count it. See the complete inline example in the eval-spec reference.

A stimulus may declare a golden_patch, a golden_trajectory, or both: the patch drives file-state graders (against the patched workspace) while the trajectory drives behavior/metric graders. With a golden trajectory that has user-delimited turns (at least one user step), event graders may also be scoped to a specific turn. The diff graders (diff-contains, diff-not-contains, diff-empty) remain rejected with turn: oracle has one golden patch, not one workspace diff per agent turn. A user-less (autonomous-agent) trajectory has no turn boundaries, so all turn-scoped graders are rejected against it.

The --no-golden-input baseline withholds every golden input (patch, trajectory, and custom metrics) and grades the resulting empty workspace/trajectory as a negative control — see Baseline (negative control) below.

Golden custom metrics (validating custom-metrics graders)

The custom-metrics grader reads a JSON file from the workspace (default custom_metrics.json). Since oracle runs no agent, that file only exists if the golden patch writes it. Add a golden_custom_metrics (fake or real, inline object or a path to a JSON file) and oracle materializes it into the workspace at every path a custom-metrics grader reads:

stimuli:
  - name: tests-pass
    prompt: "Make the tests pass"
    golden_custom_metrics:
      inline:
        tests_failed: 0
    graders:
      - type: custom-metrics
        config:
          assertions:
            - metric: tests_failed
              equals: 0

Like the patch and trajectory, it is withheld by --no-golden-input.

Options

Flag	Type	Required	Description
--eval-spec, -e <path>	string	Yes	Path to eval spec file
--stimulus <name>	string	No	Select a single stimulus. Default: every stimulus that declares a golden_patch, golden_trajectory, or golden_custom_metrics
--no-golden-input	boolean	No	Materialize or grade the baseline with all golden inputs withheld; grading it is a sanity check that should fail
--output jsonl | atif	string	No	Emit JSONL outcomes, or one bare ATIF document. ATIF requires --skip-grade and exactly one selected stimulus.
--skip-grade	boolean	No	Materialize and exit without grading
--judge-model <model>	string	No	Model for LLM judge graders
--judge-reasoning-effort <level>	string	No	Reasoning effort for the judge model: low, medium, high, or xhigh
--grader-plugin <specifier>	string	No	Grader plugin to load (npm package name or local path). Repeatable.
--keep-workspace	boolean	No	Keep the materialized workspace (a temp dir) and print its path (for debugging)
--workspace <path>	string	No	Materialize into this directory instead of a temp dir; contents are preserved. Requires a single stimulus.
--verbose	boolean	No	Show detailed grader evidence

Golden patch reference for graders

When the golden patch is resolved, its diff text is passed to every grader. The prompt LLM judge includes it as a Reference Solution section so the judge can compare the agent’s work against the known-good answer — except in oracle mode, where the applied patch is also reported as the agent output. Handing the judge identical reference and output is circular, so the Reference Solution section is omitted whenever the two are the same.

Oracle works best with static / file-based graders

Oracle applies the golden patch to files, so file-state graders (file-exists, file-contains, run-command, …) verify the harness directly. An LLM judge under oracle instead sees the patch diff as the agent output and scores it against the rubric — there is no separate reference solution to compare against, so it’s a weaker harness-verification signal. To have an LLM judge compare a real agent run against the golden patch, use vally grade instead, where the reference and the agent output differ.

Grader support under oracle

Oracle grades a single materialized workspace. Which graders are supported depends on whether the stimulus declares a golden_trajectory:

Grader category	Grades	Requires golden_trajectory?	Turn-scopable?
File-state (file-exists, file-contains, run-command)	The patched workspace (final file state)	No	No
Diff (diff-contains, diff-not-contains, diff-empty)	The golden patch text itself	No	No
Output-scoped (output-contains, output-matches, and their negations)	The trajectory output	Yes	Yes
Trajectory / behavior / metric / transcript (completed, exit-success, tool-calls, skill-invocation, system-event, max-repeat, token-budget, tool-call-count, turn-count, step-count, error-count, wall-time, transcript-contains, transcript-not-contains, transcript-matches, transcript-not-matches)	The agent’s events, metrics, and transcript	Yes	Yes

Without a golden_trajectory, oracle’s synthesized trajectory has no events, zeroed metrics, and no assistant messages, so output-scoped and trajectory-scoped graders would grade an empty run — in apply mode they are rejected, along with any grader scoped to a specific turn. With a golden_trajectory, those graders grade its real events and metrics, and may be scoped to a turn when the trajectory has user-delimited turns (at least one user step); turn-scoped graders are rejected against a user-less trajectory, which has no turn boundaries. The three diff graders (diff-contains, diff-not-contains, diff-empty) also cannot be turn-scoped under oracle: a golden trajectory supplies events per turn, but oracle has only one cumulative golden patch. (A workspace-reading grader such as file-exists still cannot be turn-scoped, since it grades final state.) Copying a typical eval spec into oracle surfaces a clear error naming the offending graders and pointing at golden_trajectory.

The --no-golden-input baseline is the exception: output-scoped and trajectory-scoped graders are not rejected there. They run against the empty baseline as the negative control (see below). Turn-scoped and agent-scoped (scope:) graders are still rejected in baseline — an empty baseline has no turns and no per-agent events to slice, so those graders could only error rather than produce a healthy failure.

Baseline (negative control)

--no-golden-input is a negative control: it withholds every golden input and verifies that the graders actually depend on the solution. A healthy grader should now fail — if it still passes, it isn’t really checking anything.

For file-state and behavior graders this is direct: withholding the golden patch leaves an unmodified workspace and an empty trajectory, so solution-dependent checks — file-exists, run-command, tool-calls with required:, skill-invocation with required:, and the like — run and fail. Each such failure is healthy, and the run exits 0 when every grader either fails, is N/A (below), or has no outstanding problem. (Graders whose condition an empty run trivially satisfies — e.g. a max:-only tool-call-count — do not fall here; see the trivially-passing case below.)

Absence assertions are reported N/A. Some graders pass precisely because nothing happened — a tool-calls or skill-invocation grader with only disallowed: (and no required:), output-not-contains / output-not-matches on empty output, max-repeat (an empty run repeats nothing), loop-outcome with should_loop: false (the default — an empty run didn’t loop), or a custom-metrics grader whose assertions are all absent:. On the empty baseline there is nothing to withhold, so passing is correct, not a broken grader. Oracle reports these as N/A and excludes them from the control. A stimulus whose graders are all absence assertions (a “no action needed” task) is therefore reported N/A and passes.

A grader that passes on the baseline but does not assert absence (e.g. file-not-exists, or a tool-call-count with a max an empty run trivially satisfies) is flagged as trivially passing and fails the run (exit 1) — the negative control caught a grader that doesn’t depend on the solution.

custom-metrics on the baseline

When a stimulus defines golden_custom_metrics, the baseline synthesizes an empty metrics file in place of the withheld golden data, so a custom-metrics value/presence assertion fails cleanly on the missing key — a healthy expected-fail, not a file-not-found error. An absent:-only grader instead passes and is reported N/A. A custom-metrics grader with no golden_custom_metrics to withhold has no file to synthesize, so it genuinely errors (exit 1) — the negative control needs golden metrics to withhold to be meaningful.

Note

The metric-threshold family — token-budget, tool-call-count, turn-count, error-count, and wall-time — checks a numeric ceiling (max:). An empty baseline scores zero on every metric, so a max: threshold is trivially satisfied and the grader is flagged as trivially passing (exit 1). These graders are not absence assertions and cannot serve as the negative control on their own; pair them with a solution-dependent grader (e.g. tool-calls with required:) if a stimulus must produce a healthy baseline failure.

A grader that errors on the baseline (e.g. an invalid regex, or a thrown exception) is a harness fault, not the control holding — it is reported separately and also fails the run (exit 1), rather than being mistaken for a healthy failure.

Exit codes

The meaning of a “good” run is inverted for the baseline sanity check, which is expected to fail. Baseline graders that legitimately pass on empty input (absence assertions like disallowed-only tool-calls or output-not-contains) are reported N/A and do not fail the run:

Code	Apply mode (default)	Baseline (--no-golden-input)
0	All graders passed	Every grader failed as expected, or is N/A (an absence assertion)
1	A grader failed, or an error occurred	A grader trivially passed without depending on the solution, or error

Baseline JSONL carries an explicit verdict

With --output jsonl, baseline outcomes include an oracleBaseline: { mode: "baseline", ok } field. The negative-control verdict is a per-grader classification, not the aggregate score — a healthy baseline emits score: 0, so a threshold-aware re-reader (score >= threshold) would otherwise contradict the exit code. Consumers that re-read baseline outcomes (e.g. vally grade, the analytics server) MUST prefer oracleBaseline.ok over any score/threshold rule.

Examples

# Verify the harness: apply each golden patch and assert the graders pass
vally oracle --eval-spec eval.yaml

# Oracle-grade a single stimulus, verbose
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --verbose

# Baseline sanity check: graders should FAIL without the patch
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --no-golden-input

# CI gate, machine-readable
vally oracle --eval-spec eval.yaml --output jsonl > oracle.jsonl

# Materialize a golden solution without grading and preserve it at a known path
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug \
  --skip-grade --workspace ./oracle-ws

# Materialize one golden solution as a standalone ATIF document
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug \
  --skip-grade --output atif > trajectory.json

# Inspect the materialized + patched workspace
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --keep-workspace

# Materialize into a directory you control (preserved after the run)
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --workspace ./oracle-ws

Re-grading a real agent run

oracle verifies graders against the correct answer (the golden patch). To re-grade an existing agent trajectory without re-running the agent — e.g. after tweaking grader configs — use vally grade instead.

Output format

--output atif writes exactly one ATIF JSON document to stdout and sends status messages and diagnostics to stderr. It requires --skip-grade and exactly one selected stimulus. A supplied golden_trajectory is emitted unchanged after validation and must contain at least one step. Without one, oracle emits a single agent step containing the golden patch when present. For a custom-metrics-only solution or --no-golden-input baseline, it emits a system placeholder. That step has no model_name or per-step token metrics; the document still identifies the top-level agent.model_name as oracle and reports zero-valued aggregate token totals in final_metrics. ATIF has no field for oracle’s separate diff, so a consumer that grades diff evidence must transport the golden patch separately.

The default human-readable output is:

✅ fix-the-bug [oracle] (1/1 graders passed)

Score: 100.0% | PASSED

Baseline (--no-golden-input) — expected to fail, so a correct failure is reported as OK:

✅ fix-the-bug [baseline (no golden inputs)] (1 failed as expected)
  ✓ [run-command] failed as expected — Command "npm test" exited with code 1

Score: 0.0% | baseline OK (graders correctly FAILED on the baseline (no golden inputs))

A trajectory-scoped grader (e.g. tool-calls with required:) fails the same way on the empty baseline. Absence-asserting graders that legitimately pass are reported N/A:

✅ no-op-task [baseline (no golden inputs)] (0 failed as expected, 1 n/a)
  ⊘ [tool-calls] N/A — asserts absence, satisfied by the empty baseline

Score: 100.0% | baseline OK (all graders assert absence — nothing to withhold, reported N/A)

A grader that trivially passes without depending on the solution fails the run. Its evidence is always shown (even without --verbose), since a trivially-passing grader is the defect the negative control exists to surface:

❌ fix-the-bug [baseline (no golden inputs)] (0 failed as expected, 1 unexpectedly passed)
  ✗ [file-not-exists] passed without the golden solution (trivially passing) — File "result.txt" does not exist

Score: 100.0% | baseline PROBLEM (some graders PASSED on the baseline without depending on the golden solution)