CLI: oracle

Usage

vally oracle --eval-spec eval.yaml [options]
vally oracle --eval-spec eval.yaml --stimulus <name>

Description

Re-grade a stimulus against its golden patch (a reference-solution diff) without running an agent. oracle materializes the stimulus’s starting environment into a fresh workspace, applies the golden patch, and runs the eval spec’s graders against the result.

Use it to:

• Verify the harness. If a grader doesn’t pass on the correct answer, the grader is wrong. Run oracle to catch that before shipping.
• Gate CI before publishing. Require every task to pass an oracle run so you never ship an unscorable benchmark.

To re-grade an existing agent trajectory (e.g. after tweaking grader configs) without re-running the agent, use vally grade instead.

Re-grading the same inputs is deterministic for static graders, so an oracle run is safe to gate CI on.

A stimulus declares its golden patch in eval.yaml:

stimuli:
  - name: fix-the-bug
    prompt: "Fix the off-by-one error in pagination"
    golden_patch:
      path: solutions/fix-the-bug.patch # or use `inline:` for an embedded diff
    graders:
      - type: run-command
        config:
          command: "npm test"

Options

Flag	Type	Required	Description
--eval-spec, -e <path>	string	Yes	Path to eval spec file (grader + golden-patch source of truth)
--stimulus <name>	string	No	Grade a single stimulus. Default: every stimulus that declares a golden_patch
--no-golden-patch	boolean	No	Grade the unpatched baseline — a sanity check that should fail
--output jsonl	string	No	Emit graded results as JSONL (one outcome per line)
--judge-model <model>	string	No	Model for LLM judge graders
--judge-reasoning-effort <level>	string	No	Reasoning effort for the judge model: low, medium, high, or xhigh
--grader-plugin <specifier>	string	No	Grader plugin to load (npm package name or local path)
--keep-workspace	boolean	No	Keep the materialized workspace (a temp dir) and print its path (for debugging)
--workspace <path>	string	No	Materialize into this directory instead of a temp dir; contents are preserved. Requires a single stimulus.
--verbose	boolean	No	Show detailed grader evidence

Golden patch reference for graders

When the golden patch is resolved, its diff text is passed to every grader. The prompt LLM judge includes it as a Reference Solution section so the judge can compare the agent’s work against the known-good answer — except in oracle mode, where the applied patch is also reported as the agent output. Handing the judge identical reference and output is circular, so the Reference Solution section is omitted whenever the two are the same.

Oracle works best with static / file-based graders

Oracle applies the golden patch to files, so file-state graders (file-exists, file-contains, run-command, …) verify the harness directly. An LLM judge under oracle instead sees the patch diff as the agent output and scores it against the rubric — there is no separate reference solution to compare against, so it’s a weaker harness-verification signal. To have an LLM judge compare a real agent run against the golden patch, use vally grade instead, where the reference and the agent output differ.

Graders scoped to a specific turn are rejected: oracle grades a single materialized workspace with no agent trajectory to slice by turn. Output-scoped graders (output-contains, output-matches, and their negations) are likewise rejected: oracle reports the golden patch diff as the trajectory output, so they would grade the diff text rather than an agent answer. Trajectory-scoped graders that inspect the agent’s events, metrics, or conversation transcript — completed, exit-success, tool-calls, skill-invocation, max-repeat, token-budget, tool-call-count, turn-count, error-count, wall-time, transcript-contains, transcript-not-contains, transcript-matches, and transcript-not-matches — are rejected too: oracle’s synthesized trajectory has no events, zeroed metrics, and no assistant messages, so they would grade an empty run. Copying a typical eval spec into oracle will surface a clear error naming the offending graders. Use file-state graders (file-exists, file-contains, run-command) — or diff graders (diff-contains, diff-not-contains, diff-empty), which grade the golden patch text itself — to verify the patched workspace.

Exit codes

The meaning of a “good” run is inverted for the baseline sanity check, which is expected to fail:

Code	Apply mode (default)	Baseline (--no-golden-patch)
0	All graders passed	Graders correctly failed (the sanity check holds)
1	A grader failed, or an error occurred	A stimulus unexpectedly passed, or an error occurred

Examples

# Verify the harness: apply each golden patch and assert the graders pass
vally oracle --eval-spec eval.yaml

# Oracle-grade a single stimulus, verbose
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --verbose

# Baseline sanity check: graders should FAIL without the patch
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --no-golden-patch

# CI gate, machine-readable
vally oracle --eval-spec eval.yaml --output jsonl > oracle.jsonl

# Inspect the materialized + patched workspace
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --keep-workspace

# Materialize into a directory you control (preserved after the run)
vally oracle --eval-spec eval.yaml --stimulus fix-the-bug --workspace ./oracle-ws

Re-grading a real agent run

oracle verifies graders against the correct answer (the golden patch). To re-grade an existing agent trajectory without re-running the agent — e.g. after tweaking grader configs — use vally grade instead.

Output format

✅ fix-the-bug [oracle] (1/1 graders passed)

Score: 100.0% | PASSED

Baseline (no patch) — expected to fail, so a correct failure is reported as OK:

✅ fix-the-bug [baseline (no patch)] (0/1 graders passed, 1 failed)
  ✗ [run-command] Command "npm test" exited with code 1

Score: 0.0% | baseline OK (graders correctly FAILED on the unpatched baseline)