CLI: compare

Usage

vally compare <experiment-dir> [options]
vally compare --baseline <path> --treatment <path> [options]

Description

Compares a treatment run against a baseline and reports a signed, treatment-relative verdict plus metric deltas and statistics. The prompt judge compares each stimulus against its rubric, which is read from the trajectories — so no eval spec is required.

Comparison mode always supplies each run’s output, metrics, and session timeline. It does not compare workspace diffs, golden patches, or repo snapshots. If an eval spec configures evidence with diff, golden_patch, or repo, compare warns that those selections are ignored.

Vally judges each pair twice, swapping which run is A and B to reduce order bias. Both passes share one temporary evidence directory outside both agents’ working directories. Each run is copied once under a neutral name (run-1 or run-2); the prompt maps those files to A and B differently for each pass. The files contain the captured output, Vally’s recorded timeline events, and metrics.

The initial prompt contains the task, rubric, instructions, and file paths, never evidence content. Each pass has its own read budget. See prompt evidence delivery for read-only restrictions, read limits, and custom-client requirements.

Vally deletes the shared directory after both judge calls finish or fail. A comparison timeout can return while a call is still running; Vally keeps the files until both calls settle and reports VALLY_JUDGE_WORKSPACE_CLEANUP_PENDING. Retries get a new directory so they cannot remove evidence still used by an earlier attempt. If shutting down the judge session fails or times out, Vally still attempts file cleanup but cannot confirm that the session has stopped. Cleanup failures use VALLY_JUDGE_WORKSPACE_CLEANUP_FAILED; forced termination can also leave files behind. These copies are not saved for re-grading.

Read usage and cleanup details are available in result metadata for library callers, but not in compare --output files. CLI users receive cleanup warnings. metadata.evidence_usage reports each pass separately; metadata.judge_cleanup reports the shared directory’s cleanup, with session failures labeled by pass.

There are two input modes:

• Experiment mode — vally compare <experiment-dir>. Reads each variant’s results.jsonl from an experiment output directory, discovers the baseline variant from the embedded provenance, and fans out into one baseline-vs-treatment comparison per other variant.
• Two-run mode — vally compare --baseline <path> --treatment <path>. Compares two independent runs of the same eval spec (for example, a CI regression check across time). Trajectories are paired by stimulus name and trial index.

Options

Flag	Type	Required	Description
<experiment-dir>	string	mode	Experiment output directory (experiment mode). Mutually exclusive with the two-run flags.
--baseline <path>	string	mode	Baseline run directory or JSONL file (two-run mode).
--treatment <path>	string	mode	Treatment run directory or JSONL file (two-run mode).
--baseline-variant <name>	string	No	Experiment mode: override the baseline variant recorded in the output.
-e, --eval-spec <path>	string	No	Optional eval spec to override the rubric embedded in the trajectories.
--judge-model <model>	string	No	Model for the comparison judge.
--judge-timeout <duration>	duration	No	Timeout for each comparison judge turn, including reminders (default: 2m).
--judge-reasoning-effort <level>	string	No	Non-empty, model-supported reasoning effort for the comparison judge. Accepted values depend on the model.
--output <path>	string	No	Write per-treatment comparison records to this JSONL file.
--fail-on-regression	boolean	No	Exit non-zero when a treatment produces a valid verdict that is a significant regression — a negative mean score whose 95% CI is entirely below zero. This flag gates on regressions only; a treatment that could not be judged at all fails regardless (see Exit codes).
--param <key=value>	string	No	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.
--verbose	boolean	No	Show per-stimulus and per-trial detail.

Provide either an experiment directory or both --baseline and --treatment — not both.

Standalone compare does not inherit judge settings from --eval-spec or saved output. Use --judge-model, --judge-reasoning-effort, and --judge-timeout to override its defaults.

Output

For each treatment, compare prints:

• Win rate — wins / ties / losses across the paired trials.
• Mean score — the signed treatment-relative score in [-1, 1] (+ means the treatment is better) with a 95% confidence interval.
• McNemar test — a paired test on pass/fail flips between baseline and treatment, when the trajectories carry grade results.
• Metric deltas — per-metric treatment − baseline differences (tokens, turns, tool calls, wall time, errors) with confidence intervals and a high-variance flag.

Retries and partial comparisons

compare retries judge failures up to 2 times per pair, with exponential backoff. Set --judge-timeout as needed to give slow judges more time or stop waiting for stalled judges. This timeout applies to each judge turn, not the entire comparison; retries can cause the full comparison to take longer than --judge-timeout.

The built-in Copilot LLM client recognizes SDK timeout, rate-limit, and connection failures and marks them retryable. The comparison grader also explicitly marks its own comparison deadlines retryable. The runner reads only the top-level thrown error’s retryable boolean or returned failure metadata. Unmarked failures from other clients or graders are terminal.

If a pair returns a judge failure, either immediately because it is non-retryable or after retries stop, the comparison is partial: a banner above the statistics reports how many of the matched pairs were actually judged, and every statistic below it covers only the judged pairs.

⚠ Judged 9 of 10 matched pairs — 1 failed during judging.
    All statistics below — score, CI, grader pass/fail, and metric deltas —
    cover only the 9 judged pairs.

If every matched pair fails, compare prints “No statistics available — every matched pair failed” instead of a fabricated 0% win rate.

A partial comparison fails the command — it exits 1. Judging that could not complete is an operational failure, not a verdict, so it is never reported as success.

The three outcomes gate as follows:

Outcome	Exit
Every matched pair judged	0
Partial — some pairs failed after retries	1
No signal — every pair failed, or the run errored	1

With --output, each failed trial’s diagnostics (error class, whether it was retryable, attempt count, message, and model) are persisted under that trial’s failure field in the JSONL. The error class is diagnostic, not a retry decision. For example, an SDK timeout recognized by the built-in client can exhaust its retries:

{
  "trialIndex": 3,
  "errored": true,
  "failure": {
    "errorClass": "timeout",
    "retryable": true,
    "attempts": 3,
    "message": "Timeout after 120000ms waiting for session.idle",
    "model": "claude-opus-4.8"
  }
}

If the forward and reverse judge calls fail with different errors, errorClass/message describe only the representative one; the sibling’s message is preserved in otherMessages. A pair is retryable only when every failed direction is explicitly retryable:

{
  "errorClass": "timeout",
  "retryable": false,
  "otherMessages": ["Model \"gpt-9\" is not available"]
}

A pair that recovers on retry leaves no trace in the report or the JSONL — a retry-recovered run is byte-identical to a clean one.

Scoring scale

The comparison verdict uses a signed five-bucket magnitude, from the treatment’s perspective:

Verdict	Score
much better	+1.0
slightly better	+0.4
equal / tie	0
slightly worse	-0.4
much worse	-1.0

The judge runs each comparison twice with the two responses swapped (position-swap debiasing); if the two directions disagree on the winner, the result is forced to a tie.

Exit codes

Code	Meaning
0	Every treatment produced a complete verdict over every matched pair and, with --fail-on-regression, none was a significant regression.
1	An operational failure — a treatment could not be judged at all (its comparison errored, or every trial for a treatment errored, e.g. an unavailable or rate-limited judge), or some (but not all) pairs failed judging — or, with --fail-on-regression, a valid verdict was a significant regression.

Note

A treatment with no usable verdict at all, and a partial treatment (some pairs judged, some failed even after retries), both exit 1 regardless of --fail-on-regression — those are operational failures, not quality verdicts. --fail-on-regression only adds gating on valid regression verdicts. (A single un-judged or partial treatment is enough; it need not be every treatment.)

Examples

# Compare every variant of an experiment against its baseline
vally compare ./experiments/output/2026-06-26T12-00-00

# Ad-hoc comparison of two independent runs (CI regression check)
vally compare \
  --baseline ./results/main \
  --treatment ./results/pr-branch \
  --fail-on-regression

# Override the rubric with an eval spec and pick a judge model
vally compare ./experiments/output/latest \
  --eval-spec eval.yaml \
  --judge-model gpt-5.5 \
  --verbose

# Write machine-readable comparison records
vally compare \
  --baseline ./results/v1.jsonl \
  --treatment ./results/v2.jsonl \
  --output ./comparison.jsonl

Workflow

A typical A/B comparison:

1. Run eval on the baseline: vally eval --eval-spec eval.yaml --output-dir ./baseline
2. Make your change (update a skill, change a model, etc.).
3. Run eval on the treatment: vally eval --eval-spec eval.yaml --output-dir ./treatment
4. Compare: vally compare --baseline ./baseline --treatment ./treatment

For multi-variant experiments, run vally experiment run and point compare at the experiment output directory instead — it compares every variant against the declared baseline in one pass.

Comparing multi-turn evals

Multi-turn stimuli (those that use turns instead of prompt) compare exactly like single-turn ones — there is no special flag. compare pairs trajectories by stimulus name and trial index, and the prompt judge reads the full recorded conversation for each side, so it judges the end-to-end interaction (memory, follow-ups, recovery) against the stimulus rubric.

# Baseline and treatment specs both contain the same multi-turn stimuli
vally eval --eval-spec eval.yaml --output-dir ./baseline
# ...change a skill/model/prompt...
vally eval --eval-spec eval.yaml --output-dir ./treatment

# Same command as any other comparison — turns are handled automatically
vally compare --baseline ./baseline --treatment ./treatment --verbose

Notes for multi-turn comparisons:

• Give each multi-turn stimulus a stable name — pairing is by name, so renaming a stimulus makes it look like a removed/added case rather than a change to compare.
• A rubric that describes the whole conversation (not just the last turn) gives the judge the context it needs; --verbose shows the per-stimulus verdict.
• The turns metric in the metric deltas reflects agent conversation turns during execution, which is independent of how many prompt turns you authored.

See Writing Eval Specs → Multi-turn conversations for how to author these stimuli, and samples/multi-turn-memory/ for a runnable example.