CLI: compare

Usage

vally compare <experiment-dir> [options]
vally compare --baseline <path> --treatment <path> [options]

Description

Compares a treatment run against a baseline and reports a signed, treatment-relative verdict plus metric deltas and statistics. The prompt judge compares each stimulus against its rubric, which is read from the trajectories — so no eval spec is required.

Comparison mode always supplies each run’s output, metrics, and session timeline. It does not compare workspace diffs, golden patches, or repo snapshots. If an eval spec configures evidence with diff, golden_patch, or repo, compare warns that those selections are ignored.

There are two input modes:

• Experiment mode — vally compare <experiment-dir>. Reads each variant’s results.jsonl from an experiment output directory, discovers the baseline variant from the embedded provenance, and fans out into one baseline-vs-treatment comparison per other variant.
• Two-run mode — vally compare --baseline <path> --treatment <path>. Compares two independent runs of the same eval spec (for example, a CI regression check across time). Trajectories are paired by stimulus name and trial index.

Options

Flag	Type	Required	Description
<experiment-dir>	string	mode	Experiment output directory (experiment mode). Mutually exclusive with the two-run flags.
--baseline <path>	string	mode	Baseline run directory or JSONL file (two-run mode).
--treatment <path>	string	mode	Treatment run directory or JSONL file (two-run mode).
--baseline-variant <name>	string	No	Experiment mode: override the baseline variant recorded in the output.
-e, --eval-spec <path>	string	No	Optional eval spec to override the rubric embedded in the trajectories.
--judge-model <model>	string	No	Model for the comparison judge.
--judge-reasoning-effort <level>	string	No	Reasoning effort for the comparison judge model: low, medium, high, or xhigh.
--output <path>	string	No	Write per-treatment comparison records to this JSONL file.
--fail-on-regression	boolean	No	Exit non-zero when a treatment produces a valid verdict that is a significant regression — a negative mean score whose 95% CI is entirely below zero. This flag gates on regressions only; a treatment that could not be judged at all fails regardless (see Exit codes).
--param <key=value>	string	No	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.
--verbose	boolean	No	Show per-stimulus and per-trial detail.

Provide either an experiment directory or both --baseline and --treatment — not both.

Output

For each treatment, compare prints:

• Win rate — wins / ties / losses across the paired trials.
• Mean score — the signed treatment-relative score in [-1, 1] (+ means the treatment is better) with a 95% confidence interval.
• McNemar test — a paired test on pass/fail flips between baseline and treatment, when the trajectories carry grade results.
• Metric deltas — per-metric treatment − baseline differences (tokens, turns, tool calls, wall time, errors) with confidence intervals and a high-variance flag.

Retries and partial comparisons

A judge call that fails transiently (timeout, rate limit, dropped connection) is retried automatically — up to 2 retries (3 attempts total) with exponential backoff. Non-retryable failures (for example, an unavailable model) are never retried.

If a pair still fails after retries are exhausted, the comparison is partial: a banner above the statistics reports how many of the matched pairs were actually judged, and every statistic below it covers only the judged pairs.

⚠ Judged 9 of 10 matched pairs — 1 failed during judging.
    All statistics below — score, CI, grader pass/fail, and metric deltas —
    cover only the 9 judged pairs.

If every matched pair fails, compare prints “No statistics available — every matched pair failed” instead of a fabricated 0% win rate.

A partial comparison fails the command — it exits 1. Judging that could not complete is an operational failure, not a verdict, so it is never reported as success.

The three outcomes gate as follows:

Outcome	Exit
Every matched pair judged	0
Partial — some pairs failed after retries	1
No signal — every pair failed, or the run errored	1

With --output, each failed trial’s diagnostics (error class, whether it was retryable, attempt count, message, and model) are persisted under that trial’s failure field in the JSONL:

{
  "trialIndex": 3,
  "errored": true,
  "failure": {
    "errorClass": "timeout",
    "retryable": true,
    "attempts": 3,
    "message": "Timeout after 120000ms waiting for session.idle",
    "model": "claude-opus-4.8"
  }
}

If the forward and reverse judge calls fail with different errors, errorClass/message describe only the representative one; the sibling’s message is preserved in otherMessages:

{
  "errorClass": "timeout",
  "retryable": false,
  "otherMessages": ["Model \"gpt-9\" is not available"]
}

A pair that recovers on retry leaves no trace in the report or the JSONL — a retry-recovered run is byte-identical to a clean one.

Scoring scale

The comparison verdict uses a signed five-bucket magnitude, from the treatment’s perspective:

Verdict	Score
much better	+1.0
slightly better	+0.4
equal / tie	0
slightly worse	-0.4
much worse	-1.0

The judge runs each comparison twice with the two responses swapped (position-swap debiasing); if the two directions disagree on the winner, the result is forced to a tie.

Exit codes

Code	Meaning
0	Every treatment produced a complete verdict over every matched pair and, with --fail-on-regression, none was a significant regression.
1	An operational failure — a treatment could not be judged at all (its comparison errored, or every trial for a treatment errored, e.g. an unavailable or rate-limited judge), or some (but not all) pairs failed judging — or, with --fail-on-regression, a valid verdict was a significant regression.

Note

A treatment with no usable verdict at all, and a partial treatment (some pairs judged, some failed even after retries), both exit 1 regardless of --fail-on-regression — those are operational failures, not quality verdicts. --fail-on-regression only adds gating on valid regression verdicts. (A single un-judged or partial treatment is enough; it need not be every treatment.)

Examples

# Compare every variant of an experiment against its baseline
vally compare ./experiments/output/2026-06-26T12-00-00

# Ad-hoc comparison of two independent runs (CI regression check)
vally compare \
  --baseline ./results/main \
  --treatment ./results/pr-branch \
  --fail-on-regression

# Override the rubric with an eval spec and pick a judge model
vally compare ./experiments/output/latest \
  --eval-spec eval.yaml \
  --judge-model gpt-5.5 \
  --verbose

# Write machine-readable comparison records
vally compare \
  --baseline ./results/v1.jsonl \
  --treatment ./results/v2.jsonl \
  --output ./comparison.jsonl

Workflow

A typical A/B comparison:

1. Run eval on the baseline: vally eval --eval-spec eval.yaml --output-dir ./baseline
2. Make your change (update a skill, change a model, etc.).
3. Run eval on the treatment: vally eval --eval-spec eval.yaml --output-dir ./treatment
4. Compare: vally compare --baseline ./baseline --treatment ./treatment

For multi-variant experiments, run vally experiment run and point compare at the experiment output directory instead — it compares every variant against the declared baseline in one pass.

Comparing multi-turn evals

Multi-turn stimuli (those that use turns instead of prompt) compare exactly like single-turn ones — there is no special flag. compare pairs trajectories by stimulus name and trial index, and the prompt judge reads the full recorded conversation for each side, so it judges the end-to-end interaction (memory, follow-ups, recovery) against the stimulus rubric.

# Baseline and treatment specs both contain the same multi-turn stimuli
vally eval --eval-spec eval.yaml --output-dir ./baseline
# ...change a skill/model/prompt...
vally eval --eval-spec eval.yaml --output-dir ./treatment

# Same command as any other comparison — turns are handled automatically
vally compare --baseline ./baseline --treatment ./treatment --verbose

Notes for multi-turn comparisons:

• Give each multi-turn stimulus a stable name — pairing is by name, so renaming a stimulus makes it look like a removed/added case rather than a change to compare.
• A rubric that describes the whole conversation (not just the last turn) gives the judge the context it needs; --verbose shows the per-stimulus verdict.
• The turns metric in the metric deltas reflects agent conversation turns during execution, which is independent of how many prompt turns you authored.

See Writing Eval Specs → Multi-turn conversations for how to author these stimuli, and samples/multi-turn-memory/ for a runnable example.