CLI: compare

Usage

vally compare <experiment-dir> [options]
vally compare --baseline <path> --treatment <path> [options]

Description

Compares a treatment run against a baseline and reports a signed, treatment-relative verdict plus metric deltas and statistics. The prompt judge compares each stimulus against its rubric, which is read from the trajectories — so no eval spec is required.

Comparison mode always supplies each run’s output, metrics, and session timeline. It does not compare workspace diffs or golden patches. If an eval spec configures evidence with diff or golden_patch, compare warns that those selections are ignored.

There are two input modes:

• Experiment mode — vally compare <experiment-dir>. Reads each variant’s results.jsonl from an experiment output directory, discovers the baseline variant from the embedded provenance, and fans out into one baseline-vs-treatment comparison per other variant.
• Two-run mode — vally compare --baseline <path> --treatment <path>. Compares two independent runs of the same eval spec (for example, a CI regression check across time). Trajectories are paired by stimulus name and trial index.

Options

Flag	Type	Required	Description
<experiment-dir>	string	mode	Experiment output directory (experiment mode). Mutually exclusive with the two-run flags.
--baseline <path>	string	mode	Baseline run directory or JSONL file (two-run mode).
--treatment <path>	string	mode	Treatment run directory or JSONL file (two-run mode).
--baseline-variant <name>	string	No	Experiment mode: override the baseline variant recorded in the output.
-e, --eval-spec <path>	string	No	Optional eval spec to override the rubric embedded in the trajectories.
--judge-model <model>	string	No	Model for the comparison judge.
--judge-reasoning-effort <level>	string	No	Reasoning effort for the comparison judge model: low, medium, high, or xhigh.
--output <path>	string	No	Write per-treatment comparison records to this JSONL file.
--fail-on-regression	boolean	No	Exit non-zero if a treatment scores significantly worse than the baseline — a negative mean score whose 95% CI is entirely below zero (or every comparison fails).
--param <key=value>	string	No	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.
--verbose	boolean	No	Show per-stimulus and per-trial detail.

Provide either an experiment directory or both --baseline and --treatment — not both.

Output

For each treatment, compare prints:

• Win rate — wins / ties / losses across the paired trials.
• Mean score — the signed treatment-relative score in [-1, 1] (+ means the treatment is better) with a 95% confidence interval.
• McNemar test — a paired test on pass/fail flips between baseline and treatment, when the trajectories carry grade results.
• Metric deltas — per-metric treatment − baseline differences (tokens, turns, tool calls, wall time, errors) with confidence intervals and a high-variance flag.

Failed judge calls are excluded from the statistics.

Scoring scale

The comparison verdict uses a signed five-bucket magnitude, from the treatment’s perspective:

Verdict	Score
much better	+1.0
slightly better	+0.4
equal / tie	0
slightly worse	-0.4
much worse	-1.0

The judge runs each comparison twice with the two responses swapped (position-swap debiasing); if the two directions disagree on the winner, the result is forced to a tie.

Exit codes

Code	Meaning
0	Comparison completed (and no regression, when --fail-on-regression)
1	Error occurred, or a regression was detected with --fail-on-regression

Examples

# Compare every variant of an experiment against its baseline
vally compare ./experiments/output/2026-06-26T12-00-00

# Ad-hoc comparison of two independent runs (CI regression check)
vally compare \
  --baseline ./results/main \
  --treatment ./results/pr-branch \
  --fail-on-regression

# Override the rubric with an eval spec and pick a judge model
vally compare ./experiments/output/latest \
  --eval-spec eval.yaml \
  --judge-model gpt-5.5 \
  --verbose

# Write machine-readable comparison records
vally compare \
  --baseline ./results/v1.jsonl \
  --treatment ./results/v2.jsonl \
  --output ./comparison.jsonl

Workflow

A typical A/B comparison:

1. Run eval on the baseline: vally eval --eval-spec eval.yaml --output-dir ./baseline
2. Make your change (update a skill, change a model, etc.).
3. Run eval on the treatment: vally eval --eval-spec eval.yaml --output-dir ./treatment
4. Compare: vally compare --baseline ./baseline --treatment ./treatment

For multi-variant experiments, run vally experiment run and point compare at the experiment output directory instead — it compares every variant against the declared baseline in one pass.

Comparing multi-turn evals

Multi-turn stimuli (those that use turns instead of prompt) compare exactly like single-turn ones — there is no special flag. compare pairs trajectories by stimulus name and trial index, and the prompt judge reads the full recorded conversation for each side, so it judges the end-to-end interaction (memory, follow-ups, recovery) against the stimulus rubric.

# Baseline and treatment specs both contain the same multi-turn stimuli
vally eval --eval-spec eval.yaml --output-dir ./baseline
# ...change a skill/model/prompt...
vally eval --eval-spec eval.yaml --output-dir ./treatment

# Same command as any other comparison — turns are handled automatically
vally compare --baseline ./baseline --treatment ./treatment --verbose

Notes for multi-turn comparisons:

• Give each multi-turn stimulus a stable name — pairing is by name, so renaming a stimulus makes it look like a removed/added case rather than a change to compare.
• A rubric that describes the whole conversation (not just the last turn) gives the judge the context it needs; --verbose shows the per-stimulus verdict.
• The turns metric in the metric deltas reflects agent conversation turns during execution, which is independent of how many prompt turns you authored.

See Writing Eval Specs → Multi-turn conversations for how to author these stimuli, and samples/multi-turn-memory/ for a runnable example.