CLI: experiment

Usage

vally experiment run <experiment-file> [options]
vally experiment merge <shard-dir...> --output-dir <path> [options]

Description

Run the same eval set across multiple variants (e.g. different models, skill sets, MCP server configurations) for controlled A/B comparison. Variants are declared in an experiment file — see Experiment File Schema for the YAML format.

Every (eval × variant) combination flows through a single shared worker pool, so one slow variant does not block the others. Each variant gets its own output directory containing results.jsonl (one trial-result record per trial, with an experiment provenance block) and run-summary.jsonl. An unsharded run also writes a single experiment-level markdown report (report.md) alongside the per-variant subdirectories; a sharded run defers the report to vally experiment merge.

The console comparison includes only variants with observed results. If the baseline was not run (for example, with --variant treatment), or variants observed different stimuli, it reports that comparison is unavailable and omits deltas. A tie is reported for all variants only when every variant matches the baseline; partial ties are distinguished from regressions.

Sharded runs label the console table as partial, including the shard index and total. They show observed counts without winner claims or comparison deltas. Run vally experiment merge for the full experiment comparison.

run

Run the declared eval set across every variant in the experiment file.

Arguments

Argument	Required	Default	Description
<experiment-file>	Yes	—	Path to the experiment YAML file.

Options

Flag	Type	Default	Description
--otlp-endpoint <url>	string	—	Export OpenTelemetry traces to an OTLP/HTTP collector.
--variant <name>	string	—	Run only the named variant. Useful for CI splits or partial reruns.
--eval-filter <pattern>	path/glob	—	Run only declared evals matching this path/glob. Repeatable. Narrows (intersects) the file’s evals: set — it can never add an eval the file did not declare. Each pattern must match at least one declared eval, or the run fails fast (so a stale filter can’t silently shrink a CI run).
--output-dir <path>	path	./vally-experiment-results	Directory for output files. A timestamped subdirectory is created inside.
--workers <n>	integer	5	Max concurrent trials across all variants.
--backend <name-or-package>	string	Project backend or local	Backend that controls where trials run: a built-in name (local or docker), or a backend plugin package/path that registers exactly one backend. Relative paths resolve from the experiment file directory. Defaults to .vally.yaml defaults.backend.name when configured, otherwise local.
--backend-args <key=value>	string	—	Argument passed to the selected backend. Repeatable; repeated keys are collected as arrays. Requires --backend.
--shard <i/n>	string	—	Run only shard i of n (e.g. 1/4). Requires --run-id.
--run-id <id>	string	Generated UUID	Stable experiment run identifier (a UUID is recommended). Accepted for unsharded runs and required with --shard.
--shard-strategy <specifier>	string	round-robin	Shard selection strategy: a built-in (round-robin, by-stimulus) or a module specifier.
--param <key=value>	string	—	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml. See vally eval — Parameter resolution for the full precedence chain.
--grader-plugin <specifier>	string	—	Grader plugin package to load (npm package or local path; relative paths resolve from the experiment file directory; repeatable). Additive with .vally.yaml plugins.grader; duplicate modules are loaded once.
--executor-plugin <specifier>	string	—	Executor plugin package to load (npm package or local path; relative paths resolve from the experiment file directory; repeatable). Additive with .vally.yaml plugins.executor; duplicate modules are loaded once.
--reporter-plugin <specifier>	string	—	Reporter plugin package to load (npm package or local path; relative paths resolve from the experiment file directory; repeatable). Additive with .vally.yaml plugins.reporter; duplicate modules are loaded once.
--eval-plugin <specifier>	string	—	Eval provider plugin package to load and select (npm package or local path; relative paths resolve from the experiment file directory). Replaces project eval-provider loading and .vally.yaml defaults.eval_provider.
--workspace <path>	path	—	Root directory for agent workspaces. Contents are preserved after the run for inspection. Paths are keyed by (variant, stimulus), so variants never collide.
--dry-run	flag	false	Resolve the experiment and print the plan without executing.
--require-pass	flag	false	Exit 1 when a valid experiment verdict fails. Without this flag, valid passing and failing verdicts both exit 0. Planning, execution, grading, configuration, cleanup, and required artifact-write errors always exit 1. Under --shard, gates only that shard’s partial verdict.
--compare	flag	false	After the run completes, compare each variant against the baseline using embedded rubrics and shared eval judge settings (see below). Skipped for sharded runs and single---variant runs.
--judge-model <model>	string	inherited	Override the post-run comparison judge model. Requires --compare; does not change trial grading.
--judge-timeout <duration>	duration	inherited (or 2m)	Override the post-run comparison judge turn timeout. Requires --compare.
--judge-reasoning-effort <level>	string	inherited	Override the post-run comparison judge with a non-empty, model-supported reasoning effort. Requires --compare; does not change trial grading.
--verbose	flag	false	Increase console logging verbosity and show agent output and additional evidence; safety caps still apply (see eval reporting).

Exit codes

Code	Meaning
0	The experiment completed without errors, regardless of whether valid experiment verdicts passed or failed.
1	A planning, execution, grading, configuration, cleanup, or required artifact-write error occurred; or --require-pass was set and a valid experiment verdict failed. With --compare, also when a variant could not be compared (see below).

Cleanup errors include backend shutdown failures, executor shutdown failures or timeouts, and session-log finalization failures. LLM client teardown is best-effort and does not change the exit code.

Reporter callback failures and failures to render the unsharded best-effort report.md are warnings and do not change the exit code. Required run artifacts such as per-variant JSONL, summaries, and shard metadata are not best-effort; write failures exit 1.

Note

Under --shard, an eval slice that produces no trial verdict is neutral for --require-pass. Every remaining partial eval verdict must pass. These shard verdicts are not authoritative; use vally experiment merge --require-pass after all shards finish to gate the complete experiment. Errors always exit 1.

Comparing variants after a run (--compare)

--compare runs vally compare over the output at the end of a completed run (including a run whose aggregate verdict failed), so a two-variant experiment produces its head-to-head comparison in one command instead of a separate invocation against the generated, timestamped output path.

It is opt-in because the comparison is LLM-judged — see vally compare for how it works and what it costs. It is skipped, with a printed reason, for sharded runs (no single comparable output until merge) and single---variant runs (nothing to compare). For three or more variants it fans out into one baseline-vs-treatment comparison per other variant.

--compare inherits judge_model, judge_reasoning_effort, judge_provider, and judge_timeout from the selected evals after experiment and variant overrides. Model and effort fall back to EVAL_JUDGE_MODEL and EVAL_JUDGE_REASONING_EFFORT when not configured. --judge-model, --judge-reasoning-effort, and --judge-timeout override comparison only, not trial grading. The judge timeout defaults to 2m when unset. A model override does not change the inherited provider.

vally experiment run experiment.yaml --compare --judge-model gpt-5.6-sol --judge-timeout 5m

Selected evals and variants must agree on the comparison settings. On a conflict, trials finish and results are saved, but comparison exits 1. Use the flags to choose a model, reasoning effort, or timeout, or set a common judge_provider.

Use standalone vally compare for --eval-spec, comparison --output, or --fail-on-regression.

--compare does not gate on regression

--compare reports the comparison but never fails the build on a regression — a variant that scores significantly worse than the baseline still exits 0. Exit 1 from --compare means the comparison did not fully succeed: a variant that could not be compared at all (the comparison errored, or every trial for a variant errored — e.g. an unavailable or rate-limited judge), or one that was only partially judged (some pairs failed even after retries). To gate CI on a regression, run vally compare <output-dir> --fail-on-regression as a separate step.

Note

The --compare verdict is judged against each stimulus’s rubric (or a generic default rubric when none is set) — it is not derived from the experiment’s configured graders. It is a separate, rubric-based signal from the per-variant pass/fail printed above it, and can disagree with it. A comparison failure sets the exit code but does not change the run’s own aggregate verdict.

Output layout

The output written by vally experiment run:

<output-dir>/
└── 2026-06-09T20-41-10-885Z/         # timestamped run directory
    ├── report.md                      # experiment-level markdown report
    ├── otel-spans.jsonl                # absent when --otlp-endpoint is specified
    ├── otel-context.json               # experiment parent identity, also in OTLP mode
    ├── <variant-a>/                    # one subdirectory per variant (named after the variant)
    │   ├── results.jsonl              # trial-result records + experiment block
    │   ├── run-summary.jsonl
    │   └── <eval>/<stimulus>/<model>/<trial>/   # preserved per-trial session logs
    │       ├── events.jsonl           # best-effort (may be absent)
    │       ├── metadata.json          # always written
    │       └── artifacts/             # copied stimulus artifacts (if configured)
    └── <variant-b>/
        ├── results.jsonl
        ├── run-summary.jsonl
        └── <eval>/<stimulus>/<model>/<trial>/
            └── metadata.json

Each results.jsonl line is a TrialResultRecord (the same shape vally eval writes) with a top-level runId and an extra experiment field carrying the same run ID, variant name, and content hashes of the eval and resolved config.

Like vally eval, each trial’s executor session log is preserved. Because one experiment run spans multiple variants, the tree is partitioned by variant first, and each variant directory holds its results.jsonl / run-summary.jsonl alongside the flattened per-trial session directories (<variant>/<eval>/<stimulus>/<model>/<trial>/); the metadata.json / events.jsonl semantics match vally eval, and configured stimulus artifacts are copied into an artifacts/ subdirectory of that same trial directory. In a sharded run the logs live inside that shard’s directory (<run-id>/shard-NN-of-NN/<variant>/...); vally experiment merge leaves them in place per shard rather than consolidating them.

Sensitive data

Executor session logs and traces may contain prompts, model outputs, tool-call payloads, and other sensitive data. Treat the output directory accordingly.

Examples

# Run every variant
vally experiment run experiments/skill-comparison.yaml

# Gate on the experiment verdict as well as operational errors
vally experiment run experiments/skill-comparison.yaml --require-pass

# Run a single variant (e.g. for a CI matrix split)
vally experiment run experiments/skill-comparison.yaml --variant treatment

# Run only a subset of the declared evals (e.g. one plugin's evals per CI shard).
# The pattern is intersected with the file's evals: set, so the file's own
# exclusions still apply. A pattern matching no declared eval fails fast.
vally experiment run experiments/skill-comparison.yaml \
  --eval-filter 'tests/my-plugin/**/eval.vally.yaml'

# Run a single eval (targeted local run); repeat --eval-filter to add more.
vally experiment run experiments/skill-comparison.yaml \
  --eval-filter tests/my-plugin/some-eval/eval.vally.yaml

# Print the resolved plan without executing
vally experiment run experiments/skill-comparison.yaml --dry-run

# Custom output location
vally experiment run experiments/skill-comparison.yaml \
  --output-dir ./runs/skill-comparison

# Run a single cell of a matrix experiment (names are generated as axis=label,…)
vally experiment run experiments/model-x-skills.yaml \
  --variant "model=gpt-5.5,skills=none"

Note

--eval-filter patterns use the exact on-disk casing: matching is case-sensitive.

merge

Reconcile the output directories from a sharded run into a single result and report.

Arguments

Argument	Required	Default	Description
<shard-dir...>	Yes	—	Shard output directories (one per shard of the run).

Options

Flag	Type	Default	Description
-o, --output-dir <path>	path	—	Directory for the merged output. Required.
--require-pass	flag	false	Exit 1 when a valid merged verdict fails. Without this flag, valid passing and failing verdicts both exit 0. Reconstructed plan-time, execution, and grading errors, plus shard-validation and required output-write errors, always exit 1.
--otlp-endpoint <url>	string	—	Export merge traces to an OTLP/HTTP collector.

Exit codes

Code	Meaning
0	The shards reconciled without errors, regardless of whether valid merged verdicts passed or failed.
1	A shard-validation, required output-write, or reconstructed plan-time, execution, or grading error occurred; or --require-pass was set and a valid merged verdict failed.

Output layout

The output written by vally experiment merge:

<output-dir>/
├── report.md                  # authoritative experiment report
├── plan-snapshot.json
├── experiment-manifest.json   # promoted run identity (for a future `experiment compare`)
├── otel-spans.jsonl            # merge's own spans; absent in OTLP mode
├── otel-context.json           # merge span identity
├── <variant-a>/
│   ├── results.jsonl          # all shards' trials, concatenated
│   └── run-summary.jsonl
└── <variant-b>/
    ├── results.jsonl
    └── run-summary.jsonl

Any other per-shard output (e.g. preserved workspaces or executor session logs) is left in place per-shard and is not consolidated into the merged directory.

Sharded runs and merging

For large experiments you can split the work across N machines with --shard i/n. Each shard plans the whole run but executes only its slice of the trials, writing its output under a stable <run-id>/shard-<i>-of-<n>/ layout (the index and total are zero-padded to equal width, so the two-digit form only appears once n >= 10):

<output-dir>/
└── <run-id>/                         # stable, shared across all shards
    ├── shard-1-of-3/
    │   ├── plan-snapshot.json        # planned evals/stimuli/graders (machine-independent)
    │   ├── shard-manifest.json       # run + shard identity (planDigest, selected/completed keys)
    │   ├── otel-spans.jsonl           # this shard's trace; absent in OTLP mode
    │   ├── otel-context.json          # parent identity for merge span links
    │   ├── <variant-a>/results.jsonl  # only this shard's trials
    │   └── <variant-b>/results.jsonl
    ├── shard-2-of-3/
    └── shard-3-of-3/

A sharded run writes no report.md — a single shard’s slice is incomplete, and its per-variant pass/fail is partial. Run vally experiment merge to reconcile the shards into one authoritative result.

vally experiment merge <shard-dir...> -o <path>

Merge reads every shard’s shard-manifest.json and plan-snapshot.json, validates that their persisted experiment identity and plan snapshots match and that their trial selections partition the plan exactly, then concatenates the per-variant results.jsonl, regenerates the run summaries, and renders the final report.md. Merge is pure data wrangling — no agents are executed and no grading is performed (each shard graded its own trials during run). The merged report reproduces an unsharded run of the same spec apart from the render-time timestamp (and any diagnostics emitted during execution, which aren’t persisted per shard — see below).

Plugin consistency is not validated

Merge does not compare plugin sources, versions, or configuration across shards. Run every shard with the same backend, grader, executor, and reporter plugin setup; otherwise merge can accept shards that produced results under different plugin behavior.

Shard directories are passed explicitly — there is no auto-discovery, so a partial merge can’t happen by accident. Merge fails (exit 1) with a specific message if a shard is missing or duplicated, the persisted experiment identities or plan snapshots differ, a shard did not finish, or their slices don’t cover the planned run exactly.

Sharded example (CI matrix)

# Each CI job runs one shard, sharing a stable run id:
vally experiment run experiments/skill-comparison.yaml \
  --shard 1/3 --run-id "$RUN_ID" --output-dir ./out
# ... shards 2/3 and 3/3 run on other machines ...

# A final job merges the collected shard directories:
vally experiment merge \
  ./out/$RUN_ID/shard-1-of-3 \
  ./out/$RUN_ID/shard-2-of-3 \
  ./out/$RUN_ID/shard-3-of-3 \
  -o ./out/merged

If a shard is lost, re-run just that shard into the same --run-id and merge again — sharding is deterministic, so the re-run reproduces the same slice.

Workflow

1. Author an experiment YAML file declaring the eval set, the vary axis, and the named variants — or a matrix of axes that expands into them automatically. See Experiment File Schema.
2. Run vally experiment run <file> (optionally with --dry-run first to inspect the plan). To parallelize across machines, run each shard with --shard i/n --run-id <id>, then reconcile them with vally experiment merge.
3. Inspect per-variant results.jsonl directly, ingest the run directory with vally ingest, or browse it via vally serve.