Skip to content
Vally

Run Your First Experiment

This workflow starts from an existing eval and runs it under two controlled configurations. The experiment file defines what may differ; Vally rejects undeclared drift before execution.

Prerequisites: complete Runtime Setup and have a working evals/hello/eval.yaml from Run Your First Eval.

  1. Create two named variants

    Create experiments/model-comparison.experiment.yaml:

    name: starter-model-comparison
    evals:
      - ../evals/hello/eval.yaml
    

    overrides:
      runs: 5
    

    vary:
      - /defaults/model
    

    baseline: current
    variants:
      current:
        overrides:
          model: gpt-5.5
      candidate:
        overrides:
          model: gpt-5.6-terra

    Paths in evals resolve from the experiment file. The shared runs: 5 override keeps this first experiment manageable while providing a useful comparison. Increase it when you need greater statistical confidence. baseline must name a variant, and vary authorizes only the listed differences. See the normative Experiment File Schema.

  2. Preview the resolved plan

    Terminal window
    vally experiment run experiments/model-comparison.experiment.yaml --dry-run

    The preview executes no trials. It should identify current as the baseline and print two plans: current with gpt-5.5, and candidate with gpt-5.6-terra. Fix a validation or drift error before continuing.

  3. Run and compare the variants

    Terminal window
    vally experiment run experiments/model-comparison.experiment.yaml \
      --output-dir ./experiment-results \
      --require-pass \
      --compare

    --require-pass gates each variant’s normal grader verdict. --compare then uses an LLM judge to compare every treatment with the declared baseline. It spends model tokens and uses the default judge model.

  4. Inspect the output

    The command prints the exact timestamped run directory. For example:

    experiment-results/
    └── 2026-09-08T20-41-10-885Z/
        ├── report.md
        ├── current/
        │   ├── results.jsonl
        │   └── run-summary.jsonl
        └── candidate/
            ├── results.jsonl
            └── run-summary.jsonl

    Each results.jsonl contains trial records with experiment provenance. The comparison appears in command output; use the run directory for repeatable standalone comparison.

  5. Gate a regression

    Terminal window
    vally compare ./experiment-results/<timestamp> --fail-on-regression

    This exits 1 when a valid treatment verdict is a significant regression: its mean signed score is negative and its 95% confidence interval is entirely below zero. It also exits 1 when judging is partial or produces no signal. A complete comparison without a significant regression exits 0.

The experiment run itself exits 1 for planning, execution, grading, configuration, or required artifact-write errors. A valid failing grader verdict exits 1 only when --require-pass is set. See the experiment exit-code reference for the normative contract.

Expand a matrix

Section titled “Expand a matrix”

Use matrix instead of hand-writing every combination. This example expands model x timeout into four variants:

name: model-and-timeout-comparison
evals:
  - ../evals/hello/eval.yaml


matrix:
  model:
    path: /defaults/model
    values: [gpt-5.5, gpt-5.6-terra]
  timeout:
    path: /defaults/timeout
    values: [2m, 3m]


baseline:
  model: gpt-5.5
  timeout: 2m

The stable generated names are:

model=gpt-5.5,timeout=2m
model=gpt-5.5,timeout=3m
model=gpt-5.6-terra,timeout=2m
model=gpt-5.6-terra,timeout=3m

The baseline resolves to model=gpt-5.5,timeout=2m, and vary is derived from the two axis paths. Preview the expansion with --dry-run; pass a generated name to --variant to run one cell.

Section titled “Related reference”