CLI: eval

Usage

vally eval [options]

Description

Run stimuli against an agent and grade the results. Stimuli come from one of three sources:

1. --eval-spec / -e — load stimuli from one or more eval spec files.
2. --suite — run a named suite of evals defined in .vally.yaml.
3. --tag — discover and filter eval stimuli by tag.

Options

Input

Flag	Type	Default	Description
--eval-spec, -e <path>	string	—	Path to eval spec file (repeatable). Cannot be combined with --suite.
--suite <name>	string	—	Run a named suite of evals from .vally.yaml. Cannot be combined with --eval-spec.
--tag <key=values>	string	—	Filter stimuli by tag (repeatable, e.g., --tag cost=free,low). Cannot be combined with --suite. A filter that matches zero stimuli exits with code 1.
--param <key=value>	string	—	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.

Execution

Flag	Type	Default	Description
--executor <name>	string	From eval spec	Executor to run stimuli with. The name must resolve to a built-in executor or one registered via --executor-plugin. Overrides defaults.executor for every eval in the run.
--model <model>	string	From eval spec	Model for agent execution. Comma-separated list runs evals against each model (e.g., --model gpt-5.5,claude-opus-4.8).
--reasoning-effort <level>	string	From eval spec	Reasoning effort for the agent: low, medium, high, or xhigh. Overrides defaults.reasoning_effort; falls back to EVAL_REASONING_EFFORT env var.
--timeout <duration>	duration	2m	Timeout per trial (e.g. 5m, 300s, 30000ms). Unit suffix required. Falls back to defaults.timeout, then to 2m.
--runs <n>	number	From eval spec	Positive integer specifying the number of trials per stimulus (overrides defaults.runs). 1 runs once; ≥ 2 enables multi-trial mode with pass@k metrics.
--workers <n>	number	5	Number of concurrent stimulus sessions.
--max-retries <n>	number	2	Max retries per stimulus on retryable errors (timeouts, rate limits, transient failures). Each retry uses a fresh workspace with exponential backoff. Set to 0 to disable. Ignored (with a warning) when --runs is 2 or greater.
--skill-dir <path>	string	—	Opt-in: discover and load skills from this directory for the whole run. A missing/non-directory path fails the command. Omit to load only skills specified in environment.skills.
--work-dir <path>	string	cwd	Working directory for agent runs.
--workspace <path>	string	—	Workspace directory for the agent (preserved after run, useful for debugging).
--shutdown-timeout <duration>	duration	20s	Max time to wait for executors to shut down during cleanup, including on Ctrl+C (e.g. 5s, 500ms).

Grading

Flag	Type	Default	Description
--threshold <number>	number	From eval spec	Override the scoring threshold for this run. Must be between 0 and 1. When no threshold is configured, evals use binary grader pass/fail.
--judge-model <model>	string	From eval spec	Model for LLM judge graders. Overrides defaults.judge_model.
--judge-reasoning-effort <level>	string	From eval spec	Reasoning effort for the judge model: low, medium, high, or xhigh. Overrides defaults.judge_reasoning_effort.
--skip-grade	boolean	false	Run agents without grading. Grade later with vally grade.
--skip-validate	boolean	false	Skip eval spec validation (validation runs automatically by default).

Reporting

Flag	Type	Default	Description
--output-dir <path>	string	./vally-results	Parent directory for eval outputs. Each run is written to a timestamped subdirectory.
--output <format>	jsonl	—	Emit one trial-result record per trial to stdout (so --runs 3 emits three records per stimulus), followed by a run-summary. Useful for piping into vally grade.
--junit	boolean	false	Write a JUnit XML report (eval-results.junit.xml) to the run’s output directory.
--verbose	boolean	false	Increase logging verbosity and show full agent output.

Plugins & Integrations

Flag	Type	Default	Description
--grader-plugin <specifier>	string	—	Grader plugin to load (npm package or local path). Registers external graders before validation. Repeatable.
--executor-plugin <specifier>	string	—	Executor plugin to load (npm package or local path). Repeatable.
--eval-plugin <specifier>	string	—	Eval provider plugin to load. Requires --eval-spec; cannot be combined with --suite.
--reporter-plugin <specifier>	string	—	Reporter plugin to load (npm package or local path; relative paths resolve from the --eval-spec directory when present, otherwise from --skill-dir / --work-dir). Repeatable.
--backend <name-or-package>	string	local	Backend controlling where trials run: a built-in name (local) or a backend plugin package/path that registers exactly one backend.
--backend-args <key=value>	string	—	Argument passed to the selected backend. Repeatable; repeated keys are collected as arrays. Requires --backend.
--otlp-endpoint <url>	string	—	Export OpenTelemetry traces to an OTLP/HTTP collector. Must be an http(s) URL with no embedded credentials.

Exit codes

Code	Meaning
0	All eval verdicts passed, or verdict failures were suppressed by an override such as --threshold 0
1	One or more eval verdicts failed; an execution/tooling error occurred; or a --tag filter matched zero stimuli

When a threshold is configured via --threshold or scoring.threshold, the eval verdict is based on the aggregate score meeting that threshold. If no threshold is configured, the command uses the binary grader pass/fail verdict. Execution and tooling errors always exit with code 1, even when --threshold 0 is used.

Parameter resolution

When an eval file contains ${KEY} placeholders, values are resolved from multiple sources in this precedence order (highest wins):

1. CLI — --param KEY=value flags
2. Param file — eval.params.yaml (eval-level) or .vally.yaml params: (suite-level)
3. Inline defaults — ${KEY=fallback} in the YAML itself

To include a literal ${...} in your eval (e.g., a bash variable or another template syntax), use the double-dollar escape: $${HOME} produces ${HOME} in the output without being treated as a param placeholder.

Skills

A run is skill-free by default. Skills load only when you opt in through one of two explicit mechanisms:

• environment.skills (recommended) — list skill directories in an eval spec’s environment block. A top-level block applies to all stimuli in the spec; an individual stimulus can declare its own. Paths resolve relative to the eval file. Each listed directory is copied into the agent’s isolated workspace, so the agent sees exactly the declared skills and nothing else. A missing directory, a directory without a SKILL.md, or an unparseable SKILL.md fails the run fast rather than silently loading nothing.
• --skill-dir <path> (whole run, opt-in) — discover and load skills from <path> (the same detection vally lint <path> uses) for all stimuli in the run. Useful for ad-hoc “load this folder of skills” runs. A missing or non-directory path fails the command.

The two mechanisms don’t stack: when environment.skills is declared (in an eval spec, applying to all its stimuli, or on an individual stimulus), that list is the complete skill set and the --skill-dir base is ignored (replace, not merge). Stimuli with no environment.skills fall back to the --skill-dir set.

Behavioral change

Earlier versions auto-discovered skills from the working directory. They no longer do: omit --skill-dir and declare nothing in environment.skills, and the agent runs with no skills. To restore the old “load skills next to me” behavior, pass --skill-dir . explicitly.

Examples

With eval spec

# Run all stimuli from eval.yaml
vally eval \
  --eval-spec eval.yaml \
  --skill-dir ./skills/test-writer \
  --output-dir ./results \
  --model gpt-5.5

# Run multiple eval spec files
vally eval -e auth.eval.yaml -e perf.eval.yaml \
  --skill-dir ./skills/test-writer

# Compare models side-by-side
vally eval \
  --eval-spec eval.yaml \
  --model gpt-5.5,claude-opus-4.8 \
  --output-dir ./results

# Keep workflow steps running for poor eval performance, but still fail on execution errors
vally eval \
  --eval-spec eval.yaml \
  --threshold 0

Capture without grading

# Capture trajectories now, grade later
vally eval \
  --eval-spec eval.yaml \
  --skip-grade \
  --output jsonl > outcomes.jsonl

# Later: grade with updated graders
vally grade --eval-spec eval.yaml < outcomes.jsonl

Export OpenTelemetry to an OTLP collector

vally eval \
  --eval-spec eval.yaml \
  --otlp-endpoint http://localhost:4318

This works with any telemetry-capable executor. In OTLP mode, Vally trial/attempt spans and executor runtime spans are exported to the same collector. Each logical trial is one trace: a vally.trial parent span with one child vally.attempt span per attempt (executor runtime spans nest under the attempt). Filter traces by the vally.trial span and its vally.* attributes to inspect a specific trial. The agent underlying the executor determines the shape and content of the child spans beneath the vally spans.

OTLP mode does not write local span files.

vally eval \
  --eval-spec eval.yaml \
  --executor-plugin @microsoft/vally-executor-claude-cli \
  --otlp-endpoint http://localhost:4318

Preserve workspace

# Keep agent's workspace for inspection
vally eval \
  --eval-spec eval.yaml \
  --workspace ./debug-workspace \
  --verbose

When the eval uses environment.git, the preserved workspace is a valid git checkout, so you can cd into it and run git status / git diff to inspect what the agent changed (a type: worktree environment is preserved as a linked worktree of its source repo; a type: clone environment is a standalone clone).

A preserved type: worktree workspace stays registered in its source repo, so it keeps appearing in git worktree list until you remove it. To clean one up, run git -C <source-repo> worktree remove <preserved-dir>; if you already deleted the directory by hand, run git -C <source-repo> worktree prune.

Trajectory output

Output is saved in a timestamped subdirectory under the output directory (default: ./vally-results/).

Single run or JSONL mode:

results/
└── <timestamp>/
    ├── results.jsonl
    ├── eval-results.md
    ├── otel-spans.jsonl       (absent when --otlp-endpoint specified)
    └── <eval-name>/<stimulus-name>/<model>/0/
        ├── metadata.json
        ├── events.jsonl   (best-effort, may be absent)
        └── artifacts/     (present only when the stimulus configures `artifacts`)

Session logs are written per (eval, stimulus, model, trial) under the run directory in <eval>/<stimulus>/<model>/<trial>/ directories. metadata.json is always written and includes a logSource field ("native", "raw-event-fallback", or "none") indicating what was captured. events.jsonl is best-effort — it is present when the executor emits native session state or when raw SDK events were captured as a fallback, but may be absent for executors that don’t emit events. When a stimulus configures artifacts, the copied files are placed in an artifacts/ subdirectory of that same trial directory.

Sensitive data

Executor session logs may contain prompts, model outputs, tool-call payloads, and other sensitive data.

Multi-trial (--runs K where K ≥ 2):

results/
└── <timestamp>/
    ├── results.jsonl
    ├── eval-results.md
    ├── otel-spans.jsonl       (absent when --otlp-endpoint specified)
    └── <eval-name>/<stimulus-name>/<model>/
        ├── 0/
        │   ├── events.jsonl
        │   ├── metadata.json
        │   └── artifacts/   (present only when the stimulus configures `artifacts`)
        └── 1/
            ├── events.jsonl
            ├── metadata.json
            └── artifacts/

Multi-trial mode

When --runs is ≥ 2 (or defaults.runs is ≥ 2 in the eval spec), the eval command runs each stimulus multiple times and aggregates results:

vally eval --eval-spec eval.yaml --runs 5 --verbose

The console shows a per-trial glyph row and a pass tally; --verbose adds a per-trial Trials breakdown. pass@k / pass^k and flakiness appear in the saved markdown report, not the console.

━━━ test-writer · basic-test-generation ━━━
  ✔ ✔ ✘ ✔ ✔  4/5 trial(s) passed

  Trials
  ─────────────────────────────────────────
  Trial 1  8.3s  1,204 tokens
  Trial 2  7.1s  1,180 tokens
  Trial 3  9.2s  1,320 tokens
  Trial 4  6.8s  1,150 tokens
  Trial 5  8.0s  1,205 tokens

  Graders (trials passed per grader)
  ─────────────────────────────────────────
  ✔ file-exists      5/5
  ✘ output-contains  4/5

Trials within a stimulus run sequentially. Parallelism across different stimuli is controlled by --workers.

Workspace caching

When runs ≥ 2 and the stimulus has an environment with setup work (files, skills, or commands), Vally prepares the workspace once and restores it for each trial. This avoids redundant npm install, builds, and other expensive setup.

All trials execute at the same workspace path, so command outputs that embed absolute paths (virtualenvs, build caches, etc.) remain valid. On filesystems that support it (APFS, btrfs, XFS, ReFS), copies use copy-on-write cloning for near-instant restoration.

Workspace caching is skipped for git-backed workspaces (any environment.git — worktree or clone; each trial is materialized fresh) and when the executor doesn’t support local workspaces (e.g., sandbox executors).

Execution modes

runs must be a positive integer.

runs value	Mode	Behavior
1	Single run	Execute once, grade, report
≥ 2	Multi-trial	Run K times, grade each, aggregate with pass@k/pass^k

Multi-model mode

Pass a comma-separated list of models to run every eval against each model independently:

vally eval --eval-spec eval.yaml --model gpt-5.5,claude-opus-4.8

Each eval is executed once per model. With 3 evals and 2 models, that’s 6 total runs. Results are reported separately per model — the console shows which model each eval ran against:

  ✔ my-eval [gpt-5.5]  all graders passed
  ✘ my-eval [claude-opus-4.8]  grader(s) failed

The exit code is 0 only when all model variants pass.

Workspace preservation

--workspace <path> cannot be combined with multiple models in a single command. The workspace path is keyed by (variant, stimulus) and does not include the model, so two models running the same eval would both claim the same workspace directory, which the planner rejects as a collision:

Workspace path collision at '<workspace>/main/<stimulus>': claimed by
'... × gpt-5.5' and '... × claude-opus-4.8'. Rename one of the stimuli
or use distinct --workspace roots.

If you need preserved workspaces with multi-model, run each model separately with its own --workspace root:

vally eval --eval-spec eval.yaml --model gpt-5.5 --workspace ./ws/gpt-5.5
vally eval --eval-spec eval.yaml --model claude-opus-4.8 --workspace ./ws/claude-opus-4.8

Combining with other options

Multi-model works with --runs, --suite, --tag, and --workers.

# 2 models × 5 trials per stimulus
vally eval --eval-spec eval.yaml \
  --model gpt-5.5,claude-opus-4.8 \
  --runs 5

# Suite + multi-model
vally eval --suite regression --model gpt-5.5,claude-opus-4.8