CLI: eval

Usage

vally eval [options]

Description

Run stimuli against an agent and grade the results. Stimuli come from one of three sources:

1. --eval-spec / -e — load stimuli from one or more eval spec files.
2. --suite — run a named suite of evals defined in .vally.yaml.
3. --tag — discover and filter eval stimuli by tag.

To run trials in containers, see the Docker backend for requirements, image options, and examples.

Options

Input

Flag	Type	Default	Description
--eval-spec, -e <path>	string	—	Path to eval spec file (repeatable). Cannot be combined with --suite.
--suite <name>	string	—	Run a named suite of evals from .vally.yaml. Cannot be combined with --eval-spec.
--tag <key=values>	string	—	Filter stimuli by tag (repeatable). Use key=values to include or key!=values to exclude, e.g. --tag cost=free,low --tag flaky!=yes. Cannot be combined with --suite. A filter that matches zero stimuli exits with code 1.
--param <key=value>	string	—	Set a param value (repeatable, e.g. --param MODEL=gpt-4o). Overrides eval param files and .vally.yaml.

Execution

Flag	Type	Default	Description
--executor <name>, --agent <name>	string	From eval spec	Executor to run stimuli with. The name must resolve to a built-in executor or one registered via --executor-plugin. Overrides defaults.executor for every eval in the run. --agent is an alias for --executor.
--model <model>	string	From eval spec	Model for agent execution. Comma-separated list runs evals against each model (e.g., --model gpt-5.5,claude-opus-4.8).
--reasoning-effort <level>	string	From eval spec	Non-empty, model-supported reasoning effort for the agent. Common values include none, minimal, low, medium, high, xhigh, and max. Overrides defaults.reasoning_effort; falls back to EVAL_REASONING_EFFORT env var.
--timeout <duration>	duration	2m	Timeout per trial (e.g. 5m, 300s, 30000ms). Unit suffix required. Falls back to defaults.timeout, then to 2m.
--runs <n>	number	From eval spec	Positive integer specifying the number of trials per stimulus (overrides defaults.runs). 1 runs once; ≥ 2 enables multi-trial mode with pass@k metrics.
--workers <n>	number	5	Number of concurrent stimulus sessions.
--max-retries <n>	number	2	Max retries per stimulus on retryable failures. Each retry uses a fresh workspace with exponential backoff. Set to 0 to disable. Ignored (with a warning) when --runs is 2 or greater.
--skill-dir <path>	string	—	Opt-in: discover and load skills from this directory for the whole run. A missing/non-directory path fails the command. Omit to load only skills specified in agent_environment.skills.
--work-dir <path>	string	cwd	Working directory for agent runs.
--workspace <path>	string	—	Root for preserved trial workspaces, or the exact existing directory used for execution with --in-place.
--in-place	boolean	false	Use the existing --workspace directory without setup or isolation. Requires --workspace <path>. See Run in an existing directory.
--shutdown-timeout <duration>	duration	20s	Max time to wait for executor and backend shutdown during cleanup, including on Ctrl+C (e.g. 5s, 500ms).

Grading

Flag	Type	Default	Description
--threshold <number>	number	From eval spec	Override the scoring threshold for this run. Must be between 0 and 1. When no threshold is configured, evals use binary grader pass/fail.
--require-pass	boolean	false	Exit 1 when any valid eval verdict fails. Without this flag, valid passing and failing verdicts both exit 0. Grader, execution, configuration, and tooling errors always exit 1.
--judge-model <model>	string	From eval spec	Model for LLM judge graders. Overrides defaults.judge_model.
--judge-timeout <duration>	duration	2m	Override the timeout for each prompt or panel judge turn. Bounded by the remaining scoring.timeout.
--judge-reasoning-effort <level>	string	From eval spec	Non-empty, model-supported reasoning effort for the judge model. Accepted values depend on the model. Overrides defaults.judge_reasoning_effort.
--skip-grade	boolean	false	Run agents without grading. Grade later with vally grade.
--skip-validate	boolean	false	Skip eval spec validation (validation runs automatically by default).

On a spec with no scoring block and at least one grader, --threshold emits a scoring-defaults-applied warning even with a single grader, because the override switches each trial’s verdict to score >= threshold instead of the all-graders-passed flag. This applies only when grading runs; with --skip-grade the override has no effect and the warning is suppressed.

Reporting

Flag	Type	Default	Description
--output-dir <path>	string	./vally-results	Parent directory for eval outputs. Each run is written to a timestamped subdirectory.
--output <format>	jsonl	—	Emit one trial-result record per trial to stdout (so --runs 3 emits three records per stimulus), followed by a run-summary. Useful for piping into vally grade.
--run-id <id>	string	Generated UUID	Stable identifier placed on every trial-result and run-summary record. Must be a non-empty path-safe token; UUIDs are recommended.
--junit	boolean	false	Write a JUnit XML report (eval-results.junit.xml) to the run’s output directory.
--verbose	boolean	false	Increase logging verbosity and show agent output and additional evidence; safety caps still apply.

Plugins & Integrations

Flag	Type	Default	Description
--grader-plugin <specifier>	string	—	Grader plugin to load from an npm package or local path, additive with project plugins.grader. Registers external graders before validation. Repeatable.
--executor-plugin <specifier>	string	—	Executor plugin to load from an npm package or local path, additive with project plugins.executor. Repeatable.
--eval-plugin <specifier>	string	—	Eval provider plugin to load and select, replacing project eval-provider loading and defaults.eval_provider. Requires --eval-spec; cannot be combined with --suite.
--reporter-plugin <specifier>	string	—	Reporter plugin to load from an npm package or local path, additive with project plugins.reporter. Relative CLI paths resolve from the --eval-spec directory when present, otherwise from --skill-dir / --work-dir. Repeatable.
--backend <name-or-package>	string	Project backend or local	Backend controlling where trials run: a built-in name (local or docker) or a plugin package/path registering exactly one backend. Defaults to .vally.yaml defaults.backend.name when configured, otherwise local.
--backend-args <key=value>	string	—	Argument passed to the selected backend. Repeatable; repeated keys are collected as arrays. Requires --backend.
--otlp-endpoint <url>	string	—	Export OpenTelemetry traces to an OTLP/HTTP collector. Must be an http(s) URL with no embedded credentials.

Exit codes

Code	Meaning
0	Eval execution and grading completed without errors, regardless of whether valid eval verdicts passed or failed.
1	A grader, execution, configuration, or tooling error occurred; a --tag or --suite filter matched no evals or stimuli; or --require-pass was set and one or more eval verdicts failed.

--require-pass requires every eval verdict to pass. When a threshold is configured via --threshold or scoring.threshold and at least one stimulus is scored, the verdict is based on the aggregate score meeting that threshold. If no threshold is configured, the command uses the binary grader pass/fail verdict.

Terminal output labels invalid trial grades separately from valid failing verdicts. Non-blocking grader errors are reported as warnings: the aggregate verdict remains valid, but the command still exits 1 because grading encountered an error.

When an eval’s stimuli run successfully without any graders, the eval passes vacuously.

Parameter resolution

When an eval file contains ${KEY} placeholders, values are resolved from multiple sources in this precedence order (highest wins):

1. CLI — --param KEY=value flags
2. Param file — eval.params.yaml (eval-level) or .vally.yaml params: (suite-level)
3. Inline defaults — ${KEY=fallback} in the YAML itself

To include a literal ${...} in your eval (e.g., a bash variable or another template syntax), use the double-dollar escape: $${HOME} produces ${HOME} in the output without being treated as a param placeholder.

Skills

A run is skill-free by default, aside from any skills the trial workspace itself carries under copilot-sdk (see the note below). Skills load only when you opt in through one of two explicit mechanisms:

• agent_environment.skills (recommended) — list skill directories in an eval spec’s agent_environment block. A top-level block applies to all stimuli in the spec; an individual stimulus can declare its own. Paths resolve relative to the eval file. Each listed directory is copied into the agent’s isolated workspace, so the agent sees the declared skills plus any the workspace itself carries under copilot-sdk (see below). A missing directory, a directory without a SKILL.md, or an unparseable SKILL.md fails the run fast rather than silently loading nothing.
• --skill-dir <path> (whole run, opt-in) — discover and load skills from <path> (the same detection vally lint <path> uses) for all stimuli in the run. Useful for ad-hoc “load this folder of skills” runs. A missing or non-directory path fails the command.

The two mechanisms don’t stack: when agent_environment.skills is declared (in an eval spec, applying to all its stimuli, or on an individual stimulus), that list is the complete skill set and the --skill-dir base is ignored (replace, not merge). Stimuli with no agent_environment.skills fall back to the --skill-dir set.

Behavioral change

Earlier versions auto-discovered skills from the working directory. They no longer do: omit --skill-dir and declare nothing in agent_environment.skills, and the agent runs with no skills. To restore the old “load skills next to me” behavior, pass --skill-dir . explicitly.

Note

The copilot-sdk executor enables the Copilot runtime’s workspace config discovery so staged .github/agents profiles load natively. As a side effect, skill directories present in the trial workspace itself are also loaded, in addition to the skills declared above — so under this executor agent_environment.skills is a floor rather than the exact set. Keep the workspace clean when you need a strictly declared-only baseline. A small number of builtin runtime skills load regardless of these settings.

Examples

With eval spec

# Run all stimuli from eval.yaml
vally eval \
  --eval-spec eval.yaml \
  --skill-dir ./skills/test-writer \
  --output-dir ./results \
  --model gpt-5.5

# Run multiple eval spec files
vally eval -e auth.eval.yaml -e perf.eval.yaml \
  --skill-dir ./skills/test-writer

# Compare models side-by-side
vally eval \
  --eval-spec eval.yaml \
  --model gpt-5.5,claude-opus-4.8 \
  --output-dir ./results

# Gate a workflow step on eval performance as well as errors
vally eval \
  --eval-spec eval.yaml \
  --require-pass

Capture without grading

# Capture trajectories now, grade later
vally eval \
  --eval-spec eval.yaml \
  --skip-grade \
  --output jsonl > outcomes.jsonl

# Later: grade with updated graders
vally grade --eval-spec eval.yaml < outcomes.jsonl

Export OpenTelemetry to an OTLP collector

vally eval \
  --eval-spec eval.yaml \
  --otlp-endpoint http://localhost:4318

Vally supplies the same run-scoped configuration to tracing-capable backends and executors before they initialize. In OTLP mode, Vally trial/attempt spans and runtime spans are exported to the same collector. Each logical trial is one trace: a vally.trial parent span with one child vally.attempt span per attempt (runtime spans nest under the attempt). Filter traces by the vally.trial span and its vally.* attributes to inspect a specific trial. The agent underlying the executor determines the shape and content of the child spans beneath the vally spans.

OTLP mode does not write local span files.

Preserve workspace

# Keep agent's workspace for inspection
vally eval \
  --eval-spec eval.yaml \
  --workspace ./debug-workspace \
  --verbose

When the eval uses agent_environment.git, the preserved workspace is a valid git checkout, so you can cd into it and run git status / git diff to inspect what the agent changed (a type: worktree environment is preserved as a linked worktree of its source repo; a type: clone environment is a standalone clone).

A preserved type: worktree workspace stays registered in its source repo, so it keeps appearing in git worktree list until you remove it. To clean one up, run git -C <source-repo> worktree remove <preserved-dir>; if you already deleted the directory by hand, run git -C <source-repo> worktree prune.

Run in an existing directory

For large repositories that are expensive to copy or check out, opt into running directly in an already-prepared directory:

vally eval --eval-spec eval.yaml --workspace /path/to/checkout --in-place \
  --output-dir /path/outside/checkout/results

--in-place requires --workspace <path> and uses that exact directory, rather than creating separate trial directories under it. The directory must already exist. Unless diff evidence is requested, it need not be a Git repository or have a clean working tree. Relative paths resolve from where you launch Vally. Every stimulus, model, and trial uses that same directory; Vally never copies, moves, deletes, or restores it, even on failure. Keep --output-dir outside the checkout (or ignored by Git) so logs and reports do not affect grading.

This mode uses one worker and disables automatic retries. An explicit --workers value other than 1, --max-retries other than 0, or a non-local --backend is an error.

The selected stimuli must not configure agent_environment.git, non-empty agent_environment.files, agent_environment.skills, agent_environment.commands, or grading_environment.files. These restrictions apply even with --skip-validate. Environment variables and MCP servers are still supported. --skill-dir can load skills without copying them into the checkout.

No workspace.patch is written automatically. Transcript, tool-call, and ordinary file-based graders do not require Git.

Whole-run diff grading (diff-contains, diff-not-contains, diff-empty, or LLM diff evidence) requires a Git checkout with a HEAD commit and no staged, unstaged, or non-ignored untracked changes before each trial. Linked worktrees, sparse checkouts, and SHA-1 or SHA-256 repositories are supported. These requirements also apply with --skip-grade, so diff evidence remains available for later grading.

Diff evidence compares the final workspace against the starting commit, including changes the agent commits and new non-ignored files. Vally leaves the Git index and configuration unchanged. Git’s normal ignore, line-ending, and filter settings apply. For a repository subdirectory, cleanliness and diffs cover only that directory. Turn-scoped diff graders are unsupported. An invalid starting state fails the trial before the agent runs. Diff-capture errors are recorded on the trajectory and fail diff graders; with --skip-grade, they remain available for later grading. Edits from an earlier trial can cause a later diff-graded trial to fail the clean-checkout requirement.

Caution

This is not a sandbox. Agent and grader edits persist, and later trials or models see those edits. One worker prevents concurrent trials within this command, but does not reset state or coordinate with other Vally processes. Use a disposable checkout or restore it yourself when independent trials are required.

Trajectory output

Output is saved in a timestamped subdirectory under the output directory (default: ./vally-results/).

Single run or JSONL mode:

results/
└── <timestamp>/
    ├── results.jsonl
    ├── eval-results.md
    ├── otel-spans.jsonl       (absent when --otlp-endpoint specified)
    └── <eval-name>/<stimulus-name>/<model>/0/
        ├── metadata.json
        ├── events.jsonl   (best-effort, may be absent)
        └── artifacts/     (present only when the stimulus configures `artifacts`)

Session logs are written per (eval, stimulus, model, trial) under the run directory in <eval>/<stimulus>/<model>/<trial>/ directories. metadata.json is always written and includes a logSource field ("native", "raw-event-fallback", or "none") indicating what was captured. events.jsonl is best-effort — it is present when the executor emits native session state or when raw SDK events were captured as a fallback, but may be absent for executors that don’t emit events. When a stimulus configures artifacts, the copied files are placed in an artifacts/ subdirectory of that same trial directory.

Sensitive data

Executor session logs may contain prompts, model outputs, tool-call payloads, and other sensitive data.

Multi-trial (--runs K where K ≥ 2):

results/
└── <timestamp>/
    ├── results.jsonl
    ├── eval-results.md
    ├── otel-spans.jsonl       (absent when --otlp-endpoint specified)
    └── <eval-name>/<stimulus-name>/<model>/
        ├── 0/
        │   ├── events.jsonl
        │   ├── metadata.json
        │   └── artifacts/   (present only when the stimulus configures `artifacts`)
        └── 1/
            ├── events.jsonl
            ├── metadata.json
            └── artifacts/

Multi-trial mode

When --runs is ≥ 2 (or defaults.runs is ≥ 2 in the eval spec), the eval command runs each stimulus multiple times and aggregates results:

vally eval --eval-spec eval.yaml --runs 5 --verbose

The console shows live progress and summarizes trial outcomes with aggregate counts and verdicts such as PASS, FAIL, and FLAKY. Mixed passing and failing trials can produce FLAKY in streamed cards and the final summary. pass@k / pass^k appear in the saved markdown report, not the console.

Use --verbose for per-worker progress, expanded failure reasons and per-trial critiques, nested grader evidence, and failing trials’ agent output. Safety caps still apply: each nested evidence block is limited to 4,096 characters and 40 rendered lines. Truncated evidence is marked; consult saved results and session logs for further detail.

Successful --skip-grade executions remain passing in both single-trial and repeated runs; skipped trials are neutral rather than grading failures. Multi-model summaries keep same-named eval files separate and use a distinct default model (executor) column when no model was configured.

────────────────────────────────────────────────────────

Summary
  FLAKY  test-writer · 5 trials each
    ! basic-test-generation   4 passed · 1 failed   output-contains failed on trial 3

Weighted evals show a stimulus ratio and score comparator instead of the unweighted state-word heading, for example test-writer ✓ 1/1 (score 90% ≥ 80%).

Trials within a stimulus run sequentially. Parallelism across different stimuli is controlled by --workers.

Workspace caching

When runs ≥ 2 and the stimulus has an agent_environment with setup work (files, skills, or commands), Vally prepares the workspace once and restores it for each trial. This avoids redundant npm install, builds, and other expensive setup.

All trials execute at the same workspace path, so command outputs that embed absolute paths (virtualenvs, build caches, etc.) remain valid. On filesystems that support it (APFS, btrfs, XFS, ReFS), copies use copy-on-write cloning for near-instant restoration.

Workspace caching is skipped for git-backed workspaces (any agent_environment.git — worktree or clone), agent environments that stage any file with dest_root: assets, and executors that don’t support local workspaces (e.g., sandbox executors). Each affected trial is materialized and staged independently.

Execution modes

runs must be a positive integer.

runs value	Mode	Behavior
1	Single run	Execute once, grade, report
≥ 2	Multi-trial	Run K times, grade each, aggregate with pass@k/pass^k

Multi-model mode

Pass a comma-separated list of models to run every eval against each model independently:

vally eval --eval-spec eval.yaml --model gpt-5.5,claude-opus-4.8

Each eval is executed once per model. With 3 evals and 2 models, that’s 6 total runs. The final grid keeps models in separate columns, with settled stimulus ratios and verdicts (shown here for an eval with one stimulus):

eval       gpt-5.5   claude-opus-4.8
my-eval    ✓ 1/1     ✘ 0/1

By default, the exit code is 0 even when some model variants receive valid failing verdicts. Pass --require-pass to exit 1 for those verdicts. Grader, execution, configuration, or tooling errors on any variant always exit 1.

Workspace preservation

Without --in-place, --workspace <path> cannot be combined with multiple models in a single command. The workspace path is keyed by (variant, stimulus) and does not include the model, so two models running the same eval would both claim the same workspace directory, which the planner rejects as a collision:

Workspace path collision at '<workspace>/main/<stimulus>': claimed by
'... × gpt-5.5' and '... × claude-opus-4.8'. Rename one of the stimuli
or use distinct --workspace roots.

If you need preserved workspaces with multi-model, run each model separately with its own --workspace root:

vally eval --eval-spec eval.yaml --model gpt-5.5 --workspace ./ws/gpt-5.5
vally eval --eval-spec eval.yaml --model claude-opus-4.8 --workspace ./ws/claude-opus-4.8

With --workspace <path> --in-place, models run sequentially in the same directory; changes persist between models.

Combining with other options

Multi-model works with --runs, --suite, --tag, and --workers.

# 2 models × 5 trials per stimulus
vally eval --eval-spec eval.yaml \
  --model gpt-5.5,claude-opus-4.8 \
  --runs 5

# Suite + multi-model
vally eval --suite regression --model gpt-5.5,claude-opus-4.8