Eval Spec Reference

The eval.yaml file defines what to test and how to grade it. This page documents every field.

Note

Eval spec files are discovered by filename pattern. The default patterns are eval.yaml and eval.yml. Custom patterns can be configured via paths.evalFilenames in .vally.yaml.

Tip

Run vally lint --eval-spec eval.yaml to check your spec for errors before execution. Validation also runs automatically in eval and grade commands. See the lint reference for all checks.

Top-level fields

Field	Type	Required	Description
name	string	No	Human-readable name for this eval
description	string	No	What this eval tests
version	string	No	Version for tracking
type	"capability" | "regression"	No	Eval intent
environment	EnvironmentConfig	No	Root environment (merged into all stimuli)
config	EvalConfig	No	Execution configuration
stimuli	Stimulus[]	Yes	Test cases (at least one required)
scoring	ScoringConfig	No	Score aggregation settings

Config

config:
  runs: 3 # Trials per stimulus (default: 1)
  timeout: 2m # Per trial (e.g. 5m, 300s, 30000ms); unit suffix required (default: 2m)
  model: gpt-5.5 # Model for agent execution
  judge_model: gpt-5.5 # Default model for LLM graders
  executor: copilot-sdk # or custom executor via --executor-plugin

Field	Type	Default	Description
runs	number	1	Number of trials per stimulus. 0 = lint-only (skip execution). 1 = single run. ≥ 2 = multi-trial with pass@k/pass^k aggregation.
timeout	duration	2m	Time before a trial times out. Requires a unit suffix (e.g. 5m, 300s, 30000ms).
model	string	—	Model identifier for the executor
judge_model	string	—	Default model for LLM judge graders (prompt and pairwise). Graders can override this in their own config.
executor	string	—	Name of the executor to use for this eval. A spec selects a single executor. Built-in: copilot-sdk. Additional executors can be registered by passing --executor-plugin on the CLI, then referenced here by name. The flag is repeatable so a single vally eval invocation can run multiple specs that select different executors.

Stimulus

stimuli:
  - name: my-test-case
    prompt: "Do something"
    environment: { ... }
    graders: [...]
    rubric: ["criterion 1", "criterion 2"]
    constraints: { ... }

Field	Type	Required	Description
name	string	Yes	Unique identifier for this stimulus
prompt	string	Yes	The prompt sent to the agent
environment	EnvironmentConfig	No	Stimulus-specific environment (merged with root)
graders	GraderConfig[]	No	Graders to run on the trajectory
rubric	string[]	No	Evaluation criteria for LLM judge graders. Passed to prompt and pairwise graders as context for their judgment.
constraints	Constraints	No	Resource limits for the trial
tags	Record<string, string | string[]>	No	Key-value pairs for filtering into suites

Tags

Tags are key-value pairs used for filtering stimuli into suites. They can be defined at the eval level (inherited by all stimuli) or the stimulus level (overrides eval-level tags on the same key).

tags:
  priority: p0
  area: [auth, security]

Filtering semantics:

• AND across keys — a stimulus must match all specified tag keys
• OR within values — a stimulus matches a key if it has any of the specified values
• Stimulus tags override eval tags on the same key

See Authoring Eval Suites for tagging strategies and suite configuration.

Environment

Environment Fields

environment:
  skills:
    - ./path/to/SKILL.md
  files:
    - src: fixtures/input.txt
      dest: input.txt
    - src: fixtures/test-data # directories are copied recursively
      dest: test-data
  commands:
    - npm install
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  mcpServers:
    db:
      type: stdio
      command: db-serve
      args: ["--port", "5432"]
    api:
      type: http
      url: http://localhost:3000/mcp

Field	Type	Description
commands	string[]	Shell commands to run during setup
files	{src, dest}[]	Files or directories to copy into the workspace before execution
git	object	Git worktree configuration for fixture data (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to SKILL.md files to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Git config

The git field sets up a Git worktree as the evaluation workspace, checking out a specific commit, tag, or branch from a local repository.

environment:
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  commands:
    - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Note

When both root and stimulus environments are defined, they are merged with conflict detection:

• Skills: Duplicates across root and stimulus are deduplicated with a warning.
• Commands: Duplicate commands across root and stimulus throw an error.
• Files: Files with the same dest but different src throw a conflict error. Identical entries are also rejected as redundant.

Only cross-array conflicts are detected — duplicates within the same array (root-only or stimulus-only) are not flagged.

Grader config

graders:
  - type: output-contains
    name: "output includes hello"
    config:
      substring: "hello"
      case_sensitive: false

Field	Type	Required	Description
type	string	Yes	Registered grader name (e.g., output-contains, file-exists)
name	string	No	Human-readable display name for this grader instance. Defaults to type when omitted.
config	object	No	Grader-specific configuration (varies by type)

Constraints

constraints:
  max_turns: 10
  max_tokens: 5000
  max_duration: 1m
  expect_tools: ["write_file", "read_file"]
  reject_tools: ["delete_file"]
  expect_skills: ["my-skill"]
  reject_skills: []

Field	Type	Description
max_turns	number	Maximum conversation turns
max_tokens	number	Maximum total tokens
max_duration	duration	Maximum wall-clock time (e.g. 2m, 30s, 500ms). Unit suffix required.
expect_tools	string[]	Tools the agent must call
reject_tools	string[]	Tools the agent must not call
expect_skills	string[]	Skills that must be activated
reject_skills	string[]	Skills that must not be activated

Scoring

Caution

scoring.threshold is applied to the aggregate score when present, and can be overridden for a run with vally eval --threshold <number>. scoring.weights is accepted and validated, but is not yet applied by the evaluation pipeline. Currently all grader scores are averaged equally. See the Scoring concept page for details.

scoring:
  weights:
    file-exists: 1.0
    output-contains: 0.5
  threshold: 0.7

Field	Type	Description
weights	Record<string, number>	Weight per grader type. If omitted, all graders weighted equally.
threshold	number	Minimum aggregate score to pass (0.0–1.0)

Complete example

eval.yaml

name: test-writer-eval
description: Evaluates the test-writer skill
version: "1.0"
type: capability

environment:
  skills:
    - ./SKILL.md

config:
  runs: 3
  timeout: 2m
  model: gpt-5.5
  executor: copilot-sdk

stimuli:
  - name: basic-test-generation
    prompt: |
      Write unit tests for this function:
      function add(a, b) { return a + b; }
      Save the tests to add.test.js.
    graders:
      - type: file-exists
        name: "test file was created"
        config:
          path: "add.test.js"
      - type: output-contains
        config:
          substring: "test"
    constraints:
      max_turns: 10
      expect_tools: ["write_file"]

  - name: edge-case-empty-input
    prompt: "Write tests for a function that takes no arguments."
    graders:
      - type: file-exists
        config:
          path: "*.test.js"

scoring:
  weights:
    file-exists: 1.0
    output-contains: 0.5
  threshold: 0.7