Writing Eval Specs

An eval spec (eval.yaml or eval.yml) defines what to test and how to grade it. This guide walks through the common authoring path: start with one stimulus and grader, add workspace setup as needed, then configure evaluation and scoring. The eval spec reference is the exhaustive field contract.

Note

By default, vally eval discovers files named eval.yaml or eval.yml in your configured eval directories. You can customize this via paths.evalFilenames in .vally.yaml, or target specific files with vally eval -e <path>.

Minimal eval spec

The simplest possible eval spec:

eval.yaml

name: hello-world
defaults:
  runs: 1

stimuli:
  - name: hello-world
    prompt: "Say hello"
    graders:
      - type: output-contains
        config:
          substring: "hello"

This defines one stimulus with one grader. The agent will be prompted with “Say hello”, and the grader will check if the output contains “hello”.

Run and inspect the eval

Run the spec directly while authoring it:

vally eval --eval-spec eval.yaml

vally eval validates the spec before running the agent. The output shows each grader’s verdict and the aggregate result. --verbose adds per-trial grader details; on multi-trial runs, it shows agent output only for failing trials. Inspect results.jsonl for passing-trial output. See Debugging Evals for the systematic troubleshooting workflow and the eval command reference for all options.

Expanded example

The following example combines the fields used by many evals. It is not an exhaustive schema; later sections explain these choices, and the eval spec reference documents every supported field.

eval.yaml

name: my-skill-eval # Human-readable name
description: Evaluates X # Optional: what this eval tests
version: "1.0" # Optional: version tracking
type: capability # capability | regression

# Root-level environment (merged into all stimuli)
agent_environment:
  skills:
    - ./path/to/my-skill # Skill directory (containing SKILL.md)
  files:
    - src: fixtures/input.txt
      dest: input.txt
  commands:
    - npm install

# Root-level grader-only files (merged into all stimuli)
grading_environment:
  files:
    - src: hidden-tests/shared-fixtures.json
      dest: .vally-grading/shared-fixtures.json

# Execution configuration
defaults:
  runs: 3 # Trials per stimulus
  timeout: 120s # Duration per trial (requires a unit suffix)
  model: gpt-5.5 # Model for agent execution
  executor: copilot-sdk # Which executor to use

# Test cases
stimuli:
  - name: basic-usage
    prompt: |
      Use the skill to generate unit tests for the
      functions in input.txt and basic.txt.
    agent_environment: # Stimulus-level env (merged with root)
      files:
        - src: fixtures/basic.txt
          dest: basic.txt
    graders:
      - type: file-exists
        config:
          path: "*.test.js"
      - type: output-contains
        config:
          substring: "test"
      - type: tool-calls
        config:
          required: ["write_file"]
    constraints:
      max_turns: 10
      max_tokens: 5000

  - name: edge-case
    prompt: "Handle an empty input file."
    graders:
      - type: file-exists
        config:
          path: "*.test.js"
      - type: output-contains
        config:
          substring: "empty"
      - type: tool-calls
        config:
          required: ["write_file"]

# How to aggregate scores
scoring:
  weights:
    file-exists: 0.5
    output-contains: 0.3
    tool-calls: 0.2
  threshold: 0.7

Multi-turn conversations

Every example so far has used a single prompt — one instruction, one response. To test behavior across a back-and-forth conversation, replace prompt with turns: an ordered list of prompts sent sequentially to the same agent session. Later turns can rely on context established earlier (memory, follow-ups, corrections).

eval.yaml

stimuli:
  - name: recall-across-turns
    turns:
      - "Remember this code name for later: ALPHA-7."
      - "Acknowledged? Just say ready."
      - "What was the code name I gave you at the start?"
    graders:
      # Scope a grader to one turn with `turn:` (0-based index into `turns`).
      - type: output-contains
        turn: 1 # the "say ready" acknowledgement
        config:
          substring: "ready"
          case_sensitive: false
      - type: output-contains
        turn: 2 # the recall question
        config:
          substring: "ALPHA-7"

Key rules:

• Provide either prompt or turns, not both. When turns is set, prompt is synthesized from the joined turns, so reporting and LLM judges still work.
• turn: is 0-based — turn: 0 is the first prompt. A grader without turn: runs against the entire trajectory (all turns combined).
• turn: applies to event-based graders and to the static diff graders (diff-contains, diff-not-contains, diff-empty). A scoped diff is the change made during that one turn; file-based graders still inspect only the final workspace and cannot use turn:.

Tip

Runnable version: samples/multi-turn-memory/. Full field list: eval spec reference → Stimulus. A/B comparison: vally compare → multi-turn.

Agent and grading environments

Use agent_environment for setup the agent may see and grading_environment for files that must remain hidden until grading.

Root agent environment

Applied to all stimuli. Define here what’s shared:

agent_environment:
  skills:
    - ./skills/my-skill # Skill directory (containing SKILL.md)
  files:
    - src: fixtures/shared.txt # Files to copy into workspace
      dest: shared.txt
    - src: fixtures/test-data # Directories are copied recursively
      dest: test-data
  commands:
    - npm install # Commands to run in workspace (/bin/sh on Unix, cmd.exe on Windows)

Stimulus agent environment

Per-stimulus overrides. Merged with the root environment (arrays are concatenated, not replaced):

stimuli:
  - name: test-with-config
    agent_environment:
      files:
        - src: fixtures/config.json
          dest: config.json # Added to root files
    prompt: "Use the config file..."

Note

When both root and stimulus agent environments are defined, they are merged with conflict detection:

• Skills: Duplicates are deduplicated with a warning.
• Commands: Duplicate commands throw an error (probably a copy-paste mistake).
• Files: Files with the same dest but different src throw a conflict error. Identical entries also throw (redundant duplicate).

Grader-only files

Stage answer keys, hidden tests, and golden data with grading_environment. These files are added only to a disposable copy of the completed workspace used by graders. The agent never sees them, they do not appear in the recorded diff, and they are not retained in preserved workspaces.

agent_environment:
  files:
    - src: fixtures/starter/
      dest: .

stimuli:
  - name: slugify
    prompt: "Implement slugify() in src/slugify.ts"
    grading_environment:
      files:
        - src: hidden-tests/slugify.test.ts
          dest: .vally-grading/slugify.test.ts
    graders:
      - type: run-command
        config:
          command: npx
          args: [vitest, run, .vally-grading/slugify.test.ts]

grading_environment supports only files. Its dest paths must be confined, relative paths without .., and root/stimulus destinations may not overlap. Keep each grader src outside every directory staged through agent_environment (including files, skills, and local git sources) so the hidden content cannot be copied into the agent-visible workspace. See the grading environment reference for all staging and export semantics.

Named agent environments

Instead of defining agent environments inline in each eval, you can define shared agent-visible environments in .vally.yaml and reference them by name:

.vally.yaml

environments:
  auth-workspace:
    skills: [skills/auth]
    files:
      - src: fixtures/auth-data.json
        dest: test-data.json
    commands:
      - npm install

Then reference in your eval spec:

evals/auth/eval.yaml

agent_environment: auth-workspace
stimuli:
  - name: login-test
    prompt: "Test the login flow"

This keeps your eval specs DRY and makes it easy to update shared setup across multiple evals. Named environments apply only to agent_environment; grading_environment is always declared inline so hidden grader inputs remain explicit in the eval spec. See the .vally.yaml reference for the full agent environment schema.

Grader configuration

Each grader in a stimulus specifies a type (matching a registered grader name) and optional config:

graders:
  - type: output-contains
    config:
      substring: "success"
      case_sensitive: false # Grader-specific option

  - type: file-exists
    config:
      path: "output/*.txt" # Supports glob patterns

  - type: run-command
    config:
      command: "node validate.js"
      expected_exit_code: 0

  - type: prompt
    config:
      prompt: "Ignore unrelated refactoring." # supplemental; criteria come from the stimulus rubric
      scoring: scale_1_5

See the Grader catalog for all built-in graders and their config options, including the prompt and panel LLM judges.

Using LLM judges

Prompt grader with rubric

Use a rubric on the stimulus alongside a prompt grader to give the LLM judge evaluation criteria:

defaults:
  judge_model: gpt-5.6-terra # default model for all LLM graders

stimuli:
  - name: code-quality
    prompt: "Refactor this function for readability"
    rubric:
      - "Code is well-structured with clear variable names"
      - "Comments explain non-obvious logic"
      - "No dead code or redundant operations"
    graders:
      - type: prompt
        config:
          prompt: "Evaluate the refactored code against the rubric criteria"
          scoring: scale_1_5

A/B comparison

Comparison is a mode of the prompt judge — any stimulus with a rubric can be compared. Run vally compare over an experiment’s output or two independent runs, and the prompt judge compares the baseline against each treatment using that rubric.

# Compare two independent runs of the same eval spec
vally compare --baseline ./results/main --treatment ./results/pr-branch

Runtime constraints

Constraints bound agent execution during a trial:

constraints:
  max_turns: 10 # Maximum conversation turns
  max_tokens: 5000 # Maximum total tokens
  max_duration: 1m # Hard cap on the agent's run — overrunning fails the eval
  max_agent_duration: 45s # Agent working-limit — stop the agent, then grade what it finished

Use max_duration when exceeding the limit should fail the trial. Use max_agent_duration when Vally should stop the agent cleanly and grade its partial work. Keep max_agent_duration below the effective hard cap: --timeout when supplied, otherwise max_duration, then defaults.timeout.

Scoring configuration

When scoring.weights is provided, weights apply per grader type: when a stimulus has multiple graders with the same type (e.g. two output-contains checks), their scores are averaged first, then the type weight is applied once — Σ(weight_t × avg_score_t). Weights must be a normalized distribution (sum to 1.0, ±0.01 tolerance). The vally lint command reports a scoring-weight-sum error when they don’t. Graders absent from the map receive weight 0 and do not contribute to the score. scoring.threshold is applied to the aggregate score when present, and can be overridden for a run with vally eval --threshold <number>.

scoring:
  weights:
    file-exists: 0.7 # Weight for each grader type
    output-contains: 0.3
  threshold: 0.7 # Minimum aggregate score to pass

If weights is omitted, all graders are averaged equally. See Scoring for how pass@k and pass^k are computed.

Patterns and tips

Multiple stimuli testing the same capability

stimuli:
  - name: simple-case
    prompt: "Generate tests for function add(a,b)"
    graders: [{ type: file-exists, config: { path: "*.test.js" } }]

  - name: complex-case
    prompt: "Generate tests for an async API client"
    graders: [{ type: file-exists, config: { path: "*.test.js" } }]

  - name: edge-case
    prompt: "Generate tests for a function with no arguments"
    graders: [{ type: file-exists, config: { path: "*.test.js" } }]

Regression testing

Use type: regression to signal this eval is checking for regressions, not new capabilities:

type: regression

defaults:
  runs: 5 # More trials for statistical confidence

scoring:
  threshold: 0.9 # Higher bar for regressions

Next steps

• eval.yaml schema reference — complete field specification
• Grader catalog — all available graders
• Authoring eval suites — organize and filter larger eval collections
• Debugging evals — when evals fail unexpectedly