.vally.yaml Configuration

The .vally.yaml file defines project-level configuration for eval discovery, named environments, and eval suites. Place it at your project root.

Minimal Example

paths:
  evals: evals/

paths

Configures where to find skills, evals, and results.

Field	Type	Default	Description
skills	string or string[]	["skills/"]	Directories containing SKILL.md files
evals	string or string[]	["evals/"]	Directories to scan for eval files
results	string	"vally-results/"	Directory for output results
evalFilenames	string or string[]	["eval.yaml", "eval.yml"]	Filename patterns for eval discovery (supports globs)

Custom Filename Patterns

paths:
  evals: [evals/, tests/evals/]
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

Note

When evalFilenames is set, it replaces the defaults entirely. Include eval.yaml explicitly if you still want it discovered.

suites

Named groups of evals for running specific subsets. Each suite must have at least one of filter or evals.

Filter-Only Suite

Select stimuli by tags across all discovered evals:

suites:
  fast:
    description: "P0 tests for inner loop"
    filter:
      priority: p0

Filter semantics: AND across keys, OR within values.

Evals-Only Suite

Scope to specific eval files by path or glob:

suites:
  safety:
    description: "All safety evals"
    evals:
      - "evals/safety/**/*.eval.yaml"
      - "evals/shared/baseline.eval.yaml"

Note

Suite evals globs resolve directly against the filesystem, relative to the project root. They can reference any eval file, not just those matching evalFilenames patterns, and may reach a sibling tree outside the project root via .. (e.g. ../../tools/**/eval.yaml) or an absolute path. A pattern that resolves to the filesystem root is refused.

Use the exact on-disk casing: matching is case-sensitive.

Combined Suite

Scope by file path, then filter by tags:

suites:
  safety-p0:
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

File scoping narrows which eval files are loaded. Tag filtering narrows which stimuli within those files run.

environments

Named environments for reuse across eval specs.

environments:
  auth-workspace:
    skills:
      - skills/auth
    files:
      - src: fixtures/users.json
        dest: test-data/users.json
    commands:
      - npm install

Reference in eval specs by name:

environment: auth-workspace
stimuli:
  - name: login-test
    prompt: "Test the login flow"

Environment Fields

environment:
  skills:
    - ./path/to/my-skill # Skill directory (containing SKILL.md)
  files:
    - src: fixtures/input.txt
      dest: input.txt
    - src: fixtures/test-data # directories are copied recursively
      dest: test-data
  commands:
    - npm install
  commandTimeout: 2m
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  mcpServers:
    db:
      type: stdio
      command: db-serve
      args: ["--port", "5432"]
    api:
      type: http
      url: http://localhost:3000/mcp
  env:
    COPILOT_AGENT_ACTION: review
    LOG_LEVEL: debug

Field	Type	Description
commands	string[]	Shell commands to run during setup (uses /bin/sh on Unix, cmd.exe on Windows)
commandTimeout	duration	Per-command timeout for commands (e.g. 2m, 90s). Must be positive (0 is rejected). Defaults to 60s
env	Record<string, string>	Environment variables set on the agent process (see below)
files	{src, dest}[]	Files or directories to copy into the workspace before execution
git	object	Git configuration for fixture data — local worktree or remote clone (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to skill directories (each containing a SKILL.md) to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Note

commands run from the workspace directory, not the directory you launched vally from. Use files (or a git worktree) to copy any scripts or assets a command needs into the workspace, and reference them with paths relative to the workspace root.

Each command is executed in the system shell (/bin/sh on Unix, cmd.exe on Windows) with a per-command timeout set by commandTimeout (defaults to 60s). Shell features such as pipes, redirects, and variable expansion are available.

Agent env

The env field sets environment variables on the agent process for every run of this environment. All process-spawning executors (claude-cli, cca, copilot-sdk) merge these over the inherited environment. The built-in mock executor has no agent process and ignores env; Vally prints a warning listing the ignored variable names so the misconfiguration is visible.

environment:
  env:
    COPILOT_AGENT_ACTION: review
    LOG_LEVEL: debug

Field	Type	Description
env	Record<string, string>	Variable name → value, set on the agent process. In experiment variant overrides, values support ${…} interpolation like other override fields.

Note

This sets the agent’s runtime environment; the values are not substituted into other spec fields, and are distinct from an MCP server’s own env (which configures that server’s child process rather than the agent).

Caution

env values run inside the agent process. Variables such as NODE_OPTIONS, LD_PRELOAD, or PATH can change what code the agent executes, so treat authoring env as trust-equivalent to running code on the eval host — be cautious with specs from less-trusted collaborators.

Note

In an experiment variant override, env deep-merges by key with the inherited environment: set a single key to null (e.g. FOO: null) to delete an inherited variable, or set the whole field to null to clear all inherited variables.

Caution

Reports and provenance artifacts show only the names of these variables, never their values. However, trajectory files (the raw agent output) are not filtered — if the agent process echoes an env value (e.g. in a debug log or tool output), it will appear verbatim in the trajectory. Avoid using env to pass long-lived secrets; prefer tokens the agent already has from the host environment.

Note

Like every other config field, env values are part of the environment’s config identity — they feed the config hash recorded in provenance and are compared by experiment drift detection (two variants whose env differs on a path that isn’t a declared vary axis are flagged as drift, with the values redacted). A consequence: a value that changes between runs — e.g. a rotated token interpolated into env — changes that identity, which is a further reason to keep long-lived or rotating secrets out of env.

Git config

The git field sets up the evaluation workspace from a Git repository. It has two modes, discriminated by type:

• worktree — check out a ref from a local repository as a detached worktree.
• clone — clone a remote repository at a given ref into the workspace.

For Harbor-exported tasks, both worktree and clone exports are finalized as self-contained, pinned checkouts baked into the task image. Before shipping the checkout, Harbor strips the transient clone origin (for worktrees, an implementation-created file:// remote), reflogs, fetch record, and hooks so remote credentials and local source paths are not shipped. Git LFS-tracked files remain pointers rather than materialized content and emit warnings; submodules are not materialized and emit warnings; and symlink and executable-mode fidelity depend on the export host OS.

type: worktree

For Harbor-exported tasks, environment.git with type: worktree is materialized at export time from the local source repository as a self-contained checkout of the pinned commit baked into the task image at /app. The export retains history reachable from that commit, which can increase output size for repositories with large histories.

Caution

Harbor uses host Git configuration for authentication but overrides line-ending settings for a deterministic LF checkout. It rejects non-LFS custom filter declarations because their commands cannot be reproduced safely in the image.

Caution

A Harbor Git checkout replaces the base image’s entire /app directory. Do not keep required runtime files there in a custom baseImage.

environment:
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  commands:
    - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source

type: clone

environment:
  git:
    type: clone
    url: https://github.com/octocat/hello.git
    ref: v2.1.0 # optional; defaults to the remote's default branch
    shallow: true # optional; true → depth 1, or an explicit integer depth
    sparse: # optional; cone-mode paths to materialize
      - src/core

Field	Type	Required	Description
type	"clone"	Yes	Must be "clone"
url	string	Yes	Remote repository URL to clone: http(s), ssh, git, file://, or scp-style git@host:path
ref	string	No	Commit-ish to check out; defaults to the remote’s default branch
shallow	boolean | integer	No	Shallow-clone history: true fetches depth 1, an integer sets an explicit depth
sparse	string[]	No	Sparse-checkout (cone mode) paths — only these directories are materialized

HTTP(S) clone URLs must not contain userinfo, query/fragment parameters, or backslashes, and SSH URLs must not contain passwords. Use host Git authentication such as Git Credential Manager, an SSH agent, or GIT_ASKPASS.

For Harbor-exported tasks, environment.git with type: clone contacts the remote repository during export and uses the host’s Git configuration, including credential helpers. After materialization, the resulting checkout is a self-contained, pinned checkout (commit SHA) baked into the task image. The Docker build may still require network access for the base image, runtime installation, or environment commands.

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Complete Example

paths:
  skills: [skills/]
  evals: [evals/]
  results: vally-results/
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

environments:
  default-workspace:
    skills: [skills/copilot]
    commands: [npm install]
    mcpServers:
      db:
        type: stdio
        command: db-serve
        args: ["--port", "5432"]

suites:
  ci-gate:
    description: "Fast checks for every PR"
    filter:
      priority: [p0, p1]

  safety:
    description: "All safety evals"
    evals: ["evals/safety/**/*.eval.yaml"]

  safety-p0:
    description: "Critical safety evals only"
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

Schema Validation

The JSON schema is at packages/core/src/config/vally-config.schema.json. For VS Code autocompletion, add to settings:

{
  "yaml.schemas": {
    "./packages/core/src/config/vally-config.schema.json": ".vally.yaml"
  }
}