.vally.yaml Configuration

The .vally.yaml file defines project-level configuration for eval discovery, named environments, and eval suites. Place it at your project root.

Minimal Example

paths:
  evals: evals/

paths

Configures where to find skills, evals, and results.

Field	Type	Default	Description
skills	string or string[]	["skills/"]	Directories containing SKILL.md files
evals	string or string[]	["evals/"]	Directories to scan for eval files
results	string	"vally-results/"	Directory for output results
evalFilenames	string or string[]	["eval.yaml", "eval.yml"]	Filename patterns for eval discovery (supports globs)

Custom Filename Patterns

paths:
  evals: [evals/, tests/evals/]
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

Note

When evalFilenames is set, it replaces the defaults entirely. Include eval.yaml explicitly if you still want it discovered.

suites

Named groups of evals for running specific subsets. Each suite must have at least one of filter or evals.

Filter-Only Suite

Select stimuli by tags across all discovered evals:

suites:
  fast:
    description: "P0 tests for inner loop"
    filter:
      priority: p0

Filter semantics: AND across keys, OR within values.

Evals-Only Suite

Scope to specific eval files by path or glob:

suites:
  safety:
    description: "All safety evals"
    evals:
      - "evals/safety/**/*.eval.yaml"
      - "evals/shared/baseline.eval.yaml"

Note

Suite evals globs resolve directly against the filesystem relative to the project root. They can reference any eval file, not just those matching evalFilenames patterns.

Combined Suite

Scope by file path, then filter by tags:

suites:
  safety-p0:
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

File scoping narrows which eval files are loaded. Tag filtering narrows which stimuli within those files run.

environments

Named environments for reuse across eval specs.

environments:
  auth-workspace:
    skills:
      - skills/auth
    files:
      - src: fixtures/users.json
        dest: test-data/users.json
    commands:
      - npm install

Reference in eval specs by name:

environment: auth-workspace
stimuli:
  - name: login-test
    prompt: "Test the login flow"

Environment Fields

environment:
  skills:
    - ./path/to/my-skill # Skill directory (containing SKILL.md)
  files:
    - src: fixtures/input.txt
      dest: input.txt
    - src: fixtures/test-data # directories are copied recursively
      dest: test-data
  commands:
    - npm install
  commandTimeout: 2m
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  mcpServers:
    db:
      type: stdio
      command: db-serve
      args: ["--port", "5432"]
    api:
      type: http
      url: http://localhost:3000/mcp

Field	Type	Description
commands	string[]	Shell commands to run during setup (uses /bin/sh on Unix, cmd.exe on Windows)
commandTimeout	duration	Per-command timeout for commands (e.g. 2m, 90s). Must be positive (0 is rejected). Defaults to 60s
files	{src, dest}[]	Files or directories to copy into the workspace before execution
git	object	Git configuration for fixture data — local worktree or remote clone (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to skill directories (each containing a SKILL.md) to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Note

commands run from the workspace directory, not the directory you launched vally from. Use files (or a git worktree) to copy any scripts or assets a command needs into the workspace, and reference them with paths relative to the workspace root.

Each command is executed in the system shell (/bin/sh on Unix, cmd.exe on Windows) with a per-command timeout set by commandTimeout (defaults to 60s). Shell features such as pipes, redirects, and variable expansion are available.

Git config

The git field sets up the evaluation workspace from a Git repository. It has two modes, discriminated by type:

• worktree — check out a ref from a local repository as a detached worktree.
• clone — clone a remote repository at a given ref into the workspace.

type: worktree

environment:
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  commands:
    - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source

type: clone

environment:
  git:
    type: clone
    url: https://github.com/octocat/hello.git
    ref: v2.1.0 # optional; defaults to the remote's default branch
    shallow: true # optional; true → depth 1, or an explicit integer depth
    sparse: # optional; cone-mode paths to materialize
      - src/core

Field	Type	Required	Description
type	"clone"	Yes	Must be "clone"
url	string	Yes	Remote repository URL to clone: http(s), ssh, git, file://, or scp-style git@host:path
ref	string	No	Commit-ish to check out; defaults to the remote’s default branch
shallow	boolean | integer	No	Shallow-clone history: true fetches depth 1, an integer sets an explicit depth
sparse	string[]	No	Sparse-checkout (cone mode) paths — only these directories are materialized

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Complete Example

paths:
  skills: [skills/]
  evals: [evals/]
  results: vally-results/
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

environments:
  default-workspace:
    skills: [skills/copilot]
    commands: [npm install]
    mcpServers:
      db:
        type: stdio
        command: db-serve
        args: ["--port", "5432"]

suites:
  ci-gate:
    description: "Fast checks for every PR"
    filter:
      priority: [p0, p1]

  safety:
    description: "All safety evals"
    evals: ["evals/safety/**/*.eval.yaml"]

  safety-p0:
    description: "Critical safety evals only"
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

Schema Validation

The JSON schema is at packages/core/src/config/vally-config.schema.json. For VS Code autocompletion, add to settings:

{
  "yaml.schemas": {
    "./packages/core/src/config/vally-config.schema.json": ".vally.yaml"
  }
}