.vally.yaml Configuration

The .vally.yaml file defines project-level configuration for eval discovery, named environments, and eval suites. Place it at your project root.

Minimal Example

paths:
  evals: evals/

paths

Configures where to find skills, evals, and results.

Field	Type	Default	Description
skills	string or string[]	["skills/"]	Directories containing SKILL.md files
evals	string or string[]	["evals/"]	Directories to scan for eval files
results	string	"results/"	Directory for output results
evalFilenames	string or string[]	["eval.yaml", "eval.yml"]	Filename patterns for eval discovery (supports globs)

Custom Filename Patterns

paths:
  evals: [evals/, tests/evals/]
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

Note

When evalFilenames is set, it replaces the defaults entirely. Include eval.yaml explicitly if you still want it discovered.

suites

Named groups of evals for running specific subsets. Each suite must have at least one of filter or evals.

Filter-Only Suite

Select stimuli by tags across all discovered evals:

suites:
  fast:
    description: "P0 tests for inner loop"
    filter:
      priority: p0

Filter semantics: AND across keys, OR within values.

Evals-Only Suite

Scope to specific eval files by path or glob:

suites:
  safety:
    description: "All safety evals"
    evals:
      - "evals/safety/**/*.eval.yaml"
      - "evals/shared/baseline.eval.yaml"

Note

Suite evals globs resolve directly against the filesystem relative to the project root. They can reference any eval file, not just those matching evalFilenames patterns.

Combined Suite

Scope by file path, then filter by tags:

suites:
  safety-p0:
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

File scoping narrows which eval files are loaded. Tag filtering narrows which stimuli within those files run.

environments

Named environments for reuse across eval specs.

environments:
  auth-workspace:
    skills:
      - skills/auth
    files:
      - src: fixtures/users.json
        dest: test-data/users.json
    commands:
      - npm install

Reference in eval specs by name:

environment: auth-workspace
stimuli:
  - name: login-test
    prompt: "Test the login flow"

Environment Fields

environment:
  skills:
    - ./path/to/SKILL.md
  files:
    - src: fixtures/input.txt
      dest: input.txt
    - src: fixtures/test-data # directories are copied recursively
      dest: test-data
  commands:
    - npm install
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  mcpServers:
    db:
      type: stdio
      command: db-serve
      args: ["--port", "5432"]
    api:
      type: http
      url: http://localhost:3000/mcp

Field	Type	Description
commands	string[]	Shell commands to run during setup
files	{src, dest}[]	Files or directories to copy into the workspace before execution
git	object	Git worktree configuration for fixture data (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to SKILL.md files to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Git config

The git field sets up a Git worktree as the evaluation workspace, checking out a specific commit, tag, or branch from a local repository.

environment:
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  commands:
    - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Complete Example

paths:
  skills: [skills/]
  evals: [evals/]
  results: results/
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

environments:
  default-workspace:
    skills: [skills/copilot]
    commands: [npm install]
    mcpServers:
      db:
        type: stdio
        command: db-serve
        args: ["--port", "5432"]

suites:
  ci-gate:
    description: "Fast checks for every PR"
    filter:
      priority: [p0, p1]

  safety:
    description: "All safety evals"
    evals: ["evals/safety/**/*.eval.yaml"]

  safety-p0:
    description: "Critical safety evals only"
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

Schema Validation

The JSON schema is at packages/core/src/config/vally-config.schema.json. For VS Code autocompletion, add to settings:

{
  "yaml.schemas": {
    "./packages/core/src/config/vally-config.schema.json": ".vally.yaml"
  }
}