.vally.yaml Configuration

The .vally.yaml file defines project-level configuration for eval discovery, named environments, eval suites, and plugins. Place it at your project root.

Minimal Example

paths:
  evals: evals/

paths

Configures where to find skills, evals, and results.

Field	Type	Default	Description
skills	string or string[]	["skills/"]	Directories containing SKILL.md files
evals	string or string[]	["evals/"]	Directories to scan for eval files
results	string	"vally-results/"	Directory for output results
evalFilenames	string or string[]	["eval.yaml", "eval.yml"]	Filename patterns for eval discovery (supports globs)

Custom Filename Patterns

paths:
  evals: [evals/, tests/evals/]
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

Note

When evalFilenames is set, it replaces the defaults entirely. Include eval.yaml explicitly if you still want it discovered.

suites

Named groups of evals for running specific subsets. Each suite must have at least one of filter or evals.

Filter-Only Suite

Select stimuli by tags across all discovered evals:

suites:
  fast:
    description: "P0 tests for inner loop"
    filter:
      priority: p0

Filter semantics: AND across keys, OR within values. Suite filter: blocks are include-only — the --tag key!=value exclusion form is available only on the CLI, and a bare key like flaky! here is treated as a literal tag key, not a negation.

Evals-Only Suite

Scope to specific eval files by path or glob:

suites:
  safety:
    description: "All safety evals"
    evals:
      - "evals/safety/**/*.eval.yaml"
      - "evals/shared/baseline.eval.yaml"

Note

Suite evals globs resolve directly against the filesystem, relative to the project root. They can reference any eval file, not just those matching evalFilenames patterns, and may reach a sibling tree outside the project root via .. (e.g. ../../tools/**/eval.yaml) or an absolute path. A pattern that resolves to the filesystem root is refused.

Combined Suite

Scope by file path, then filter by tags:

suites:
  safety-p0:
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

File scoping narrows which eval files are loaded. Tag filtering narrows which stimuli within those files run.

Path casing

Paths written in suite or experiment evals patterns and --eval-filter must match on-disk casing. The project or experiment directory used to resolve relative patterns follows the filesystem’s casing rules.

Metadata

Root metadata provides passive, JSON-compatible defaults to integrations such as reporter plugins:

metadata:
  dashboard:
    project: agent-platform
  owners: [evals]

suites:
  nightly:
    evals: ["evals/**/*.eval.yaml"]
    metadata:
      dashboard:
        project: nightly-evals
      cadence: nightly

Metadata does not affect eval discovery or stimulus filtering. Reporters receive the effective metadata in PlanRunStartContext.metadata. When a named suite is selected, its metadata is shallow-merged over the root metadata, replacing matching top-level keys. The selected suite and the same effective metadata are also available in PlanRunStartContext.suite.

Namespace integration-specific keys to avoid collisions. Do not store secrets in metadata. Reporter plugins may persist or display the values.

environments

Named agent environments for reuse throughout an eval spec via its agent_environment field. These blocks are agent-visible setup; grader-only files belong in the eval spec’s grading_environment block and cannot use a named reference.

environments:
  auth-workspace:
    skills:
      - skills/auth
    files:
      - src: fixtures/users.json
        dest: test-data/users.json
    commands:
      - npm install

Reference in eval specs by name:

agent_environment: auth-workspace
stimuli:
  - name: login-test
    prompt: "Test the login flow"

Place the fields shown below directly under environments.<name> as shown above.

Agent environment fields

skills:
  - ./path/to/my-skill # Skill directory (containing SKILL.md)
files:
  - src: fixtures/input.txt
    dest: input.txt
  - src: fixtures/test-data # directories are copied recursively
    dest: test-data
  - src: fixtures/session-log # staged outside the graded workspace
    dest: session-state/session-1
    dest_root: assets
commands:
  - npm install
commandTimeout: 2m
git:
  type: worktree
  ref: v2.1.0
  source: ../my-repo
mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
  api:
    type: http
    url: http://localhost:3000/mcp
env:
  COPILOT_AGENT_ACTION: review
  LOG_LEVEL: debug

Field	Type	Description
commands	string[]	Shell commands to run during setup (uses /bin/sh on Unix, cmd.exe on Windows)
commandTimeout	duration	Per-command timeout for commands (e.g. 2m, 90s). Must be positive (0 is rejected). Defaults to 60s
env	Record<string, string>	Environment variables set on the agent process (see below)
files	{src, dest, dest_root?}[]	Files or directories to copy during setup (see below)
git	object	Git configuration for fixture data — local worktree or remote clone (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to skill directories (each containing a SKILL.md) to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Staging files

Each files entry copies src to a destination chosen by dest_root:

Field	Type	Required	Description
src	string	Yes	File or directory to copy, relative to the eval file’s directory (or absolute)
dest	string	Yes	Destination path. Under dest_root: assets it must stay inside the assets root once normalized (no absolute paths, and no traversal that ends up above the root); under the default workdir it is resolved relative to the workspace root
dest_root	"workdir" | "assets"	No	Which root dest is resolved under. Defaults to workdir

• dest_root: workdir (default) — the file lands inside the graded workspace, exactly as before. It stays out of diffs — the workspace baseline is captured after setup — but it is a real file in workDir, so tree- and file-scanning graders (file-exists, file-contains, …) do see it.
• dest_root: assets — the file is staged into a managed, per-trial assets directory that sits outside the graded workspace. Nothing is copied into workDir, so neither diff- nor tree-scanning graders see it, and the directory is cleaned up automatically at the end of the trial (no manual delete step). Use this for setup inputs that must exist on disk but should not contaminate the graded tree — recorded session logs to seed a session, or large fixtures a grader reads.

The assets directory is exposed as the EVALUATE_ASSETS environment variable (see Agent env) and to graders as trajectory.assetsDir (the program grader also receives it as EVALUATE_ASSETS). It is a host path allocated fresh per trial; treat it as absent when re-grading a stored trajectory on a different host.

Caution

The assets directory is agent-visible: its path is passed to the agent as EVALUATE_ASSETS, and its contents are not made read-only. Only workspace and diff scanning exclude it — the agent can read, and modify, anything staged there. Do not stage grader or test scripts as assets, and do not rely on an asset being unchanged at grade time.

Caution

To place a setup file outside the graded workspace, use dest_root: assets — it stages into the managed, auto-cleaned assets dir and keeps workDir clean. Under dest_root: assets, dest is normalized first and must still land inside the assets root: absolute paths are rejected, and so is any .. traversal that escapes it. An interior .. that cancels out — nested/../seed.txt is just seed.txt — is fine.

A workdir-rooted dest is joined onto the workspace with path.join, so an absolute-looking dest keeps the workspace prefix (/etc/seed.txt lands at <workDir>/etc/seed.txt). Only a leading .. actually escapes, landing in a sibling of the per-trial workspace that trial cleanup never removes. Because that location is shared across trials, a leftover can bleed into later runs, so vally lint flags it with an environment-file-workdir-escape warning and points you at dest_root: assets. Under dest_root: assets both an absolute dest and an escaping .. are an environment-file-assets-escape error, since assets staging rejects them outright.

Harness customization files

Use files to stage agent customizations at the location expected by the selected executor:

GitHub Copilot

files:
  - src: ./agents/reviewer.agent.md
    dest: .github/agents/reviewer.agent.md

Claude Code

files:
  - src: ./agents/reviewer.md
    dest: .claude/agents/reviewer.md

Both harnesses discover staged profiles natively — Vally does not parse agent files, so the full frontmatter each harness supports works as documented upstream. Staging the file is the entire setup: there is no flag to turn on and no agent field in the eval spec. For Copilot, a Git-backed workspace with a GitHub remote may additionally expose organization or enterprise agents to the authenticated identity, so the available roster can differ between runners.

Neither harness replaces the default root agent. A matching profile is invoked as a model-routed subagent: the harness compares the stimulus against each profile’s name and description and decides whether to delegate. Write a stimulus that clearly matches the profile you want to exercise.

Because that routing is a model decision, gate it with a grader rather than assuming it happened. For Copilot, given an agent file whose frontmatter sets name: Reviewer:

stimuli:
  - name: delegates-to-reviewer
    prompt: "Delegate this to the Reviewer specialist and report its findings."
    graders:
      - type: tool-calls
        config:
          required:
            - name: "^task$"
              args:
                agent_type: "^Reviewer$"

The eval now fails when the agent is never used. agent_type is the agent’s display name: the frontmatter name when present, otherwise the filename stem — a reviewer.agent.md with no name is matched as ^reviewer$. Argument names are harness-specific, so inspect a recorded trajectory before writing the matcher for another executor. Subagent events also carry an agentId, which grader-level scope can target to assert what the subagent actually did, not just that it ran.

Caution

For Copilot, staged agents are loaded through the runtime’s workspace config discovery, which also picks up skill directories present in the trial workspace. Explicit agent_environment.skills still takes precedence on name collision, but additional workspace-provided skills will load, so treat agent_environment.skills as a floor rather than an exact set and keep the workspace clean when you need strict isolation. Workspace MCP configuration (.mcp.json, .github/mcp.json) is not part of session discovery, so MCP servers still come only from agent_environment.mcpServers. The hosted github-mcp-server is disabled so evals do not silently gain a network dependency; declare your own server under agent_environment.mcpServers if you need GitHub tools, including under that same name, which replaces the hosted one rather than being suppressed. For Codex, AGENTS.md provides workspace instructions, not named subagents.

Note

commands run from the workspace directory, not the directory you launched vally from. Use files (or a git worktree) to copy any scripts or assets a command needs into the workspace, and reference them with paths relative to the workspace root.

Each command is executed in the system shell (/bin/sh on Unix, cmd.exe on Windows) with a per-command timeout set by commandTimeout (defaults to 60s). Shell features such as pipes, redirects, and variable expansion are available.

Agent env

The env field sets environment variables on the agent process for every run of this environment. The built-in copilot-sdk executor and custom process-spawning executors that support environment variables merge these over the inherited environment. The built-in mock executor has no agent process and ignores env; Vally prints a warning listing the ignored variable names so the misconfiguration is visible.

env:
  COPILOT_AGENT_ACTION: review
  LOG_LEVEL: debug

Field	Type	Description
env	Record<string, string>	Variable name → value, set on the agent process. In experiment variant overrides, values support ${…} interpolation like other override fields.

Note

This sets the agent’s runtime environment; the values are not substituted into other spec fields, and are distinct from an MCP server’s own env (which configures that server’s child process rather than the agent).

Note

When the environment stages at least one file with dest_root: assets, Vally sets EVALUATE_ASSETS on the agent process to the absolute path of the per-trial assets directory, and expands the literal ${EVALUATE_ASSETS} token inside your env values. This lets a spec point the agent at a staged input machine-independently, for example:

files:
  - src: fixtures/session-log
    dest: session-state/session-1
    dest_root: assets
env:
  RESUME_SESSION_FIXTURE_PATH: "${EVALUATE_ASSETS}/session-state/session-1"

${EVALUATE_ASSETS} is the only token expanded, and only when assets are staged; no other interpolation is performed on env values here.

Caution

env values run inside the agent process. Variables such as NODE_OPTIONS, LD_PRELOAD, or PATH can change what code the agent executes, so treat authoring env as trust-equivalent to running code on the eval host — be cautious with specs from less-trusted collaborators.

Note

In an experiment variant override, env deep-merges by key with the inherited environment: set a single key to null (e.g. FOO: null) to delete an inherited variable, or set the whole field to null to clear all inherited variables.

Caution

Reports and provenance artifacts show only the names of these variables, never their values. However, trajectory files (the raw agent output) are not filtered — if the agent process echoes an env value (e.g. in a debug log or tool output), it will appear verbatim in the trajectory. Avoid using env to pass long-lived secrets; prefer tokens the agent already has from the host environment.

Note

Like every other config field, env values are part of the environment’s config identity — they feed the config hash recorded in provenance and are compared by experiment drift detection (two variants whose env differs on a path that isn’t a declared vary axis are flagged as drift, with the values redacted). A consequence: a value that changes between runs — e.g. a rotated token interpolated into env — changes that identity, which is a further reason to keep long-lived or rotating secrets out of env.

Git config

The git field sets up the evaluation workspace from a Git repository. It has two modes, discriminated by type:

• worktree — check out a ref from a local repository as a detached worktree.
• clone — clone a remote repository at a given ref into the workspace.

type: worktree

git:
  type: worktree
  ref: v2.1.0
  source: ../my-repo
  timeout: 10m # optional; raise for very large repos
commands:
  - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source
timeout	duration	No	Timeout for the local git steps that materialize the worktree (add, remove), e.g. 10m. Must be positive. Defaults to 5m

Note

Worktree checkout runs non-interactively (GIT_TERMINAL_PROMPT=0, and a batch-mode ssh unless you export GIT_SSH_COMMAND), so an unattended run fails fast instead of blocking on a credential or passphrase prompt from an LFS/smudge filter or post-checkout hook. A caller-provided GIT_SSH_COMMAND is honored.

type: clone

git:
  type: clone
  url: https://github.com/octocat/hello.git
  ref: v2.1.0 # optional; defaults to the remote's default branch
  shallow: true # optional; true → depth 1, or an explicit integer depth
  sparse: # optional; cone-mode paths to materialize
    - src/core
  timeout: 10m # optional; local clone steps only (not the network fetch)

Field	Type	Required	Description
type	"clone"	Yes	Must be "clone"
url	string	Yes	Remote repository URL to clone: http(s), ssh, git, file://, or scp-style git@host:path
ref	string	No	Commit-ish to check out; defaults to the remote’s default branch
shallow	boolean | integer	No	Shallow-clone history: true fetches depth 1, an integer sets an explicit depth
sparse	string[]	No	Sparse-checkout (cone mode) paths — only these directories are materialized
timeout	duration	No	Timeout for each local git step (init, remote add, sparse-checkout, checkout), e.g. 10m. Does not cap the network fetch. Must be positive. Defaults to 5m

HTTP(S) clone URLs must not contain userinfo, query/fragment parameters, or backslashes, and SSH URLs must not contain passwords. Use host Git authentication such as Git Credential Manager, an SSH agent, or GIT_ASKPASS.

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

The copilot-sdk executor supports MCP sampling (model requests made by an MCP server) automatically, with no additional configuration. These requests count toward the trial’s time budget and can incur additional model usage.

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

plugins

Loads plugins for commands operating on the project. Package names and local paths are supported; local paths resolve relative to the directory containing .vally.yaml.

plugins:
  grader:
    - ./plugins/grader.mjs
  executor:
    - ./plugins/executor.mjs
  reporter:
    - ./plugins/reporter.mjs
  eval_provider:
    - ./plugins/eval-provider.mjs
  backend:
    - ./plugins/backend.mjs

defaults:
  eval_provider: custom-eval-provider
  backend:
    name: custom-backend
    args: [pool=research]

plugins lists the modules to load and register with Vally. All fields are additive lists, and duplicate modules are loaded once. Loading makes implementations available; where selection is required, it happens in the configuration that uses them. For example, an eval selects an executor through defaults.executor and a grader through a stimulus’s graders entries.

The matching grader, executor, and reporter CLI options add modules. --eval-plugin replaces both project eval-provider loading and defaults.eval_provider. --backend and --backend-args replace project backend loading and the entire defaults.backend selection.

For vally eval, defaults.eval_provider requires --eval-spec and cannot be used with --suite.

Command	Project plugins used
eval	grader, executor, reporter, eval_provider, backend
experiment run	grader, executor, reporter, eval_provider, backend
grade, oracle, lint, export	grader

When eval receives explicit eval files from multiple projects, additive plugins are loaded from each owning project. The run can use only one backend. If --backend is omitted, all projects must select the same defaults.backend name and arguments. A project without a backend selection uses local, so it conflicts with a project that selects a custom backend. Vally reports these conflicts before starting the run. Pass --backend to choose one backend for all projects.

Complete Example

paths:
  skills: [skills/]
  evals: [evals/]
  results: vally-results/
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

metadata:
  dashboard:
    project: vally
  owner: eval-platform

environments:
  default-workspace:
    skills: [skills/copilot]
    commands: [npm install]
    mcpServers:
      db:
        type: stdio
        command: db-serve
        args: ["--port", "5432"]

suites:
  ci-gate:
    description: "Fast checks for every PR"
    metadata:
      dashboard:
        project: vally-ci
    filter:
      priority: [p0, p1]

  safety:
    description: "All safety evals"
    evals: ["evals/safety/**/*.eval.yaml"]

  safety-p0:
    description: "Critical safety evals only"
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

plugins:
  grader:
    - ./plugins/project-graders.mjs
  executor:
    - ./plugins/project-executor.mjs
  reporter:
    - ./plugins/ci-reporter.mjs
  backend:
    - ./plugins/project-backend.mjs

defaults:
  backend:
    name: project-backend
    args: [pool=research]

Schema Validation

The JSON schema is at packages/core/src/config/vally-config.schema.json. For VS Code autocompletion, add to settings:

{
  "yaml.schemas": {
    "./packages/core/src/config/vally-config.schema.json": ".vally.yaml"
  }
}