.vally.yaml Configuration

The .vally.yaml file defines project-level configuration for eval discovery, named environments, and eval suites. Place it at your project root.

Minimal Example

paths:
  evals: evals/

paths

Configures where to find skills, evals, and results.

Field	Type	Default	Description
skills	string or string[]	["skills/"]	Directories containing SKILL.md files
evals	string or string[]	["evals/"]	Directories to scan for eval files
results	string	"vally-results/"	Directory for output results
evalFilenames	string or string[]	["eval.yaml", "eval.yml"]	Filename patterns for eval discovery (supports globs)

Custom Filename Patterns

paths:
  evals: [evals/, tests/evals/]
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

Note

When evalFilenames is set, it replaces the defaults entirely. Include eval.yaml explicitly if you still want it discovered.

suites

Named groups of evals for running specific subsets. Each suite must have at least one of filter or evals.

Filter-Only Suite

Select stimuli by tags across all discovered evals:

suites:
  fast:
    description: "P0 tests for inner loop"
    filter:
      priority: p0

Filter semantics: AND across keys, OR within values. Suite filter: blocks are include-only — the --tag key!=value exclusion form is available only on the CLI, and a bare key like flaky! here is treated as a literal tag key, not a negation.

Evals-Only Suite

Scope to specific eval files by path or glob:

suites:
  safety:
    description: "All safety evals"
    evals:
      - "evals/safety/**/*.eval.yaml"
      - "evals/shared/baseline.eval.yaml"

Note

Suite evals globs resolve directly against the filesystem, relative to the project root. They can reference any eval file, not just those matching evalFilenames patterns, and may reach a sibling tree outside the project root via .. (e.g. ../../tools/**/eval.yaml) or an absolute path. A pattern that resolves to the filesystem root is refused.

Use the exact on-disk casing: matching is case-sensitive.

Combined Suite

Scope by file path, then filter by tags:

suites:
  safety-p0:
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

File scoping narrows which eval files are loaded. Tag filtering narrows which stimuli within those files run.

environments

Named environments for reuse across eval specs.

environments:
  auth-workspace:
    skills:
      - skills/auth
    files:
      - src: fixtures/users.json
        dest: test-data/users.json
    commands:
      - npm install

Reference in eval specs by name:

agent_environment: auth-workspace
stimuli:
  - name: login-test
    prompt: "Test the login flow"

Environment Fields

environment:
  skills:
    - ./path/to/my-skill # Skill directory (containing SKILL.md)
  files:
    - src: fixtures/input.txt
      dest: input.txt
    - src: fixtures/test-data # directories are copied recursively
      dest: test-data
    - src: fixtures/session-log # staged outside the graded workspace
      dest: session-state/session-1
      dest_root: assets
  commands:
    - npm install
  commandTimeout: 2m
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  mcpServers:
    db:
      type: stdio
      command: db-serve
      args: ["--port", "5432"]
    api:
      type: http
      url: http://localhost:3000/mcp
  env:
    COPILOT_AGENT_ACTION: review
    LOG_LEVEL: debug

Field	Type	Description
commands	string[]	Shell commands to run during setup (uses /bin/sh on Unix, cmd.exe on Windows)
commandTimeout	duration	Per-command timeout for commands (e.g. 2m, 90s). Must be positive (0 is rejected). Defaults to 60s
env	Record<string, string>	Environment variables set on the agent process (see below)
files	{src, dest, dest_root?}[]	Files or directories to copy during setup (see below)
git	object	Git configuration for fixture data — local worktree or remote clone (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to skill directories (each containing a SKILL.md) to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Staging files

Each files entry copies src to a destination chosen by dest_root:

Field	Type	Required	Description
src	string	Yes	File or directory to copy, relative to the eval file’s directory (or absolute)
dest	string	Yes	Destination path. Under dest_root: assets it must stay inside the assets root once normalized (no absolute paths, and no traversal that ends up above the root); under the default workdir it is resolved relative to the workspace root
dest_root	"workdir" | "assets"	No	Which root dest is resolved under. Defaults to workdir

• dest_root: workdir (default) — the file lands inside the graded workspace, exactly as before. It stays out of diffs — the workspace baseline is captured after setup — but it is a real file in workDir, so tree- and file-scanning graders (file-exists, file-contains, …) do see it.
• dest_root: assets — the file is staged into a managed, per-trial assets directory that sits outside the graded workspace. Nothing is copied into workDir, so neither diff- nor tree-scanning graders see it, and the directory is cleaned up automatically at the end of the trial (no manual delete step). Use this for setup inputs that must exist on disk but should not contaminate the graded tree — recorded session logs to seed a session, or large fixtures a grader reads.

The assets directory is exposed as the EVALUATE_ASSETS environment variable (see Agent env) and to graders as trajectory.assetsDir (the program grader also receives it as EVALUATE_ASSETS). It is a host path allocated fresh per trial; treat it as absent when re-grading a stored trajectory on a different host.

Caution

The assets directory is agent-visible: its path is passed to the agent as EVALUATE_ASSETS, and its contents are not made read-only. Only workspace and diff scanning exclude it — the agent can read, and modify, anything staged there. Do not stage grader or test scripts as assets, and do not rely on an asset being unchanged at grade time.

Caution

To place a setup file outside the graded workspace, use dest_root: assets — it stages into the managed, auto-cleaned assets dir and keeps workDir clean. Under dest_root: assets, dest is normalized first and must still land inside the assets root: absolute paths are rejected, and so is any .. traversal that escapes it. An interior .. that cancels out — nested/../seed.txt is just seed.txt — is fine.

A workdir-rooted dest is joined onto the workspace with path.join, so an absolute-looking dest keeps the workspace prefix (/etc/seed.txt lands at <workDir>/etc/seed.txt). Only a leading .. actually escapes, landing in a sibling of the per-trial workspace that trial cleanup never removes. Because that location is shared across trials, a leftover can bleed into later runs, so vally lint flags it with an environment-file-workdir-escape warning and points you at dest_root: assets. Under dest_root: assets both an absolute dest and an escaping .. are an environment-file-assets-escape error, since assets staging rejects them outright.

Harness customization files

Use files to stage agent customizations at the location expected by the selected executor:

GitHub Copilot

environment:
  files:
    - src: ./agents/reviewer.agent.md
      dest: .github/agents/reviewer.agent.md

Claude Code

environment:
  files:
    - src: ./agents/reviewer.md
      dest: .claude/agents/reviewer.md

Both harnesses discover staged profiles natively — Vally does not parse agent files, so the full frontmatter each harness supports works as documented upstream. Staging the file is the entire setup: there is no flag to turn on and no agent field in the eval spec. For Copilot, a Git-backed workspace with a GitHub remote may additionally expose organization or enterprise agents to the authenticated identity, so the available roster can differ between runners.

Neither harness replaces the default root agent. A matching profile is invoked as a model-routed subagent: the harness compares the stimulus against each profile’s name and description and decides whether to delegate. Write a stimulus that clearly matches the profile you want to exercise.

Because that routing is a model decision, gate it with a grader rather than assuming it happened. For Copilot, given an agent file whose frontmatter sets name: Reviewer:

stimuli:
  - name: delegates-to-reviewer
    prompt: "Delegate this to the Reviewer specialist and report its findings."
    graders:
      - type: tool-calls
        config:
          required:
            - name: "^task$"
              args:
                agent_type: "^Reviewer$"

The eval now fails when the agent is never used. agent_type is the agent’s display name: the frontmatter name when present, otherwise the filename stem — a reviewer.agent.md with no name is matched as ^reviewer$. Argument names are harness-specific, so inspect a recorded trajectory before writing the matcher for another executor. Subagent events also carry an agentId, which grader-level scope can target to assert what the subagent actually did, not just that it ran.

Caution

For Copilot, staged agents are loaded through the runtime’s workspace config discovery, which also picks up skill directories present in the trial workspace. Explicit environment.skills still takes precedence on name collision, but additional workspace-provided skills will load, so treat environment.skills as a floor rather than an exact set and keep the workspace clean when you need strict isolation. Workspace MCP configuration (.mcp.json, .github/mcp.json) is not part of session discovery, so MCP servers still come only from environment.mcpServers. The hosted github-mcp-server is disabled so evals do not silently gain a network dependency; declare your own server under environment.mcpServers if you need GitHub tools, including under that same name, which replaces the hosted one rather than being suppressed. For Codex, AGENTS.md provides workspace instructions, not named subagents.

Note

commands run from the workspace directory, not the directory you launched vally from. Use files (or a git worktree) to copy any scripts or assets a command needs into the workspace, and reference them with paths relative to the workspace root.

Each command is executed in the system shell (/bin/sh on Unix, cmd.exe on Windows) with a per-command timeout set by commandTimeout (defaults to 60s). Shell features such as pipes, redirects, and variable expansion are available.

Agent env

The env field sets environment variables on the agent process for every run of this environment. All process-spawning executors (claude-cli, cca, copilot-sdk) merge these over the inherited environment. The built-in mock executor has no agent process and ignores env; Vally prints a warning listing the ignored variable names so the misconfiguration is visible.

environment:
  env:
    COPILOT_AGENT_ACTION: review
    LOG_LEVEL: debug

Field	Type	Description
env	Record<string, string>	Variable name → value, set on the agent process. In experiment variant overrides, values support ${…} interpolation like other override fields.

Note

This sets the agent’s runtime environment; the values are not substituted into other spec fields, and are distinct from an MCP server’s own env (which configures that server’s child process rather than the agent).

Note

When the environment stages at least one file with dest_root: assets, Vally sets EVALUATE_ASSETS on the agent process to the absolute path of the per-trial assets directory, and expands the literal ${EVALUATE_ASSETS} token inside your env values. This lets a spec point the agent at a staged input machine-independently, for example:

environment:
  files:
    - src: fixtures/session-log
      dest: session-state/session-1
      dest_root: assets
  env:
    RESUME_SESSION_FIXTURE_PATH: "${EVALUATE_ASSETS}/session-state/session-1"

${EVALUATE_ASSETS} is the only token expanded, and only when assets are staged; no other interpolation is performed on env values here.

Caution

env values run inside the agent process. Variables such as NODE_OPTIONS, LD_PRELOAD, or PATH can change what code the agent executes, so treat authoring env as trust-equivalent to running code on the eval host — be cautious with specs from less-trusted collaborators.

Note

In an experiment variant override, env deep-merges by key with the inherited environment: set a single key to null (e.g. FOO: null) to delete an inherited variable, or set the whole field to null to clear all inherited variables.

Caution

Reports and provenance artifacts show only the names of these variables, never their values. However, trajectory files (the raw agent output) are not filtered — if the agent process echoes an env value (e.g. in a debug log or tool output), it will appear verbatim in the trajectory. Avoid using env to pass long-lived secrets; prefer tokens the agent already has from the host environment.

Note

Like every other config field, env values are part of the environment’s config identity — they feed the config hash recorded in provenance and are compared by experiment drift detection (two variants whose env differs on a path that isn’t a declared vary axis are flagged as drift, with the values redacted). A consequence: a value that changes between runs — e.g. a rotated token interpolated into env — changes that identity, which is a further reason to keep long-lived or rotating secrets out of env.

Git config

The git field sets up the evaluation workspace from a Git repository. It has two modes, discriminated by type:

• worktree — check out a ref from a local repository as a detached worktree.
• clone — clone a remote repository at a given ref into the workspace.

For Harbor-exported tasks, both worktree and clone exports are finalized as self-contained, pinned checkouts baked into the task image at /app on Linux or C:\app on Windows. Before shipping the checkout, Harbor strips the transient clone origin (for worktrees, an implementation-created file:// remote), reflogs, fetch record, and hooks so remote credentials and local source paths are not shipped. Git LFS-tracked files remain pointers rather than materialized content and emit warnings; submodules are not materialized and emit warnings; and symlink and executable-mode fidelity depend on the export host OS. When the Harbor exporter installs the runtime, it also installs Git for Git-backed tasks; with runtime installation disabled, the custom base image must provide Git for agent-side Git commands.

type: worktree

For Harbor-exported tasks, environment.git with type: worktree is materialized at export time from the local source repository as a self-contained checkout of the pinned commit baked into the task image at the OS-specific workspace path. The export retains history reachable from that commit, which can increase output size for repositories with large histories.

Caution

Harbor uses host Git configuration for authentication but overrides line-ending settings for a deterministic LF checkout. It rejects non-LFS custom filter declarations because their commands cannot be reproduced safely in the image.

Caution

A Harbor Git checkout replaces the base image’s entire workspace directory (/app on Linux or C:\app on Windows). Do not keep required runtime files there in a custom baseImage.

environment:
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
    timeout: 10m # optional; raise for very large repos
  commands:
    - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source
timeout	duration	No	Timeout for the local git steps that materialize the worktree (add, remove), e.g. 10m. Must be positive. Defaults to 5m

Note

Worktree checkout runs non-interactively (GIT_TERMINAL_PROMPT=0, and a batch-mode ssh unless you export GIT_SSH_COMMAND), so an unattended run fails fast instead of blocking on a credential or passphrase prompt from an LFS/smudge filter or post-checkout hook. A caller-provided GIT_SSH_COMMAND is honored.

type: clone

environment:
  git:
    type: clone
    url: https://github.com/octocat/hello.git
    ref: v2.1.0 # optional; defaults to the remote's default branch
    shallow: true # optional; true → depth 1, or an explicit integer depth
    sparse: # optional; cone-mode paths to materialize
      - src/core
    timeout: 10m # optional; local clone steps only (not the network fetch)

Field	Type	Required	Description
type	"clone"	Yes	Must be "clone"
url	string	Yes	Remote repository URL to clone: http(s), ssh, git, file://, or scp-style git@host:path
ref	string	No	Commit-ish to check out; defaults to the remote’s default branch
shallow	boolean | integer	No	Shallow-clone history: true fetches depth 1, an integer sets an explicit depth
sparse	string[]	No	Sparse-checkout (cone mode) paths — only these directories are materialized
timeout	duration	No	Timeout for each local git step (init, remote add, sparse-checkout, checkout), e.g. 10m. Does not cap the network fetch. Must be positive. Defaults to 5m

HTTP(S) clone URLs must not contain userinfo, query/fragment parameters, or backslashes, and SSH URLs must not contain passwords. Use host Git authentication such as Git Credential Manager, an SSH agent, or GIT_ASKPASS.

For Harbor-exported tasks, environment.git with type: clone contacts the remote repository during export and uses the host’s Git configuration, including credential helpers. After materialization, the resulting checkout is a self-contained, pinned checkout (commit SHA) baked into the task image. The Docker build may still require network access for the base image, runtime installation, or environment commands.

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Complete Example

paths:
  skills: [skills/]
  evals: [evals/]
  results: vally-results/
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]

environments:
  default-workspace:
    skills: [skills/copilot]
    commands: [npm install]
    mcpServers:
      db:
        type: stdio
        command: db-serve
        args: ["--port", "5432"]

suites:
  ci-gate:
    description: "Fast checks for every PR"
    filter:
      priority: [p0, p1]

  safety:
    description: "All safety evals"
    evals: ["evals/safety/**/*.eval.yaml"]

  safety-p0:
    description: "Critical safety evals only"
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }

Schema Validation

The JSON schema is at packages/core/src/config/vally-config.schema.json. For VS Code autocompletion, add to settings:

{
  "yaml.schemas": {
    "./packages/core/src/config/vally-config.schema.json": ".vally.yaml"
  }
}