Eval Spec Reference

The eval.yaml file defines what to test and how to grade it. This page documents every field.

Note

Eval spec files are discovered by filename pattern. The default patterns are eval.yaml and eval.yml. Custom patterns can be configured via paths.evalFilenames in .vally.yaml.

Tip

Run vally lint --eval-spec eval.yaml to check your spec for errors before execution. Validation also runs automatically in eval and grade commands. See the lint reference for all checks.

Top-level fields

Field	Type	Required	Description
name	string	No	Human-readable name for this eval
description	string	No	What this eval tests
version	string	No	Version for tracking
type	"capability" | "regression"	No	Eval intent
environment	EnvironmentConfig	No	Root environment (merged into all stimuli)
defaults	EvalDefaults	No	Execution configuration
stimuli	Stimulus[]	Yes	Test cases (at least one required)
scoring	ScoringConfig	No	Score aggregation settings

Defaults

defaults:
  runs: 3 # Trials per stimulus (default: 1)
  timeout: 2m # Per trial (e.g. 5m, 300s, 30000ms); unit suffix required (default: 2m)
  model: gpt-5.5 # Model for agent execution
  reasoning_effort: high # Reasoning effort for the agent under test
  judge_model: gpt-5.5 # Default model for LLM graders
  judge_reasoning_effort: high # Reasoning effort for the judge model
  executor: copilot-sdk # or custom executor via --executor-plugin

Field	Type	Default	Description
runs	number	1	Positive integer specifying the number of trials per stimulus. 1 = single run. ≥ 2 = multi-trial with pass@k/pass^k aggregation.
timeout	duration	2m	Time before a trial times out. Requires a unit suffix (e.g. 5m, 300s, 30000ms).
model	string	—	Model identifier for the executor
reasoning_effort	string	—	Reasoning effort for the agent under test: low, medium, high, or xhigh. Forwarded to the executor (including custom executor plugins) via ExecutorOptions.reasoningEffort. When unset, the agent runs at the model’s own default effort. Only takes effect on models that support reasoning effort; unsupported models may ignore or reject the setting. Overridden by the --reasoning-effort CLI flag; falls back to the EVAL_REASONING_EFFORT env var.
judge_model	string	—	Default model for LLM judge graders (prompt and panel). Graders can override this in their own config.
judge_reasoning_effort	string	—	Reasoning effort for the judge model: low, medium, high, or xhigh. Applies only to the eval-level judge_model — not to per-grader model overrides or the panel’s models. When unset, the judge runs at the model’s own default effort. Only takes effect on models that support reasoning effort; unsupported models may ignore or reject the setting.
executor	string | object	—	Executor for this eval. Either a bare name (copilot-sdk), or an object { name, config } where config is executor-specific (see Executor config & BYOK). Built-in: copilot-sdk. Register more via --executor-plugin and reference them by name. A spec selects a single executor. Omit the field entirely to use the default executor — an empty string is rejected at load time.
judge_provider	object	—	BYOK model provider for the LLM judge graders (prompt, panel) — the judge counterpart to executor.config.provider. Points every judge call at a custom OpenAI-compatible endpoint. Either a declarative block or the opt-in { source: copilot-env } selector. See Judge provider (BYOK) below.

Executor config & BYOK

The executor field can be an object that pairs the executor name with executor-specific config. The config shape is defined by the selected executor, and vally fails closed — supplying config for an executor that doesn’t accept it is an error, so a misdirected block never silently no-ops.

The copilot-sdk executor accepts a provider block for bring-your-own-key (BYOK): point the agent under test at your own model endpoint (Azure AI Foundry, OpenAI, Anthropic, Ollama, vLLM, or any OpenAI-compatible server) instead of authenticating through the GitHub/Copilot token chain. This is useful in CI, where a long-lived provider key is more durable than a Copilot-seated token.

defaults:
  model: gpt-5.2-codex
  executor:
    name: copilot-sdk
    config:
      provider:
        type: azure # "openai" | "azure" | "anthropic" (default: "openai")
        baseUrl: https://your-resource.openai.azure.com/openai/v1/
        wireApi: responses # "completions" | "responses" (default: "completions")
        apiKeyEnv: FOUNDRY_API_KEY # name of the env var holding the key
        azure:
          apiVersion: "2024-10-21"

Fields under config.provider:

Field	Type	Default	Description
baseUrl	string	— (required)	Provider API endpoint. Must be an absolute http/https URL (e.g. http://localhost:11434/v1); a scheme-less value like localhost:11434/v1 is rejected.
type	string	openai	Provider type: openai, azure, or anthropic. type selects the wire protocol, not the hosting product — use openai for Ollama / vLLM / LiteLLM / Foundry Local and any other OpenAI-compatible server.
wireApi	string	completions	Wire protocol (openai/azure only). Use responses for newer models.
transport	string	http	Transport for the OpenAI Responses API (wireApi: responses): http or websockets.
apiKeyEnv	string	—	Name of an environment variable holding the API key, read at run time. Credentials are only ever referenced by env-var name — there is no literal apiKey field, so keys never land in the spec or in persisted run artifacts.
bearerTokenEnv	string	—	Name of an environment variable holding a bearer token (Authorization header), read at run time. Takes precedence over apiKeyEnv when both are set (and apiKeyEnv is then not read at all).
azure	object	—	Azure-specific options. Only apiVersion (string) is accepted; unknown keys are rejected.
headers	object	—	Extra HTTP headers on outbound provider requests (string values). Header values are treated as secret and masked in persisted artifacts and error output.
maxPromptTokens	number	—	Positive integer. Overrides the resolved model’s default max prompt tokens (triggers compaction before a request would exceed it).
maxOutputTokens	number	—	Positive integer. Overrides the resolved model’s default max output tokens.

Unknown keys under provider (and under provider.azure) are rejected, not ignored — a typo such as apiKeyENV fails validation at plan/lint time rather than silently disabling the credential.

For most setups you only need baseUrl, defaults.model, and one credential env var — the block below is the common case:

defaults:
  model: qwen3
  executor:
    name: copilot-sdk
    config:
      provider:
        baseUrl: http://localhost:11434/v1 # e.g. a local Ollama server
        apiKeyEnv: OLLAMA_KEY # omit entirely if the endpoint needs no key

The names in apiKeyEnv / bearerTokenEnv are read from the environment of the vally process (the shell or CI job running vally) when each trial runs — not from the eval’s environment.env, which is applied only to the spawned agent and is invisible here. Export the variable before running; a trial fails if its referenced variable is unset (this is checked per trial, at execution time). bearerTokenEnv takes precedence over apiKeyEnv. Because executor is part of defaults, experiment variants can vary it — e.g. compare a local Ollama model against the GHCP API in a single run by overriding /defaults/executor (and /defaults/model) per variant; a variant that omits the provider config runs on the default GHCP auth chain.

Advanced: modelId and wireModel

Most authors set only defaults.model, which serves as both the runtime’s model identity and the name sent to the provider. Override these two (under config.provider) only when those roles must differ:

Field	Type	Default	Description
modelId	string	falls back to model	Well-known model name the runtime looks up to resolve agent configuration (tools, prompts, reasoning behavior) and default token limits. Also used as the provider-facing model name when wireModel is unset.
wireModel	string	falls back to modelId	Model name sent to the provider API for inference, when it differs from modelId.

The resolution order for the name sent to the provider is wireModel → modelId → defaults.model. You need this split when the provider’s name for a model differs from the well-known one — for example an Azure deployment name or a custom fine-tune name (a gateway that renames models, such as LiteLLM or vLLM, is another possible case):

defaults:
  model: gpt-4o
  executor:
    name: copilot-sdk
    config:
      provider:
        type: azure
        baseUrl: https://your-resource.openai.azure.com/openai/v1/
        apiKeyEnv: FOUNDRY_API_KEY
        modelId: gpt-4o # runtime uses this for agent config + token limits
        wireModel: my-gpt4o-deployment # what the Azure API actually expects

Judge provider (BYOK)

defaults.judge_provider points the LLM judge graders (prompt, panel) at a custom OpenAI-compatible endpoint — the judge counterpart to the executor’s config.provider for the agent. judge_model picks which model the judge uses; judge_provider picks where the request is sent. It applies to every LLM judge call in the eval (including each judge in a panel).

defaults:
  judge_model: qwen3
  judge_provider:
    baseUrl: http://localhost:8080/v1 # e.g. a local OpenAI-compatible server
    apiKeyEnv: JUDGE_API_KEY # env-var name; omit if the endpoint needs no key

The block accepts the same fields as the executor provider (see the provider table above): type, baseUrl (required, absolute http/https, no embedded credentials), wireApi, transport, apiKeyEnv / bearerTokenEnv (env-var names — literal secrets are not accepted), azure, headers, modelId, wireModel, maxPromptTokens, maxOutputTokens. The same validation and secret-redaction rules apply, and it can be varied in experiments via /defaults/judge_provider.

The agent’s executor.config.provider and the judge’s judge_provider are independent: you can point the agent at one endpoint and the judge at another (or leave the judge on the default GitHub/Copilot auth chain by omitting judge_provider). Credentials are read from the environment of the vally process, not the eval’s environment.env.

Resolving from environment variables (opt-in)

Instead of a declarative block, the judge provider can be resolved from the ambient COPILOT_PROVIDER_* variables that Copilot CLI itself uses. This suits environments that already inject those variables and don’t author a provider block. Because it changes where judge requests are sent, it is strictly opt-in — the mere presence of COPILOT_PROVIDER_* never repoints the judge on its own.

Opt in one of two ways:

# In the spec:
defaults:
  judge_provider:
    source: copilot-env

# Or without editing the spec, via a gate on the vally process:
export VALLY_JUDGE_PROVIDER_SOURCE=copilot-env

When active, the provider is built from these variables (mirroring Copilot CLI’s buildProviderConfigFromEnv()): COPILOT_PROVIDER_BASE_URL (required), COPILOT_PROVIDER_TYPE, COPILOT_PROVIDER_API_KEY, COPILOT_PROVIDER_BEARER_TOKEN, COPILOT_PROVIDER_WIRE_API, COPILOT_PROVIDER_TRANSPORT, COPILOT_PROVIDER_AZURE_API_VERSION, COPILOT_PROVIDER_MODEL_ID, COPILOT_PROVIDER_WIRE_MODEL, COPILOT_PROVIDER_MAX_PROMPT_TOKENS, COPILOT_PROVIDER_MAX_OUTPUT_TOKENS, and COPILOT_PROVIDER_HEADERS (newline-separated Name: Value pairs).

Literal values vs. names. This is the key difference from the declarative block: COPILOT_PROVIDER_API_KEY / COPILOT_PROVIDER_BEARER_TOKEN hold the literal key/token, whereas the block’s apiKeyEnv / bearerTokenEnv hold the name of a variable. Both paths redact resolved secrets from errors and never persist them; what’s recorded in run artifacts is { source: copilot-env }, not the resolved endpoint or credentials.

Precedence. The declarative block and the env-source selector are mutually exclusive:

defaults.judge_provider	VALLY_JUDGE_PROVIDER_SOURCE	Result
absent	unset	Default GitHub/Copilot auth
absent	copilot-env	Resolve from COPILOT_PROVIDER_*
mapping	unset	Declarative provider
{ source: copilot-env }	unset	Resolve from COPILOT_PROVIDER_*
mapping	copilot-env	Error (conflict)
{ source: copilot-env }	copilot-env	Resolve from COPILOT_PROVIDER_*
any	any other value	Error (invalid gate value)

Failure modes. With the selector active but COPILOT_PROVIDER_BASE_URL unset, the run fails fast rather than silently falling back to GitHub auth (missing key/token is allowed — some local endpoints need none). Invalid enum, integer, header, or URL values also fail with a clear message.

Scope. The selector is honored by eval, grade, oracle, and experiment; it applies only to the LLM judge. The agent executor (executor.config.provider) is unaffected and has no env-source mode. The judge process also strips COPILOT_PROVIDER_* from the environment handed to the Copilot SDK unless resolved through this opt-in, so ambient variables can’t bypass it.

Stimulus

stimuli:
  - name: my-test-case
    prompt: "Do something"
    environment: { ... }
    artifacts: { ... }
    graders: [...]
    rubric: ["criterion 1", "criterion 2"]
    constraints: { ... }

For multi-turn conversations, use turns instead of prompt:

stimuli:
  - name: recall-across-turns
    turns:
      - "Remember this code name: ALPHA-7."
      - "What was the code name I gave you?"
    graders:
      - type: output-contains
        turn: 1
        config:
          substring: "ALPHA-7"

Tip

See Writing Eval Specs → Multi-turn conversations for a walkthrough, samples/multi-turn-memory/ for a runnable example, and vally compare for A/B comparing multi-turn runs.

Field	Type	Required	Description
name	string	Yes	Unique identifier for this stimulus
prompt	string	One of prompt or turns	The prompt sent to the agent. When turns is provided, prompt is synthesized automatically and any explicit value is ignored.
turns	string[]	One of prompt or turns	Ordered list of prompts for a multi-turn conversation. Each entry is sent sequentially to the same agent session.
attachments	string[]	No	File paths (binary images or native documents) attached to the agent’s initial prompt, resolved relative to the eval file (absolute paths allowed). Must be a non-empty array of non-empty strings; existence is checked at validation time. Only attachment-capable executors (e.g. copilot-sdk) accept them; others reject the stimulus.
environment	EnvironmentConfig	No	Stimulus-specific environment (merged with root)
artifacts	ArtifactsConfig	No	File copy filters applied before workspace cleanup to capture output artifacts in run results.
graders	GraderConfig[]	No	Graders to run on the trajectory
rubric	string[]	No	Evaluation criteria for LLM judge graders. Each entry must be a non-empty string. Passed to prompt and panel graders as the criteria they score, and used by vally compare for comparisons (overrideable with --eval-spec). Omitted or empty, LLM judges fall back to a generic built-in rubric.
constraints	Constraints	No	Resource limits for the trial
simulation	Simulation	No	Tool-call simulation: canned responses for specific tools (deterministic, side-effect-free evals)
supported_executors	string[]	No	Allow-list of executor IDs (e.g. copilot-sdk) this stimulus can run under. When the active executor isn’t listed, the stimulus is skipped (recorded with status skipped, not run or failed). Omit to run under any executor. An empty list is rejected at load time.
golden_patch	GoldenPatch	No	Reference-solution diff for oracle grading
golden_trajectory	GoldenTrajectory	No	Reference-solution trajectory (ATIF) for oracle grading of trajectory-/metric-/transcript-scoped graders
golden_custom_metrics	GoldenCustomMetrics	No	Reference-solution custom metrics for oracle grading of custom-metrics graders
tags	Record<string, string | string[]>	No	Key-value pairs for filtering into suites

Artifacts

artifacts:
  include:
    - "**/Cargo.toml"
    - "**/src/**/*.rs"
  exclude:
    - "**/target/**"

Field	Type	Required	Description
include	string[]	Yes	Non-empty list of path patterns to copy.
exclude	string[]	No	Path patterns to remove from the include matches.

Precedence: if a path matches both include and exclude, it is excluded.

Pattern semantics:

• Patterns are path globs: * matches any run of characters within a single path segment, and ** matches across path segments (directories at any depth).
• Patterns without glob tokens are treated as exact file or directory matches. Example: foo.txt matches only that file, while dir matches files under dir/ recursively.
• Copying artifacts recursively scans the workspace to evaluate include/exclude rules, so this option can add noticeable overhead in very large repositories.
• .git is always excluded regardless of include patterns, even if you glob **/*.

Destination layout:

For vally eval, copied files are placed in an artifacts/ subdirectory of the trial’s session-log directory — <runDir>/<eval>/<stimulus>/<model>/<trial>/artifacts/ — so a trial’s logs (events.jsonl/metadata.json) and its copied artifacts sit together. Each copied file keeps its workspace-relative path under artifacts/.

vally experiment co-locates artifacts the same way, under each variant’s per-trial session directory: <runDir>/<variant>/<eval>/<stimulus>/<model>/<trial>/artifacts/.

Golden patch

A golden patch is a unified-diff reference solution for the stimulus. It powers vally oracle, which materializes the stimulus’s starting environment, applies the patch, and runs the graders against the result — so you can verify the harness (graders should pass on the correct answer) and gate CI before publishing a benchmark.

Provide exactly one of inline or path:

stimuli:
  # Patch stored in a file (resolved relative to the eval file's directory)
  - name: fix-the-bug
    prompt: "Fix the off-by-one error in pagination"
    golden_patch:
      path: solutions/fix-the-bug.patch

  # Patch embedded inline
  - name: add-readme
    prompt: "Add a README"
    golden_patch:
      inline: |
        --- /dev/null
        +++ b/README.md
        @@ -0,0 +1 @@
        +# My Project

Field	Type	Required	Description
inline	string	One of the two	The unified-diff patch text, embedded directly in the spec
path	string	One of the two	Path to a patch file, resolved relative to the eval file’s directory

The resolved diff is passed to graders under vally oracle and vally grade, including when a prompt/panel grader selects config.evidence: [golden_patch]. Normal vally eval does not resolve reference solutions, so the selected evidence is unavailable there; use diff to judge the agent’s workspace changes during an inline eval run.

Golden trajectory

A golden trajectory is a reference-solution ATIF trajectory (fake or real) for a stimulus. A golden patch only captures the final file state, so under vally oracle trajectory-, metric-, and transcript-scoped graders (tool-calls, skill-invocation, max-repeat, token-budget, tool-call-count, turn-count, error-count, wall-time, transcript-*, and output-scoped graders) have no events or metrics to validate against and are rejected. Supplying a golden_trajectory gives oracle real events and metrics to grade those graders offline — you can hand-author the events you expect a passing run to produce and confirm the graders agree, no agent run required.

Provide exactly one of inline (an embedded ATIF document) or path (an ATIF JSON file resolved relative to the eval file’s directory). The example below is complete and passes the tool-calls grader as written:

stimuli:
  # Trajectory stored in a file
  - name: runs-the-tests
    prompt: "Run the test suite"
    golden_trajectory:
      path: solutions/runs-the-tests.trajectory.json
    graders:
      - type: tool-calls
        config:
          required: ["bash"]

  # The same trajectory embedded inline
  - name: runs-the-tests-inline
    prompt: "Run the test suite"
    golden_trajectory:
      inline:
        schema_version: "ATIF-v1.6"
        session_id: sess-runs-the-tests
        agent: { name: oracle, version: "1.0" }
        steps:
          - step_id: 0
            source: user
            message: "Run the test suite"
          - step_id: 1
            source: agent
            message: "Running the tests with bash."
            tool_calls:
              - tool_call_id: call-1
                function_name: bash
                arguments: { command: "npm test" }
            observation:
              results:
                # Its source_call_id matches the tool_call in THIS same step.
                - source_call_id: call-1
                  content: "All tests passed"
    graders:
      - type: tool-calls
        config:
          required: ["bash"]

Co-locate each observation with its tool_call

The tool-calls grader (and other tool-result checks) only counts a tool once it sees the matching tool result, which oracle derives from an ATIF observation. Each observation result is matched to a tool_call within the same agent step, so its source_call_id must reference a tool_call_id declared in that step. A source_call_id pointing at a call in a different step resolves to tool "unknown" and never counts — the grader then reports the required tools “were not called” even though the call appears in the trajectory.

Field	Type	Required	Description
inline	object	One of the two	An embedded ATIF trajectory document
path	string	One of the two	Path to an ATIF JSON file, resolved relative to the eval file’s directory

A stimulus can declare a golden_patch, a golden_trajectory, or both: the patch drives file-state graders (against the patched workspace) while the trajectory drives behavior/metric graders. The document is validated as ATIF at oracle time, not at load time. Only vally oracle consumes it — a normal vally eval grades the real agent run.

Golden custom metrics

The custom-metrics grader reads a JSON file the run writes into the workspace (default custom_metrics.json). Under vally oracle no agent runs, so that file only exists if the golden patch creates it. A golden_custom_metrics supplies the file directly (fake or real) so custom-metrics graders can be validated offline without baking the metrics into a patch.

Oracle writes the resolved metrics into the workspace at every path a custom-metrics grader reads (its config.path, or the default custom_metrics.json). Provide exactly one of inline (an embedded metrics object) or path (a JSON file resolved relative to the eval file’s directory):

stimuli:
  - name: tests-pass
    prompt: "Make the tests pass"
    golden_custom_metrics:
      inline:
        tests_failed: 0
        quality_score: 0.9
    graders:
      - type: custom-metrics
        config:
          assertions:
            - metric: tests_failed
              equals: 0
            - metric: quality_score
              min: 0.8

Field	Type	Required	Description
inline	object	One of the two	The metrics object, embedded directly in the spec
path	string	One of the two	Path to a JSON metrics file, resolved relative to the eval file’s directory

The metrics object may be flat ({ "tests_failed": 0 }) or an envelope nesting them under values — the same shapes the custom-metrics grader accepts. The --no-golden-input baseline withholds it, so custom-metrics graders correctly fail on the baseline. Only vally oracle consumes it.

Tags

Tags are key-value pairs used for filtering stimuli into suites. They can be defined at the eval level (inherited by all stimuli) or the stimulus level (overrides eval-level tags on the same key).

tags:
  priority: p0
  area: [auth, security]

Filtering semantics:

• AND across keys — a stimulus must match all specified tag keys
• OR within values — a stimulus matches a key if it has any of the specified values
• Stimulus tags override eval tags on the same key

See Authoring Eval Suites for tagging strategies and suite configuration.

Environment

Environment Fields

environment:
  skills:
    - ./path/to/my-skill # Skill directory (containing SKILL.md)
  files:
    - src: fixtures/input.txt
      dest: input.txt
    - src: fixtures/test-data # directories are copied recursively
      dest: test-data
  commands:
    - npm install
  commandTimeout: 2m
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  mcpServers:
    db:
      type: stdio
      command: db-serve
      args: ["--port", "5432"]
    api:
      type: http
      url: http://localhost:3000/mcp
  env:
    COPILOT_AGENT_ACTION: review
    LOG_LEVEL: debug

Field	Type	Description
commands	string[]	Shell commands to run during setup (uses /bin/sh on Unix, cmd.exe on Windows)
commandTimeout	duration	Per-command timeout for commands (e.g. 2m, 90s). Must be positive (0 is rejected). Defaults to 60s
env	Record<string, string>	Environment variables set on the agent process (see below)
files	{src, dest}[]	Files or directories to copy into the workspace before execution
git	object	Git configuration for fixture data — local worktree or remote clone (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to skill directories (each containing a SKILL.md) to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Note

commands run from the workspace directory, not the directory you launched vally from. Use files (or a git worktree) to copy any scripts or assets a command needs into the workspace, and reference them with paths relative to the workspace root.

Each command is executed in the system shell (/bin/sh on Unix, cmd.exe on Windows) with a per-command timeout set by commandTimeout (defaults to 60s). Shell features such as pipes, redirects, and variable expansion are available.

Agent env

The env field sets environment variables on the agent process for every run of this environment. All process-spawning executors (claude-cli, cca, copilot-sdk) merge these over the inherited environment. The built-in mock executor has no agent process and ignores env; Vally prints a warning listing the ignored variable names so the misconfiguration is visible.

environment:
  env:
    COPILOT_AGENT_ACTION: review
    LOG_LEVEL: debug

Field	Type	Description
env	Record<string, string>	Variable name → value, set on the agent process. In experiment variant overrides, values support ${…} interpolation like other override fields.

Note

This sets the agent’s runtime environment; the values are not substituted into other spec fields, and are distinct from an MCP server’s own env (which configures that server’s child process rather than the agent).

Caution

env values run inside the agent process. Variables such as NODE_OPTIONS, LD_PRELOAD, or PATH can change what code the agent executes, so treat authoring env as trust-equivalent to running code on the eval host — be cautious with specs from less-trusted collaborators.

Note

In an experiment variant override, env deep-merges by key with the inherited environment: set a single key to null (e.g. FOO: null) to delete an inherited variable, or set the whole field to null to clear all inherited variables.

Caution

Reports and provenance artifacts show only the names of these variables, never their values. However, trajectory files (the raw agent output) are not filtered — if the agent process echoes an env value (e.g. in a debug log or tool output), it will appear verbatim in the trajectory. Avoid using env to pass long-lived secrets; prefer tokens the agent already has from the host environment.

Note

Like every other config field, env values are part of the environment’s config identity — they feed the config hash recorded in provenance and are compared by experiment drift detection (two variants whose env differs on a path that isn’t a declared vary axis are flagged as drift, with the values redacted). A consequence: a value that changes between runs — e.g. a rotated token interpolated into env — changes that identity, which is a further reason to keep long-lived or rotating secrets out of env.

Git config

The git field sets up the evaluation workspace from a Git repository. It has two modes, discriminated by type:

• worktree — check out a ref from a local repository as a detached worktree.
• clone — clone a remote repository at a given ref into the workspace.

For Harbor-exported tasks, both worktree and clone exports are finalized as self-contained, pinned checkouts baked into the task image. Before shipping the checkout, Harbor strips the transient clone origin (for worktrees, an implementation-created file:// remote), reflogs, fetch record, and hooks so remote credentials and local source paths are not shipped. Git LFS-tracked files remain pointers rather than materialized content and emit warnings; submodules are not materialized and emit warnings; and symlink and executable-mode fidelity depend on the export host OS.

type: worktree

For Harbor-exported tasks, environment.git with type: worktree is materialized at export time from the local source repository as a self-contained checkout of the pinned commit baked into the task image at /app. The export retains history reachable from that commit, which can increase output size for repositories with large histories.

Caution

Harbor uses host Git configuration for authentication but overrides line-ending settings for a deterministic LF checkout. It rejects non-LFS custom filter declarations because their commands cannot be reproduced safely in the image.

Caution

A Harbor Git checkout replaces the base image’s entire /app directory. Do not keep required runtime files there in a custom baseImage.

environment:
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  commands:
    - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source

type: clone

environment:
  git:
    type: clone
    url: https://github.com/octocat/hello.git
    ref: v2.1.0 # optional; defaults to the remote's default branch
    shallow: true # optional; true → depth 1, or an explicit integer depth
    sparse: # optional; cone-mode paths to materialize
      - src/core

Field	Type	Required	Description
type	"clone"	Yes	Must be "clone"
url	string	Yes	Remote repository URL to clone: http(s), ssh, git, file://, or scp-style git@host:path
ref	string	No	Commit-ish to check out; defaults to the remote’s default branch
shallow	boolean | integer	No	Shallow-clone history: true fetches depth 1, an integer sets an explicit depth
sparse	string[]	No	Sparse-checkout (cone mode) paths — only these directories are materialized

HTTP(S) clone URLs must not contain userinfo, query/fragment parameters, or backslashes, and SSH URLs must not contain passwords. Use host Git authentication such as Git Credential Manager, an SSH agent, or GIT_ASKPASS.

For Harbor-exported tasks, environment.git with type: clone contacts the remote repository during export and uses the host’s Git configuration, including credential helpers. After materialization, the resulting checkout is a self-contained, pinned checkout (commit SHA) baked into the task image. The Docker build may still require network access for the base image, runtime installation, or environment commands.

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Note

When both root and stimulus environments are defined, they are merged with conflict detection:

• Skills: Duplicates across root and stimulus are deduplicated with a warning.
• Commands: Duplicate commands across root and stimulus throw an error.
• Files: Files with the same dest but different src throw a conflict error. Identical entries are also rejected as redundant.

Only cross-array conflicts are detected — duplicates within the same array (root-only or stimulus-only) are not flagged.

Grader config

graders:
  - type: output-contains
    name: "output includes hello"
    config:
      substring: "hello"
      case_sensitive: false

Field	Type	Required	Description
type	string	Yes	Registered grader name (e.g., output-contains, file-exists)
name	string	No	Human-readable display name for this grader instance. Defaults to a name derived from this instance’s config (e.g. output-matches /Cosmos/). Set it explicitly for a name that doesn’t change when the config does.
turn	integer	No	Scope this grader to a specific conversation turn (0-based). The pipeline slices the trajectory to that turn before grading. Also valid for diff-contains, diff-not-contains, and diff-empty, which inspect that turn’s recorded workspace diff; other workspace graders remain unsupported.
scope	string | object	No	Scope this grader to the parent agent or a specific subagent. See Subagent scope. Only valid for non-workspace graders.
config	object	No	Grader-specific configuration (varies by type)

LLM judge evidence

The prompt and panel graders accept a grader-level evidence selector:

graders:
  - type: prompt
    config:
      evidence: [trajectory, diff, golden_patch, repo]

trajectory includes the agent output, execution metrics, and session timeline; diff includes the agent’s cumulative workspace changes; golden_patch includes the stimulus’s reference solution; and repo includes a bounded snapshot of the final workspace filesystem (a file tree plus text-file contents), letting the judge read the agent’s output directly. Sensitive filenames (.env*, private keys, package-manager auth configs) are listed in the tree but their contents are omitted and not sent to the judge. When omitted or empty ([]), evidence defaults to trajectory evidence. When evidence is a non-empty list, it must contain unique supported values and cannot select diff or repo together with grader-level turn or scope. Lint warns when a stimulus declares golden_patch but no prompt or panel grader can send it to an LLM judge. This does not prevent oracle-only patch use.

repo reads the live workspace, so it is available whenever the workspace is on disk (vally eval, and vally grade with a materialized workspace); an absent or unreadable workspace fails the grader with a clear message before the judge is called.

This is separate from the stimulus-level artifacts field: stimulus artifacts copy files out of the workspace, while grader config.evidence controls evidence sent to an LLM judge.

Subagent scope

scope restricts a grader to one agent’s slice of the trajectory, so you can assert behavior of a subagent separately from the parent (root) agent. It composes with turn and, like turn, works generically for any non-workspace grader — the pipeline slices the trajectory before grading.

scope value	Selects
parent	Only the parent/root agent (events with no subagent identity)
subagent	Any subagent (regardless of which)
{ agent: "<name/id>" }	Events whose agent identity equals the given value

Agent identity comes from the adapter’s own labeling, which is not guaranteed to be a unique instance id. So { agent } matches all events sharing that identity: if a parent fans out several same-named subagents, they’re selected together as one aggregate slice, not addressed individually.

graders:
  # The parent delegated the edit; assert it did not edit directly.
  - type: tool-calls
    scope: parent
    config:
      disallowed: [edit]
  # Assert the delegated subagent actually performed the edit.
  - type: tool-calls
    scope: subagent
    config:
      required: [edit]

Per-adapter identity

Per-call agent identity comes from the trajectory adapter, and its meaning — and therefore whether { agent } is usable — differs by source:

• Copilot stamps each event with the subagent agentId from the SDK envelope. scope: parent and scope: subagent work reliably. That id is the opaque launching tool-call id (e.g. toolu_01Biq…), not a friendly agent name, so a { agent } value can’t be authored ahead of a run and isn’t portable across runs — prefer parent / subagent on Copilot.
• ATIF trajectories support subagent scoping when the document inlines its subagent trajectories (subagent_trajectories linked from a step’s subagent_trajectory_ref): the adapter inlines those events stamped with the subagent’s declared agent name, so { agent: "<name>" } is authorable. ATIF subagents referenced only as external files (by session_id / trajectory_path, not inlined) carry no events in the document, so scope: subagent / { agent } selects nothing for those.

When a scope (or scope + turn) selects no events, the grader fails loud with the list of subagent ids that are present (or a note that none were captured), rather than passing vacuously.

Constraints

constraints:
  max_turns: 10
  max_tokens: 5000
  max_duration: 1m
  max_agent_duration: 45s
  expect_tools: ["write_file", "read_file"]
  reject_tools: ["delete_file"]
  expect_skills: ["my-skill"]
  reject_skills: []

Field	Type	Description
max_turns	number	Maximum conversation turns
max_tokens	number	Maximum total tokens
max_duration	duration	Hard wall-clock cap on the agent’s run (e.g. 2m, 30s, 500ms). Unit suffix required. It bounds the in-flight agent work only — workspace setup and teardown/disconnect are bounded separately, not by this cap. Exceeding it aborts the run and fails the eval.
max_agent_duration	duration	Agent working-time limit (e.g. 10m, 90s). The agent gets up to this long to work; on expiry it is stopped and the trial proceeds through normal teardown, grading whatever was produced as a success.
expect_tools	string[]	Tools the agent must call
reject_tools	string[]	Tools the agent must not call
expect_skills	string[]	Skills that must be activated
reject_skills	string[]	Skills that must not be activated

max_duration vs max_agent_duration — two different ways to limit run time:

• max_duration is a hard ceiling on the agent’s wall-clock run. Overrun it and the run is aborted and fails. It bounds the in-flight agent work only — workspace setup and teardown/disconnect are bounded separately, not by this cap.
• max_agent_duration caps only the agent’s working time: “work for up to 10 minutes, then we’ll stop you and grade whatever you finished.” On expiry the agent is stopped and the trial runs through normal teardown, graded as a success.

A few details:

• The working-limit covers the whole stimulus run (all turns), not a single turn.
• Both limits stop the agent the same way under the hood (the executors abort the in-flight agent run — the Copilot SDK and Claude CLI offer no gentler mechanism). The difference is the verdict: max_duration fails the run, while max_agent_duration keeps it a success and grades the partial work.
• When the working-limit stops a run, its trajectory records endReason: "agent_timeout" (vs "completed" for a normal finish) so graders and the analytics server can account for it.
• Setting only max_agent_duration is enough — the run is allowed to last the full working-limit.
• If you set both, keep the working-limit below the hard cap. The effective hard cap is max_duration (or, if unset, the eval’s defaults.timeout, then the --timeout override). When the working-limit is >= that cap, the hard cap fails the run before the working-limit can stop it cleanly. vally lint warns when it can detect this.

Simulation

Run the agent against a deterministic, mocked toolset instead of the real environment. Useful for loop/repetition detection, failure-mode recovery tests, and behavioral evals that must not cause real side effects. Requires a simulation-capable executor (copilot-sdk); other executors reject a stimulus that declares simulation.

Scope: first-party tools only (MCP server tools are not simulated).

stimuli:
  - name: npm-registry-down
    prompt: "Install the project dependencies"
    simulation:
      max_iterations: 20
      tool_overrides:
        # Shell tools can simulate success OR failure, and support ordered
        # regex patterns matched against the command (first match wins).
        bash:
          patterns:
            - match: "npm install|npm ci"
              output: |-
                npm error code ETIMEDOUT
                npm error network request to registry.npmjs.org failed
              is_error: true
            - match: "node -v"
              output: "v14.21.3" # successful stdout (is_error defaults to false)
        # Non-shell tools can only simulate failures.
        web_fetch:
          output: "Error: HTTP 403 Forbidden"
          is_error: true

Field	Type	Required	Description
tool_overrides	Record<string, Override>	No	Per-tool canned responses, keyed by tool name (bash, web_fetch, …).
max_iterations	number	No	Cap on model iterations (assistant turns). Exceeding it stops the run cleanly with endReason: simulation_cap; this is not a run error — graders still run on the collected trajectory.

Each override value is one of:

• string — shorthand for successful stdout ({ output: <str>, is_error: false }). Because it resolves to success, this form is only valid for shell tools (see Shell vs non-shell below); non-shell tools must use { output, is_error: true }.
• object — { output: string, is_error?: boolean }. is_error defaults to false.
• patterns — { patterns: [{ match, output, is_error? }], default? } for input-dependent shell tools. match is a case-insensitive regex tested against the command string; the first match wins. If nothing matches, default is used, or — when omitted — the real tool runs.

Shell vs non-shell: shell tools (bash, shell, powershell, treated as aliases) can simulate a successful or failed result because the command is rewritten to emit the canned output. All other first-party tools can only simulate failures — a non-shell override that resolves to success (e.g. a plain string, or is_error: false) is rejected by vally lint. Pattern overrides are shell-only.

Recording: simulated tool calls are flagged in the trajectory (simulated: true) and counted in metrics (simulatedToolCallCount), so graders and reports can tell simulated calls apart from real ones.

Scoring

scoring:
  weights:
    file-exists: 0.7
    output-contains: 0.3
  threshold: 0.7

Field	Type	Description
weights	Record<string, number>	Weight per grader type. Must be a normalized distribution (values sum to 1.0 ±0.01). When multiple graders share the same type, their scores are averaged before the weight is applied. Graders absent from the map receive weight 0. If omitted, all graders are averaged equally.
threshold	number	Optional. Minimum aggregate score to pass (0.0–1.0). When omitted, the verdict falls back to binary all-graders-pass and scoring.weights has no effect on the verdict. Can be overridden per-run with vally eval --threshold <number>.

Complete example

eval.yaml

name: test-writer-eval
description: Evaluates the test-writer skill
version: "1.0"
type: capability

environment:
  skills:
    - ./my-skill # Skill directory (containing SKILL.md)

defaults:
  runs: 3
  timeout: 2m
  model: gpt-5.5
  executor: copilot-sdk

stimuli:
  - name: basic-test-generation
    prompt: |
      Write unit tests for this function:
      function add(a, b) { return a + b; }
      Save the tests to add.test.js.
    graders:
      - type: file-exists
        name: "test file was created"
        config:
          path: "add.test.js"
      - type: output-contains
        config:
          substring: "test"
    constraints:
      max_turns: 10
      expect_tools: ["write_file"]

  - name: edge-case-empty-input
    prompt: "Write tests for a function that takes no arguments."
    graders:
      - type: file-exists
        config:
          path: "*.test.js"

scoring:
  weights:
    file-exists: 0.7
    output-contains: 0.3
  threshold: 0.7