Eval Spec Reference

The eval.yaml file defines what to test and how to grade it. This page documents every field.

Note

Eval spec files are discovered by filename pattern. The default patterns are eval.yaml and eval.yml. Custom patterns can be configured via paths.evalFilenames in .vally.yaml.

Tip

Run vally lint --eval-spec eval.yaml to check your spec for errors before execution. Validation also runs automatically in eval and grade commands. See the lint reference for all checks.

Top-level fields

Field	Type	Required	Description
name	string	Yes	Human-readable name for this eval
description	string	No	What this eval tests
version	string	No	Version for tracking
type	"capability" | "regression"	No	Eval intent
agent_environment	EnvironmentConfig	No	Root agent-visible environment (merged into all stimuli). Deprecated alias: environment
grading_environment	GradingEnvironmentConfig	No	Root grader-only files, hidden from the agent (merged into all stimuli)
defaults	EvalDefaults	No	Execution configuration
stimuli	Stimulus[]	Yes	Test cases (at least one required)
scoring	ScoringConfig	No	Score aggregation settings

Defaults

defaults:
  runs: 3 # Trials per stimulus (default: 1)
  timeout: 2m # Per trial (e.g. 5m, 300s, 30000ms); unit suffix required (default: 2m)
  model: gpt-5.5 # Model for agent execution
  reasoning_effort: high # Reasoning effort for the agent under test
  judge_model: gpt-5.6-terra # Default model for LLM graders
  judge_timeout: 2m # Per prompt/panel judge turn (default: 2m)
  judge_reasoning_effort: high # Reasoning effort for the judge model
  executor: copilot-sdk # or custom executor via --executor-plugin

Field	Type	Default	Description
runs	number	1	Positive integer specifying the number of trials per stimulus. 1 = single run. ≥ 2 = multi-trial with pass@k/pass^k aggregation.
timeout	duration	2m	Time before a trial times out. Requires a unit suffix (e.g. 5m, 300s, 30000ms).
model	string	—	Model identifier for the executor
reasoning_effort	string	—	Non-empty, model-supported reasoning effort for the agent under test. Common values include none, minimal, low, medium, high, xhigh, and max; accepted values depend on the model. Forwarded to the executor (including custom executor plugins) via ExecutorOptions.reasoningEffort. When unset, the agent runs at the model’s own default effort. Unsupported models may reject the setting. Overridden by the --reasoning-effort CLI flag; falls back to the EVAL_REASONING_EFFORT env var.
judge_model	string	—	Default model for LLM judge graders (prompt and panel). Graders can override this in their own config.
judge_timeout	duration	2m	Timeout for each prompt or panel judge turn.
judge_reasoning_effort	string	—	Non-empty, model-supported reasoning effort for the judge model. Common values include none, minimal, low, medium, high, xhigh, and max; accepted values depend on the model. Applies only to the eval-level judge_model — not to per-grader model overrides or the panel’s models. When unset, the judge runs at the model’s own default effort. Unsupported models may reject the setting.
executor	string | object	—	Executor for this eval. Either a bare name (copilot-sdk), or an object { name, config } where config is executor-specific (see Executor config & BYOK). Built-in: copilot-sdk. Register more via --executor-plugin and reference them by name. A spec selects a single executor. Omit the field entirely to use the default executor — an empty string is rejected at load time.
judge_provider	object	—	BYOK model provider for the LLM judge graders (prompt, panel) — the judge counterpart to executor.config.provider. Points every judge call at a custom OpenAI-compatible endpoint. Either a declarative block or the opt-in { source: copilot-env } selector. See Judge provider (BYOK) below.

Reasoning-effort values are model-dependent; surrounding whitespace is trimmed, while case is preserved. Blank values are rejected rather than treated as unset. The same rules apply to per-grader and panel judge values.

Executor config & BYOK

The executor field can be an object that pairs the executor name with executor-specific config. The config shape is defined by the selected executor, and vally fails closed — supplying config for an executor that doesn’t accept it is an error, so a misdirected block never silently no-ops.

HydraFusion (Copilot)

Specify model: auto and autoTier to run evals with HydraFusion:

defaults:
  model: auto
  executor:
    name: copilot-sdk
    config:
      autoTier: hydrafusion # or hydrafusion-max

BYOK provider

The copilot-sdk executor accepts a provider block for bring-your-own-key (BYOK): point the agent under test at your own model endpoint (Azure AI Foundry, OpenAI, Anthropic, Ollama, vLLM, or any OpenAI-compatible server) instead of authenticating through the GitHub/Copilot token chain. This is useful in CI, where a long-lived provider key is more durable than a Copilot-seated token.

defaults:
  model: gpt-5.2-codex
  executor:
    name: copilot-sdk
    config:
      provider:
        type: azure # "openai" | "azure" | "anthropic" (default: "openai")
        baseUrl: https://your-resource.openai.azure.com/openai/v1/
        wireApi: responses # "completions" | "responses" (default: "completions")
        apiKeyEnv: FOUNDRY_API_KEY # name of the env var holding the key
        azure:
          apiVersion: "2024-10-21"

Fields under config.provider:

Field	Type	Default	Description
baseUrl	string	— (required)	Provider API endpoint. Must be an absolute http/https URL (e.g. http://localhost:11434/v1); a scheme-less value like localhost:11434/v1 is rejected.
type	string	openai	Provider type: openai, azure, or anthropic. type selects the wire protocol, not the hosting product — use openai for Ollama / vLLM / LiteLLM / Foundry Local and any other OpenAI-compatible server.
wireApi	string	completions	Wire protocol (openai/azure only). Use responses for newer models.
transport	string	http	Transport for the OpenAI Responses API (wireApi: responses): http or websockets.
apiKeyEnv	string	—	Name of an environment variable holding the API key, read at run time. Credentials are only ever referenced by env-var name — there is no literal apiKey field, so keys never land in the spec or in persisted run artifacts.
bearerTokenEnv	string	—	Name of an environment variable holding a bearer token (Authorization header), read at run time. Takes precedence over apiKeyEnv when both are set (and apiKeyEnv is then not read at all).
azure	object	—	Azure-specific options. Only apiVersion (string) is accepted; unknown keys are rejected.
headers	object	—	Extra HTTP headers on outbound provider requests (string values). Header values are treated as secret and masked in persisted artifacts and error output.
maxPromptTokens	number	—	Positive integer. Overrides the resolved model’s default max prompt tokens (triggers compaction before a request would exceed it).
maxOutputTokens	number	—	Positive integer. Overrides the resolved model’s default max output tokens.

Unknown keys under provider (and under provider.azure) are rejected, not ignored — a typo such as apiKeyENV fails validation at plan/lint time rather than silently disabling the credential.

For most setups you only need baseUrl, defaults.model, and one credential env var — the block below is the common case:

defaults:
  model: qwen3
  executor:
    name: copilot-sdk
    config:
      provider:
        baseUrl: http://localhost:11434/v1 # e.g. a local Ollama server
        apiKeyEnv: OLLAMA_KEY # omit entirely if the endpoint needs no key

The names in apiKeyEnv / bearerTokenEnv are read from the environment of the vally process (the shell or CI job running vally) when each trial runs — not from the eval’s agent_environment.env, which is applied only to the spawned agent and is invisible here. Export the variable before running; a trial fails if its referenced variable is unset (this is checked per trial, at execution time). bearerTokenEnv takes precedence over apiKeyEnv. Because executor is part of defaults, experiment variants can vary it — e.g. compare a local Ollama model against the GHCP API in a single run by overriding /defaults/executor (and /defaults/model) per variant; a variant that omits the provider config runs on the default GHCP auth chain.

Advanced: modelId and wireModel

Most authors set only defaults.model, which serves as both the runtime’s model identity and the name sent to the provider. Override these two (under config.provider) only when those roles must differ:

Field	Type	Default	Description
modelId	string	falls back to model	Well-known model name the runtime looks up to resolve agent configuration (tools, prompts, reasoning behavior) and default token limits. Also used as the provider-facing model name when wireModel is unset.
wireModel	string	falls back to modelId	Model name sent to the provider API for inference, when it differs from modelId.

The resolution order for the name sent to the provider is wireModel → modelId → defaults.model. You need this split when the provider’s name for a model differs from the well-known one — for example an Azure deployment name or a custom fine-tune name (a gateway that renames models, such as LiteLLM or vLLM, is another possible case):

defaults:
  model: gpt-4o
  executor:
    name: copilot-sdk
    config:
      provider:
        type: azure
        baseUrl: https://your-resource.openai.azure.com/openai/v1/
        apiKeyEnv: FOUNDRY_API_KEY
        modelId: gpt-4o # runtime uses this for agent config + token limits
        wireModel: my-gpt4o-deployment # what the Azure API actually expects

Judge provider (BYOK)

defaults.judge_provider points the LLM judge graders (prompt, panel) at a custom OpenAI-compatible endpoint — the judge counterpart to the executor’s config.provider for the agent. judge_model picks which model the judge uses; judge_provider picks where the request is sent. It applies to every LLM judge call in the eval (including each judge in a panel).

defaults:
  judge_model: qwen3
  judge_provider:
    baseUrl: http://localhost:8080/v1 # e.g. a local OpenAI-compatible server
    apiKeyEnv: JUDGE_API_KEY # env-var name; omit if the endpoint needs no key

The block accepts the same fields as the executor provider (see the provider table above): type, baseUrl (required, absolute http/https, no embedded credentials), wireApi, transport, apiKeyEnv / bearerTokenEnv (env-var names — literal secrets are not accepted), azure, headers, modelId, wireModel, maxPromptTokens, maxOutputTokens. The same validation and secret-redaction rules apply, and it can be varied in experiments via /defaults/judge_provider.

The agent’s executor.config.provider and the judge’s judge_provider are independent: you can point the agent at one endpoint and the judge at another (or leave the judge on the default GitHub/Copilot auth chain by omitting judge_provider). Credentials are read from the environment of the vally process, not the eval’s agent_environment.env.

Resolving from environment variables (opt-in)

Instead of a declarative block, the judge provider can be resolved from the ambient COPILOT_PROVIDER_* variables that Copilot CLI itself uses. This suits environments that already inject those variables and don’t author a provider block. Because it changes where judge requests are sent, it is strictly opt-in — the mere presence of COPILOT_PROVIDER_* never repoints the judge on its own.

Opt in one of two ways:

# In the spec:
defaults:
  judge_provider:
    source: copilot-env

# Or without editing the spec, via a gate on the vally process:
export VALLY_JUDGE_PROVIDER_SOURCE=copilot-env

When active, the provider is built from these variables (mirroring Copilot CLI’s buildProviderConfigFromEnv()): COPILOT_PROVIDER_BASE_URL (required), COPILOT_PROVIDER_TYPE, COPILOT_PROVIDER_API_KEY, COPILOT_PROVIDER_BEARER_TOKEN, COPILOT_PROVIDER_WIRE_API, COPILOT_PROVIDER_TRANSPORT, COPILOT_PROVIDER_AZURE_API_VERSION, COPILOT_PROVIDER_MODEL_ID, COPILOT_PROVIDER_WIRE_MODEL, COPILOT_PROVIDER_MAX_PROMPT_TOKENS, COPILOT_PROVIDER_MAX_OUTPUT_TOKENS, and COPILOT_PROVIDER_HEADERS (newline-separated Name: Value pairs).

Literal values vs. names. This is the key difference from the declarative block: COPILOT_PROVIDER_API_KEY / COPILOT_PROVIDER_BEARER_TOKEN hold the literal key/token, whereas the block’s apiKeyEnv / bearerTokenEnv hold the name of a variable. Both paths redact resolved secrets from errors and never persist them; what’s recorded in run artifacts is { source: copilot-env }, not the resolved endpoint or credentials.

Precedence. The declarative block and the env-source selector are mutually exclusive:

defaults.judge_provider	VALLY_JUDGE_PROVIDER_SOURCE	Result
absent	unset	Default GitHub/Copilot auth
absent	copilot-env	Resolve from COPILOT_PROVIDER_*
mapping	unset	Declarative provider
{ source: copilot-env }	unset	Resolve from COPILOT_PROVIDER_*
mapping	copilot-env	Error (conflict)
{ source: copilot-env }	copilot-env	Resolve from COPILOT_PROVIDER_*
any	any other value	Error (invalid gate value)

Failure modes. With the selector active but COPILOT_PROVIDER_BASE_URL unset, the run fails fast rather than silently falling back to GitHub auth (missing key/token is allowed — some local endpoints need none). Invalid enum, integer, header, or URL values also fail with a clear message.

Scope. The selector is honored by eval, grade, oracle, and experiment; it applies only to the LLM judge. The agent executor (executor.config.provider) is unaffected and has no env-source mode. The judge process also strips COPILOT_PROVIDER_* from the environment handed to the Copilot SDK unless resolved through this opt-in, so ambient variables can’t bypass it.

Stimulus

stimuli:
  - name: my-test-case
    prompt: "Do something"
    agent_environment: { ... }
    artifacts: { ... }
    graders: [...]
    rubric: ["criterion 1", "criterion 2"]
    constraints: { ... }

For multi-turn conversations, use turns instead of prompt:

stimuli:
  - name: recall-across-turns
    turns:
      - "Remember this code name: ALPHA-7."
      - "What was the code name I gave you?"
    graders:
      - type: output-contains
        turn: 1
        config:
          substring: "ALPHA-7"

Tip

See Writing Eval Specs → Multi-turn conversations for a walkthrough, samples/multi-turn-memory/ for a runnable example, and vally compare for A/B comparing multi-turn runs.

Field	Type	Required	Description
name	string	Yes	Unique identifier for this stimulus
prompt	string	One of prompt or turns	The prompt sent to the agent. When turns is provided, prompt is synthesized automatically and any explicit value is ignored.
turns	string[]	One of prompt or turns	Ordered list of prompts for a multi-turn conversation. Each entry is sent sequentially to the same agent session.
attachments	string[]	No	File paths (binary images or native documents) attached to the agent’s initial prompt, resolved relative to the eval file (absolute paths allowed). Must be a non-empty array of non-empty strings; existence is checked at validation time. Only attachment-capable executors (e.g. copilot-sdk) accept them; others reject the stimulus.
agent_environment	EnvironmentConfig	No	Stimulus agent-visible environment (merged with root). Deprecated alias: environment
grading_environment	GradingEnvironmentConfig	No	Stimulus grader-only files, hidden from the agent (merged with root)
artifacts	ArtifactsConfig	No	File copy filters applied before workspace cleanup to capture output artifacts in run results.
graders	GraderConfig[]	No	Graders to run on the trajectory
rubric	string[]	No	Evaluation criteria for LLM judge graders. Each entry must be a non-empty string. Passed to prompt and panel graders as the criteria they score, and used by vally compare for comparisons (overrideable with --eval-spec). Omitted or empty, LLM judges fall back to a generic built-in rubric.
constraints	Constraints	No	Resource limits for the trial
simulation	Simulation	No	Tool-call simulation: canned responses for specific tools (deterministic, side-effect-free evals)
supported_executors	string[]	No	Allow-list of executor IDs (e.g. copilot-sdk) this stimulus can run under. When the active executor isn’t listed, the stimulus is skipped (recorded with status skipped, not run or failed). Omit to run under any executor. An empty list is rejected at load time.
golden_patch	GoldenPatch	No	Reference-solution diff for oracle grading
golden_command	non-empty string	No	Reference-solution shell command run in the materialized workspace by oracle
golden_trajectory	GoldenTrajectory	No	Reference-solution trajectory (ATIF) for oracle grading of trajectory-/metric-/transcript-scoped graders
golden_custom_metrics	GoldenCustomMetrics	No	Reference-solution custom metrics for oracle grading of custom-metrics graders
tags	Record<string, string | string[]>	No	Key-value pairs for filtering into suites

Fields not listed above are rejected when the eval spec is loaded.

Artifacts

artifacts:
  include:
    - "**/Cargo.toml"
    - "**/src/**/*.rs"
  exclude:
    - "**/target/**"

Field	Type	Required	Description
include	string[]	Yes	Non-empty list of path patterns to copy.
exclude	string[]	No	Path patterns to remove from the include matches.

Precedence: if a path matches both include and exclude, it is excluded.

Pattern semantics:

• Patterns are path globs: * matches any run of characters within a single path segment, and ** matches across path segments (directories at any depth).
• Patterns without glob tokens are treated as exact file or directory matches. Example: foo.txt matches only that file, while dir matches files under dir/ recursively.
• Copying artifacts recursively scans the workspace to evaluate include/exclude rules, so this option can add noticeable overhead in very large repositories.
• .git is always excluded regardless of include patterns, even if you glob **/*.

Destination layout:

For vally eval, copied files are placed in an artifacts/ subdirectory of the trial’s session-log directory — <runDir>/<eval>/<stimulus>/<model>/<trial>/artifacts/ — so a trial’s logs (events.jsonl/metadata.json) and its copied artifacts sit together. Each copied file keeps its workspace-relative path under artifacts/.

vally experiment co-locates artifacts the same way, under each variant’s per-trial session directory: <runDir>/<variant>/<eval>/<stimulus>/<model>/<trial>/artifacts/.

Golden patch

A golden patch is a unified-diff reference solution for the stimulus. It powers vally oracle, which materializes the stimulus’s starting environment, applies the patch, and runs the graders against the result — so you can verify the harness (graders should pass on the correct answer) and gate CI before publishing a benchmark.

Provide exactly one of inline or path:

stimuli:
  # Patch stored in a file (resolved relative to the eval file's directory)
  - name: fix-the-bug
    prompt: "Fix the off-by-one error in pagination"
    golden_patch:
      path: solutions/fix-the-bug.patch

  # Patch embedded inline
  - name: add-readme
    prompt: "Add a README"
    golden_patch:
      inline: |
        --- /dev/null
        +++ b/README.md
        @@ -0,0 +1 @@
        +# My Project

Field	Type	Required	Description
inline	string	One of the two	The unified-diff patch text, embedded directly in the spec
path	string	One of the two	Path to a patch file, resolved relative to the eval file’s directory

The resolved diff is passed to graders under vally oracle and vally grade, including when a prompt/panel grader selects config.evidence: [golden_patch]. When a prompt/panel grader selects golden_patch, that LLM evidence is omitted when it is identical to the trajectory output — always when oracle synthesizes the trajectory from the patch, and with a golden_trajectory only when that trajectory’s own output is the patch text. When it is omitted and nothing else is selected, the grader fails with nothing to judge; pair golden_patch with trajectory, diff, or repo. Under oracle, trajectory is what presents the patch to the judge as the output being graded. Normal vally eval does not resolve reference solutions, so the selected evidence is unavailable there; use diff to judge the agent’s workspace changes during an inline eval run.

Golden command

A golden_command is a reference-solution shell command for cases where a patch cannot reproduce the state that graders need. Oracle runs it through the platform shell with the materialized workspace as its working directory. Its value must be a non-empty, non-whitespace string:

stimuli:
  - name: reproduces-ui-bug
    prompt: "Reproduce the bug and collect the interaction log"
    golden_patch:
      path: solutions/instrumentation.patch
    golden_command: node scripts/reproduce-ui-bug.mjs
    graders:
      - type: file-contains
        config:
          path: interaction.log
          value: opened-settings

The command runs after golden_patch is applied and before explicit golden_custom_metrics are written or graders run. It must exit successfully within five minutes; a nonzero exit, startup failure, or timeout fails the oracle run. Command output is discarded and is not added to the synthesized trajectory, so behavior and transcript graders still require a golden_trajectory.

The command and any relative paths it contains are interpreted inside the materialized workspace, not relative to eval.yaml. Scripts and files may come from agent_environment.files, an agent_environment.git checkout, or an earlier golden_patch; external tools and runtimes must be installed by agent_environment.commands or already be available on PATH. The command must remain in the foreground and must not intentionally detach or background child processes. --no-golden-input withholds the command as part of the oracle negative control. Normal vally eval does not run it.

When a grader consumes the workspace diff, Oracle captures one cumulative diff from the materialized environment to the final state, including changes made by both golden_patch and golden_command.

For a command-only stimulus, a prompt or panel grader with default or trajectory-only evidence cannot see command-created workspace changes. Oracle warns about this ambiguity without rejecting it. Select diff or repo evidence when the judge should inspect those changes, or provide a golden_patch or golden_trajectory that supplies judge-visible content.

Golden trajectory

A golden trajectory is a reference-solution ATIF trajectory (fake or real) for a stimulus. A golden patch only captures the final file state, so under vally oracle trajectory-, metric-, and transcript-scoped graders (tool-calls, skill-invocation, system-event, max-repeat, token-budget, tool-call-count, turn-count, step-count, error-count, wall-time, transcript-*, and output-scoped graders) have no events or metrics to validate against and are rejected unless --no-golden-input is used. Supplying a golden_trajectory gives oracle real events and metrics to grade those graders offline — you can hand-author the events you expect a passing run to produce and confirm the graders agree, no agent run required.

Provide exactly one of inline (an embedded ATIF document) or path (an ATIF JSON file resolved relative to the eval file’s directory). Every trajectory document, including an inlined subagent trajectory, must contain at least one step. The example below is complete and passes the tool-calls grader as written:

stimuli:
  # Trajectory stored in a file
  - name: runs-the-tests
    prompt: "Run the test suite"
    golden_trajectory:
      path: solutions/runs-the-tests.trajectory.json
    graders:
      - type: tool-calls
        config:
          required: ["bash"]

  # The same trajectory embedded inline
  - name: runs-the-tests-inline
    prompt: "Run the test suite"
    golden_trajectory:
      inline:
        schema_version: "ATIF-v1.6"
        session_id: sess-runs-the-tests
        agent: { name: oracle, version: "1.0" }
        steps:
          - step_id: 0
            source: user
            message: "Run the test suite"
          - step_id: 1
            source: agent
            message: "Running the tests with bash."
            tool_calls:
              - tool_call_id: call-1
                function_name: bash
                arguments: { command: "npm test" }
            observation:
              results:
                # Its source_call_id matches the tool_call in THIS same step.
                - source_call_id: call-1
                  content: "All tests passed"
    graders:
      - type: tool-calls
        config:
          required: ["bash"]

Co-locate each observation with its tool_call

The tool-calls grader (and other tool-result checks) only counts a tool once it sees the matching tool result, which oracle derives from an ATIF observation. Each observation result is matched to a tool_call within the same agent step, so its source_call_id must reference a tool_call_id declared in that step. A source_call_id pointing at a call in a different step resolves to tool "unknown" and never counts — the grader then reports the required tools “were not called” even though the call appears in the trajectory.

Field	Type	Required	Description
inline	object	One of the two	An embedded ATIF trajectory document
path	string	One of the two	Path to an ATIF JSON file, resolved relative to the eval file’s directory

A stimulus can combine a golden_patch, golden_command, and golden_trajectory: the patch and command drive file-state graders against the materialized workspace, while the trajectory drives behavior/metric graders. The document is validated as ATIF at oracle time, not at load time. Only vally oracle consumes these inputs — a normal vally eval grades the real agent run.

Golden custom metrics

The custom-metrics grader reads a JSON file the run writes into the workspace (default custom_metrics.json). Under vally oracle no agent runs, so that file only exists if the golden patch or golden command creates it. A golden_custom_metrics supplies the file directly (fake or real) so custom-metrics graders can be validated offline without baking the metrics into a patch.

Oracle supplies the resolved metrics at every path a custom-metrics grader reads (its config.path, or the default custom_metrics.json). Provide exactly one of inline (an embedded metrics object) or path (a JSON file resolved relative to the eval file’s directory):

stimuli:
  - name: tests-pass
    prompt: "Make the tests pass"
    golden_custom_metrics:
      inline:
        tests_failed: 0
        quality_score: 0.9
    graders:
      - type: custom-metrics
        config:
          assertions:
            - metric: tests_failed
              equals: 0
            - metric: quality_score
              min: 0.8

Field	Type	Required	Description
inline	object	One of the two	The metrics object, embedded directly in the spec
path	string	One of the two	Path to a JSON metrics file, resolved relative to the eval file’s directory

The metrics object may be flat ({ "tests_failed": 0 }) or an envelope nesting them under values — the same shapes the custom-metrics grader accepts. The --no-golden-input baseline withholds it: value/presence assertions then fail cleanly (a healthy negative control), while an absent-only grader is reported N/A. Only vally oracle consumes it.

Tags

Tags are key-value pairs used for filtering stimuli into suites. They can be defined at the eval level (inherited by all stimuli) or the stimulus level (overrides eval-level tags on the same key).

tags:
  priority: p0
  area: [auth, security]

Filtering semantics:

• AND across keys — a stimulus must match all specified tag keys
• OR within values — a stimulus matches a key if it has any of the specified values
• Stimulus tags override eval tags on the same key
• Include vs. exclude — --tag key=value includes matching stimuli; --tag key!=value excludes them (a stimulus without the excluded key is kept). Exclusion takes precedence: a stimulus is dropped if any exclude condition matches, even when it also satisfies an include on a different key or value. Suite filter: blocks in .vally.yaml are include-only.

See Authoring Eval Suites for tagging strategies and suite configuration.

Agent environment

Note

agent_environment is the agent-visible setup block. The deprecated alias environment is still accepted for a few releases and emits a deprecation warning. Set only one of the two at each level.

Place the fields shown below under agent_environment at the eval or stimulus level:

agent_environment:
  files:
    - src: fixtures/input.txt
      dest: input.txt

Agent environment fields

skills:
  - ./path/to/my-skill # Skill directory (containing SKILL.md)
files:
  - src: fixtures/input.txt
    dest: input.txt
  - src: fixtures/test-data # directories are copied recursively
    dest: test-data
  - src: fixtures/session-log # staged outside the graded workspace
    dest: session-state/session-1
    dest_root: assets
commands:
  - npm install
commandTimeout: 2m
git:
  type: worktree
  ref: v2.1.0
  source: ../my-repo
mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
  api:
    type: http
    url: http://localhost:3000/mcp
env:
  COPILOT_AGENT_ACTION: review
  LOG_LEVEL: debug

Field	Type	Description
commands	string[]	Shell commands to run during setup (uses /bin/sh on Unix, cmd.exe on Windows)
commandTimeout	duration	Per-command timeout for commands (e.g. 2m, 90s). Must be positive (0 is rejected). Defaults to 60s
env	Record<string, string>	Environment variables set on the agent process (see below)
files	{src, dest, dest_root?}[]	Files or directories to copy during setup (see below)
git	object	Git configuration for fixture data — local worktree or remote clone (see below)
mcpServers	Record<string, McpServerConfig>	Named MCP servers to start or connect to
skills	string[]	Paths to skill directories (each containing a SKILL.md) to load

Tip

files entries can be individual files or entire directories. When src points to a directory, its contents are copied recursively into dest. Paths are relative to the eval file’s directory.

Staging files

Each files entry copies src to a destination chosen by dest_root:

Field	Type	Required	Description
src	string	Yes	File or directory to copy, relative to the eval file’s directory (or absolute)
dest	string	Yes	Destination path. Under dest_root: assets it must stay inside the assets root once normalized (no absolute paths, and no traversal that ends up above the root); under the default workdir it is resolved relative to the workspace root
dest_root	"workdir" | "assets"	No	Which root dest is resolved under. Defaults to workdir

• dest_root: workdir (default) — the file lands inside the graded workspace, exactly as before. It stays out of diffs — the workspace baseline is captured after setup — but it is a real file in workDir, so tree- and file-scanning graders (file-exists, file-contains, …) do see it.
• dest_root: assets — the file is staged into a managed, per-trial assets directory that sits outside the graded workspace. Nothing is copied into workDir, so neither diff- nor tree-scanning graders see it, and the directory is cleaned up automatically at the end of the trial (no manual delete step). Use this for setup inputs that must exist on disk but should not contaminate the graded tree — recorded session logs to seed a session, or large fixtures a grader reads.

The assets directory is exposed as the EVALUATE_ASSETS environment variable (see Agent env) and to graders as trajectory.assetsDir (the program grader also receives it as EVALUATE_ASSETS). It is a host path allocated fresh per trial; treat it as absent when re-grading a stored trajectory on a different host.

Caution

The assets directory is agent-visible: its path is passed to the agent as EVALUATE_ASSETS, and its contents are not made read-only. Only workspace and diff scanning exclude it — the agent can read, and modify, anything staged there. Do not stage grader or test scripts as assets, and do not rely on an asset being unchanged at grade time.

Caution

To place a setup file outside the graded workspace, use dest_root: assets — it stages into the managed, auto-cleaned assets dir and keeps workDir clean. Under dest_root: assets, dest is normalized first and must still land inside the assets root: absolute paths are rejected, and so is any .. traversal that escapes it. An interior .. that cancels out — nested/../seed.txt is just seed.txt — is fine.

A workdir-rooted dest is joined onto the workspace with path.join, so an absolute-looking dest keeps the workspace prefix (/etc/seed.txt lands at <workDir>/etc/seed.txt). Only a leading .. actually escapes, landing in a sibling of the per-trial workspace that trial cleanup never removes. Because that location is shared across trials, a leftover can bleed into later runs, so vally lint flags it with an environment-file-workdir-escape warning and points you at dest_root: assets. Under dest_root: assets both an absolute dest and an escaping .. are an environment-file-assets-escape error, since assets staging rejects them outright.

Harness customization files

Use files to stage agent customizations at the location expected by the selected executor:

GitHub Copilot

files:
  - src: ./agents/reviewer.agent.md
    dest: .github/agents/reviewer.agent.md

Claude Code

files:
  - src: ./agents/reviewer.md
    dest: .claude/agents/reviewer.md

Both harnesses discover staged profiles natively — Vally does not parse agent files, so the full frontmatter each harness supports works as documented upstream. Staging the file is the entire setup: there is no flag to turn on and no agent field in the eval spec. For Copilot, a Git-backed workspace with a GitHub remote may additionally expose organization or enterprise agents to the authenticated identity, so the available roster can differ between runners.

Neither harness replaces the default root agent. A matching profile is invoked as a model-routed subagent: the harness compares the stimulus against each profile’s name and description and decides whether to delegate. Write a stimulus that clearly matches the profile you want to exercise.

Because that routing is a model decision, gate it with a grader rather than assuming it happened. For Copilot, given an agent file whose frontmatter sets name: Reviewer:

stimuli:
  - name: delegates-to-reviewer
    prompt: "Delegate this to the Reviewer specialist and report its findings."
    graders:
      - type: tool-calls
        config:
          required:
            - name: "^task$"
              args:
                agent_type: "^Reviewer$"

The eval now fails when the agent is never used. agent_type is the agent’s display name: the frontmatter name when present, otherwise the filename stem — a reviewer.agent.md with no name is matched as ^reviewer$. Argument names are harness-specific, so inspect a recorded trajectory before writing the matcher for another executor. Subagent events also carry an agentId, which grader-level scope can target to assert what the subagent actually did, not just that it ran.

Caution

For Copilot, staged agents are loaded through the runtime’s workspace config discovery, which also picks up skill directories present in the trial workspace. Explicit agent_environment.skills still takes precedence on name collision, but additional workspace-provided skills will load, so treat agent_environment.skills as a floor rather than an exact set and keep the workspace clean when you need strict isolation. Workspace MCP configuration (.mcp.json, .github/mcp.json) is not part of session discovery, so MCP servers still come only from agent_environment.mcpServers. The hosted github-mcp-server is disabled so evals do not silently gain a network dependency; declare your own server under agent_environment.mcpServers if you need GitHub tools, including under that same name, which replaces the hosted one rather than being suppressed. For Codex, AGENTS.md provides workspace instructions, not named subagents.

Note

commands run from the workspace directory, not the directory you launched vally from. Use files (or a git worktree) to copy any scripts or assets a command needs into the workspace, and reference them with paths relative to the workspace root.

Each command is executed in the system shell (/bin/sh on Unix, cmd.exe on Windows) with a per-command timeout set by commandTimeout (defaults to 60s). Shell features such as pipes, redirects, and variable expansion are available.

Agent env

The env field sets environment variables on the agent process for every run of this environment. The built-in copilot-sdk executor and custom process-spawning executors that support environment variables merge these over the inherited environment. The built-in mock executor has no agent process and ignores env; Vally prints a warning listing the ignored variable names so the misconfiguration is visible.

env:
  COPILOT_AGENT_ACTION: review
  LOG_LEVEL: debug

Field	Type	Description
env	Record<string, string>	Variable name → value, set on the agent process. In experiment variant overrides, values support ${…} interpolation like other override fields.

Note

This sets the agent’s runtime environment; the values are not substituted into other spec fields, and are distinct from an MCP server’s own env (which configures that server’s child process rather than the agent).

Note

When the environment stages at least one file with dest_root: assets, Vally sets EVALUATE_ASSETS on the agent process to the absolute path of the per-trial assets directory, and expands the literal ${EVALUATE_ASSETS} token inside your env values. This lets a spec point the agent at a staged input machine-independently, for example:

files:
  - src: fixtures/session-log
    dest: session-state/session-1
    dest_root: assets
env:
  RESUME_SESSION_FIXTURE_PATH: "${EVALUATE_ASSETS}/session-state/session-1"

${EVALUATE_ASSETS} is the only token expanded, and only when assets are staged; no other interpolation is performed on env values here.

Caution

env values run inside the agent process. Variables such as NODE_OPTIONS, LD_PRELOAD, or PATH can change what code the agent executes, so treat authoring env as trust-equivalent to running code on the eval host — be cautious with specs from less-trusted collaborators.

Note

In an experiment variant override, env deep-merges by key with the inherited environment: set a single key to null (e.g. FOO: null) to delete an inherited variable, or set the whole field to null to clear all inherited variables.

Caution

Reports and provenance artifacts show only the names of these variables, never their values. However, trajectory files (the raw agent output) are not filtered — if the agent process echoes an env value (e.g. in a debug log or tool output), it will appear verbatim in the trajectory. Avoid using env to pass long-lived secrets; prefer tokens the agent already has from the host environment.

Note

Like every other config field, env values are part of the environment’s config identity — they feed the config hash recorded in provenance and are compared by experiment drift detection (two variants whose env differs on a path that isn’t a declared vary axis are flagged as drift, with the values redacted). A consequence: a value that changes between runs — e.g. a rotated token interpolated into env — changes that identity, which is a further reason to keep long-lived or rotating secrets out of env.

Git config

The git field sets up the evaluation workspace from a Git repository. It has two modes, discriminated by type:

• worktree — check out a ref from a local repository as a detached worktree.
• clone — clone a remote repository at a given ref into the workspace.

type: worktree

git:
  type: worktree
  ref: v2.1.0
  source: ../my-repo
  timeout: 10m # optional; raise for very large repos
commands:
  - dotnet restore

Field	Type	Required	Description
type	"worktree"	Yes	Must be "worktree"
ref	string	Yes	A commit-ish value (tag, commit SHA, or branch name) to check out
source	string	Yes	Path to the local repo used as the worktree source
timeout	duration	No	Timeout for the local git steps that materialize the worktree (add, remove), e.g. 10m. Must be positive. Defaults to 5m

Note

Worktree checkout runs non-interactively (GIT_TERMINAL_PROMPT=0, and a batch-mode ssh unless you export GIT_SSH_COMMAND), so an unattended run fails fast instead of blocking on a credential or passphrase prompt from an LFS/smudge filter or post-checkout hook. A caller-provided GIT_SSH_COMMAND is honored.

type: clone

git:
  type: clone
  url: https://github.com/octocat/hello.git
  ref: v2.1.0 # optional; defaults to the remote's default branch
  shallow: true # optional; true → depth 1, or an explicit integer depth
  sparse: # optional; cone-mode paths to materialize
    - src/core
  timeout: 10m # optional; local clone steps only (not the network fetch)

Field	Type	Required	Description
type	"clone"	Yes	Must be "clone"
url	string	Yes	Remote repository URL to clone: http(s), ssh, git, file://, or scp-style git@host:path
ref	string	No	Commit-ish to check out; defaults to the remote’s default branch
shallow	boolean | integer	No	Shallow-clone history: true fetches depth 1, an integer sets an explicit depth
sparse	string[]	No	Sparse-checkout (cone mode) paths — only these directories are materialized
timeout	duration	No	Timeout for each local git step (init, remote add, sparse-checkout, checkout), e.g. 10m. Does not cap the network fetch. Must be positive. Defaults to 5m

HTTP(S) clone URLs must not contain userinfo, query/fragment parameters, or backslashes, and SSH URLs must not contain passwords. Use host Git authentication such as Git Credential Manager, an SSH agent, or GIT_ASKPASS.

MCP server config

Each entry in mcpServers is either a stdio server (launched as a child process) or a remote server (connected over HTTP/SSE).

The copilot-sdk executor supports MCP sampling (model requests made by an MCP server) automatically, with no additional configuration. These requests count toward the trial’s time budget and can incur additional model usage.

Stdio (child process)

mcpServers:
  db:
    type: stdio
    command: db-serve
    args: ["--port", "5432"]
    env:
      DB_HOST: localhost
    cwd: ./services/db
    timeout: 5000

Field	Type	Required	Description
type	"stdio"	Yes	Launch as a child process
command	string	Yes	Executable to run
args	string[]	No	Arguments passed to the command
env	Record<string, string>	No	Extra environment variables for the child process
cwd	string	No	Working directory for the child process
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Remote (HTTP/SSE)

mcpServers:
  api:
    type: http # or "sse"
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer ${API_TOKEN}"
    timeout: 10000

Field	Type	Required	Description
type	"http" | "sse"	Yes	Connect to a remote server
url	string	Yes	Server endpoint URL
headers	Record<string, string>	No	Extra HTTP headers (e.g. auth tokens)
timeout	number	No	Timeout in milliseconds for connecting to / invoking the server

Note

When both root and stimulus agent environments are defined, they are merged with conflict detection:

• Skills: Duplicates across root and stimulus are deduplicated with a warning.
• Commands: Duplicate commands across root and stimulus throw an error.
• Files: Files with the same dest (and dest_root) but different src throw a conflict error. Identical entries are also rejected as redundant.

Only cross-array conflicts are detected — duplicates within the same array (root-only or stimulus-only) are not flagged.

Grading environment

grading_environment stages grader-only files — answer keys, golden data, hidden test suites — that the agent under test must never see. It is a top-level block, a peer of agent_environment (not nested inside it), and may be set at eval-root and/or stimulus level; the two are merged with the same dest-conflict detection as agent_environment.files.

agent_environment: # staged before the run, agent sees it
  files:
    - { src: repo/, dest: . }
grading_environment: # staged only for grading, hidden from the agent
  files:
    - { src: fixtures/answer-key.json, dest: .vally-grading/answer-key.json }
    - { src: fixtures/hidden-tests, dest: tests } # a dir, copied recursively

Field	Type	Required	Description
files	{src, dest}[]	No	Files/directories staged for graders only. Same shape as agent_environment.files minus dest_root (grader files are always workdir-rooted); dest must be a relative path with no ...

Unlike agent_environment, this block has no skills, commands, git, mcpServers, or env fields. Run grader setup through the grader itself (for example, a program grader command), and stage only the files that command needs.

How it works. Graders run against a disposable copy of the produced workspace with these files staged in; the copy is destroyed afterward and the pristine workspace is preserved. So grader files never reach the agent, the recorded diff, or a preserved/exported workspace — and a grader that mutates the workspace (e.g. a program grader) can’t contaminate the result. Workspace graders (file-exists, file-contains, custom-metrics, program, …) read them from dest like any other file. vally oracle and vally grade stage them the same way, so offline grading reproduces the run. Exporters receive grader-only files separately so formats such as Harbor can keep them in the verifier boundary and out of the agent task image.

Tip

Point dest at a directory the agent won’t also use — a dedicated .vally-grading/ is a good convention — and avoid executable-looking names for data fixtures so a grader’s own test runner or build glob doesn’t pick them up.

Caution

Keep grader src files outside everything your agent_environment stages — its files, skills, and git source, plus stimulus attachments. If any agent-visible source overlaps a grader src (e.g. agent_environment.files: [{ src: ".", dest: "." }] alongside a grader src in the same tree), the agent would see grader-designated content before grading, so vally lint rejects it (grading-file-src-agent-visible). Put grader fixtures in a dedicated directory the agent environment doesn’t stage.

Grader config

graders:
  - type: output-contains
    name: output-includes-hello
    config:
      substring: "hello"
      case_sensitive: false

Field	Type	Required	Description
type	string	Yes	Registered grader name (e.g., output-contains, file-exists)
name	string	No	Stable identifier for this grader instance — set it when something outside Vally needs to refer to this specific check. Lowercase letters, digits and hyphens, starting with a letter or digit, at most 60 characters, and unique across the eval file. It may match its own grader’s type, but not the type of a different grader in the file, since reports and the results API select a grader by name or type. When omitted, reports fall back to a label derived from this instance’s config (e.g. output-matches /Cosmos/), which is for humans only and changes whenever the config does.
turn	integer	No	Scope this grader to a specific conversation turn (0-based). The pipeline slices the trajectory to that turn before grading. Also valid for diff-contains, diff-not-contains, and diff-empty, which inspect that turn’s recorded workspace diff; other workspace graders remain unsupported.
scope	string | object	No	Scope this grader to the parent agent or a specific subagent. See Subagent scope. Only valid for non-workspace graders.
config	object	No	Grader-specific configuration (varies by type)

LLM judge evidence

The prompt and panel graders accept a grader-level evidence selector:

graders:
  - type: prompt
    config:
      evidence: [trajectory, diff, golden_patch, repo]

trajectory includes the agent output, execution metrics, and session timeline; diff includes the agent’s cumulative workspace changes; golden_patch includes the stimulus’s reference solution; and repo includes a bounded snapshot of the final workspace filesystem (a file tree plus text-file contents), letting the judge read the agent’s output directly. Sensitive filenames (.env*, private keys, package-manager auth configs) are listed in the tree but their contents are omitted and not sent to the judge. When omitted or empty ([]), evidence defaults to trajectory evidence. When evidence is a non-empty list, it must contain unique supported values and cannot select diff or repo together with grader-level turn or scope. Lint warns when a stimulus declares golden_patch but no grader in the stimulus can use it. A prompt or panel judge (which scores the patch as the output under vally oracle, and can weigh it as evidence under vally grade), a reference-based grader, or any grader that reads the workspace (for example diff-*, file-*, run-command, program, custom-metrics, or a plugin grader declaring requiresWorkspace) clears the warning. The one exception is a stimulus that also declares golden_trajectory: then the oracle output is that trajectory’s output, so a prompt/panel judge clears the warning only when its evidence selects patch-bearing evidence — golden_patch, diff (which carries the patch here), or repo (the oracle applies the patch to the workspace before snapshotting it). It fires only when the patch has no such consumer in any run mode, which usually means a stray patch on a stimulus graded solely by plain output matchers, or one with no graders at all.

repo reads the live workspace, so it is available whenever the workspace is on disk (vally eval, and vally grade with a materialized workspace); an absent or unreadable workspace fails the grader with a clear message before the judge is called.

This is separate from the stimulus-level artifacts field: stimulus artifacts copy files out of the workspace, while grader config.evidence controls evidence sent to an LLM judge.

Both prompt and panel automatically copy selected evidence to temporary files outside the agent’s working directory. Initial judge messages contain the task, rubric, instructions, and file paths, never evidence content. Judges can only read and search the selected files.

Vally deletes the copies after all judge calls finish or fail; they are not saved for re-grading. Repository capture limits and turn/subagent restrictions are unchanged. See the prompt grader reference for file layout, retrieval budgets, custom-client capability requirements, and failure behavior. Comparison also reads files, but judges only output, metrics, and timelines.

Subagent scope

scope restricts a grader to one agent’s slice of the trajectory, so you can assert behavior of a subagent separately from the parent (root) agent. It composes with turn and, like turn, works generically for any non-workspace grader — the pipeline slices the trajectory before grading.

scope value	Selects
parent	Only the parent/root agent (events with no subagent identity)
subagent	Any subagent (regardless of which)
{ agent: "<name/id>" }	Events whose agent identity equals the given value

Agent identity comes from the adapter’s own labeling, which is not guaranteed to be a unique instance id. So { agent } matches all events sharing that identity: if a parent fans out several same-named subagents, they’re selected together as one aggregate slice, not addressed individually.

graders:
  # The parent delegated the edit; assert it did not edit directly.
  - type: tool-calls
    scope: parent
    config:
      disallowed: [edit]
  # Assert the delegated subagent actually performed the edit.
  - type: tool-calls
    scope: subagent
    config:
      required: [edit]

Per-adapter identity

Per-call agent identity comes from the trajectory adapter, and its meaning — and therefore whether { agent } is usable — differs by source:

• Copilot stamps each event with the subagent agentId from the SDK envelope. scope: parent and scope: subagent work reliably. That id is the opaque launching tool-call id (e.g. toolu_01Biq…), not a friendly agent name, so a { agent } value can’t be authored ahead of a run and isn’t portable across runs — prefer parent / subagent on Copilot.
• ATIF trajectories support subagent scoping when the document inlines its subagent trajectories (subagent_trajectories linked from a step’s subagent_trajectory_ref): the adapter inlines those events stamped with the subagent’s declared agent name by default, so { agent: "<name>" } is authorable. Integrations opting into fromAtif instance identity instead use encoded trajectory-id paths, keeping same-named instances distinct. ATIF subagents referenced only as external files (by session_id / trajectory_path, not inlined) carry no events in the document, so scope: subagent / { agent } selects nothing for those.

When the selected scope contains no recorded events, most graders report an error. Absence checks are the exception: tool-calls with only disallowed: [edit] passes because no edits were recorded in that scope. A non-empty required, sequence, or parallel list still produces an error. This also applies when scope is combined with a valid turn. With oracle --no-golden-input, empty selections are graded normally instead.

Constraints

constraints:
  max_turns: 10
  max_tokens: 5000
  max_duration: 1m
  max_agent_duration: 45s

Field	Type	Description
max_turns	number	Maximum conversation turns
max_tokens	number	Maximum total tokens
max_duration	duration	Hard wall-clock cap on the agent’s run (e.g. 2m, 30s, 500ms). Unit suffix required. It bounds the in-flight agent work only — workspace setup and teardown/disconnect are bounded separately, not by this cap. Exceeding it aborts the run and fails the eval.
max_agent_duration	duration	Agent working-time limit (e.g. 10m, 90s). The agent gets up to this long to work; on expiry it is stopped and the trial proceeds through normal teardown, grading whatever was produced as a success.

max_duration vs max_agent_duration — two different ways to limit run time:

• max_duration is a hard ceiling on the agent’s wall-clock run. Overrun it and the run is aborted and fails. It bounds the in-flight agent work only — workspace setup and teardown/disconnect are bounded separately, not by this cap.
• max_agent_duration caps only the agent’s working time: “work for up to 10 minutes, then we’ll stop you and grade whatever you finished.” On expiry the agent is stopped and the trial runs through normal teardown, graded as a success.

A few details:

• The working-limit covers the whole stimulus run (all turns), not a single turn.
• Both limits stop the agent the same way under the hood (the executors abort the in-flight agent run — the Copilot SDK and Claude CLI offer no gentler mechanism). The difference is the verdict: max_duration fails the run, while max_agent_duration keeps it a success and grades the partial work.
• When the working-limit stops a run, its trajectory records endReason: "agent_timeout" (vs "completed" for a normal finish) so graders and the analytics server can account for it.
• Setting only max_agent_duration is enough — the run is allowed to last the full working-limit.
• If you set both, keep the working-limit below the hard cap. The effective hard cap is max_duration (or, if unset, the eval’s defaults.timeout, then the --timeout override). When the working-limit is >= that cap, the hard cap fails the run before the working-limit can stop it cleanly. vally lint warns when it can detect this.

Simulation

Run the agent with deterministic, mocked responses for selected tool calls. Useful for loop/repetition detection, failure-mode recovery tests, and behavioral evals. Calls that do not match an override still run the real tool, so provide a default when an overridden tool must never cause real side effects. Requires a simulation-capable executor (copilot-sdk); other executors reject a stimulus that declares simulation.

stimuli:
  - name: npm-registry-down
    prompt: "Install the project dependencies"
    simulation:
      max_iterations: 20
      tool_overrides:
        # Shell tools can simulate success OR failure, and support ordered
        # regex patterns matched against the command (first match wins).
        bash:
          patterns:
            - match: "npm install|npm ci"
              output: |-
                npm error code ETIMEDOUT
                npm error network request to registry.npmjs.org failed
              is_error: true
            - match: "node -v"
              output: "v14.21.3" # successful stdout (is_error defaults to false)
        # Non-shell tools can only simulate failures.
        web_fetch:
          output: "Error: HTTP 403 Forbidden"
          is_error: true

        # MCP tools use the exact runtime name: <server-name>-<tool-name>.
        # Their real schema remains visible, but matching calls do not reach
        # the real tool.
        slack-search_messages:
          patterns:
            - args:
                query: "(?i)project atlas"
                channel: "^engineering$"
              output: '[{"text":"Launch moved to Friday"}]'
              structured_output:
                messages:
                  - text: "Launch moved to Friday"
            - pattern: '"query":"incident.*timeline"'
              output: "Search temporarily unavailable"
              is_error: true

Field	Type	Required	Description
tool_overrides	Record<string, Override>	No	Per-tool canned responses, keyed by tool name (bash, web_fetch, …).
max_iterations	number	No	Cap on model iterations (assistant turns). Exceeding it stops the run cleanly with endReason: simulation_cap; this is not a run error — graders still run on the collected trajectory.

Each override value is one of:

• string — shorthand for successful stdout ({ output: <str>, is_error: false }). Valid only for shell and configured MCP tools.
• object — { output: string, is_error?: boolean, structured_output?: object }. is_error defaults to false. structured_output is MCP-only.
• patterns — ordered, argument-dependent responses. The first matching rule wins. If none match, Vally uses default or runs the real tool when no default is set.

Pattern rules support these selectors:

• match: case-insensitive regex against a shell tool’s command.
• args: regexes for named string arguments. All entries must match. Supports inline (?i), (?m), and (?s) flags.
• pattern: regex against JSON.stringify(arguments). Prefer args when possible.

Rules can combine selectors; all specified selectors must match. Each rule uses the same output, is_error, and structured_output fields as an object override.

Shell vs non-shell: shell tools (bash, shell, powershell, treated as aliases) can simulate a successful or failed result because the command is rewritten to emit the canned output. Configured MCP tools can also simulate success or failure. Other first-party tools can only simulate failures — a non-shell override that resolves to success (e.g. a plain string, or is_error: false) is rejected by vally lint.

MCP tools: use the exact runtime name: <server-name>-<tool-name>. server-name is the key under agent_environment.mcpServers. Matching calls return the canned response; unmatched calls run normally. Tool definitions remain unchanged. If a tool has an outputSchema, every successful override must include structured_output that satisfies that schema: the simulation proxy validates it against the real server’s advertised schema at run time and fails the tool call otherwise, mirroring the real server. (Error overrides — is_error: true — skip this check.)

Recording: simulated tool calls are flagged in the trajectory (simulated: true) and counted in metrics (simulatedToolCallCount), so graders and reports can tell simulated calls apart from real ones.

Scoring

scoring:
  weights:
    file-exists: 0.7
    output-contains: 0.3
  threshold: 0.7
  timeout: 10m

Field	Type	Description
weights	Record<string, number>	Weight per grader type. Must be a normalized distribution (values sum to 1.0 ±0.01). When multiple graders share the same type, their scores are averaged before the weight is applied. Graders absent from the map receive weight 0. If omitted, all graders are averaged equally.
threshold	number	Optional. Minimum aggregate score to pass (0.0–1.0). When omitted, the verdict falls back to binary all-graders-pass and scoring.weights has no effect on the verdict. Can be overridden per-run with vally eval --threshold <number>.
timeout	duration	Optional. Shared wall-clock budget for the complete grading phase of one trial.

Complete example

eval.yaml

name: test-writer-eval
description: Evaluates the test-writer skill
version: "1.0"
type: capability

agent_environment:
  skills:
    - ./my-skill # Skill directory (containing SKILL.md)

defaults:
  runs: 3
  timeout: 2m
  model: gpt-5.5
  executor: copilot-sdk

stimuli:
  - name: basic-test-generation
    prompt: |
      Write unit tests for this function:
      function add(a, b) { return a + b; }
      Save the tests to add.test.js.
    graders:
      - type: file-exists
        name: test-file-was-created
        config:
          path: "add.test.js"
      - type: output-contains
        config:
          substring: "test"
      - type: tool-calls
        config:
          required: ["write_file"]
    constraints:
      max_turns: 10

  - name: edge-case-empty-input
    prompt: "Write tests for a function that takes no arguments."
    graders:
      - type: file-exists
        config:
          path: "*.test.js"

scoring:
  weights:
    file-exists: 0.5
    output-contains: 0.3
    tool-calls: 0.2
  threshold: 0.7