YAML Schema

Complete reference for YAML schema used in waza evaluations.

Separate grader-assurance reports

Assurance documents do not change native eval/task/historical result versions. Offline reports use waza.grader-assurance version 1.0; calibrated evaluator reports use version 1.1 with required assessment_mode: independent_authored_rubric_calibration and a required calibration.execution_ledger array. Default offline readers reject 1.1; explicit consumer selection is required.

The calibrated ledger distinguishes actual lifecycle/callback evidence, requested model, independent received-event and accounting attribution, sanitized diagnostics and finalized usage/credits. Incomplete usage is null, not zero; known complete accounting survives operational failure. Exact versioned schemas reject unknown members and inconsistent lifecycle shapes. Supplied-claim admission additionally checks aggregate accounting, provenance and scoped finite-case/domain consistency, but does not authenticate a reviewer or re-execute source evidence. See the grader guide and explicit inspector.

Preserved-native mechanical reports separately select version 1.2 with assessment_mode: preserved_native_mechanical_assurance. The internal assurance.VerifyPreserved operation and assurance.ParsePreservedReport reader use this profile; default CLI/dashboard readers and calibrated 1.1 reading do not admit it automatically. Its source scopes are preserved_native_selected_artifacts and preserved_file_subset. Calibration is not selected: usage/credits remain null, executions/samples are zero and no execution ledger is present. Current native grader bindings do not certify historical grader correspondence.

Portable evidence

Tasks retain their legacy format; current source writes results 1.9 (v0.40.0 writes 1.8), while published v0.39.1 writes 1.7. Historical results retain their declared 1.x versions or the missing-version 1.4 reader default. Snapshot schema 1.1 adds optional evidence and selected workspaceFiles; the portable manifest has an independent version: "1.0". Legacy artifacts still load without assessed evidence. New evidence consumers reject duplicate keys, unknown metadata, and unsupported manifest versions; historical same-major reader tolerance does not authorize unrecognized evidence.

The independent manifest schema is schemas/evidence-manifest-1.0.schema.json. It is embedded for Go consumers as schemas.EvidenceManifestSchemaJSON, not versioned with the result/task schemas.

The dashboard API omits evidenceRuns entirely when all task runs lack both evidence and requirement explanations, preserving the legacy response shape. The UI reports that absence as unassessed. If any run has either kind of metadata, the API returns every trial, including historical missing-metadata rows; absence is never inferred to be successful capture.

Manifest field	Meaning
version, sha256	Independent manifest version and canonical metadata digest
source_manifest_sha256	Original capture digest retained after regrading
origin	eval_id, task_id, run_number, attempt_count, prior_attempts; references are scoped to all fields
runtime	Requested engine/model/tool policy, actual engine-derived execution mode, dependency mode, verified enforcement, waza/SDK/effective-model/Go/platform versions
runtime.no_skills	`boolean
runtime.native_skill_control	unknown, sdk_default, requested_sdk_disable, mock_not_applicable; requested native control, not SDK enforcement proof
artifacts[]	id, kind, availability, completeness, optional content/source digest, snapshot document/pointer, reason, redacted flag
redaction	Policy, actual matched rules and count, limitations; normalization-only changes are not secret matches
diagnostics[]	Typed category/code and safe message; operational uncertainty, not causal classification

captured artifacts require a content digest and snapshot pointer. digest_only uses a separate source digest without preserved content or a content locator. unavailable / not_requested have reasons and no locator. Most tapes have unknown completeness; run errors make them partial. A selected file may be complete while the workspace remains partial/unavailable. External state, grader implementation, rubric version, and lockfile provenance are unavailable. Unknown/null versions stay unavailable rather than being guessed. Captured environment and engine-config artifacts fingerprint the snapshot’s /env allow-list/configuration and /engine configuration respectively; they do not certify the full host environment. Secret environment-key removals count as actual redaction matches.

Cached evidence keeps the original full source origin; task/run/attempt still match even if source eval ID differs. Per-run requirement_explanations contain scoped named-check observations and references with the full origin. Unique names resolve; missing/flattened collisions remain unresolved. Observations are existing grader pass/fail, qualitative prompt checks, or typed unavailable operational status, not new requirement scores or enforcement. Regrading invalidates original result/validation/grader-configuration/checkpoint links: old checkpoints never observe newly declared checks.

Canonical JSON hashes use Go map-encoded sorted UTF-8 object keys, exact numeric lexical tokens, Go encoding/json HTML escaping (<, >, &) and U+2028/U+2029 escaping, and ordered arrays (json-v1). Manifest SHA-256 omits its own member entirely and includes all recognized metadata, including timings; no cross-run hash stability is claimed. Source configuration, instruction, and fixture digests do not preserve source bytes. See Snapshot & Replay for capture safety and the narrow verified file-only regrading operation, and Schema Changes for compatibility.

Frozen prompt-judging evidence

Result writer 1.7 adds optional runs[].judge_evidence and runs[].prompt_judging, plus separate current-invocation regrade_usage.sessions. The bundle has its own strict version 1.0 and embedded judge-evidence-1.0.schema.json. It is not an evidence-manifest document variant. Presence-bearing sections retain captured empty values versus unavailable/null items, full origin, source mode, fixed limits, redaction and integrity metadata. Readers retain legacy defaults. Optional snapshot 1.2 references the frozen digest as judgeEvidenceSHA256; ordinary snapshots remain 1.1.

Final eval/task prompt config accepts an optional object evidence_context with exact slash-relative files and evaluator_only arrays. {} opts into core recorded evidence; null/wrong types, duplicate paths, traversal and glob paths are invalid. Checkpoint, pairwise and continue_session: true combinations are unsupported. The new profile rejects ambiguous keys, aliases/merges, malformed Unicode and invalid wire types before lossy decoding. The runtime verifies digest, complete binding, required sections and current selection before judging. See the canonical contract for limits, native-platform/authentication requirements and privacy boundaries.

Optional descriptive task requirements

requirements is an optional task array. Each entry has id (task-local stable string), category (outcome|boundary|recovery|quality), description, and optional checks. A check has scope: eval|task|checkpoint, grader (explicit existing grader name), and after_turn only for checkpoint scope. Absent/empty checks are uncovered/unresolved. Metadata is descriptive and keeps v1 runtime behavior unchanged; minimum-version notes are not enforcement. See Offline Eval Preflight for schema-valid examples, reference diagnostics and the independent waza.preflight JSON artifact.

Nonempty tasks_from selects CSV tasks instead of YAML tasks; either a nonempty task list or a nonempty CSV source is required by the eval schema. CSV paths are bounded to the eval directory both lexically and after symlinks resolve; filesystem/volume-root eval directories are valid. Inclusive range bounds use 1-based data rows, excluding the header. Fallback IDs/names retain the original row number (range: [2, 2] selects row-2). This is not retained-root, snapshot, or race-free-read assurance.

eval.yaml

Main evaluation configuration file.

name: code-explainer-eval # Required: eval suite name
description: "..." # Required: what this eval tests
skill: code-explainer # Required for legacy suites; optional scenario context
schemaVersion: "1.4" # Optional: schema version; defaults to the current version
version: "1.0" # Optional: version number

config:
  trials_per_task: 1 # Runs per task
  timeout_seconds: 300 # Task timeout
  parallel: false # Concurrent execution
  workers: 0 # Auto-size parallel workers
  model: claude-sonnet-4.6 # Default model
  executor: mock # mock or copilot-sdk

graders: # Validation rules
  - type: text
    name: checks_logic
    config:
      regex_match:
        - "(?i)(function)"

tasks: # Test cases
  - "tasks/*.yaml" # From files

mcp_mocks: # Optional: deterministic local MCP server mocks
  - name: github
    tools:
      list_issues:
        responses:
          - match: { owner: microsoft, repo: waza }
            return:
              issues: []

adversarial: # Optional: fault-injection packs for waza adversarial
  packs:
    - prompt-injection
    - scope-bypass
  on_unsafe_outcome: fail

Top-Level Fields

name

Type: string
Required: yes

The unique name of this evaluation suite. Used in reports and result files.

name: code-explainer-eval

description

Type: string
Required: yes

Describes what this evaluation tests. Appears in reports.

description: "Evaluates agent's ability to explain Python code"

skill

Type: string
Required: yes unless scenario is present

The skill or custom agent being evaluated; optional context for a scenario.

skill: code-explainer

scenario

Type: nonempty string Requires: explicit eval schemaVersion: "2.0" (only this exact version)

Workflow identity without a required skill. A scenario may retain optional skill context using existing discovery precedence. No target or explicit skill directories means no ambient skill loading. An empty task skill_directories: [] disables discovery; --no-skills overrides contexts. Legacy suites retain existing behavior. Tasks and results remain 1.x. Native YAML/JSON admission checks field presence, not its truthiness. A present scenario must be a nonblank string and explicitly use 2.0; null, numbers, booleans, objects and arrays are rejected. This intentionally corrects earlier native-reader acceptance of scenario: null in versionless/1.x suites and scalar-to-string coercion. To retain a legacy suite, remove the scenario field; to select a scenario, supply a nonblank string with schemaVersion: "2.0".

name: inventory-eval
scenario: inventory
schemaVersion: "2.0"

version

Type: string
Required: no
Default: (empty)

Version number of the evaluation suite.

version: "1.0"

schemaVersion

Type: string
Required: no
Default: legacy eval schema version (1.4); current source result writers independently emit 1.9 (v0.40.0 writes 1.8; published v0.39.1 writes 1.7)

Schema version for the public artifact shape. It uses MAJOR.MINOR format with no patch component. Same-major minor additions are backward-compatible; readers ignore unknown fields and warn. Different major versions are rejected with a migration hint.

Scenario-bearing evals explicitly opt into exact 2.0, ensuring older executables reject their new discovery semantics. Eval 2.0 requires a nonempty scenario and rejects unknown eval/config fields; it does not allow future minor versions. Legacy eval/task/result defaults remain unchanged. Every nested baseline_outcome result is checked against the same result-version boundary as its parent. Missing or blank nested versions retain the legacy 1.4 reader default; declared same-major versions are preserved, while malformed or different-major versions are rejected rather than silently reserialized.

schemaVersion: "1.2"

See Schema Changes for the versioning policy and changelog.

mcp_mocks

Type: array Required: no Requires: schemaVersion: "1.1" or newer

Defines deterministic MCP server mocks for Copilot SDK evals. Waza exposes each mock as a local stdio MCP server, so tests do not need live network services, credentials, or fixed ports.

schemaVersion: "1.1"
mcp_mocks:
  - name: github
    fixtures: fixtures/mcp/github

You can define tools inline:

mcp_mocks:
  - name: github
    tools:
      list_issues:
        description: Return issues from a fixture
        input_schema:
          type: object
          properties:
            owner: { type: string }
            repo: { type: string }
          required: [owner, repo]
        responses:
          - match:
              owner: microsoft
              repo: waza
            return:
              issues:
                - number: 363
                  title: MCP server mocks for hermetic eval
          - match_regex:
              repo: "^waza-.*"
            return:
              issues: []
          - match_schema:
              type: object
              required: [owner, repo]
            error: "No fixture for this repository"

Field	Type	Description
name	string	MCP server name made available to the agent
fixtures	string	Directory of JSON tool fixtures, resolved relative to eval.yaml
tools	object	Inline tool definitions keyed by tool name
description	string	Tool description returned from tools/list
input_schema	object	JSON Schema returned from tools/list
responses	array	Ordered responses for tools/call
match	object	Exact full-argument match
match_schema	object	JSON Schema that the call arguments must satisfy
match_regex	object	Per-field regular expression match
return	any	JSON value returned as the mocked tool result
error	string	Tool error text returned with isError: true
sequence	array	Finite scenario-2.0 response steps with return/error and optional delay_ms
delay_ms	integer	Per-step delay from 0 to 300,000 ms; not a host/process timeout guarantee

Response entries are checked in order. Unknown tools and unmatched calls fail loudly with a tool error that names the missing mock fixture. Finite sequence responses are available only in scenario evals using exact schemaVersion: "2.0"; each matching call consumes one step, sequences contain 1–100 steps, and optional step delays are bounded at 300,000 ms.

In v0.40.0, a task can replace an eval-level mock by name with its own mcp_mocks entry. Published v0.39.1 does not apply task-level mocks. Other eval-level mocks remain available to that task.

Unmatched-call diagnostics omit argument values, including nested payloads. A canceled or expired context at the in-process handler’s tool-call admission boundary produces an MCP isError result before matching or returning a fixture. This is not per-request stdio cancellation. An error text containing “timeout” models a dependency-reported error, not transport cancellation. See Dependency Recovery.

command_mocks

Type: array Required: no Requires: schemaVersion: "1.3" or newer and config.executor: copilot-sdk

Declares temporary executable shims for commands invoked by the agent’s shell. The shims are task-isolated, appear before the host PATH, and leave the rest of the host PATH available. Only declared executable names are intercepted; command mocks do not sandbox arbitrary shell commands. Host commands reached through shim pass-through do not inherit the internal WAZA_COMMAND_MOCK_* variables.

schemaVersion: "1.3"
config:
  executor: copilot-sdk
  model: claude-sonnet-4.6

command_mocks:
  - name: az
    expect_calls: 2
    responses:
      - args: ["account", "show", "--output", "json"]
        stdout:
          subscriptionId: "00000000-0000-0000-0000-000000000000"
      - args_regex: ["group", "show", "--name", ".+"]
        fixture: fixtures/commands/az-group-show.json

Field	Type	Description
name	string	Executable name to intercept, such as az or gh.
expect_calls	integer	Optional exact invocation count; a mismatch fails the task.
responses	array	Ordered responses; the first matching response is used.
args	array of strings	Exact arguments, excluding the executable name.
args_regex	array of strings	Positional, full-string regular expressions for the argument vector.
environment	object	Optional environment-variable values that must match exactly.
workdir	string	Optional exact working directory relative to the task workspace.
stdout	any	Output written to stdout; structured values are encoded as JSON.
stderr	string	Output written to stderr.
exit_code	integer	Process exit code from 0 through 255; defaults to 0.
fixture	string	Relative path to raw stdout fixture bytes, resolved from eval.yaml.
sequence	array	Finite response steps, only for scenario eval schema 2.0; at most 100 steps
delay_ms	integer	Per-step delay from 0 to 300,000 ms

Task-level command_mocks replace the eval-level list; an empty list disables the inherited mocks. Configured commands with unmatched calls fail closed and report the executable, sanitized argument vector, and working directory that need a response fixture. Sanitized calls appear in runs[].command_invocations and the grader context. Real credentials should not be placed in mock responses. Only declared executable names are intercepted; other commands may resolve through the host PATH, and shims are not host sandboxing.

adversarial

Type: object Required: no Requires: schemaVersion: "1.2" or newer

Declares built-in adversarial / fault-injection packs for waza adversarial --spec. The normal waza run command ignores this block; it is consumed by the dedicated adversarial harness.

schemaVersion: "1.2"
adversarial:
  packs:
    - prompt-injection
    - scope-bypass
  on_unsafe_outcome: fail

Field	Type	Required	Description
packs	array of strings	yes	Built-in pack identifiers such as prompt-injection and scope-bypass
on_unsafe_outcome	string	no	fail (default, exits 2 on unsafe) or warn (records unsafe outcomes but exits 0)

Unknown pack names and empty packs lists are rejected with a configuration error.

config Section

trials_per_task

Type: integer
Default: 1

How many times each task is run. Use > 1 for statistical confidence.

config:
  trials_per_task: 3

timeout_seconds

Type: integer
Default: 300

Maximum seconds per task before timeout.

config:
  timeout_seconds: 300 # 5 minutes

Common values:

• 60 — Quick validation tasks
• 300 — Standard code analysis
• 600 — Complex multi-file tasks

first_event_timeout_seconds

Type: integer
Default: 0 (disabled)

Maximum seconds to wait for the first session event before treating a run as a session-start hang and aborting it. The embedded engine can occasionally launch but never start the agent’s first turn (no events at all); without this guard such a run blocks until timeout_seconds, which must be sized for the slowest legitimate full turn and is therefore a poor catch-all for a fast failure. Keep this comfortably above realistic first-turn latency so a slow-but-live first turn is never aborted. 0 disables the check. Override per task with the task-level field of the same name (0 disables it for that task).

config:
  timeout_seconds: 1800 # 30 min — the slowest legitimate full turn
  first_event_timeout_seconds: 300 # but the first turn must start within 5 min

parallel

Type: boolean
Default: false

Run tasks concurrently instead of sequentially.

config:
  parallel: true

workers

Type: integer
Default: 0

Number of concurrent workers when parallel: true. Use 0 or omit the field to auto-size workers.

config:
  parallel: true
  workers: 8

model

Type: string
Default: (empty)

Default LLM model. Override with --model flag.

config:
  model: claude-sonnet-4.6

reasoning_effort and judge_reasoning_effort

Type: string (optional)

Options: low, medium, high, xhigh, max

Requires: executor: copilot-sdk

reasoning_effort pins the agent and responder effort across an evaluation, including follow-up session resumes. judge_reasoning_effort supplies the default for prompt graders, including checkpoint, pairwise, and continue_session judges. A prompt grader can override it with graders[].config.reasoning_effort, including task and checkpoint grader definitions. There is no per-task agent effort setting.

config:
  executor: copilot-sdk
  model: gpt-5
  reasoning_effort: high
  judge_model: gpt-5-mini
  judge_reasoning_effort: low

Omit these fields to leave the SDK/model defaults unchanged. With an explicit effort, select a concrete model from waza models: Waza checks the runtime’s supported-effort metadata before creating or resuming hosted Copilot sessions. Unknown models, unavailable metadata, and unsupported values fail with actionable errors instead of silently falling back. Custom providers are not described by the hosted catalog, so their effort is forwarded without this preflight check and provider errors are surfaced. Both selected settings are included in result setup metadata and cache keys. Regrading replaces the recorded judge effort with the current eval’s setting, clearing an old value when the setting is omitted.

executor

Type: string
Default: mock
Options: mock, copilot-sdk

Execution engine:

• mock — Local testing (no API calls)
• copilot-sdk — Real LLM execution

config:
  executor: copilot-sdk

max_attempts

Type: integer
Default: 0

Maximum retry attempts per task on failure. Set to 0 for no retries.

config:
  max_attempts: 3

group_by

Type: string
Default: (none)

Group results by model or by each task’s complete ordered tags set. Tasks without tags are omitted from tag groups. Useful for organizing results when running many tasks. Commas and backslashes in individual tag labels are escaped so different tag sets remain distinct. An explicit empty tag label is represented as \0, while a task with no tags remains ungrouped.

config:
  group_by: tags

fail_fast

Type: boolean
Default: false

Stop the entire evaluation run on the first task failure instead of continuing.

config:
  fail_fast: true

skill_directories

Type: list[string]
Default: []

Additional directories to search for skills beyond the default skills/ directory.

config:
  skill_directories:
    - ./custom-skills
    - /opt/shared-skills

instruction_files

Type: list[string] Default: []

Instruction files to apply to every task. Paths are resolved relative to the active fixtures/context directory, copied into each task workspace, and appended to the agent system message as path-labeled instructions.

config:
  instruction_files:
    - .github/instructions/project.instructions.md

inject_skill_body

Type: boolean Default: true

Controls whether Waza injects the full target SKILL.md or .agent.md body into the system prompt when skill: is set. The injected <skill_context> block starts with Base directory for this skill: <absolute path>, so relative links in the body (for example scripts/install.sh or assets/templates/) resolve from the skill’s own directory even though the agent runs in a temp workspace. This matters for a skill with disable-model-invocation: true, which the agent cannot load through the skill tool.

config:
  inject_skill_body: false

When false, Waza keeps Copilot SDK skill discovery enabled but does not add the target <skill_context> block or a synthetic <available_skills> summary. This is useful for trigger-precision evals that use behavior or skill_invocation graders to measure whether the agent actually invokes the skill tool. disabled_skills: ["*"] takes precedence and disables all skill loading.

trigger_skill_routing

Type: boolean Default: false

When true with inject_skill_body: false, Waza appends a scoped trigger-precision routing instruction: invoke the target skill when the task is in scope, otherwise do not invoke it. The flag does not inject the skill body, does not emit a synthetic <available_skills> summary, and has no effect when normal skill-body injection is enabled.

config:
  inject_skill_body: false
  trigger_skill_routing: true

native_user_input

Type: boolean Default: false

Gives the agent the Copilot SDK ask_user tool, so it can ask the host questions mid-task (through OnUserInputRequest). Each question is answered by the task’s user_input_responses first, then by the task’s adaptive responder (inputs.responder) if it has one. Anything else fails the run. Requires a scenario eval (schemaVersion: "2.0") and executor: copilot-sdk; older Waza versions reject this field rather than ignoring it.

config:
  executor: copilot-sdk
  native_user_input: true

required_skills

Type: list[string]
Default: []

Skills that must be available before the evaluation runs. The run will fail if any required skill is not found.

config:
  required_skills:
    - code-analyzer
    - test-runner

disabled_skills

Type: list[string]
Default: []

Skills to disable for the evaluation. Use ["*"] to disable all skill loading entirely, or list specific skill directory names to exclude.

# Disable all skills
config:
  disabled_skills: ["*"]

# Disable specific skills
config:
  disabled_skills:
    - noisy-skill
    - experimental-skill

mcp_servers

Type: object
Default: (none)

MCP (Model Context Protocol) server configurations for this evaluation. Each key is a server name, and the value is its configuration. When tools is omitted or null for a live stdio or HTTP/SSE server, Waza exposes all tools by sending an explicit ["*"] allowlist to the Copilot SDK. Set tools to an explicit list, including [], to restrict or disable exposed tools.

config:
  mcp_servers:
    filesystem:
      command: /path/to/server
      args: [--root, /data]
      tools: ["*"]
    github:
      type: http
      url: http://localhost:3000

Use top-level mcp_mocks for hermetic evals that should not call live MCP services.

---

graders Section

List of validation rules. Used across tasks.

graders:
  - type: text
    name: pattern_check
    config:
      regex_match: ["success"]

  - type: code
    name: logic_check
    config:
      assertions:
        - "len(output) > 0"

Grader Fields

Field	Type	Description
type	string	Grader type: code, prompt, text, file, json_schema, program, behavior, action_sequence, skill_invocation, trigger, diff, tool_constraint, tool_calls
name	string	Unique grader name (used to reference in tasks)
weight	float	Relative importance in composite scoring (default: 1.0)
config	object	Type-specific configuration

See Validators & Graders for complete documentation.

tasks Section

List of test cases to run.

tasks:
  - "tasks/*.yaml" # Load from files
  - "tasks/basic.yaml" # Specific file

Loading from Files

tasks:
  - "tasks/*.yaml" # All YAML files
  - "tasks/basic/*.yaml" # Subdirectory
  - "tasks/important.yaml" # Single file

File path is relative to eval.yaml.

Task File Format

Individual task files (tasks/task-name.yaml).

id: basic-usage-001 # Required: unique ID
name: Basic Usage # Required: display name
description: "..." # Optional: full description

tags: # Optional: for filtering
  - basic
  - happy-path

inputs: # Required: test inputs
  prompt: "Your instruction"
  files:
    - path: sample.py

expected: # Required: validation rules
  output_contains: ["function"]
  behavior:
    max_tool_calls: 5

inputs.environment (unsupported)

This compatibility field is accepted as a string-to-string map, but its values are not applied during task execution. A non-empty map emits a task-load warning and an unsupported task.environment preflight diagnostic without printing variable names or values. Omitted or empty maps emit neither. Existing run exit policies are unchanged; preflight remains warning-only by default and retains its existing --strict failure policy.

Remove this field and configure required environment variables outside Waza before starting the process. The Copilot client process is shared across tasks, so this is not per-task environment isolation. Native snapshot execution continues to reject non-empty task environment maps.

id

Type: string
Required: yes

Unique task identifier within the eval suite.

id: basic-usage-001

name

Type: string
Required: yes

Human-readable task name.

name: "Basic Usage - Python Function"

description

Type: string
Required: no

Full description of what the task tests.

description: "Test that the agent explains a simple Python function correctly"

tags

Type: array of strings
Required: no

Tags for filtering and categorization.

tags:
  - basic
  - happy-path
  - python

Usage:

waza run eval.yaml --tags "basic"
waza run eval.yaml --tags "edge-case"

skill_directories (optional)

Override the eval-level skill_directories for a specific task. Paths are resolved relative to the eval YAML directory.

skill_directories:
  - ./skills/custom-task-skills

When specified on a task, this replaces (not merges with) the eval-level skill_directories.

instruction_files (optional)

Apply additional instruction files for a specific task. Paths are resolved relative to the active fixtures/context directory. Task-level entries are added to the eval-level config.instruction_files list.

instruction_files:
  - .github/instructions/review.instructions.md

command_mocks (optional)

Replace the eval-level command mock list for this task. The enclosing eval must use schemaVersion: "1.3" or newer and executor: copilot-sdk. Finite response sequences additionally require scenario eval schema 2.0. Use an empty list to disable inherited mocks.

command_mocks: []

mcp_mocks (optional)

Added in v0.40.0: Published v0.39.1 does not apply task-level mcp_mocks.

Replace or add MCP mocks for this task. Each entry replaces the eval-level mcp_mocks entry with the same name for this task only, and a new name adds a mock; other eval-level mocks still apply. Entries use the eval-level fields. The enclosing eval must use schemaVersion: "1.1" or newer and executor: copilot-sdk, and finite response sequences require scenario eval schema 2.0. Relative fixtures paths resolve from the eval directory. The list must not be empty; omit it to use the eval-level mocks.

An eval mock that every selected task replaces is unused by those tasks, but its fixtures are skipped only when no trigger phase needs it; its inline tools are still validated. Trigger tests use eval-level mocks and live servers, not task replacements. An eval-level mock still shadows the same-named live server in that phase. Successfully parsed trigger tests with the mock executor are synthesized without MCP; unresolved trigger discovery or parsing conservatively retains eval-level dependencies during preflight. See the preflight dependency rules.

Task loading and filter errors are reported before engine startup. Admission validates inline mock input_schema declarations offline, even on replaced eval mocks. Schemas must be self-contained (internal $ref values are supported); omission or null retains the default tool schema.

mcp_mocks:
  - name: host
    tools:
      ask_user:
        responses:
          - match_regex: { question: "(?i)publish" }
            return: { answer: Publish this version, wasFreeform: false }
          - match_schema: {}
            return: { answer: Not now, wasFreeform: false }

user_input_responses (optional)

Script answers to the agent’s native questions (the SDK ask_user tool). Requires config.native_user_input: true in the eval. With the flag set, omitting the field (or []) means only the adaptive responder, if any, can answer.

Each entry has a required answer and an optional question, an RE2 pattern that must match the whole question. Entries are used in order across the initial prompt, follow-up prompts, and responder replies of one trial; every trial and retry starts again at the first entry. An entry without question answers whatever is asked next. An answer equal to an offered choice selects it; any other answer is sent as typed text and is rejected when the request does not allow typed answers.

user_input_responses:
  - question: "(?is).*environment.*"
    answer: staging
  - answer: Fixes the login timeout.

A question that does not match the next entry, or arrives after the script is used up, returns an error to the SDK (never a default answer) and the run ends with status error. Unused entries also fail the run. Every request is recorded in runs[].user_input_requests and passed to inline code graders as user_input_requests. With the flag on, a run with no questions records []; without it the field is absent and graders receive null. This is the SDK-native callback, not an MCP ask_user tool mock, and structured elicitation forms are not supported.

Adaptive responder answers

When native_user_input is enabled and the task has an adaptive responder (inputs.responder), scripted entries are still tried first. A question that the next entry does not match goes to the responder model instead, and that entry waits for a later question. The responder uses the same session and instructions as its between-turn replies and receives the question and offered choices. Its answer follows the same choice-or-typed rule. A responder stop, abstain, or error fails the run. Answers given during a turn do not count toward max_followups. Each in-turn responder call may take up to the task timeout, and its time counts against the agent’s turn.

Each record’s source is script or responder. Without the flag, a task’s responder keeps its existing between-turn behavior only. Responder answers call a model and can differ between runs.

inputs Section

Test inputs passed to the agent.

inputs:
  prompt: "Your instruction here"
  files:
    - path: sample.py
    - path: nested/module.py
    - content: |
        def hello():
            print("Hi")

prompt

Type: string
Required: yes (unless prompt_file is used)

Instruction text sent to the agent. When a task includes workspace files, name the file and tell the agent to read it:

inputs:
  prompt: "Read sample.py and explain this Python code."
  files:
    - path: sample.py

prompt_file

Type: string
Required: no (alternative to prompt)

Path to a file containing the prompt text, resolved relative to the task YAML file’s directory. Use this when prompts are long or shared across tasks. You must specify either prompt or prompt_file, but not both.

inputs:
  prompt_file: prompts/review-instructions.md

follow_up_prompts

Type: array of strings
Required: no

Sequential follow-up prompts sent after the initial prompt. Each follow-up reuses the same session and workspace directory, preserving conversation history and file changes across turns.

inputs:
  prompt: "Create a helper function"
  follow_up_prompts:
    - "Add error handling"
    - "Write tests"

Graders evaluate only the final state after all prompts complete. If any follow-up fails, remaining prompts are skipped and the run is marked as an error.

responder

Type: object
Required: no

An LLM-backed surrogate user that answers the skill’s follow-up questions during a multi-turn run. Mutually exclusive with follow_up_prompts.

Field	Type	Required	Description
model	string	no	Responder model. Defaults to the eval-level config.model.
instructions	string	yes	Target configuration the responder represents + abstain rule.
max_followups	integer	yes	Max responder replies before the loop stops (>= 1).

inputs:
  prompt: "Add a new agent to my application"
  responder:
    instructions: "Be research-agent with tools web_search; abstain if unknown."
    max_followups: 8

The responder classifies each agent message as reply, stop, or abstain. An abstain marks the run as an error with outcome abstained, distinct from model timeouts or network errors. If max_followups is reached while the agent is still asking questions, the loop stops with outcome cap_exhausted and graders evaluate the final state.

checkpoints

Type: array of objects
Required: no

Per-turn graders that run at specific turn boundaries during multi-turn conversations. Useful for asserting intermediate state instead of only the final output. Added in schemaVersion: "1.1".

Field	Type	Required	Description
after_turn	integer	yes	1-based turn number to grade after. Turn 1 is the initial prompt; follow-ups are 2, 3, …
graders	array	yes	Inline graders (same schema as the task-level graders: field).
on_failure	string	no	continue (default) or stop. stop aborts remaining turns when this checkpoint fails.

checkpoints:
  - after_turn: 1
    graders:
      - type: text
        contains: ["analyzing", "files"]
  - after_turn: 2
    on_failure: stop
    graders:
      - type: tool_calls
        required: ["read_file"]

Each checkpoint’s outcomes appear on results.json under checkpoints[]. The final-pass validations still run independently. waza gate continues to use the final-pass status; checkpoint failures flip the task status to failed (or error when on_failure: stop fired).

files

Type: array
Required: no

Files copied into the fresh task workspace. A path is required for both fixture-backed and inline files. When content is omitted, path is resolved from the active fixtures directory (./fixtures relative to the eval spec file by default). When content is provided, Waza writes that content to path directly in the workspace. File contents are not inserted into the prompt automatically.

inputs:
  files:
    - path: sample.py # Loaded from the fixtures directory
    - path: inline.py
      content: "def foo(): ..." # Or inline content

repos

Type: array
Required: no

Local git repositories to materialize into the per-task workspace before the agent runs. Useful when developing skills inside the same repo the skills operate on — each test run gets a clean isolated checkout instead of hand-staged fixtures.

inputs:
  prompt: "Explain the repo layout"
  workdir: my-repo
  repos:
    - type: worktree        # required; only "worktree" is currently supported
      source: /path/to/local/clone   # required; local git repo to source from
      commit: main          # optional; commit SHA, branch, or tag (defaults to HEAD)
      dest: my-repo         # optional; subdir under workspace (omit to use workspace root)

Field	Required	Description
type	yes	Materialization strategy. Currently only worktree (uses git worktree add --detach against a local source).
source	yes	Local filesystem path to a git repository.
commit	no	Commit SHA, branch, or tag. Defaults to source HEAD. Branch/tag names use --detach so they don’t conflict with the source checkout.
dest	yes	Relative subdirectory under the workspace where the repo is materialized. Required because git worktree add refuses targets that already exist. Must not contain .. segments.

Worktrees are removed via git worktree remove --force when the engine shuts down, before the workspace directory is deleted.

workdir

Type: string
Required: no

Relative path inside the workspace to use as the agent’s working directory. Typically set to the same value as a repos[*].dest so the agent starts inside the checked-out repo. Must not contain path traversal.

inputs:
  workdir: my-repo

expected Section

Validation rules and constraints.

expected:
  output_contains: ["function", "parameter"]
  output_not_contains: ["error"]
  output_contains_any: ["recursion", "iteration", "loop"]
  outcomes:
    - type: task_completed
  behavior:
    max_tool_calls: 5
    max_tokens: 4096

output_contains

Type: array of strings

Strings that must appear in output.

expected:
  output_contains:
    - "function"
    - "parameter"
    - "return"

output_not_contains

Type: array of strings

Strings that must NOT appear.

expected:
  output_not_contains:
    - "error"
    - "failed"

output_contains_any

Type: array of strings

At least one of these strings must appear in the output (OR logic). Useful when an agent may express a concept in different ways. All checks are case-insensitive.

expected:
  output_contains_any:
    - "recursion"
    - "iteration"
    - "loop"

matches

Type: array of strings

Regex patterns to match.

expected:
  matches:
    - "returns\\s+.*value"
    - "def\\s+\\w+\\("

outcomes

Type: array of objects

Descriptive metadata consumed by static spec coverage. These entries are not native runtime assertions: condition is not evaluated as a state predicate, and metadata does not verify that a tool or external action happened. The native fields are type, value and condition; tool_name is unsupported.

expected:
  outcomes:
    - type: local-record
      value: record.txt
      condition: "contains complete or done (descriptive only)"

For an executable outcome, use explicit native graders and associate requirements.checks with their names. This task permits two valid final states but rejects a wrong state or a forbidden retained file:

id: local-record
name: Inspect a local record
inputs:
  prompt: "Inspect record.txt; do not create forbidden.txt."
  files:
    - path: record.txt
expected:
  outcomes:
    - type: local-record
      value: record.txt
graders:
  - name: final-state
    type: file
    config:
      must_exist: [record.txt]
      must_not_exist: [forbidden.txt]
      content_patterns:
        - path: record.txt
          must_match: ["^(complete|done)\\n?$"]
  - name: no-delete-tool
    type: tool_calls
    config:
      forbidden_tools: [delete_record]
requirements:
  - id: saved
    category: outcome
    description: Accept complete or done, not an arbitrary state.
    checks:
      - scope: task
        grader: final-state
  - id: boundary
    category: boundary
    description: No retained forbidden file or observed delete_record call.
    checks:
      - scope: task
        grader: final-state
      - scope: task
        grader: no-delete-tool

For ordered tool observations use action_sequence with expected_actions and an appropriate matching_mode; do not demand one tool sequence when alternative implementations are valid. File checks observe only the supplied final workspace, and tool graders observe recorded calls, not external state or absent transient effects. The offline regression suite runs the example with complete, done, wrong-state and forbidden-file controls; none is evidence of live agent behavior.

behavior

Type: object

Behavioral constraints on agent execution. Each constraint that is set contributes to the behavior efficiency score.

Field	Type	Description
max_tool_calls	int	Maximum tool invocations allowed
max_iterations	int	Maximum conversation rounds (turns)
max_tokens	int	Maximum tokens in the response
max_response_time_ms	int	Maximum wall-clock execution time in milliseconds
required_tools	string[]	Tools the agent must use
forbidden_tools	string[]	Tools the agent must NOT use

expected:
  behavior:
    max_tool_calls: 5
    max_iterations: 10
    max_tokens: 4096
    max_response_time_ms: 30000 # 30 seconds
    required_tools:
      - grep
    forbidden_tools:
      - rm

.waza.yaml Configuration

Optional project-level configuration file.

# Root of waza project
paths:
  skills: skills
  evals: evals
  results: results

# Generated/discovered eval and task naming
files:
  evalFile: eval.yaml
  taskGlob: tasks/*.yaml
  taskFileSuffix: .yaml

# Model defaults
defaults:
  model: claude-sonnet-4.6
  timeout: 300
  workers: 0

# Cache settings
cache:
  enabled: false
  dir: .waza-cache

# Token budget configuration
tokens:
  warningThreshold: 2500
  fallbackLimit: 1000
  limits:
    defaults:
      "SKILL.md": 500
      "references/**/*.md": 1000
      "*.md": 2000
    overrides:
      "README.md": 3000

# Cloud storage (optional)
storage:
  provider: azure-blob
  accountName: "myteamwaza"
  containerName: "waza-results"
  enabled: true

APM-managed skills are detected from <paths.skills>/<name>/.apm/skills/<name>/SKILL.md without extra configuration. Standard top-level SKILL.md files in paths.skills take precedence when both layouts exist for the same skill.

files Section

Controls the filenames generated by waza new skill and waza new eval. Workspace discovery prefers files.evalFile and falls back to eval.yaml for compatibility.

Field	Type	Default	Description
evalFile	string	eval.yaml	Eval filename to generate and discover
taskGlob	string	tasks/*.yaml	Task glob written into generated eval specs
taskFileSuffix	string	.yaml	Suffix for generated task files

For editor-friendly Waza-specific names:

files:
  evalFile: waza-eval.yaml
  taskGlob: tasks/*.waza-task.yaml
  taskFileSuffix: .waza-task.yaml

tokens Section

Per-file token budget configuration used by waza tokens check and waza tokens suggest. See the Token Limits guide for full documentation including pattern matching rules and migration from .token-limits.json.

Priority order: .waza.yaml tokens.limits → .token-limits.json (legacy, emits deprecation warning) → built-in defaults.

Field	Type	Default	Description
warningThreshold	integer	2500	Token count at which a soft warning is shown
fallbackLimit	integer	1000	Limit applied to files that match no pattern
limits.defaults	map	(built-in)	Glob patterns → token limits
limits.overrides	map	{}	Exact file paths → token limits (take precedence over defaults)

Both limits.defaults and limits.overrides are optional. When tokens.limits is present in .waza.yaml, .token-limits.json is not consulted.

storage Section

Configuration for uploading evaluation results to cloud storage.

provider

Type: string
Values: azure-blob

The cloud provider to use for result storage.

storage:
  provider: azure-blob

accountName

Type: string
Required: yes (when storage: is configured)

The Azure Storage account name. Results are uploaded to blob storage in this account.

storage:
  accountName: "myteamwaza"

containerName

Type: string
Default: waza-results

The blob container where results are stored.

storage:
  containerName: "waza-results"

enabled

Type: boolean
Default: true (when storage: is configured)

Enable or disable automatic result uploads. When false, results are only saved locally.

storage:
  enabled: true

Example Complete eval.yaml

name: code-explainer-eval
description: "Evaluation suite for code-explainer skill"
skill: code-explainer
version: "1.0"

config:
  trials_per_task: 1
  timeout_seconds: 300
  parallel: false
  workers: 0
  model: claude-sonnet-4.6
  executor: copilot-sdk

graders:
  - type: text
    name: explains_concepts
    config:
      regex_match:
        - "function"
        - "parameter"
        - "return"

  - type: code
    name: minimum_length
    config:
      assertions:
        - "len(output) > 200"

  - type: tool_calls
    name: reasonable_calls
    config:
      max_calls: 5

tasks:
  - "tasks/*.yaml"

JSON Schema (programmatic access)

See schemas/config.schema.json in the repository for complete JSON Schema.

# Validate config
jq . schemas/config.schema.json

Next Steps

• Writing Eval Specs — Full guide with examples
• CLI Reference — All commands
• GitHub Repository — Source code