Writing Eval Specs

A complete reference for writing eval.yaml specifications and task definitions.

Available in v0.39.0: scenario authoring and preflight are included in the published standalone release. Install it or run waza update; see availability and platform limits. Before the real-agent examples below, complete authentication, model access, permissions and usage prerequisites. For credential-free harness checks start with Quick Start.

Offline planning and descriptive requirements

Run waza preflight eval.yaml --format json before starting an evaluation. It inspects local task discovery/IDs, paths, configuration, locks/cache, mocks, capabilities and requirement references without starting any agent or service. Unresolved credentials and external state are not reported verified.

Task requirements optionally describe outcome, boundary, recovery and quality intent with stable task-local IDs. Each check names an explicit existing grader using scope: eval|task|checkpoint and grader; checkpoint scope alone requires after_turn. Empty checks are uncovered/unresolved. Metadata does not create assertions, enforce runtime policy, require one implementation path, or change existing v1 grading/exits. Reference resolution is not satisfaction. Use Offline Eval Preflight for complete valid examples and the static-versus-runtime evidence distinction.

Frozen independent prompt graders

Final eval graders and task graders may set config.evidence_context: {} to request frozen task/conversation/tool/output evidence in a fresh judge session. Add files: [report.json] and evaluator_only: [references] inside that object for reviewed explicit file selection. Use executor: copilot-sdk; do not use the option in checkpoints, with mode: pairwise, or with continue_session: true. Omission retains existing behavior; present null is not omission.

The evidence-backed judging reference contains the canonical runnable YAML, limits, explicit authentication and saved regrade commands. Required missing/redacted/incomplete evidence is an operational error, not a partial judgment. The persisted bundle is independent of optional snapshots; each final grader receives only its own selected file subset.

Agent-ready workflow

This guide is designed to be given directly to a coding agent. The agent should use the workflow below when creating or updating an eval suite, while developers can use the same checklist during review.

Required context

Before editing, inspect:

The workflow’s expected outcomes and boundaries; the target SKILL.md or .agent.md when evaluating one.
The existing eval.yaml, task files, and fixtures, if present.
schemas/eval.schema.json and schemas/task.schema.json.
The grader guide and existing nearby evals for repository conventions.

Treat the schemas as the source of truth for allowed fields. Do not invent YAML properties from examples or model memory.

Create an eval suite

For a workflow without a target:

waza new eval inventory --scenario --template repository
waza run evals/inventory/eval.yaml

Templates are repository (sanitized fixtures), cli (installed Git), and mcp (harness-only mocked dependency). All use a real copilot-sdk agent; MCP mocks do not establish production MCP reliability. config.executor: mock checks harness plumbing only, not agent quality. See the scenario examples.

Scenario evals require a nonempty scenario string and exact schemaVersion: "2.0". Older executables reject that major before executing; 1.x, omitted versions, 2.1 and future majors are not scenario contracts. Presence of the scenario key is explicit in YAML and JSON. Presence is checked in native YAML/JSON parsing: null, numeric/boolean values, objects, arrays and blank strings are invalid. This intentionally stops earlier versionless/1.x null acceptance and scalar coercion. Remove the field for a legacy suite, or provide a nonblank string and exact 2.0 for a scenario. Tasks and results retain 1.x; the dashboard still displays the suite’s eval_name and generic results have an empty skill. Optional skill selects existing skill/custom-agent context with unchanged precedence. Without a target or explicit eval/task skill directories, ambient skills are disabled. An explicit empty task skill_directories: [] disables discovery for that task; --no-skills wins over every opt-in. Disabling the last eval-level directory preserves target-free isolation. Whitespace-only optional skill values do not opt into workspace-default or ambient discovery. Scenario required_skills checks search enabled directories only; a disabled path, ["*"] or none cannot satisfy a required skill. Legacy discovery and validation retain their existing behavior. Scenario task files retain tasks/ when selected by files.taskGlob, including wildcard-directory globs; otherwise they use a matching literal directory. Unsupported or nonmatching glob/suffix combinations fail before writing. The glob must select only generated tasks, not evals, fixtures or metadata. fixtures/*.yaml remains supported; fixtures/* is rejected because it also selects .gitkeep and repository inventory. Narrow the glob before regenerating. Scenario scaffolding requires physical parent directories for the selected output path; symlinked roots or generated task/fixture parents are rejected. Use a physical --output path. Relative paths start at the retained working directory; absolute paths are admitted from the filesystem root, with only verified literal macOS /var and /tmp system aliases. Writes use retained directory handles rather than following replaced parent paths. Failed creation or writing rolls back this attempt’s owned files, including a partial output. Created-file handles stay open until success or rollback so unlinked ownership cannot be confused with a recycled inode. Observed replacements and unknown content are preserved, with explicit cleanup and handle-close errors. Empty directories can remain, and an admitted directory can subsequently move.

Scenario 2.0 bypasses result caching with an explicit notice, even with --cache, until external workflow dependencies can be fully fingerprinted. Legacy 1.x caching and billing attribution are unchanged. Repository scaffolds include fixtures/.gitattributes preserving inventory LF bytes through Git checkout. If a task glob selects that file, a safe root .gitattributes is used instead. When neither placement avoids an eval/task collision or extra task, generation fails before writing; narrow the task-directory glob or choose a different eval filename. For example, use tasks/*.gitattributes instead of fixtures/*.gitattributes when the eval itself is named .gitattributes. A normal eval filename with **/*.gitattributes keeps the existing runtime glob semantics and uses the root fallback. Older suites must be regenerated to adopt this policy; Git attributes do not rewrite existing working copies. Task globs and inputs.context.fixture resolve relative to the eval directory. Fixtures default to its fixtures/ directory; inputs.files and instruction_files resolve from the fixtures/context root. The repository template explicitly declares inputs.context.fixture: fixtures/inventory.txt. File grader outputs are task-workspace-relative. The repository template uses three file graders: report outcome, absence of private.txt, and byte-exact final preservation of inventory.txt. Altering, deleting or changing its line endings fails independently of the other checks. This checks final bytes, not transient writes or edit history. CLI/MCP templates retain two graders. The tracked repository fixture alone disables Git text conversion so Windows checkout preserves the expected LF bytes.

For a SKILL.md target:

# Scaffold from an existing skill.
waza new eval <skill-name>

# Generate candidate coverage without writing files.
waza suggest <path-to-SKILL.md> --dry-run

# Add generated tasks and fixtures to the scaffolded suite.
waza suggest <path-to-SKILL.md> --apply --output-dir evals/<skill-name>

The default waza new eval flow, waza suggest, and waza spec verify accept SKILL.md targets only. For a .agent.md target, create eval.yaml, tasks, and fixtures directly by following the custom agent guide and the schemas linked above.

After scaffolding, the agent should:

Map each USE FOR, DO NOT USE FOR, parameter, and important behavioral requirement to at least one task.
Include positive, negative, boundary, and realistic integration cases.
Use fixtures instead of relying on the developer’s working tree or live external state.
Prefer deterministic graders (text, file, behavior, action_sequence, or skill_invocation) when the expected behavior can be expressed mechanically. Use an LLM judge only for genuinely qualitative requirements.
Use copilot-sdk for real behavioral evaluation. The mock executor only verifies the harness and fixture plumbing; it does not evaluate agent quality.

Guide implementation with evals

When an eval accompanies an implementation change, use a test-first loop:

Add or update the task that expresses the required behavior.
Run only that task and confirm that it fails for the expected reason.
Change the skill, custom agent, or supporting implementation.
Rerun the task until it passes without weakening the grader.
Run related tasks, then the complete suite, to catch regressions.

Keep the task focused on observable behavior. Do not encode implementation details unless the requirement explicitly depends on a tool call, file change, or action sequence.

Update an existing eval suite

When the target skill or agent changes:

Read the diff and identify changed triggers, anti-triggers, parameters, tools, outputs, and failure behavior.
For a SKILL.md target, run waza spec verify to find uncovered requirements. For a .agent.md target, compare the tasks directly with the agent’s description, tools, handoffs, and behavioral instructions.
Add or update the smallest set of tasks needed to cover the changed behavior. Preserve stable task IDs unless the scenario itself is being replaced.
Add a regression task for every bug fix or previously missed behavior.
Keep unrelated tasks, fixtures, thresholds, and grader choices unchanged.
If external tools are involved, use mcp_mocks with schemaVersion: "1.1" or newer for deterministic CI coverage. Keep live-service testing in a separate integration suite.

File invariants

Preserve the scaffolded or existing schemaVersion. Do not copy a newer version unless the eval uses fields that require it. mcp_mocks requires version 1.1 or newer; adversarial requires version 1.2 or newer.
Every task requires non-empty id, name, and inputs.
inputs must contain exactly one of prompt or prompt_file.
Task IDs must be unique and stable; use lowercase kebab-case.
Relative paths are resolved from the eval or task location described by the relevant schema field. Do not depend on the shell’s current directory.
Fixtures must contain all state required by the scenario and must not contain secrets.
Unknown MCP tools and unmatched mock calls should fail; do not add permissive fallback responses that hide missing fixtures.
Before committing a behavioral suite, confirm config.executor is copilot-sdk. Never commit mock as a substitute for real evaluation.
Do not weaken thresholds merely to make a failing eval pass. Fix the skill, task, fixture, or grader that does not represent the intended behavior.

Validation ladder

Run the smallest relevant check first, then expand:

# SKILL.md only: inspect uncovered requirements without failing.
waza spec verify <skill-path> <eval-path>

# SKILL.md only: enforce complete requirement coverage.
waza spec verify <skill-path> <eval-path> --fail

# Exercise one changed task with the executor configured in eval.yaml.
waza run <eval-path> --task "<task-id>" --no-cache -v

# Run the complete suite and save reviewable output.
waza run <eval-path> --no-cache \
  --output results.json \
  --reporter junit:test-results.xml

Set config.executor: copilot-sdk before behavioral evaluation. Use config.executor: mock only to check task loading, fixture copying, and grader plumbing without a real agent session, then restore copilot-sdk before committing.

For non-deterministic behavior, increase trials_per_task only after the single-trial task is correct. Review failures individually; do not treat a passing aggregate score as proof that every scenario is valid.

Delegation prompt

Developers can give an agent this guide together with a focused request:

Create or update the Waza evals for <skill-or-agent-path>.

Follow the Agent-ready workflow in:
https://microsoft.github.io/waza/guides/eval-yaml/

Requirements:
- Treat schemas/eval.schema.json and schemas/task.schema.json as authoritative.
- Preserve unrelated eval behavior and stable task IDs.
- Cover positive triggers, negative triggers, parameters, edge cases, and the
  implementation behavior changed in <issue-or-diff>.
- Use fixtures and mcp_mocks instead of live external state where possible.
- For implementation work, add a focused failing task before changing behavior.
- For SKILL.md targets, run waza spec verify.
- Run a targeted task and the full relevant suite with config.executor set to
  copilot-sdk. Report the exact commands and results.

eval.yaml Structure

The evaluation spec defines the benchmark configuration, graders, and task files:

name: code-explainer-eval
description: Evaluation suite for code-explainer skill
skill: code-explainer
schemaVersion: "1.0"
version: "1.0"

config:
  trials_per_task: 1
  timeout_seconds: 300
  parallel: false
  executor: mock
  model: claude-sonnet-4.6

metrics:
  - name: accuracy
    weight: 1.0
    threshold: 0.8

graders:
  - type: text
    name: explains_concepts
    config:
      regex_match:
        - "(?i)(function|logic|parameter)"
  - type: code
    name: has_output
    config:
      assertions:
        - "len(output) > 100"

tasks:
  - "tasks/*.yaml"

Top-Level Fields

Field	Type	Required	Description
`name`	string	✓	Eval suite name
`description`	string	✗	What the eval tests
`skill`	string	Legacy schema	Associated skill/custom-agent target; optional context for scenario `2.0`
`scenario`	string	Scenario `2.0`	Nonblank workflow identity; legacy authoring omits this field
`schemaVersion`	string	Scenario `2.0`	Legacy defaults to `1.4`; scenarios explicitly select `2.0`
`version`	string	✗	Version number (e.g., “1.0”)
`inputs`	object	✗	Key-value map of global template variables (see Template Variables)
`tasks_from`	string	✗	Relative or absolute CSV dataset path; takes precedence over `tasks` when nonempty
`hooks`	object	✗	Lifecycle hooks that run shell commands at specific points (see Hooks)
`mcp_mocks`	array	✗	MCP server mocks for deterministic tool-call evals; requires `schemaVersion: "1.1"`; finite sequences require scenario `2.0`
`command_mocks`	array	✗	Executable shims for CLI-dependent evals; requires `schemaVersion: "1.3"` or newer and `executor: copilot-sdk`; finite sequences require scenario `2.0`
`adversarial`	object	✗	Built-in fault-injection packs consumed by `waza adversarial --spec`; requires `schemaVersion: "1.2"`
`baseline`	bool	✗	Mark this spec as a baseline for A/B comparison

Legacy JSON-schema authoring requires skill when scenario is absent. Scenario 2.0 requires a nonblank scenario and may omit skill. The native loader/model validation used by waza run is a different surface from full JSON-schema validation: legacy default/1.0-1.4 data historically loads with an absent or empty target and retains existing discovery behavior. Preserving that compatibility does not make targetless legacy data the recommended workflow; author a named legacy target or a scenario 2.0 suite. The schema’s skill constraint is length-only: whitespace passes that check but is not a meaningful target. JSON-RPC eval.validate reports schema errors; separate preflight checks are not universal runtime target enforcement.

schemaVersion uses MAJOR.MINOR format. Omit it for legacy files that should default to 1.4; scenario evals must select 2.0. See Schema Changes for the compatibility policy.

Targeting Custom Agents

Waza supports evaluating VS Code custom agents (.agent.md files) alongside traditional SKILL.md-based skills. When you target an agent with a tools: field in its frontmatter, waza automatically injects a tool_constraint grader to validate that only the declared tools are called.

Specify an agent:

name: security-agent-eval
description: Evaluate the security-reviewer custom agent
skill: security-reviewer  # Points to security-reviewer.agent.md
schemaVersion: "1.0"
version: "1.0"

config:
  model: claude-sonnet-4.6

Key differences:

Use skill: <name> to target either a skill or a custom agent
Waza discovers .agent.md files the same way as SKILL.md — in the current directory or agents/ subdirectories
If both SKILL.md and .agent.md exist in the same directory, SKILL.md takes priority
Custom agents can declare a tools: field in frontmatter, which auto-injects a tool_constraint grader

Learn more: See the Evaluating Custom Agents guide for detailed examples and the auto-injected tool constraint behavior.

Config Section

The config block controls execution behavior:

config:
  trials_per_task: 1 # Run each task this many times
  timeout_seconds: 300 # Task timeout in seconds
  parallel: false # Run tasks sequentially (true = concurrent)
  workers: 0 # Auto-size parallel workers if parallel: true
  model: claude-sonnet-4.6 # Default model (override with --model)
  reasoning_effort: high # Optional: pin task and responder reasoning effort
  judge_model: gpt-4o # Model for LLM-as-judge graders (optional)
  judge_reasoning_effort: low # Optional: default effort for prompt graders
  executor: copilot-sdk # mock (local) or copilot-sdk (real API)
  inject_skill_body: true # Inject target SKILL.md/.agent.md body into the prompt
  instruction_files:
    - .github/instructions/project.instructions.md

Field	Type	Default	Description
`trials_per_task`	int	1	Number of times each task runs (for statistical analysis)
`timeout_seconds`	int	300	Task timeout in seconds
`first_event_timeout_seconds`	int	0 (off)	Abort a run that produces no first event within N seconds (session-start hang); 0 disables
`parallel`	bool	false	Run tasks concurrently
`workers`	int	0	Number of parallel workers; 0 auto-sizes
`model`	string	required	Default model for tasks (override with `--model` flag)
`reasoning_effort`	string	model default	Copilot SDK task/responder effort: `low`, `medium`, `high`, `xhigh`, or `max`
`judge_model`	string	(same as `model`)	Model for `prompt`-type graders (LLM-as-judge)
`judge_reasoning_effort`	string	judge default	Copilot SDK default effort for `prompt` graders, including `continue_session` graders; the grader can override it
`executor`	string	`copilot-sdk`	Executor: `mock` (local, echoes task metadata and file content) or `copilot-sdk` (real API)
`max_attempts`	int	0	Maximum retry attempts per task on failure (0 = no retries)
`group_by`	string	—	Group results by a field (e.g., `tags`, `task_id`)
`fail_fast`	bool	false	Stop the entire run on first task failure
`skill_directories`	list[str]	`[]`	Additional directories to search for skills
`instruction_files`	list[str]	`[]`	Instruction files to apply to every task
`inject_skill_body`	bool	true	Inject the target `SKILL.md` or `.agent.md` body, prefixed with its absolute base directory, into the system prompt
`trigger_skill_routing`	bool	false	With `inject_skill_body: false`, add an eval-only routing instruction for trigger-precision runs
`disabled_skills`	list[str]	`[]`	Skills to disable. Use `["*"]` to disable all skills
`required_skills`	list[str]	`[]`	Skills that must be available before running
`mcp_servers`	object	—	MCP server configurations for the evaluation. When `tools` is omitted or null for a server, Waza exposes all server tools by sending `["*"]` to the Copilot SDK. Set `tools` to an explicit list, including `[]`, to restrict or disable exposed tools.

With an explicit reasoning effort, use a concrete model from waza models. Waza checks the hosted runtime’s supported-effort metadata before session creation and resumption, rejecting unsupported values rather than silently downgrading them. Custom providers receive the effort directly because they are not covered by the hosted model catalog. Prompt graders in eval, task, and checkpoint definitions may override judge_reasoning_effort; agent effort stays eval-level.

Trigger-precision evals

Set inject_skill_body: false when the eval is measuring whether the agent invokes a skill rather than whether it can complete the work after already seeing the skill body:

name: xyz-trigger
description: Trigger-precision tasks for the xyz skill
skill: xyz
schemaVersion: "1.0"
version: "1.0"

config:
  trials_per_task: 1
  timeout_seconds: 300
  parallel: false
  executor: copilot-sdk
  model: claude-sonnet-4.6
  inject_skill_body: false
  trigger_skill_routing: true

metrics:
  - name: trigger_precision
    weight: 1.0
    threshold: 0.8

tasks:
  - "tasks/trigger/*.yaml"

trigger_skill_routing is optional and scoped to trigger-precision runs. It helps distinguish skill routing from general model fluency by telling the agent to invoke the target skill when the task is in scope, while still withholding the skill body. It has no effect unless inject_skill_body is false.

With this setting, Waza still discovers skills and passes their directories to the Copilot SDK, but it does not add the full target <skill_context> body (normally prefixed with the skill’s absolute base directory) or a synthetic <available_skills> summary. This lets behavior graders with required_tools or forbidden_tools and skill_invocation graders observe whether the skill tool was used. If you also set disabled_skills: ["*"], all skill loading is disabled and this setting has no effect.

Common Timeouts:

60 — Quick tasks (single-file review, validation)
300 — Standard tasks (code explanation, analysis)
600 — Complex tasks (multi-file refactoring, design)

MCP Mock Servers

Use top-level mcp_mocks to replace live MCP dependencies with deterministic local stdio servers. This keeps Copilot SDK evals hermetic in CI: no network listener, no port allocation, and no external service credentials. Waza automatically exposes every tool declared by a mock to the Copilot CLI, so no separate tools allowlist is needed. Because this is an additive eval schema field, set schemaVersion: "1.1" or newer.

name: issue-triage-eval
skill: issue-triage
schemaVersion: "1.1"
version: "1.0"

config:
  executor: copilot-sdk
  model: claude-sonnet-4.6

mcp_mocks:
  - name: github
    tools:
      list_issues:
        description: Return matching issues for a repository
        input_schema:
          type: object
          properties:
            owner: { type: string }
            repo: { type: string }
          required: [owner, repo]
        responses:
          - match:
              owner: microsoft
              repo: waza
            return:
              issues:
                - number: 363
                  title: MCP server mocks for hermetic eval
          - match_regex:
              repo: "^waza-.*"
            return:
              issues: []
          - match_schema:
              type: object
              required: [owner, repo]
            error: "No fixture for this repository"

tasks:
  - tasks/*.yaml

Response matching is evaluated in order. match requires exact full-argument equality, match_schema validates the call arguments against an inline JSON Schema, and match_regex applies regular expressions to individual argument fields. Unknown tools or calls that do not match any response fail loudly with an MCP tool error that points to the missing mock fixture.

Unmatched MCP errors intentionally omit argument values, rather than echoing private or nested payloads. Scenario evals with exact schemaVersion: "2.0" can use finite sequence steps of return or error, with optional delays from 0 to 300,000 ms and at most 100 steps. A matching call consumes one step; exhaustion and unmatched calls fail closed. An error containing “timeout” is a dependency-reported error, not transport cancellation. A malformed application value is still encoded in a valid MCP result, not a malformed MCP/JSON-RPC envelope. See the Dependency Recovery guide for complete scenarios and isolation limits.

You can keep larger fixtures in JSON files instead of inline YAML:

mcp_mocks:
  - name: github
    fixtures: fixtures/mcp/github

Each .json fixture can either contain a single tool definition (tool name from the filename) or a { "tools": { ... } } object with multiple tools.

Command Mocks

Use command_mocks when a skill shells out to a CLI such as Azure CLI (az), GitHub CLI (gh), or kubectl. Waza materializes temporary executable shims per task and prepends their directory to the Copilot runtime’s existing PATH, preserving the host path. Command mocks require executor: copilot-sdk and schemaVersion: "1.3" or newer.

schemaVersion: "1.3"
config:
  executor: copilot-sdk
  model: claude-sonnet-4.6

command_mocks:
  - name: az
    expect_calls: 3
    responses:
      - args: ["account", "show", "--output", "json"]
        stdout:
          subscriptionId: "00000000-0000-0000-0000-000000000000"
          name: Test Subscription
      - args_regex: ["group", "show", "--name", ".+"]
        fixture: fixtures/commands/az-group-show.json
      - args: ["deployment", "group", "create"]
        stderr: "deployment failed"
        exit_code: 1

tasks:
  - "tasks/*.yaml"

args matches the entire argument vector, excluding the executable. args_regex applies a full-string regular expression to each argument at the same position. Responses are evaluated in order and the first matching response wins. stdout strings are emitted verbatim; structured values are JSON-encoded. fixture reads raw response bytes from a path relative to eval.yaml. Optional environment values must match exactly, and workdir matches a workspace-relative path. expect_calls requires an exact number of invocations for that executable; a mismatch is reported as a task error.

Task-level command_mocks replace the eval-level list. Set command_mocks: [] on a task to disable inherited mocks. A command that is configured but called with unmatched arguments fails closed with a diagnostic asking for a matching response fixture. Invocation records contain sanitized arguments, exit codes, and response indexes in runs[].command_invocations; graders can also read them from their context. Verbose runs print those sanitized details. Only declared executable names are intercepted, so command_mocks is not a command sandbox. Do not put real credentials in mock output.

Scenario evals with exact schemaVersion: "2.0" may use a response sequence of at most 100 output steps; each may delay up to 300,000 ms. A matching call consumes one step and exhaustion fails closed. Earlier failed attempts remain in runs[].command_invocations even when a later attempt succeeds. Task-level empty-list replacement is unchanged.

Choosing a test boundary

Mechanism	Use it for
`command_mocks`	Commands the agent invokes through its shell, with deterministic process output and exit status.
`mcp_mocks`	Structured MCP tool calls and server responses.
Hooks	Setup, teardown, or validation around tasks/evals; hooks do not intercept agent commands.
Live integration tests	Verifying real CLI/service behavior, authentication, or infrastructure. Keep these separate from hermetic evals.

Adversarial Packs

Use top-level adversarial to pin the built-in fault-injection packs and unsafe-outcome policy for waza adversarial --spec. This is additive in schemaVersion: "1.2" and is ignored by normal waza run executions.

schemaVersion: "1.2"
adversarial:
  packs:
    - prompt-injection
    - scope-bypass
  on_unsafe_outcome: fail  # or "warn"

packs must name one or more built-in packs. In v0.38.0 those are prompt-injection and scope-bypass. on_unsafe_outcome: fail exits 2 when an unsafe outcome is observed; warn records the unsafe result but exits 0. See the Adversarial Harness guide for pack behavior and CI examples.

Graders Section

Graders validate task outputs. Define once, reuse across tasks:

graders:
  - ref: github.com/waza-evals/fact#factuality@v1.0.0
    name: factuality_strict
    weight: 2.0
    config:
      threshold: 0.9

  - type: text
    name: checks_logic
    weight: 2.0
    config:
      regex_match:
        - "(?i)(function|variable|parameter)"

  - type: code
    name: has_minimum_output
    config:
      assertions:
        - "len(output) > 100"
        - "'success' in output.lower()"

  - type: text
    name: mentions_key_concepts
    config:
      contains:
        - "algorithm"
        - "optimization"

Each grader accepts an optional weight (default 1.0) that controls its influence on the composite score. See Validators & Graders for details.

Remote grader presets can be referenced with ref instead of type. Refs use <host>/<owner>/<repo>[/path][#export]@<version> and resolve through waza get, which writes waza.lock with the pinned commit SHA and content digest. Local name, weight, top-level grader fields, and config values override the remote preset; nested config maps are deep-merged, while lists are replaced.

waza get eval.yaml
waza run eval.yaml

waza run requires the lockfile and cached module contents to be present for remote refs. It fails closed on missing locks, missing cache entries, or digest mismatches.

All graders return:

score: 0.0 to 1.0
passed: boolean
message: human-readable result

See the Validators & Graders guide for all 13 native kinds and examples.

Mapping OpenAI Evals `modelgraded` YAML

OpenAI Evals modelgraded specs usually collapse into Waza’s prompt grader. The judge prompt carries the label semantics, while Waza handles execution and scoring.

OpenAI Evals field	Waza equivalent	Notes
`prompt`	`graders[].config.prompt`	Put the judging instructions directly in the prompt
`choice_strings`	prompt text	List the labels in the judge prompt; Waza’s prompt grader is binary, so the label choice becomes pass/fail guidance
`choice_scores`	prompt text	Encode the scoring rule in the judge prompt; use `pairwise` mode when the comparison is relative
`input_outputs`	`tasks:` entries	Turn each example into one Waza task with its own `inputs.prompt` and expected checks
`eval_type: cot_classify`	`type: prompt`	Use `mode: independent` for one-shot classification
`battle.yaml`	`type: prompt` + `mode: pairwise`	Closest grader match for head-to-head comparison; `waza compare` is still the better run-level report

Translation examples

`fact.yaml`

OpenAI’s registry uses this pattern for fixed-choice factual classification. In Waza, keep the evaluation as a single prompt grader and turn each input/output row into a task:

graders:
  - type: prompt
    name: fact_check
    config:
      prompt: |
        You are checking a multiple-choice answer.
        Valid choices: A, B, C, D, E.
        Call set_waza_grade_pass only if the model's answer matches the correct choice.
        Otherwise call set_waza_grade_fail with a short reason.
      continue_session: false

tasks:
  - id: fact-001
    name: fact-001
    inputs:
      prompt: "Which answer is correct for the fact pattern?"
    expected:
      output_contains:
        - "B"

`closedqa.yaml`

For closed-book QA, the judge prompt can encode the score mapping directly:

graders:
  - type: prompt
    name: closedqa_judge
    config:
      prompt: |
        Judge the answer against the reference.
        If the answer is fully correct, call set_waza_grade_pass.
        If it is partially correct or incorrect, call set_waza_grade_fail.
        Treat "Y" as 1.0 and "N" as 0.0 in your reasoning, but only emit pass/fail.
      model: claude-sonnet-4.5

tasks:
  - id: closedqa-001
    name: closedqa-001
    inputs:
      prompt: "Answer the question using the provided context."
    expected:
      output_contains:
        - "Y"

`battle.yaml`

Battle-style comparisons are the one place where the mapping is not 1:1. The nearest Waza translation is a pairwise prompt grader, but the run-level comparison report is usually better expressed with waza compare:

config:
  baseline: true

graders:
  - type: prompt
    name: battle_judge
    config:
      mode: pairwise
      prompt: |
        Compare the two answers and decide which one is better.
        Call set_waza_grade_pass if the skill run wins.
        Call set_waza_grade_fail if the baseline run wins.

tasks:
  - id: battle-001
    name: battle-001
    inputs:
      prompt: "Compare these two solutions and pick the better one."

Tasks Section

Tasks define individual test cases loaded from YAML files:

From Files

Load tasks from YAML files in a directory:

tasks:
  - "tasks/*.yaml" # All YAML files in tasks/
  - "tasks/basic/*.yaml" # Specific subdirectory
  - "tasks/advanced.yaml" # Single file

Task File Format

Individual task files (e.g., tasks/basic-usage.yaml):

id: basic-usage-001
name: Basic Usage - Python Function
description: Test that the skill explains a simple Python function correctly.

tags:
  - basic
  - happy-path

inputs:
  prompt: "Read sample.py and explain the function it contains."
  files:
    - path: sample.py

expected:
  output_contains:
    - "function"
    - "parameter"
    - "return"
  outcomes:
    - type: task_completed
  behavior:
    max_tool_calls: 5

Task Fields

Field	Type	Description
`id`	string	Unique task identifier
`name`	string	Human-readable task name
`description`	string	What the task tests
`tags`	array	Tags for filtering (e.g., `["basic", "edge-case"]`)
`inputs`	object	Test inputs (prompt, files)
`expected`	object	Validation rules and expected behavior
`skill_directories`	string[]	Skill directories for this task (overrides eval-level)
`instruction_files`	string[]	Instruction files for this task (adds to eval-level files)
`golden`	bool	Mark as a critical “golden” task — enforced by `waza gate` (must always pass)
`requirements`	array	Descriptive intent linked to explicitly scoped existing graders; does not add assertions or change pass/fail

Link intent to existing checks

Use requirements to identify the intent behind an existing named grader:

id: report-task
name: Report the inspected result
inputs:
  prompt: "Return a report containing the word inspected."
graders:
  - name: report-check
    type: text
    config:
      contains: ["inspected"]
requirements:
  - id: report-content
    category: outcome
    description: The report identifies the inspected result.
    checks:
      - scope: task
        grader: report-check

task selects a task grader; eval selects an eval-level grader. checkpoint selects a checkpoint grader and requires the matching after_turn. Names must resolve uniquely in their scope. A final result name shared by eval and task graders is unresolved because the stored result map does not distinguish their observations. Missing checks and requirements without checks stay unassessed.

runs[].requirement_explanations records existing grader outcomes and their uncertainty, not new requirement scores, enforcement, or causes. A passing check does not erase an overall operational error. With --snapshot, references identify the whole captured artifact and an artifact-relative result pointer; they are not file contents or proof that a grader’s workspace is reconstructible. Regrading applies the currently selected declarations and never reassesses historical checkpoints. See portable evidence and the result schema.

Inputs Section

inputs:
  prompt: "Your instruction to the agent"
  context:
    fixture: fixtures/demo # Optional: copy this fixture file/dir into the workspace
  files:
    - path: sample.py # Fixture file (relative to fixtures dir)
      content: | # Or inline content
        def hello():
            print("Hello")

inputs.files loads each path from the active fixtures directory (./fixtures relative to the eval spec file by default) and copies it into the fresh task workspace. It does not add file contents to prompt; tell the agent which workspace file to read. Use content to define a file inline instead.

inputs.context.fixture is resolved relative to the eval spec directory. When it points to a directory, Waza copies that directory’s contents into the fresh task workspace before the agent runs.

Loading prompts from a file

Use prompt_file instead of prompt to load the prompt text from an external file. The path is resolved relative to the task YAML file’s directory.

inputs:
  prompt_file: prompts/review-instructions.md
  files:
    - path: sample.py

This is useful when prompts are long, shared across tasks, or maintained separately. You must specify either prompt or prompt_file, but not both.

Follow-up Prompts

Use follow_up_prompts to send additional messages after the initial prompt. Each follow-up reuses the same session and workspace, so file changes and conversation history persist across turns.

inputs:
  prompt: "Create a Python function that reads a CSV file"
  follow_up_prompts:
    - "Add error handling for missing files"
    - "Write unit tests for the function"

This is useful for evaluating multi-turn conversations where each step builds on the previous one. Graders run only after all prompts (initial + follow-ups) have completed, so the final output reflects the full conversation.

Responder (Interactive Skills)

For skills that ask follow-up questions, configure a responder — an LLM that plays the user and answers the skill’s questions. It is mutually exclusive with follow_up_prompts.

inputs:
  prompt: "Add a new agent to my application"
  responder:
    model: gpt-4o          # optional; defaults to config.model
    instructions: |
      The agent you want is "research-agent" with system instructions
      "Search the web and summarise findings", tools web_search + url_fetch,
      and no handoffs. Answer the skill's questions consistently with this.
      If you genuinely can't infer an answer, abstain.
    max_followups: 8

After each agent turn the responder either replies (the answer is sent back, continuing the conversation), stops (the agent is done), or abstains — which fails the run with a distinct abstained outcome, signalling the brief is too vague. If max_followups is reached while the agent is still asking questions, the loop stops with outcome cap_exhausted and graders evaluate the final state. Each task carries its own responder, so the same skill can be tested against several target configurations.

Per-Turn Checkpoints

By default graders only run once, against the final state of a multi-turn conversation. For long conversations you can run graders at specific turn boundaries using a top-level checkpoints: list:

checkpoints:
  - after_turn: 1
    graders:
      - type: text
        contains: ["analyzing", "files"]
  - after_turn: 2
    on_failure: stop # abort the run if this checkpoint fails
    graders:
      - type: tool_calls
        required: ["read_file"]

Each checkpoint accepts:

Field	Type	Description
`after_turn`	int	1-based turn number this checkpoint runs after (initial prompt is turn 1).
`graders`	array	Inline graders, same schema as the task-level `graders:` / eval-level `graders:` field.
`on_failure`	string	`continue` (default) or `stop` — abort remaining turns when this checkpoint fails.

Outcomes are recorded per-checkpoint on results.json under checkpoints[], alongside the final validations. waza gate still uses final-pass status. Available with schemaVersion: "1.1" and above (additive — older 1.0 files load unchanged).

When a task includes workspace files, name the file in the prompt so the agent knows to read it:

inputs:
  prompt: "Read sample.py and explain this code."
  files:
    - path: sample.py

Expected Section

expected:
  # Strings that must appear in output
  output_contains:
    - "function"
    - "parameter"

  # Output must NOT contain these
  output_not_contains:
    - "error"
    - "failed"

  # At least one of these must appear (flexible matching)
  output_contains_any:
    - "recursion"
    - "iteration"
    - "loop"

  # Task outcomes
  outcomes:
    - type: task_completed
    - type: tool_called
      tool_name: code_analyzer

  # Behavioral constraints
  behavior:
    max_tool_calls: 5
    max_tokens: 4096

output_contains vs output_contains_any

output_contains — ALL listed strings must appear (AND logic). Use for required content.
output_contains_any — At least ONE listed string must appear (OR logic). Use when the agent may express concepts in different ways.

All checks are case-insensitive.

Fixture Isolation

Fixtures are test files (code, documents, data) that tasks reference.

Important: Each task gets a fresh temp workspace with fixtures copied in. Original fixtures are never modified.

Using Fixtures

Create a fixtures/ directory:

evals/code-explainer/
├── eval.yaml
├── tasks/
│   └── basic-usage.yaml
└── fixtures/
    ├── sample.py
    ├── complex.py
    └── README.md

Reference in tasks:

inputs:
  prompt: "Read sample.py and analyze it."
  files:
    - path: sample.py

Instruction Files

Use instruction_files for repository or task-specific *.instructions.md guidance:

config:
  instruction_files:
    - .github/instructions/project.instructions.md

instruction_files:
  - .github/instructions/review.instructions.md
inputs:
  prompt: "Review this change"
  files:
    - path: sample.py

Instruction files are resolved from the active fixtures/context directory, copied into each fresh temp workspace, and appended to the agent system message with path labels. Eval-level files apply to every task; task-level files are added for that task. Paths must be relative and cannot use directory traversal.

Directory Structure

# Project mode
evals/
└── code-explainer/
    ├── eval.yaml
    ├── tasks/
    │   ├── basic-usage.yaml
    │   ├── edge-case.yaml
    │   └── should-not-trigger.yaml
    └── fixtures/
        ├── sample.py
        ├── complex.py
        └── nested/
            └── module.py

Specify context directory when running:

waza run eval.yaml --context-dir evals/code-explainer/fixtures

Or use relative paths in eval.yaml if fixtures are adjacent.

Multi-Model Comparison

Run the same eval against multiple models:

# Run with gpt-4o
waza run eval.yaml --model gpt-4o -o gpt4.json

# Run with Claude
waza run eval.yaml --model claude-sonnet-4.6 -o sonnet.json

# Compare results
waza compare gpt4.json sonnet.json

Override the default model in eval.yaml:

waza run eval.yaml --model gpt-4o  # Overrides config.model

Filtering and Parallel Execution

Filter by Task Name

waza run eval.yaml --task "basic*" --task "edge*"

Filter by Tags

waza run eval.yaml --tags "happy-path"

Parallel Execution

# Run tasks concurrently with auto-sized workers
waza run eval.yaml --parallel

Saving Results

Save eval results for later analysis or comparison:

waza run eval.yaml -o results.json

Output format:

{
  "name": "code-explainer-eval",
  "model": "claude-sonnet-4.6",
  "pass_rate": 0.8,
  "tasks": [
    {
      "id": "basic-001",
      "name": "Basic Usage",
      "passed": true,
      "graders": [
        {
          "name": "checks_logic",
          "passed": true,
          "score": 1.0
        }
      ]
    }
  ]
}

Caching

For iterative testing, cache results:

waza run eval.yaml --cache --cache-dir .waza-cache

Only tasks with changed inputs/config re-run.

Common Patterns

Simple Validation

graders:
  - type: text
    name: format_check
    config:
      regex_match:
        - "^[A-Z].*\\.$" # Sentence starting with capital, ending with period

tasks:
  - "tasks/format/*.yaml"

Multi-Criteria Scoring

graders:
  - type: code
    name: completeness
    config:
      assertions:
        - "len(output) > 500"
        - "'function' in output"
        - "'parameter' in output"

tasks:
  - "tasks/completeness/*.yaml"

Behavioral Constraints

Behavioral constraints are defined in individual task YAML files:

id: efficient-001
name: Efficiency test
inputs:
  prompt: "Refactor this code"
expected:
  behavior:
    max_tool_calls: 3 # Efficient
    max_tokens: 1000 # Concise
    max_response_time_ms: 30000 # Must complete within 30 seconds
    required_tools: # Must use these tools
      - grep
      - edit
    forbidden_tools: # Must NOT use these tools
      - rm

Field	Type	Description
`max_tool_calls`	int	Maximum number of tool invocations allowed
`max_iterations`	int	Maximum number of conversation rounds (turns)
`max_tokens`	int	Maximum tokens in the response
`max_response_time_ms`	int	Maximum wall-clock execution time in milliseconds
`required_tools`	string[]	Tools the agent must use during the task
`forbidden_tools`	string[]	Tools the agent must NOT use during the task

Each constraint that is set (non-zero / non-empty) contributes equally to the behavior efficiency score. If all constraints pass, the score is 1.0; each failure reduces it proportionally.

Hooks

Lifecycle hooks run shell commands at specific points during an evaluation. Use them for setup, teardown, or validation.

hooks:
  before_run:
    - command: "npm install"
      working_directory: "./fixtures"
      error_on_fail: true
  after_run:
    - command: "bash cleanup.sh"
  before_task:
    - command: "echo Starting task"
  after_task:
    - command: "bash collect-metrics.sh"

Hook	When it runs
`before_run`	Once, before the entire evaluation starts
`after_run`	Once, after all tasks complete
`before_task`	Before each individual task
`after_task`	After each individual task

Each hook entry:

Field	Type	Default	Description
`command`	string	(required)	Shell command to execute
`working_directory`	string	`.`	Working directory for the command
`exit_codes`	list[int]	`[0]`	Acceptable exit codes
`error_on_fail`	bool	false	Abort the run if this hook fails

Template Variables

Use the inputs field to define global template variables that are substituted into task prompts:

inputs:
  language: python
  framework: fastapi

tasks:
  - "tasks/scaffold/*.yaml"

Fixture files are copied into the task workspace, not inlined in the prompt:

inputs:
  prompt: "Read sample.py and explain this code."
  files:
    - path: sample.py

External Task Lists

Use tasks_from to generate tasks from a CSV dataset:

name: shared-eval
tasks_from: shared-tasks.csv
range: [2, 2]

config:
  trials_per_task: 3
  model: claude-sonnet-4.6

This is useful when multiple eval specs share the same task set but differ in config or graders.

The first CSV row supplies headers; subsequent rows are 1-based data rows. Fallback IDs/names retain the original row number, so this range selects row-2 unless an explicit id or name supplies the identity. run, preflight, grade, and run analysis share selection and template semantics. Regrade the resulting task with waza grade eval.yaml --results results.json --task row-2, or omit --task to grade all selected tasks. CSV paths resolve from the eval directory unless absolute. They must be lexically inside that directory and remain inside after symlinks resolve. Contained links are allowed. An aliased eval base accepts relative contained sources, but an alternate physical absolute spelling outside the lexical alias root is rejected. The final CSV open and parse use a retained eval-directory root, so a late replacement symlink cannot redirect reads outside it. This does not freeze concurrent file contents or provide universal physical equivalence.

Best Practices

Clear task descriptions — Future reviewers should understand what’s being tested
Realistic validators — Don’t over-specify. A few key checks beat 20 strict rules
Fixture diversity — Include basic, edge case, and negative test fixtures
Tag your tasks — Makes filtering and analysis easier
Use timeout appropriately — Too short = false failures, too long = slow tests
Reuse graders — Define once, apply across multiple tasks
Version your evals — Track improvements with version numbers

Next Steps

Validators & Graders — Reference for all grader types
Web Dashboard — Explore results interactively
CLI Reference — All commands and flags