Writing Eval Specs

A complete reference for writing eval.yaml specifications and task definitions.

Available in v0.39.0: scenario authoring and preflight are included in the published standalone release. Install it or run waza update; see availability and platform limits. Before the real-agent examples below, complete authentication, model access, permissions and usage prerequisites. For credential-free harness checks start with Quick Start.

Offline planning and descriptive requirements

Run waza preflight eval.yaml --format json before starting an evaluation. It inspects local task discovery/IDs, paths, configuration, locks/cache, mocks, capabilities and requirement references without starting any agent or service. Unresolved credentials and external state are not reported verified.

Task requirements optionally describe outcome, boundary, recovery and quality intent with stable task-local IDs. Each check names an explicit existing grader using scope: eval|task|checkpoint and grader; checkpoint scope alone requires after_turn. Empty checks are uncovered/unresolved. Metadata does not create assertions, enforce runtime policy, require one implementation path, or change existing v1 grading/exits. Reference resolution is not satisfaction. Use Offline Eval Preflight for complete valid examples and the static-versus-runtime evidence distinction.

Frozen independent prompt graders

Final eval graders and task graders may set config.evidence_context: {} to request frozen task/conversation/tool/output evidence in a fresh judge session. Add files: [report.json] and evaluator_only: [references] inside that object for reviewed explicit file selection. Use executor: copilot-sdk; do not use the option in checkpoints, with mode: pairwise, or with continue_session: true. Omission retains existing behavior; present null is not omission.

The evidence-backed judging reference contains the canonical runnable YAML, limits, explicit authentication and saved regrade commands. Required missing/redacted/incomplete evidence is an operational error, not a partial judgment. The persisted bundle is independent of optional snapshots; each final grader receives only its own selected file subset.

Agent-ready workflow

This guide is designed to be given directly to a coding agent. The agent should use the workflow below when creating or updating an eval suite, while developers can use the same checklist during review.

Required context

Before editing, inspect:

1. The workflow’s expected outcomes and boundaries; the target SKILL.md or .agent.md when evaluating one.
2. The existing eval.yaml, task files, and fixtures, if present.
3. schemas/eval.schema.json and schemas/task.schema.json.
4. The grader guide and existing nearby evals for repository conventions.

Treat the schemas as the source of truth for allowed fields. Do not invent YAML properties from examples or model memory.

Create an eval suite

For a workflow without a target:

waza new eval inventory --scenario --template repository
waza run evals/inventory/eval.yaml

Templates are repository (sanitized fixtures), cli (installed Git), and mcp (harness-only mocked dependency). All use a real copilot-sdk agent; MCP mocks do not establish production MCP reliability. config.executor: mock checks harness plumbing only, not agent quality. See the scenario examples.

Scenario evals require a nonempty scenario string and exact schemaVersion: "2.0". Older executables reject that major before executing; 1.x, omitted versions, 2.1 and future majors are not scenario contracts. Presence of the scenario key is explicit in YAML and JSON. Presence is checked in native YAML/JSON parsing: null, numeric/boolean values, objects, arrays and blank strings are invalid. This intentionally stops earlier versionless/1.x null acceptance and scalar coercion. Remove the field for a legacy suite, or provide a nonblank string and exact 2.0 for a scenario. Tasks and results retain 1.x; the dashboard still displays the suite’s eval_name and generic results have an empty skill. Optional skill selects existing skill/custom-agent context with unchanged precedence. Without a target or explicit eval/task skill directories, ambient skills are disabled. An explicit empty task skill_directories: [] disables discovery for that task; --no-skills wins over every opt-in. Disabling the last eval-level directory preserves target-free isolation. Whitespace-only optional skill values do not opt into workspace-default or ambient discovery. Scenario required_skills checks search enabled directories only; a disabled path, ["*"] or none cannot satisfy a required skill. Legacy discovery and validation retain their existing behavior. Scenario task files retain tasks/ when selected by files.taskGlob, including wildcard-directory globs; otherwise they use a matching literal directory. Unsupported or nonmatching glob/suffix combinations fail before writing. The glob must select only generated tasks, not evals, fixtures or metadata. fixtures/*.yaml remains supported; fixtures/* is rejected because it also selects .gitkeep and repository inventory. Narrow the glob before regenerating. Scenario scaffolding requires physical parent directories for the selected output path; symlinked roots or generated task/fixture parents are rejected. Use a physical --output path. Relative paths start at the retained working directory; absolute paths are admitted from the filesystem root, with only verified literal macOS /var and /tmp system aliases. Writes use retained directory handles rather than following replaced parent paths. Failed creation or writing rolls back this attempt’s owned files, including a partial output. Created-file handles stay open until success or rollback so unlinked ownership cannot be confused with a recycled inode. Observed replacements and unknown content are preserved, with explicit cleanup and handle-close errors. Empty directories can remain, and an admitted directory can subsequently move.

Scenario 2.0 bypasses result caching with an explicit notice, even with --cache, until external workflow dependencies can be fully fingerprinted. Legacy 1.x caching and billing attribution are unchanged. Repository scaffolds include fixtures/.gitattributes preserving inventory LF bytes through Git checkout. If a task glob selects that file, a safe root .gitattributes is used instead. When neither placement avoids an eval/task collision or extra task, generation fails before writing; narrow the task-directory glob or choose a different eval filename. For example, use tasks/*.gitattributes instead of fixtures/*.gitattributes when the eval itself is named .gitattributes. A normal eval filename with **/*.gitattributes keeps the existing runtime glob semantics and uses the root fallback. Older suites must be regenerated to adopt this policy; Git attributes do not rewrite existing working copies. Task globs and inputs.context.fixture resolve relative to the eval directory. Fixtures default to its fixtures/ directory; inputs.files and instruction_files resolve from the fixtures/context root. The repository template explicitly declares inputs.context.fixture: fixtures/inventory.txt. File grader outputs are task-workspace-relative. The repository template uses three file graders: report outcome, absence of private.txt, and byte-exact final preservation of inventory.txt. Altering, deleting or changing its line endings fails independently of the other checks. This checks final bytes, not transient writes or edit history. CLI/MCP templates retain two graders. The tracked repository fixture alone disables Git text conversion so Windows checkout preserves the expected LF bytes.

For a SKILL.md target:

# Scaffold from an existing skill.
waza new eval <skill-name>

# Generate candidate coverage without writing files.
waza suggest <path-to-SKILL.md> --dry-run

# Add generated tasks and fixtures to the scaffolded suite.
waza suggest <path-to-SKILL.md> --apply --output-dir evals/<skill-name>

The default waza new eval flow, waza suggest, and waza spec verify accept SKILL.md targets only. For a .agent.md target, create eval.yaml, tasks, and fixtures directly by following the custom agent guide and the schemas linked above.

After scaffolding, the agent should:

1. Map each USE FOR, DO NOT USE FOR, parameter, and important behavioral requirement to at least one task.
2. Include positive, negative, boundary, and realistic integration cases.
3. Use fixtures instead of relying on the developer’s working tree or live external state.
4. Prefer deterministic graders (text, file, behavior, action_sequence, or skill_invocation) when the expected behavior can be expressed mechanically. Use an LLM judge only for genuinely qualitative requirements.
5. Use copilot-sdk for real behavioral evaluation. The mock executor only verifies the harness and fixture plumbing; it does not evaluate agent quality.

Guide implementation with evals

When an eval accompanies an implementation change, use a test-first loop:

1. Add or update the task that expresses the required behavior.
2. Run only that task and confirm that it fails for the expected reason.
3. Change the skill, custom agent, or supporting implementation.
4. Rerun the task until it passes without weakening the grader.
5. Run related tasks, then the complete suite, to catch regressions.

Keep the task focused on observable behavior. Do not encode implementation details unless the requirement explicitly depends on a tool call, file change, or action sequence.

Update an existing eval suite

When the target skill or agent changes:

1. Read the diff and identify changed triggers, anti-triggers, parameters, tools, outputs, and failure behavior.
2. For a SKILL.md target, run waza spec verify to find uncovered requirements. For a .agent.md target, compare the tasks directly with the agent’s description, tools, handoffs, and behavioral instructions.
3. Add or update the smallest set of tasks needed to cover the changed behavior. Preserve stable task IDs unless the scenario itself is being replaced.
4. Add a regression task for every bug fix or previously missed behavior.
5. Keep unrelated tasks, fixtures, thresholds, and grader choices unchanged.
6. If external tools are involved, use mcp_mocks with schemaVersion: "1.1" or newer for deterministic CI coverage. Keep live-service testing in a separate integration suite.

File invariants

• Preserve the scaffolded or existing schemaVersion. Do not copy a newer version unless the eval uses fields that require it. mcp_mocks requires version 1.1 or newer; adversarial requires version 1.2 or newer.
• Every task requires non-empty id, name, and inputs.
• inputs must contain exactly one of prompt or prompt_file.
• Task IDs must be unique and stable; use lowercase kebab-case.
• Relative paths are resolved from the eval or task location described by the relevant schema field. Do not depend on the shell’s current directory.
• Fixtures must contain all state required by the scenario and must not contain secrets.
• Unknown MCP tools and unmatched mock calls should fail; do not add permissive fallback responses that hide missing fixtures.
• Before committing a behavioral suite, confirm config.executor is copilot-sdk. Never commit mock as a substitute for real evaluation.
• Do not weaken thresholds merely to make a failing eval pass. Fix the skill, task, fixture, or grader that does not represent the intended behavior.

Validation ladder

Run the smallest relevant check first, then expand:

# SKILL.md only: inspect uncovered requirements without failing.
waza spec verify <skill-path> <eval-path>

# SKILL.md only: enforce complete requirement coverage.
waza spec verify <skill-path> <eval-path> --fail

# Exercise one changed task with the executor configured in eval.yaml.
waza run <eval-path> --task "<task-id>" --no-cache -v

# Run the complete suite and save reviewable output.
waza run <eval-path> --no-cache \
  --output results.json \
  --reporter junit:test-results.xml

Set config.executor: copilot-sdk before behavioral evaluation. Use config.executor: mock only to check task loading, fixture copying, and grader plumbing without a real agent session, then restore copilot-sdk before committing.

For non-deterministic behavior, increase trials_per_task only after the single-trial task is correct. Review failures individually; do not treat a passing aggregate score as proof that every scenario is valid.

Delegation prompt

Developers can give an agent this guide together with a focused request:

Create or update the Waza evals for <skill-or-agent-path>.

Follow the Agent-ready workflow in:
https://microsoft.github.io/waza/guides/eval-yaml/

Requirements:
- Treat schemas/eval.schema.json and schemas/task.schema.json as authoritative.
- Preserve unrelated eval behavior and stable task IDs.
- Cover positive triggers, negative triggers, parameters, edge cases, and the
  implementation behavior changed in <issue-or-diff>.
- Use fixtures and mcp_mocks instead of live external state where possible.
- For implementation work, add a focused failing task before changing behavior.
- For SKILL.md targets, run waza spec verify.
- Run a targeted task and the full relevant suite with config.executor set to
  copilot-sdk. Report the exact commands and results.

eval.yaml Structure

The evaluation spec defines the benchmark configuration, graders, and task files:

name: code-explainer-eval
description: Evaluation suite for code-explainer skill
skill: code-explainer
schemaVersion: "1.0"
version: "1.0"

config:
  trials_per_task: 1
  timeout_seconds: 300
  parallel: false
  executor: mock
  model: claude-sonnet-4.6

metrics:
  - name: accuracy
    weight: 1.0
    threshold: 0.8

graders:
  - type: text
    name: explains_concepts
    config:
      regex_match:
        - "(?i)(function|logic|parameter)"
  - type: code
    name: has_output
    config:
      assertions:
        - "len(output) > 100"

tasks:
  - "tasks/*.yaml"

Top-Level Fields

Field	Type	Required	Description
name	string	✓	Eval suite name
description	string	✗	What the eval tests
skill	string	Legacy schema	Associated skill/custom-agent target; optional context for scenario 2.0
scenario	string	Scenario 2.0	Nonblank workflow identity; legacy authoring omits this field
schemaVersion	string	Scenario 2.0	Legacy defaults to 1.4; scenarios explicitly select 2.0
version	string	✗	Version number (e.g., “1.0”)
inputs	object	✗	Key-value map of global template variables (see Template Variables)
tasks_from	string	✗	Relative or absolute CSV dataset path; takes precedence over tasks when nonempty
hooks	object	✗	Lifecycle hooks that run shell commands at specific points (see Hooks)
mcp_mocks	array	✗	MCP server mocks for deterministic tool-call evals; requires schemaVersion: "1.1"; finite sequences require scenario 2.0
command_mocks	array	✗	Executable shims for CLI-dependent evals; requires schemaVersion: "1.3" or newer and executor: copilot-sdk; finite sequences require scenario 2.0
adversarial	object	✗	Built-in fault-injection packs consumed by waza adversarial --spec; requires schemaVersion: "1.2"
baseline	bool	✗	Mark this spec as a baseline for A/B comparison

Legacy JSON-schema authoring requires skill when scenario is absent. Scenario 2.0 requires a nonblank scenario and may omit skill. The native loader/model validation used by waza run is a different surface from full JSON-schema validation: legacy default/1.0-1.4 data historically loads with an absent or empty target and retains existing discovery behavior. Preserving that compatibility does not make targetless legacy data the recommended workflow; author a named legacy target or a scenario 2.0 suite. The schema’s skill constraint is length-only: whitespace passes that check but is not a meaningful target. JSON-RPC eval.validate reports schema errors; separate preflight checks are not universal runtime target enforcement.

schemaVersion uses MAJOR.MINOR format. Omit it for legacy files that should default to 1.4; scenario evals must select 2.0. See Schema Changes for the compatibility policy.

Targeting Custom Agents

Waza supports evaluating VS Code custom agents (.agent.md files) alongside traditional SKILL.md-based skills. When you target an agent with a tools: field in its frontmatter, waza automatically injects a tool_constraint grader to validate that only the declared tools are called.

Specify an agent:

name: security-agent-eval
description: Evaluate the security-reviewer custom agent
skill: security-reviewer  # Points to security-reviewer.agent.md
schemaVersion: "1.0"
version: "1.0"

config:
  model: claude-sonnet-4.6

Key differences:

• Use skill: <name> to target either a skill or a custom agent
• Waza discovers .agent.md files the same way as SKILL.md — in the current directory or agents/ subdirectories
• If both SKILL.md and .agent.md exist in the same directory, SKILL.md takes priority
• Custom agents can declare a tools: field in frontmatter, which auto-injects a tool_constraint grader

Learn more: See the Evaluating Custom Agents guide for detailed examples and the auto-injected tool constraint behavior.

Config Section

The config block controls execution behavior:

config:
  trials_per_task: 1 # Run each task this many times
  timeout_seconds: 300 # Task timeout in seconds
  parallel: false # Run tasks sequentially (true = concurrent)
  workers: 0 # Auto-size parallel workers if parallel: true
  model: claude-sonnet-4.6 # Default model (override with --model)
  reasoning_effort: high # Optional: pin task and responder reasoning effort
  judge_model: gpt-4o # Model for LLM-as-judge graders (optional)
  judge_reasoning_effort: low # Optional: default effort for prompt graders
  executor: copilot-sdk # mock (local) or copilot-sdk (real API)
  inject_skill_body: true # Inject target SKILL.md/.agent.md body into the prompt
  instruction_files:
    - .github/instructions/project.instructions.md

Field	Type	Default	Description
trials_per_task	int	1	Number of times each task runs (for statistical analysis)
timeout_seconds	int	300	Task timeout in seconds
first_event_timeout_seconds	int	0 (off)	Abort a run that produces no first event within N seconds (session-start hang); 0 disables
parallel	bool	false	Run tasks concurrently
workers	int	0	Number of parallel workers; 0 auto-sizes
model	string	required	Default model for tasks (override with --model flag)
reasoning_effort	string	model default	Copilot SDK task/responder effort: low, medium, high, xhigh, or max
judge_model	string	(same as model)	Model for prompt-type graders (LLM-as-judge)
judge_reasoning_effort	string	judge default	Copilot SDK default effort for prompt graders, including continue_session graders; the grader can override it
executor	string	copilot-sdk	Executor: mock (local, echoes task metadata and file content) or copilot-sdk (real API)
max_attempts	int	0	Maximum retry attempts per task on failure (0 = no retries)
group_by	string	—	Group results by model or each task’s complete ordered tags set; commas/backslashes are escaped and an empty tag is shown as \0
fail_fast	bool	false	Stop the entire run on first task failure
skill_directories	list[str]	[]	Additional directories to search for skills
instruction_files	list[str]	[]	Instruction files to apply to every task
inject_skill_body	bool	true	Inject the target SKILL.md or .agent.md body, prefixed with its absolute base directory, into the system prompt
trigger_skill_routing	bool	false	With inject_skill_body: false, add an eval-only routing instruction for trigger-precision runs
disabled_skills	list[str]	[]	Skills to disable. Use ["*"] to disable all skills
required_skills	list[str]	[]	Skills that must be available before running
mcp_servers	object	—	MCP server configurations for the evaluation. When tools is omitted or null for a server, Waza exposes all server tools by sending ["*"] to the Copilot SDK. Set tools to an explicit list, including [], to restrict or disable exposed tools.

With an explicit reasoning effort, use a concrete model from waza models. Waza checks the hosted runtime’s supported-effort metadata before session creation and resumption, rejecting unsupported values rather than silently downgrading them. Custom providers receive the effort directly because they are not covered by the hosted model catalog. Prompt graders in eval, task, and checkpoint definitions may override judge_reasoning_effort; agent effort stays eval-level.

Trigger-precision evals

Set inject_skill_body: false when the eval is measuring whether the agent invokes a skill rather than whether it can complete the work after already seeing the skill body:

name: xyz-trigger
description: Trigger-precision tasks for the xyz skill
skill: xyz
schemaVersion: "1.0"
version: "1.0"

config:
  trials_per_task: 1
  timeout_seconds: 300
  parallel: false
  executor: copilot-sdk
  model: claude-sonnet-4.6
  inject_skill_body: false
  trigger_skill_routing: true

metrics:
  - name: trigger_precision
    weight: 1.0
    threshold: 0.8

tasks:
  - "tasks/trigger/*.yaml"

trigger_skill_routing is optional and scoped to trigger-precision runs. It helps distinguish skill routing from general model fluency by telling the agent to invoke the target skill when the task is in scope, while still withholding the skill body. It has no effect unless inject_skill_body is false.

With this setting, Waza still discovers skills and passes their directories to the Copilot SDK, but it does not add the full target <skill_context> body (normally prefixed with the skill’s absolute base directory) or a synthetic <available_skills> summary. This lets behavior graders with required_tools or forbidden_tools and skill_invocation graders observe whether the skill tool was used. If you also set disabled_skills: ["*"], all skill loading is disabled and this setting has no effect.

Common Timeouts:

• 60 — Quick tasks (single-file review, validation)
• 300 — Standard tasks (code explanation, analysis)
• 600 — Complex tasks (multi-file refactoring, design)

MCP Mock Servers

Use top-level mcp_mocks to replace live MCP dependencies with deterministic local stdio servers. This keeps Copilot SDK evals hermetic in CI: no network listener, no port allocation, and no external service credentials. Waza automatically exposes every tool declared by a mock to the Copilot CLI, so no separate tools allowlist is needed. Because this is an additive eval schema field, set schemaVersion: "1.1" or newer.

name: issue-triage-eval
skill: issue-triage
schemaVersion: "1.1"
version: "1.0"

config:
  executor: copilot-sdk
  model: claude-sonnet-4.6

mcp_mocks:
  - name: github
    tools:
      list_issues:
        description: Return matching issues for a repository
        input_schema:
          type: object
          properties:
            owner: { type: string }
            repo: { type: string }
          required: [owner, repo]
        responses:
          - match:
              owner: microsoft
              repo: waza
            return:
              issues:
                - number: 363
                  title: MCP server mocks for hermetic eval
          - match_regex:
              repo: "^waza-.*"
            return:
              issues: []
          - match_schema:
              type: object
              required: [owner, repo]
            error: "No fixture for this repository"

tasks:
  - tasks/*.yaml

Response matching is evaluated in order. match requires exact full-argument equality, match_schema validates the call arguments against an inline JSON Schema, and match_regex applies regular expressions to individual argument fields. Unknown tools or calls that do not match any response fail loudly with an MCP tool error that points to the missing mock fixture.

Unmatched MCP errors intentionally omit argument values, rather than echoing private or nested payloads. Scenario evals with exact schemaVersion: "2.0" can use finite sequence steps of return or error, with optional delays from 0 to 300,000 ms and at most 100 steps. A matching call consumes one step; exhaustion and unmatched calls fail closed. An error containing “timeout” is a dependency-reported error, not transport cancellation. A malformed application value is still encoded in a valid MCP result, not a malformed MCP/JSON-RPC envelope. See the Dependency Recovery guide for complete scenarios and isolation limits.

You can keep larger fixtures in JSON files instead of inline YAML:

mcp_mocks:
  - name: github
    fixtures: fixtures/mcp/github

Each .json fixture can either contain a single tool definition (tool name from the filename) or a { "tools": { ... } } object with multiple tools.

Task-level MCP mocks

Added in v0.40.0: Published v0.39.1 does not apply task-level mcp_mocks.

A task can declare mcp_mocks when one case needs a different response, for example approving a question that other tasks must decline. Each task entry replaces the eval mock with the same name for that task only, and a new name adds a mock. Other eval mocks still apply, so a task restates only the servers it changes.

tasks/publish.yaml

id: publish
name: Publish when the user confirms
inputs:
  prompt: Publish the reviewed version.
mcp_mocks:
  - name: host
    tools:
      ask_user:
        responses:
          - match_schema:
              type: object
              required: [choices]
              properties:
                choices: { type: array, contains: { const: Publish this version } }
            return: { answer: Publish this version, wasFreeform: false }
          - match_schema: {}
            return: { answer: Not now, wasFreeform: false }

Task mocks follow the enclosing eval’s schemaVersion (1.1 or newer, and scenario 2.0 for finite sequences) and require executor: copilot-sdk. Relative fixtures paths resolve from the eval directory, like task command_mocks fixtures. An eval mock that every selected task replaces has its fixtures skipped only when no trigger phase needs it; its inline tools are still validated. An empty task list is rejected; omit the field to use the eval mocks.

Fixture entries must be regular JSON files; symlinks to regular files remain supported. On Unix, admission opens fixtures nonblocking and validates the opened file, rejecting FIFOs without waiting for a writer so after_run can run. This does not bound unresponsive filesystem operations or promise race-free/nonblocking opening on Windows.

The CLI discovers trigger_tests.yaml after before_run, before starting the engine, and reuses that parsed spec during execution. Trigger tests use eval-level MCP mocks and live servers rather than task overrides. Preflight inspects those dependencies using local reads only, including when a trigger file cannot be parsed; independent mock diagnostics are retained. The mock executor synthesizes trigger results without MCP. Task filtering, --skip-graders, --no-skills, and cached task results do not skip trigger tests.

Admission validates selected fixture-backed tool input_schema declarations offline, including fixtures needed only by trigger tests. Omitted/null schemas retain the default; whole-tool overrides and genuinely unused fixture metadata are not admitted.

SDK startup cancellation releases the waiting run and cleans up partial startup without tying a successfully started client’s lifetime to its original caller. Reinitializing a healthy client remains idempotent even with a canceled caller context. A canceled shared startup is replaceable after successful cleanup; global shared-client shutdown still rejects new clients. A custom client factory must honor startup-context cancellation or expose ForceStop. Each partial-startup cleanup wait has a two-second deadline and reports an error if cleanup has not settled; it cannot guarantee goroutine termination for a noncooperating custom client. Pending or failed cleanup remains tracked.

Canceling one concurrent initialization waiter does not cache an engine-wide startup failure or prevent another waiter from authenticating successfully.

After completed startup, the lifetime context stays alive through graceful SDK cleanup. Authentication failure cleans up an owned client immediately and retains cleanup errors; engine shutdown still performs final cleanup. Shared clients are stopped only by their top-level owner. Partial-startup cleanup remains once-only.

Command Mocks

Use command_mocks when a skill shells out to a CLI such as Azure CLI (az), GitHub CLI (gh), or kubectl. Waza materializes temporary executable shims per task and prepends their directory to the Copilot runtime’s existing PATH, preserving the host path. Command mocks require executor: copilot-sdk and schemaVersion: "1.3" or newer.

schemaVersion: "1.3"
config:
  executor: copilot-sdk
  model: claude-sonnet-4.6

command_mocks:
  - name: az
    expect_calls: 3
    responses:
      - args: ["account", "show", "--output", "json"]
        stdout:
          subscriptionId: "00000000-0000-0000-0000-000000000000"
          name: Test Subscription
      - args_regex: ["group", "show", "--name", ".+"]
        fixture: fixtures/commands/az-group-show.json
      - args: ["deployment", "group", "create"]
        stderr: "deployment failed"
        exit_code: 1

tasks:
  - "tasks/*.yaml"

args matches the entire argument vector, excluding the executable. args_regex applies a full-string regular expression to each argument at the same position. Responses are evaluated in order and the first matching response wins. stdout strings are emitted verbatim; structured values are JSON-encoded. fixture reads raw response bytes from a path relative to eval.yaml. Optional environment values must match exactly, and workdir matches a workspace-relative path. expect_calls requires an exact number of invocations for that executable; a mismatch is reported as a task error.

Task-level command_mocks replace the eval-level list. Set command_mocks: [] on a task to disable inherited mocks. A command that is configured but called with unmatched arguments fails closed with a diagnostic asking for a matching response fixture. Invocation records contain sanitized arguments, exit codes, and response indexes in runs[].command_invocations; graders can also read them from their context. Verbose runs print those sanitized details. Only declared executable names are intercepted, so command_mocks is not a command sandbox. Do not put real credentials in mock output.

Scenario evals with exact schemaVersion: "2.0" may use a response sequence of at most 100 output steps; each may delay up to 300,000 ms. A matching call consumes one step and exhaustion fails closed. Earlier failed attempts remain in runs[].command_invocations even when a later attempt succeeds. Task-level empty-list replacement is unchanged.

Choosing a test boundary

Mechanism	Use it for
command_mocks	Commands the agent invokes through its shell, with deterministic process output and exit status.
mcp_mocks	Structured MCP tool calls and server responses.
native_user_input + task user_input_responses	Questions the agent asks the host through the Copilot SDK native ask_user tool.
Hooks	Setup, teardown, or validation around tasks/evals; hooks do not intercept agent commands.
Live integration tests	Verifying real CLI/service behavior, authentication, or infrastructure. Keep these separate from hermetic evals.

Adversarial Packs

Use top-level adversarial to pin the built-in fault-injection packs and unsafe-outcome policy for waza adversarial --spec. This is additive in schemaVersion: "1.2" and is ignored by normal waza run executions.

schemaVersion: "1.2"
adversarial:
  packs:
    - prompt-injection
    - scope-bypass
  on_unsafe_outcome: fail  # or "warn"

packs must name one or more built-in packs. In v0.38.0 those are prompt-injection and scope-bypass. on_unsafe_outcome: fail exits 2 when an unsafe outcome is observed; warn records the unsafe result but exits 0. See the Adversarial Harness guide for pack behavior and CI examples.

Graders Section

Graders validate task outputs. Define once, reuse across tasks:

graders:
  - ref: github.com/waza-evals/fact#factuality@v1.0.0
    name: factuality_strict
    weight: 2.0
    config:
      threshold: 0.9

  - type: text
    name: checks_logic
    weight: 2.0
    config:
      regex_match:
        - "(?i)(function|variable|parameter)"

  - type: code
    name: has_minimum_output
    config:
      assertions:
        - "len(output) > 100"
        - "'success' in output.lower()"

  - type: text
    name: mentions_key_concepts
    config:
      contains:
        - "algorithm"
        - "optimization"

Each grader accepts an optional weight (default 1.0) that controls its influence on the composite score. See Validators & Graders for details.

Remote grader presets can be referenced with ref instead of type. Refs use <host>/<owner>/<repo>[/path][#export]@<version> and resolve through waza get, which writes waza.lock with the pinned commit SHA and content digest. Local name, weight, top-level grader fields, and config values override the remote preset; nested config maps are deep-merged, while lists are replaced.

waza get eval.yaml
waza run eval.yaml

waza run requires the lockfile and cached module contents to be present for remote refs. It fails closed on missing locks, missing cache entries, or digest mismatches.

All graders return:

• score: 0.0 to 1.0
• passed: boolean
• message: human-readable result

See the Validators & Graders guide for all 13 native kinds and examples.

Mapping OpenAI Evals modelgraded YAML

OpenAI Evals modelgraded specs usually collapse into Waza’s prompt grader. The judge prompt carries the label semantics, while Waza handles execution and scoring.

OpenAI Evals field	Waza equivalent	Notes
prompt	graders[].config.prompt	Put the judging instructions directly in the prompt
choice_strings	prompt text	List the labels in the judge prompt; Waza’s prompt grader is binary, so the label choice becomes pass/fail guidance
choice_scores	prompt text	Encode the scoring rule in the judge prompt; use pairwise mode when the comparison is relative
input_outputs	tasks: entries	Turn each example into one Waza task with its own inputs.prompt and expected checks
eval_type: cot_classify	type: prompt	Use mode: independent for one-shot classification
battle.yaml	type: prompt + mode: pairwise	Closest grader match for head-to-head comparison; waza compare is still the better run-level report

Translation examples

fact.yaml

OpenAI’s registry uses this pattern for fixed-choice factual classification. In Waza, keep the evaluation as a single prompt grader and turn each input/output row into a task:

graders:
  - type: prompt
    name: fact_check
    config:
      prompt: |
        You are checking a multiple-choice answer.
        Valid choices: A, B, C, D, E.
        Call set_waza_grade_pass only if the model's answer matches the correct choice.
        Otherwise call set_waza_grade_fail with a short reason.
      continue_session: false

tasks:
  - id: fact-001
    name: fact-001
    inputs:
      prompt: "Which answer is correct for the fact pattern?"
    expected:
      output_contains:
        - "B"

closedqa.yaml

For closed-book QA, the judge prompt can encode the score mapping directly:

graders:
  - type: prompt
    name: closedqa_judge
    config:
      prompt: |
        Judge the answer against the reference.
        If the answer is fully correct, call set_waza_grade_pass.
        If it is partially correct or incorrect, call set_waza_grade_fail.
        Treat "Y" as 1.0 and "N" as 0.0 in your reasoning, but only emit pass/fail.
      model: claude-sonnet-4.5

tasks:
  - id: closedqa-001
    name: closedqa-001
    inputs:
      prompt: "Answer the question using the provided context."
    expected:
      output_contains:
        - "Y"

battle.yaml

Battle-style comparisons are the one place where the mapping is not 1:1. The nearest Waza translation is a pairwise prompt grader, but the run-level comparison report is usually better expressed with waza compare:

config:
  baseline: true

graders:
  - type: prompt
    name: battle_judge
    config:
      mode: pairwise
      prompt: |
        Compare the two answers and decide which one is better.
        Call set_waza_grade_pass if the skill run wins.
        Call set_waza_grade_fail if the baseline run wins.

tasks:
  - id: battle-001
    name: battle-001
    inputs:
      prompt: "Compare these two solutions and pick the better one."

Tasks Section

Tasks define individual test cases loaded from YAML files:

From Files

Load tasks from YAML files in a directory:

tasks:
  - "tasks/*.yaml" # All YAML files in tasks/
  - "tasks/basic/*.yaml" # Specific subdirectory
  - "tasks/advanced.yaml" # Single file

Task File Format

Individual task files (e.g., tasks/basic-usage.yaml):

id: basic-usage-001
name: Basic Usage - Python Function
description: Test that the skill explains a simple Python function correctly.

tags:
  - basic
  - happy-path

inputs:
  prompt: "Read sample.py and explain the function it contains."
  files:
    - path: sample.py

expected:
  output_contains:
    - "function"
    - "parameter"
    - "return"
  outcomes:
    - type: task_completed
  behavior:
    max_tool_calls: 5

Task Fields

Field	Type	Description
id	string	Unique task identifier
name	string	Human-readable task name
description	string	What the task tests
tags	array	Tags for filtering (e.g., ["basic", "edge-case"])
inputs	object	Test inputs (prompt, files)
expected	object	Validation rules and expected behavior
skill_directories	string[]	Skill directories for this task (overrides eval-level)
instruction_files	string[]	Instruction files for this task (adds to eval-level files)
golden	bool	Mark as a critical “golden” task — enforced by waza gate (must always pass)
requirements	array	Descriptive intent linked to explicitly scoped existing graders; does not add assertions or change pass/fail
user_input_responses	array	Ordered scripted answers (answer, optional question regex) to native Copilot SDK ask_user questions; requires eval config.native_user_input: true. See schema reference

Answer native agent questions

When the agent should ask the host questions through the Copilot SDK ask_user tool, set config.native_user_input: true in a scenario eval and script the answers in order. Unscripted or mismatched questions fail the run instead of receiving a default answer, and unused answers also fail it. Each entry has a required answer and an optional question regex; an answer equal to an offered choice selects it:

user_input_responses:
  - question: "(?is).*environment.*"
    answer: staging
  - answer: Fixes the login timeout.
graders:
  - name: answered
    type: code
    config:
      language: javascript
      assertions:
        - "user_input_requests.length === 2 && user_input_requests.every(r => !r.error)"

Each request is recorded under runs[].user_input_requests with its session ID, offered choices, answer, was_freeform, source, and error when rejected. Omit user_input_responses or use [] when there are no scripted answers; null is invalid. Questions arriving after the agent’s active send ends are rejected, including during session teardown.

If the task also has an adaptive responder (inputs.responder), questions that the next scripted entry does not match are answered by the responder model instead of failing (source: responder); see adaptive responder answers.

Link intent to existing checks

Use requirements to identify the intent behind an existing named grader:

id: report-task
name: Report the inspected result
inputs:
  prompt: "Return a report containing the word inspected."
graders:
  - name: report-check
    type: text
    config:
      contains: ["inspected"]
requirements:
  - id: report-content
    category: outcome
    description: The report identifies the inspected result.
    checks:
      - scope: task
        grader: report-check

task selects a task grader; eval selects an eval-level grader. checkpoint selects a checkpoint grader and requires the matching after_turn. Names must resolve uniquely in their scope. A final result name shared by eval and task graders is unresolved because the stored result map does not distinguish their observations. Missing checks and requirements without checks stay unassessed.

runs[].requirement_explanations records existing grader outcomes and their uncertainty, not new requirement scores, enforcement, or causes. A passing check does not erase an overall operational error. With --snapshot, references identify the whole captured artifact and an artifact-relative result pointer; they are not file contents or proof that a grader’s workspace is reconstructible. Regrading applies the currently selected declarations and never reassesses historical checkpoints. See portable evidence and the result schema.

Inputs Section

inputs:
  prompt: "Your instruction to the agent"
  context:
    fixture: fixtures/demo # Optional: copy this fixture file/dir into the workspace
  files:
    - path: sample.py # Fixture file (relative to fixtures dir)
      content: | # Or inline content
        def hello():
            print("Hello")

inputs.files loads each path from the active fixtures directory (./fixtures relative to the eval spec file by default) and copies it into the fresh task workspace. It does not add file contents to prompt; tell the agent which workspace file to read. Use content to define a file inline instead.

inputs.context.fixture is resolved relative to the eval spec directory. When it points to a directory, Waza copies that directory’s contents into the fresh task workspace before the agent runs.

Loading prompts from a file

Use prompt_file instead of prompt to load the prompt text from an external file. The path is resolved relative to the task YAML file’s directory.

tasks/complex-review.yaml

inputs:
  prompt_file: prompts/review-instructions.md
  files:
    - path: sample.py

This is useful when prompts are long, shared across tasks, or maintained separately. You must specify either prompt or prompt_file, but not both.

Caution

prompt_file paths must be relative. Absolute paths and directory traversal (../) are rejected for security.

Follow-up Prompts

Use follow_up_prompts to send additional messages after the initial prompt. Each follow-up reuses the same session and workspace, so file changes and conversation history persist across turns.

inputs:
  prompt: "Create a Python function that reads a CSV file"
  follow_up_prompts:
    - "Add error handling for missing files"
    - "Write unit tests for the function"

This is useful for evaluating multi-turn conversations where each step builds on the previous one. Graders run only after all prompts (initial + follow-ups) have completed, so the final output reflects the full conversation.

Responder (Interactive Skills)

For skills that ask follow-up questions, configure a responder — an LLM that plays the user and answers the skill’s questions. It is mutually exclusive with follow_up_prompts.

inputs:
  prompt: "Add a new agent to my application"
  responder:
    model: gpt-4o          # optional; defaults to config.model
    instructions: |
      The agent you want is "research-agent" with system instructions
      "Search the web and summarise findings", tools web_search + url_fetch,
      and no handoffs. Answer the skill's questions consistently with this.
      If you genuinely can't infer an answer, abstain.
    max_followups: 8

After each agent turn the responder either replies (the answer is sent back, continuing the conversation), stops (the agent is done), or abstains — which fails the run with a distinct abstained outcome, signalling the brief is too vague. If max_followups is reached while the agent is still asking questions, the loop stops with outcome cap_exhausted and graders evaluate the final state. Each task carries its own responder, so the same skill can be tested against several target configurations.

Each successful decision tool ends the responder’s classification turn without another model round. This does not discard conflicting calls that were already issued: multiple decisions in one classification still fail the run.

Complete multi-turn example

Choose scripted follow-ups when the next user messages are known in advance. Choose an adaptive responder when the evaluated agent decides which question to ask next. The examples below use the same two-question interview without a skill or external tools, so you can check conversation setup independently of your product workflow.

Create eval.yaml and a tasks/ directory. This is a scenario eval, so it uses exact schema version 2.0 and does not discover ambient skills.

name: multi-turn-demo
scenario: two-question-interview
schemaVersion: "2.0"
version: "1.0"
config:
  executor: copilot-sdk
  model: gpt-4.1
  trials_per_task: 1
  timeout_seconds: 120
metrics:
  - name: task_completion
    weight: 1.0
    threshold: 1.0
graders:
  - type: text
    name: final-preferences
    config:
      regex_match: ["(?i)blue", "(?i)cat"]
tasks:
  - tasks/scripted.yaml
  - tasks/adaptive.yaml

Create tasks/scripted.yaml:

id: scripted-preferences
name: Scripted preferences
inputs:
  prompt: |
    Ask for my favorite color, then my favorite animal, one question at a time.
    After both answers, summarize the color and animal without another question.
  follow_up_prompts:
    - "My favorite color is blue."
    - "My favorite animal is a cat."

The scripted messages are sent in order after successive agent turns; they do not adapt to what the agent actually asked.

Create tasks/adaptive.yaml:

id: adaptive-preferences
name: Adaptive preferences
inputs:
  prompt: |
    Ask for my favorite color, then my favorite animal, one question at a time.
    After both answers, summarize the color and animal without another question.
  responder:
    instructions: |
      Your favorite color is blue and your favorite animal is a cat.
      Answer only the question asked. Do not invent other preferences.
      Stop when the agent has summarized both preferences and asks nothing else.
      Abstain if an answer cannot be inferred from these facts.
    max_followups: 3

The responder model defaults to config.model; set inputs.responder.model only when you intentionally want a different model. max_followups counts responder replies, not model requests or the initial actor turn, and must be at least one. Leave room beyond the expected two replies for the completion check: reaching the cap stops further classification and reports cap_exhausted, which is not proof the conversation completed successfully.

preflight validates the configuration without starting a model:

waza preflight eval.yaml

Before real execution, configure SDK authentication, verify model access, and approve any runtime/tool permissions required by your environment. Both actor and responder inference may incur usage; the follow-up limit is not a provider spending cap. These commands exercise the real configured model:

waza run eval.yaml --task "Scripted preferences" --output scripted-results.json
waza run eval.yaml --task "Adaptive preferences" --output adaptive-results.json

For an agent-free scripted harness control only, change config.executor from copilot-sdk to mock in eval.yaml, then run only the scripted task:

waza run eval.yaml --task "Scripted preferences" --output scripted-smoke.json

Expect exit 1: generic mock output does not satisfy both preference predicates. This checks wiring, not real conversation quality. The mock engine does not call responder decision tools, so it is not an adaptive responder substitute. Restore config.executor: copilot-sdk before using the real-execution commands above.

Inspect tasks[].runs[].responder in the adaptive results: stopped means the responder chose to stop, abstained means the brief could not supply an answer, and cap_exhausted means the reply limit was reached. Also check the run status and graders; a normal stop is not automatically a passing evaluation.

If execution fails before the responder records a decision, an available execution error is reported as responder execution failed: ... in the run’s error and responder reason. A successful response that contains only text still reports responder did not call a decision tool. That message also remains the fallback when no decision and no execution error detail are available. This distinction improves diagnosis; it does not repair provider failures or change grading. Execution responses snapshot collected diagnostic output safely alongside SDK event delivery; this does not wait for future events or change final usage collection.

Duplicate decision errors

An error such as responder called responder_stop after a decision was already recorded is not an abstention or a follow-up limit. A nonterminal decision can allow an unnecessary model round before the accepted reply is delivered. The lifecycle fix in #749 marks the three internal decision tools terminal; check release notes before assuming an installed binary includes it. IsTerminal is an SDK registration option, not an eval YAML field or a prompt workaround. Terminality ends only the current classification; the responder session remains available for the next actor message. Conflicting decisions already issued in the same response remain errors.

Per-Turn Checkpoints

By default graders only run once, against the final state of a multi-turn conversation. For long conversations you can run graders at specific turn boundaries using a top-level checkpoints: list:

checkpoints:
  - after_turn: 1
    graders:
      - type: text
        contains: ["analyzing", "files"]
  - after_turn: 2
    on_failure: stop # abort the run if this checkpoint fails
    graders:
      - type: tool_calls
        required: ["read_file"]

Each checkpoint accepts:

Field	Type	Description
after_turn	int	1-based turn number this checkpoint runs after (initial prompt is turn 1).
graders	array	Inline graders, same schema as the task-level graders: / eval-level graders: field.
on_failure	string	continue (default) or stop — abort remaining turns when this checkpoint fails.

Outcomes are recorded per-checkpoint on results.json under checkpoints[], alongside the final validations. waza gate still uses final-pass status. Available with schemaVersion: "1.1" and above (additive — older 1.0 files load unchanged).

When a task includes workspace files, name the file in the prompt so the agent knows to read it:

inputs:
  prompt: "Read sample.py and explain this code."
  files:
    - path: sample.py

Expected Section

expected:
  # Strings that must appear in output
  output_contains:
    - "function"
    - "parameter"

  # Output must NOT contain these
  output_not_contains:
    - "error"
    - "failed"

  # At least one of these must appear (flexible matching)
  output_contains_any:
    - "recursion"
    - "iteration"
    - "loop"

  # Task outcomes
  outcomes:
    - type: task_completed
    - type: tool_called
      tool_name: code_analyzer

  # Behavioral constraints
  behavior:
    max_tool_calls: 5
    max_tokens: 4096

output_contains vs output_contains_any

• output_contains — ALL listed strings must appear (AND logic). Use for required content.
• output_contains_any — At least ONE listed string must appear (OR logic). Use when the agent may express concepts in different ways.

All checks are case-insensitive.

Fixture Isolation

Fixtures are test files (code, documents, data) that tasks reference.

Important: Each task gets a fresh temp workspace with fixtures copied in. Original fixtures are never modified.

Using Fixtures

Create a fixtures/ directory:

evals/code-explainer/
├── eval.yaml
├── tasks/
│   └── basic-usage.yaml
└── fixtures/
    ├── sample.py
    ├── complex.py
    └── README.md

Reference in tasks:

inputs:
  prompt: "Read sample.py and analyze it."
  files:
    - path: sample.py

Instruction Files

Use instruction_files for repository or task-specific *.instructions.md guidance:

eval.yaml

config:
  instruction_files:
    - .github/instructions/project.instructions.md

tasks/review.yaml

instruction_files:
  - .github/instructions/review.instructions.md
inputs:
  prompt: "Review this change"
  files:
    - path: sample.py

Instruction files are resolved from the active fixtures/context directory, copied into each fresh temp workspace, and appended to the agent system message with path labels. Eval-level files apply to every task; task-level files are added for that task. Paths must be relative and cannot use directory traversal.

Directory Structure

# Project mode
evals/
└── code-explainer/
    ├── eval.yaml
    ├── tasks/
    │   ├── basic-usage.yaml
    │   ├── edge-case.yaml
    │   └── should-not-trigger.yaml
    └── fixtures/
        ├── sample.py
        ├── complex.py
        └── nested/
            └── module.py

Specify context directory when running:

waza run eval.yaml --context-dir evals/code-explainer/fixtures

Or use relative paths in eval.yaml if fixtures are adjacent.

Multi-Model Comparison

Run the same eval against multiple models:

# Run with gpt-4o
waza run eval.yaml --model gpt-4o -o gpt4.json

# Run with Claude
waza run eval.yaml --model claude-sonnet-4.6 -o sonnet.json

# Compare results
waza compare gpt4.json sonnet.json

Override the default model in eval.yaml:

waza run eval.yaml --model gpt-4o  # Overrides config.model

Filtering and Parallel Execution

Filter by Task Name

waza run eval.yaml --task "basic*" --task "edge*"

Filter by Tags

waza run eval.yaml --tags "happy-path"

Parallel Execution

# Run tasks concurrently with auto-sized workers
waza run eval.yaml --parallel

Saving Results

Save eval results for later analysis or comparison:

waza run eval.yaml -o results.json

Output format:

{
  "name": "code-explainer-eval",
  "model": "claude-sonnet-4.6",
  "pass_rate": 0.8,
  "tasks": [
    {
      "id": "basic-001",
      "name": "Basic Usage",
      "passed": true,
      "graders": [
        {
          "name": "checks_logic",
          "passed": true,
          "score": 1.0
        }
      ]
    }
  ]
}

Caching

For iterative testing, cache results:

waza run eval.yaml --cache --cache-dir .waza-cache

Only tasks with changed inputs/config re-run.

Common Patterns

Simple Validation

graders:
  - type: text
    name: format_check
    config:
      regex_match:
        - "^[A-Z].*\\.$" # Sentence starting with capital, ending with period

tasks:
  - "tasks/format/*.yaml"

Multi-Criteria Scoring

graders:
  - type: code
    name: completeness
    config:
      assertions:
        - "len(output) > 500"
        - "'function' in output"
        - "'parameter' in output"

tasks:
  - "tasks/completeness/*.yaml"

Behavioral Constraints

Behavioral constraints are defined in individual task YAML files:

tasks/efficient-001.yaml

id: efficient-001
name: Efficiency test
inputs:
  prompt: "Refactor this code"
expected:
  behavior:
    max_tool_calls: 3 # Efficient
    max_tokens: 1000 # Concise
    max_response_time_ms: 30000 # Must complete within 30 seconds
    required_tools: # Must use these tools
      - grep
      - edit
    forbidden_tools: # Must NOT use these tools
      - rm

Field	Type	Description
max_tool_calls	int	Maximum number of tool invocations allowed
max_iterations	int	Maximum number of conversation rounds (turns)
max_tokens	int	Maximum tokens in the response
max_response_time_ms	int	Maximum wall-clock execution time in milliseconds
required_tools	string[]	Tools the agent must use during the task
forbidden_tools	string[]	Tools the agent must NOT use during the task

Each constraint that is set (non-zero / non-empty) contributes equally to the behavior efficiency score. If all constraints pass, the score is 1.0; each failure reduces it proportionally.

Hooks

Lifecycle hooks run process commands at specific points during an evaluation. Use them for setup, teardown, or validation.

hooks:
  before_run:
    - command: "npm install"
      working_directory: "./fixtures"
      error_on_fail: true
  after_run:
    - command: "bash cleanup.sh"
  before_task:
    - command: "echo Starting task"
  after_task:
    - command: "bash collect-metrics.sh"

Hook	When it runs
before_run	Once, before the entire evaluation starts
after_run	Once, after all tasks complete
before_task	Before each individual task
after_task	After each individual task

before_run runs before task files are loaded, MCP mocks are admitted, or the engine starts, so it can generate task YAML, prompt_file inputs, or CSV datasets. The runner repeats admission after engine initialization and after each new pass’s setup, retaining trigger-only dependencies and returning detected configuration errors before task execution without repeating hooks. In --baseline mode each pass runs before_run, MCP admission, and after_run, including when the second pass generates changed task or fixture files. If the run is interrupted (Ctrl+C), after_run still runs once, for at most one minute.

Hook commands can reference string values from the eval’s top-level inputs map with Go template syntax. This works for both legacy skill evals and scenario evals:

inputs:
  marker: DOGFOOD_VARIABLE

hooks:
  before_run:
    - command: "bash marker.sh {{.Vars.marker}}"
      error_on_fail: true

Waza splits the configured command into process arguments, then renders each argument before starting the hook. A standalone placeholder remains one argument when its value contains whitespace or is empty. An invalid template or a missing key fails the hook instead of passing the placeholder literally. Waza does not invoke a shell implicitly; use an explicit shell command when shell syntax is required. CSV row columns are not added to lifecycle hook context.

An empty rendered executable is rejected even when error_on_fail is false. When hooks are configured, hook configuration and eval inputs are included in result cache identity, so changing either prevents reuse of an old task result.

Each hook entry:

Field	Type	Default	Description
command	string	(required)	Shell command to execute
working_directory	string	.	Working directory for the command
exit_codes	list[int]	[0]	Acceptable exit codes
error_on_fail	bool	false	Abort the run if this hook fails

---

Template Variables

For tasks_from CSV datasets, use the top-level inputs field to define fallback template variables for generated task prompts. The current row’s columns override same-named eval inputs and are also available as {{.Vars.column_name}}. CSV prompt rendering additionally supports {{.JobID}}, {{.TaskName}}, and {{.Timestamp}}.

inputs:
  language: python
  framework: fastapi

tasks_from: tasks.csv

Fixture files are copied into the task workspace, not inlined in the prompt:

inputs:
  prompt: "Read sample.py and explain this code."
  files:
    - path: sample.py

---

External Task Lists

Use tasks_from to generate tasks from a CSV dataset:

name: shared-eval
tasks_from: shared-tasks.csv
range: [2, 2]

config:
  trials_per_task: 3
  model: claude-sonnet-4.6

This is useful when multiple eval specs share the same task set but differ in config or graders.

The first CSV row supplies headers; subsequent rows are 1-based data rows. Fallback IDs/names retain the original row number, so this range selects row-2 unless an explicit id or name supplies the identity. run, preflight, grade, and run analysis share selection and template semantics. Regrade the resulting task with waza grade eval.yaml --results results.json --task row-2, or omit --task to grade all selected tasks. CSV paths resolve from the eval directory unless absolute. They must be lexically inside that directory and remain inside after symlinks resolve. Contained links are allowed. An aliased eval base accepts relative contained sources, but an alternate physical absolute spelling outside the lexical alias root is rejected. The final CSV open and parse use a retained eval-directory root, so a late replacement symlink cannot redirect reads outside it. This does not freeze concurrent file contents or provide universal physical equivalence.

---

Best Practices

1. Clear task descriptions — Future reviewers should understand what’s being tested
2. Realistic validators — Don’t over-specify. A few key checks beat 20 strict rules
3. Fixture diversity — Include basic, edge case, and negative test fixtures
4. Tag your tasks — Makes filtering and analysis easier
5. Use timeout appropriately — Too short = false failures, too long = slow tests
6. Reuse graders — Define once, apply across multiple tasks
7. Version your evals — Track improvements with version numbers

Next Steps

• Validators & Graders — Reference for all grader types
• Web Dashboard — Explore results interactively
• CLI Reference — All commands and flags