Graders

Graders are the scoring engine behind every waza evaluation. After an agent executes a task, one or more graders inspect the result and produce a verdict.

How graders work

1. The agent runs a task inside an isolated workspace.
2. waza collects the output text, transcript (tool calls, events), session digest (token counts, tools used), and the workspace directory (files on disk).
3. Each grader receives that context and returns a result:

Field	Type	Description
score	float	0.0 – 1.0 (proportion of checks passed)
passed	bool	Whether the grader considers the task successful
feedback	string	Human-readable explanation
details	object	Structured metadata for debugging

You can attach graders globally (applied to every task) or per-task in your eval YAML. Each grader also accepts an optional weight field that controls its influence on the composite score (see Weighted Scoring below).

Global graders can also reference remote config-only presets:

graders:
  - ref: github.com/waza-evals/fact#factuality@v1.0.0
    name: factuality_strict
    config:
      threshold: 0.9

Run waza get eval.yaml after adding a ref; it writes waza.lock with the commit SHA and content digest. waza run expands locked refs from the local module cache and fails if the lock or digest is missing.

Challenge the checks, not just the answer

New authors can start with the catalog and Text, then return to these explicitly selected advanced profiles. Assurance/calibration profiles below are available in v0.39.0 with separate explicit selection, paid-call acknowledgment where applicable, and platform limits; see release availability.

Advanced assurance profiles and observational limits

Before relying on a grader, compare its actual verdict against independently reviewed evidence. A correct outcome and a different valid tool path should pass. Plausible success prose with wrong resulting state, and a correct outcome with a forbidden side effect, should fail the relevant check. Keep the grader configuration fixed across all cases.

The contributor baseline corpus demonstrates those checks on 12 synthetic scenarios across MCP, CLI and repository state, with four candidates each:

NO_COLOR=1 go test ./internal/graders -run '^TestBaselineChallenge' -count=1

It uses existing file and tool-call graders, runs offline without an agent or model, and does not change current commands, defaults or golden-task semantics. The challenge inventory describes its state predicates and limitations.

Candidate coverage is not reviewed assurance

These labels are not independently human-reviewed. MCP/CLI state is synthetic, not proof of a live external operation, and recorded calls do not establish authoritative resulting state. Missing required evidence is insufficient evidence, not a successful bad-case rejection proving assurance. A passing finite corpus does not establish universal grader correctness or model-backed calibration.

Strict preserved-file assurance

waza assure eval.yaml --references labels.json compares actual native file grader outputs with fixed labels and emits a scoped waza.grader-assurance report. It is offline and never runs a task agent. A supplied current-review document needs separate explicit source acceptance; bundled labels remain unreviewed. Exact source/declaration bindings and an observed executable-path digest, complete unredacted files and good/alternative/critical-bad coverage are independent gates.

Both assure and assure calibrate materialize locked grader presets from the local cache only, rechecking digest, trust and offline schema/override admission after preflight. Missing or changed pins/cache fail without fetching. Original eval bytes retain the reference; resolved-config and declaration digests bind the materialized configuration. Direct API references remain unsupported, even with a supplied native type. This neither authenticates presets nor authorizes paid calls.

The executable digest hashes the on-disk file opened at os.Executable() at observation time, not the mapped running image, RAM or authenticity. Replacement can change the observed bytes or make acquisition fail. Missing applicable provenance remains unavailable, with no zero or startup-cache digest fallback. The existing portable algorithm and report domain are unchanged.

Finite authored output separately supports native text and inline JSON-schema checks. Labels select exactly one historical snapshot or authored document; original authored envelope bytes are externally bound by SHA-256. Empty and whitespace strings are supplied input, while null is unavailable. This scope does not establish historical execution, tool completeness or billing. The exact envelope requires outer schemaVersion: "2.0" and kind: "waza.grader-reference-input" to prevent native historical import; the inner payload 1.0 and evidence 1.1 retain their independent meanings.

Unpreserved paths cannot prove absence. Unknown snapshot output/event completeness is not upgraded by success text, hashes or stored grader verdicts. Missing evidence and operational errors never count as correct negative rejection. Any wrongly accepted targeted critical bad case fails independently of average agreement. These finite cases supply no universal confidence guarantee.

The internal mechanical observation resolver has no eval-context fallback: snapshot-backed Diff observations need a caller-supplied evaluator-resolved context_dir. Omission returns not_assessed before consulting unrelated working-directory files. Ordinary eval Diff defaults, explicitly supplied relative bases and snapshot-free observations are unchanged.

The existing offline assure --calibrate remains unavailable: it reports not assessed, makes zero paid calls and leaves usage/credits null. See the CLI reference and exact contract.

The separate evaluator API assurance.Calibrate requires explicit selection, a bound local rubric, frozen native declarations and inputs, a construction-only factory and successful pre-initialization notice. It has no default factory. Freezing copies matcher declarations without compiling regular expressions or loading external schema references, including in unselected eval, task and checkpoint graders. Nested JSON numbers retain their exact values; compilation and resource policy remain the responsibility of the selected validation path. Preserved workspace verification reuses validated manifest metadata and one path policy within each call while still checking every required file’s locator, raw and typed content digests, completeness and redaction state. Independent native callbacks, lifecycle diagnostics and independent received-event/ accounting attribution remain distinct from native candidate verdicts; missing or operational evidence cannot establish strict pass. Report-1.1 claims require separate explicit dashboard inspection. Neither schema admission nor a finite agreement authenticates human review, proves exhaustive server isolation or authorizes paid calls.

The distinctly selected waza assure calibrate command supplies an owned factory only for that operation. Required references/review/current-source acceptance and rubric root remain independent of --accept-paid-calls, which defaults to false. Without acknowledgment its notice refuses execution with an empty ledger/null billing, on both interactive and noninteractive input. There is no prompt or generic --yes fallback. Its admitted job count is not a provider spending cap. The positive cooperative timeout defaults to 10m; partial reports remain available with execution errors. Explicit --output requires Unix atomic replacement; other platforms, including Windows, refuse it before preflight or paid calls. Omit the flag and capture stdout instead. See the explicit calibration flags for error exits, source/provider boundaries and output persistence.

Private calibration and retained-source decoding use bounded, content-bound acquired copies through the same native parsers and formatter, without reopening audit-copy paths. Selected rubric metadata is acquired:sha256:<digest>; ordinary path/builtin APIs retain file:/builtin: metadata. Selected acquired task decoding additionally requires an explicit nonblank id and refuses prompt_file rather than reopening a pathname; ordinary task-loading APIs retain their existing behavior.

The shared audit store retains worktree, .cache and private-directory os.Root handles for creation and cleanup rather than reopening absolute paths. Cache and private-child identities are compared before and after audit writes; replacement refuses further writes, and cleanup cannot delete replacement-namespace files. POSIX checks reject group/world-writable cache parents and private children with group/other mode bits; changed permissions that violate these checks fail closed. Unknown residual files are retained with an explicit cleanup failure. Judge/source decoding still consumes digest-bound acquired bytes, not audit paths.

Consumption integrity does not certify cache confidentiality. Retained-root namespace custody and POSIX-mode checks do not establish whole-filesystem ancestry or Windows ACLs. Windows mode bits are not ACL certification, and cross-platform source support is not evidence of actual Windows execution.

Separately selected preserved-native assurance

The offline evaluator API assurance.VerifyPreserved selects report 1.2 and preserved_native_mechanical_assurance; assurance.ParsePreservedReport is its separate strict reader. Default CLI assurance, offline Verify, calibrated 1.1 reading and dashboard inspection do not select this profile.

It challenges the current selected grader against preserved artifacts. The native-declaration binding identifies that current declaration, not the grader or source version used in an earlier execution. Historical grader correspondence is not certified.

It checks actual snapshot bytes, exact native origins/artifact locators and digests, complete unredacted selected evidence and consumed raw field presence before invoking existing tool-call, tool-constraint, action-sequence and required-file existence/content graders. Canonical event arguments and normalized digest arguments retain their native differences. Complete explicit empty tapes are specific to this operation; the nonempty-tape importer is unchanged. Missing/null arguments are not an empty object.

Selected files cannot prove absence. Adapter limits are 16 MiB raw snapshot, 4 MiB decoded bytes per file, 8 MiB selected-file total and 256 selected paths. Unsupported modes remain not assessed. Unknown historical collector completeness cannot be promoted by synthetic complete fixtures.

Strict pass requires eligible supplied review, verified provenance, complete scoped coverage and actual mechanical agreement; targeted critical false acceptance independently fails. Calibration remains not selected with null billing and no ledger, even if requested. No engine factory or paid execution is involved. See the preserved-native boundary.

Internal engine-free qualification helpers additionally check bounded original source custody, immutable projections and capture-only equivalence. They are unexported preparation primitives, not a new CLI mode, public report format, deployed backend, production review/currentness authority or task execution path. Private local journal and injected freshness controls preserve original artifact bytes, sync ordering and nonpass uncertainty within an accepted local-host scope; they do not establish global reservation or authenticate current human review. See the internal boundary.

---

At a glance

waza ships with 13 native grader kinds. Pick the right one for the job:

Type	YAML key	What it checks
Inline Script	code	Python/JS assertion expressions against output
Text	text	Text and regex matching against output
File	file	File existence and content patterns in workspace
Diff	diff	Workspace files vs. expected snapshots or fragments
JSON Schema	json_schema	Output validates against a JSON Schema
Prompt (LLM-as-judge)	prompt	A second LLM grades the result
Behavior	behavior	Agent metrics — tool calls, tokens, duration
Action Sequence	action_sequence	Tool call ordering and completeness
Skill Invocation	skill_invocation	Which skills were invoked and in what order
Program	program	External command (any language) grades via exit code
Tool Constraint	tool_constraint	Validate tool usage constraints (expect/reject lists and argument patterns)
Tool Calls	tool_calls	Validate required/forbidden tools and call count bounds
Trigger	trigger	Heuristic grader for validating whether a prompt should activate a skill

---

Inline Script (code)

Evaluates Python or JavaScript assertion expressions against the execution context. Each assertion is a one-liner that must evaluate to True.

- type: code
  name: output_quality
  config:
    language: python # or "javascript" — default is python
    assertions:
      - "len(output) > 100"
      - "'function' in output.lower()"
      - "len(transcript) > 0"

Context variables

Variable	Type	Description
output	str	Agent’s final text output
outcome	dict	Structured outcome state
transcript	list	Full execution transcript events
tool_calls	list	Tool calls extracted from transcript
tool_events	list	Canonical per-call evidence including arbitrary tool arguments, success, error, and result
command_invocations	list	Sanitized mocked CLI attempts; omitted exit_code means no completed result was captured
errors	list	Errors from transcript
duration_ms	int	Execution wall-clock time

The evidence arrays default to empty when unavailable; absence is not proof that a dependency succeeded. Use tool_events for MCP operation arguments rather than the legacy typed tool_calls view. command_invocations preserves completed exit zero distinctly from an interrupted attempt with no exit_code.

Built-in functions: len, any, all, str, int, float, bool, list, dict, re

Scoring: passed_assertions / total_assertions

No generator expressions

Python’s eval() does not support generator expressions in a restricted scope.

# ❌ Fails at runtime
assertions:
  - "any(kw in output for kw in ['azure', 'deploy'])"

# ✅ Use explicit boolean chains instead
assertions:
  - "'azure' in output.lower() or 'deploy' in output.lower()"

JavaScript example

- type: code
  name: js_checks
  config:
    language: javascript
    assertions:
      - "output.length > 50"
      - "output.includes('hello')"

---

Text

Validates the agent output using substring matching and regex patterns. Supports case-insensitive and case-sensitive substring checks, plus regex pattern matching.

- type: text
  name: format_checker
  config:
    contains:
      - "deployed to"
      - "Resource group"
    not_contains:
      - "permission denied"
    regex_match:
      - "https?://.+"
    regex_not_match:
      - "(?i)error|failed|exception"

Option	Type	Description
contains	list[str]	Substrings that must appear (case-insensitive)
not_contains	list[str]	Substrings that must not appear (case-insensitive)
contains_cs	list[str]	Substrings that must appear (case-sensitive)
not_contains_cs	list[str]	Substrings that must not appear (case-sensitive)
regex_match	list[str]	Regex patterns that must match in output
regex_not_match	list[str]	Regex patterns that must not match

Scoring: passed_checks / total_checks

Example: code quality gate

- type: text
  name: code_quality
  config:
    contains:
      - "def "
      - "return"
    not_contains:
      - "TODO"
      - "FIXME"
    regex_match:
      - "def \\w+\\(.*\\):" # Has function definitions
    regex_not_match:
      - "print\\(" # No debug prints

---

File

Validates file existence and content patterns in the agent’s workspace directory. Use when the agent creates, modifies, or should avoid certain files.

- type: file
  name: project_structure
  config:
    must_exist:
      - "src/index.ts"
      - "package.json"
      - "tsconfig.json"
    must_not_exist:
      - "node_modules/"
      - ".env"
    content_patterns:
      - path: "package.json"
        must_match:
          - '"name":\\s*"my-app"'
        must_not_match:
          - '"version":\\s*"0\\.0\\.0"'

Option	Type	Description
must_exist	list[str]	Workspace-relative paths that must be present
must_not_exist	list[str]	Paths that must not be present
content_patterns	list	Regex checks against file contents (see below)

Each content_patterns entry:

Field	Type	Description
path	string	Workspace-relative file path
must_match	list[str]	Regex patterns the file content must match
must_not_match	list[str]	Regex patterns the file must not match

Note

At least one of must_exist, must_not_exist, or content_patterns is required.

---

Diff

Compares post-execution workspace files against expected snapshots or content fragments. Ideal for testing file-editing tasks where you know the expected output.

- type: diff
  name: code_edits
  config:
    expected_files:
      - path: "src/main.py"
        contains:
          - "+def new_function():"
          - "+    return 42"
          - "-def old_function():"
      - path: "README.md"
        snapshot: "expected/README.md"

Each expected_files entry supports:

Field	Type	Description
path	string	Workspace-relative file path (required)
snapshot	string	Path to expected file for exact matching
contains	list[str]	Content fragments to check (see prefix rules)

Contains prefix rules

Prefix	Meaning
+	Fragment must be present in the file
-	Fragment must not be present
(none)	Fragment must be present (same as +)

Scoring: passed_checks / total_checks

Fragment checking

- type: diff
  name: api_updates
  config:
    expected_files:
      - path: "src/api.py"
        contains:
          - "+from fastapi import FastAPI"
          - "+@app.get('/health')"
          - "-import flask"

Snapshot matching

- type: diff
  name: exact_config
  config:
    expected_files:
      - path: "config.json"
        snapshot: "snapshots/expected_config.json"

---

JSON Schema (json_schema)

Validates that the agent output is valid JSON conforming to a given schema. Supports both inline schemas and schema files.

- type: json_schema
  name: api_response
  config:
    schema:
      type: object
      required: [status, data]
      properties:
        status:
          type: string
          enum: [success, error]
        data:
          type: object

Option	Type	Description
schema	object	Inline JSON Schema definition
schema_file	string	Path to a .json schema file
extract_json	boolean	Extract exactly one JSON object or array from prose or a Markdown JSON code block before validation (default: false)

One of schema or schema_file is required.

By default, the output must be pure JSON. Set extract_json: true when the agent may add prose or wrap a response in a JSON code block. Extraction requires exactly one JSON object or array; responses with zero or multiple candidate JSON documents fail.

Example: schema file

- type: json_schema
  name: validate_manifest
  config:
    schema_file: "schemas/manifest.schema.json"

---

Prompt (LLM-as-judge)

Uses a second LLM to evaluate the agent’s work. The judge LLM calls set_waza_grade_pass or set_waza_grade_fail tool functions to render its verdict. This is the most flexible grader — it can assess quality, correctness, style, or anything you can describe in natural language.

This is the closest Waza match for OpenAI Evals modelgraded specs. See the eval YAML guide for concrete fact.yaml, closedqa.yaml, and battle.yaml translations.

- type: prompt
  name: quality_check
  config:
    prompt: |
      Review the agent's response. Check that the explanation is:
      1. Technically accurate
      2. Easy to understand
      3. Includes code examples

      If all criteria are met, call set_waza_grade_pass.
      Otherwise, call set_waza_grade_fail with your reasoning.
    model: "gpt-4o-mini"

Option	Type	Default	Description
prompt	string	(required¹)	Instructions for the judge LLM
rubric	string	(optional)	Reusable rubric by name (e.g. groundedness) or path to a local .md file. See Rubric library.
model	string	eval judge_model	Optional per-grader judge model override. See evidence-model fallback below.
reasoning_effort	string	eval judge_reasoning_effort	Judge effort: low, medium, high, xhigh, or max. Requires executor: copilot-sdk.
continue_session	bool	false	Resume the agent’s session (judge sees full context)
mode	string	independent	Judging mode: independent (standard) or pairwise (compare two, requires --baseline)
evidence_context	object	omitted	Opt into bounded frozen evidence in a separately owned, fresh judge session; final eval/task graders only.

¹ Either prompt or rubric is required. When only rubric is set, the rubric’s body becomes the judge prompt. When both are set, the inline prompt wins (useful for ad-hoc overrides while keeping the rubric metadata attached for reporting).

Reasoning-effort overrides work on eval, task, and checkpoint prompt graders, including continue_session and pairwise judging. These are judge settings, not per-task agent settings. Pin the agent effort once per eval to compare the same suite consistently:

config:
  executor: copilot-sdk
  model: gpt-5
  reasoning_effort: high
  judge_model: gpt-5-mini
  judge_reasoning_effort: low
graders:
  - type: prompt
    name: quality
    config:
      reasoning_effort: medium
      prompt: Evaluate the response and call the grading tool.

Omitting an effort leaves the SDK/model default unchanged for new sessions; on a resumed session, the runtime retains its existing setting. With an explicit effort, select a concrete model from waza models. Waza validates hosted Copilot model support against the runtime catalog before execution rather than allowing a silent downgrade. For custom providers, effort is forwarded directly and provider errors are surfaced.

Evidence-backed independent judging

Available in v0.39.0 and opt-in: install the published standalone release or run waza update; see release availability. Older CLIs do not implement the evidence-isolation profile.

Set evidence_context to judge a recorded execution without resuming the actor or giving the judge its live workspace. Omit it to retain existing prompt-grader behavior. An empty object {} requests the recorded task, visible user/assistant messages, canonical tool arguments/results, and final actor output, with no files. Hidden reasoning and ambient instructions are not evidence sections.

For evidence-backed judging, model precedence is the grader’s model, waza grade --judge-model when regrading, eval judge_model, eval model, then auto. Only the configured model choice is inherited, never actor runtime state. An explicit hosted reasoning effort is checked against the isolated judge client’s own model catalog before creating its session; auto, unsupported efforts and unverifiable model metadata are rejected. Custom providers retain direct effort passthrough.

This is the grader from the runnable evidence-judge example:

- type: prompt
  name: grounded_report
  config:
    model: gpt-5-mini
    prompt: |
      Evaluate the recorded task using the supplied evidence.
      Check the reported status, item count, and sum against report.json.
      Prefer observed tool results over unsupported actor claims.
      Call exactly one of set_waza_grade_pass or set_waza_grade_fail once.
    evidence_context:
      files: [report.json]
      evaluator_only: [references]

Use executor: copilot-sdk. files contains exact slash-relative, agent-visible file paths: no globs, directories, traversal, absolute paths, or whole-workspace capture. evaluator_only excludes explicitly reviewed files/directories before selected-file reads. Do not select answer keys, labels, grader prompts, or private instructions. Existing sensitive-path and symlink protections still apply. Exclusions from all final eval and task graders apply to the capture union; saved regrading revalidates those exclusions and the union limits before any judge starts, without recapturing files. Conflicting declarations fail rather than weakening an exclusion. Each judge receives only its own selected files, not another grader’s files or exclusions.

Scope / setting	Evidence context
Final eval grader / final task validator	Supported
Saved-result waza grade	Supported with a valid persisted bundle
Checkpoint grader / assurance calibration	Rejected
mode: pairwise / continue_session: true	Rejected
Controlled execution / --evidence-snapshot	Rejected

Required evidence and limits

Every requested section and file is required. Explicit empty output, empty files, and an observed zero-tool execution are valid. An absent event source is not an observed empty tool list. Missing tool completions/results, unavailable sections, redacted required content, and unsupported capture fail before judge startup. There is no truncation, partial-grading fallback, or live-file recovery. Present null/scalar options, aliases/merges in evidence-enabled grader configuration, duplicate declarations, and malformed new wire fields are rejected.

The fixed limits are 64 files in the run’s union, 64 KiB per file, and 256 KiB for the entire compact JSON-encoded bundle, including core evidence and metadata. Exclusions are also limited to 64 paths. Source values, structure traversal and sanitized/encoded bytes are bounded; JSON escaping and metadata consume the encoded budget. Formatting whitespace outside JSON strings does not consume this budget; whitespace within strings does. Pretty-printed saved results use the same limit as compact capture. Each filtered delivery must fit too. Rubric text has a separate 256 KiB cap. These bounds do not guarantee sufficient model context. Selected-file capture requires Linux/macOS; it also retains the existing filesystem admission limits.

Evidence freezes after the last actor turn and command-mock finalization, before any final grader can resume or mutate the actor. Result schema 1.7 stores runs[].judge_evidence (independent bundle version 1.0) and typed runs[].prompt_judging observations. An independently requested snapshot 1.2 references that same frozen digest as judgeEvidenceSHA256; it does not recapture judging files or duplicate the bundle. Ordinary snapshots remain 1.1. Tasks with an effective evidence grader bypass cache reads and writes.

Runtime, credentials, and claim boundaries

Each judgment, including a retry or regrade, creates a fresh session on a separately owned pinned SDK runtime/client. It uses private empty working/config/ state roots, replacement judge instructions, and only the two grade tools. Actor sessions, resources, skills, instructions, hooks, MCP, command mocks, on-demand discovery and session retrieval are not inherited. Tool-source impersonation and permissions outside the grade tools fail closed. Exactly one valid grade callback is required; a later execution, policy, cancellation or cleanup error still makes the operation fail.

Hosted authentication requires an explicit environment credential, in priority order: COPILOT_GITHUB_TOKEN, GH_TOKEN, GITHUB_TOKEN. Stored CLI/gh login and keychain credentials are not copied. Keep credentials out of evals and results. Explicit BYOK uses COPILOT_BASE_URL, optional COPILOT_PROVIDER, COPILOT_WIRE_API, COPILOT_API_KEY, and COPILOT_BEARER_TOKEN; existing COPILOT_PROVIDER_* aliases remain accepted and canonical names win. Arbitrary actor environment, PATH, proxy and certificate variables are not inherited; endpoints requiring those ambient settings are not supported by this profile. COPILOT_CLI_PATH, when set, must identify the same installed native executable as the pinned embedded runtime, not a standalone CLI or copied binary.

The isolation contract is about exposed capabilities, discovery, and supplied evidence, not an OS sandbox, protection from hostile same-user processes, absence of provider/authentication traffic, unbiased judging, or better accuracy. The pinned-runtime canary test uses synthetic credentials and loopback inference; native-platform execution evidence must be reported separately.

Digests establish integrity of supplied bytes, not authenticity against a writer who can replace both bytes and hashes, nor truth of actor claims. Submission is observed from a real grade callback, not merely constructing a request. inspection: not_observable explicitly means Waza cannot tell whether the model read or understood every item. The dashboard’s existing Evidence view shows metadata only. The saved result itself contains the bounded sanitized bundle: review it before sharing; redaction does not certify absence of secrets.

Regrade without the original workspace

From the repository root:

waza preflight examples/evidence-judge/eval.yaml --context-dir examples/evidence-judge/fixtures
waza run examples/evidence-judge/eval.yaml --context-dir examples/evidence-judge/fixtures --output results.json
waza grade examples/evidence-judge/eval.yaml --results results.json --task report --output regraded.json

Provide the explicit hosted/BYOK credentials above before model-backed commands. The evidence-only regrade never inspects --workspace, resumes a saved session, follows snapshot_path, or reruns the actor. It verifies original types, digest, eval/task/run/attempt binding, completeness and the current selection before model startup. It uses the recorded task even if task YAML changed. A changed rubric is allowed; newly requested files must already be captured. Historical, cached, invalid or incomplete bundles fail operationally. Do not mix evidence regrading with legacy prompt graders; mechanical graders retain their own inputs. Saving cached-source rejection diagnostics does not make that evidence eligible for a later regrade.

Ordinary failed judgments retain grade’s existing zero exit with passed: false and status: failed; operational errors return nonzero, status: error, and no overall_score in the grading summary. Once evidence grading starts, --output retains safe per-run failure diagnostics before returning an error; output-write errors also propagate. Malformed results that cannot be decoded fail before regrading. Valid, run-bound legacy snapshot manifests remain provenance only: old validations, result, grader-configuration and checkpoint artifacts become unavailable for the new assessment. Invalid or mismatched unused manifests are omitted with a warning; the frozen judge bundle is validated independently. Original execution billing is unchanged. Each actual new judge session is counted once under regrade_usage.sessions, separate from historical evaluation_usage. In a mixed-task invocation, a later mechanical grader’s operational failure still saves earlier judge observations and current charges to --output; the overall grading status is error and its score is withheld. If verbose diagnostic output fails after judging, the saved observations and charges are retained and the affected run is marked as an error. Mechanical errors remain fail-fast: an error before the first evidence attempt starts no judge, and a later mechanical error does not start further judges, including within a task that mixes mechanical and evidence graders in either order. Local evidence-admission and diagnostic-output errors also stop further judges; prompt-execution failures can still aggregate across independent tasks. This field describes only the current regrade; a later mechanical-only regrade omits it instead of repeating earlier judge usage. Missing usage/credits remain unavailable; explicit reported zero remains zero. This is model-backed regrading and can incur new charges, not offline replay.

Using a reusable rubric

- type: prompt
  name: is_grounded
  config:
    rubric: groundedness   # built-in rubric by name
    model: "gpt-4o-mini"

- type: prompt
  name: house_style
  config:
    rubric: ./rubrics/my-house-style.md   # local file
    model: "gpt-4o-mini"

The rubric markdown ships with a YAML frontmatter (name, version, scale, description, optional goldens) so each rubric is versioned, self-documenting, and testable. See Rubric library for the full schema and the list of built-in rubrics.

How it works

1. waza starts a new Copilot session (or resumes the agent’s session if continue_session: true).
2. The judge receives your prompt plus two tool definitions: set_waza_grade_pass and set_waza_grade_fail.
3. The judge calls one of the tools. If it calls set_waza_grade_pass, score is 1.0; if set_waza_grade_fail, score is 0.0.

When to use continue_session

Set continue_session: true when the judge needs to inspect files on disk or see the full conversation history. The judge resumes the same session the agent used, so it has access to the workspace and prior context.

Example: file review with continue_session

- type: prompt
  name: file_review
  config:
    prompt: |
      Check that the files on disk are properly updated.
      Verify the code compiles and follows best practices.
      If correct, call set_waza_grade_pass.
      If not, call set_waza_grade_fail with your reasoning.
    model: "claude-sonnet-4.5"
    continue_session: true

---

Behavior

Validates agent behavior metrics — how many tool calls were made, token consumption, required/forbidden tools, and execution duration. Use this to enforce efficiency and safety guardrails.

- type: behavior
  name: efficiency_check
  config:
    max_tool_calls: 15
    max_tokens: 50000
    max_duration_ms: 60000
    required_tools:
      - bash
      - edit
    forbidden_tools:
      - rm
      - sudo

Option	Type	Description
max_tool_calls	int	Maximum allowed tool calls (0 = no limit)
max_tokens	int	Maximum total token usage (0 = no limit)
max_duration_ms	int	Maximum execution time in ms (0 = no limit)
required_tools	list[str]	Tool names that must be used
forbidden_tools	list[str]	Tool names that must not be used

At least one option must be configured. Each configured rule counts as one check.

Scoring: passed_checks / total_checks

Note

The behavior grader reads from the session digest, which tracks aggregate tool call counts, token usage, and tools used during the session.

---

Action Sequence (action_sequence)

Validates the sequence of tool calls the agent made against an expected action path. Supports three matching modes for different levels of strictness.

- type: action_sequence
  name: deploy_workflow
  config:
    matching_mode: in_order_match
    expected_actions:
      - bash
      - edit
      - bash
      - git

Option	Type	Description
expected_actions	list[str]	The expected tool call sequence
matching_mode	string	How to compare actual vs. expected (see below)

Matching modes

Mode	Description
exact_match	Actual tool calls must exactly match the expected list (same tools, same order, same count)
in_order_match	Expected tools must appear in order, but extra tools between them are allowed
any_order_match	All expected tools must appear, but order doesn’t matter

Scoring: F1 score computed from precision (correct calls / total actual) and recall (matched / total expected).

---

Skill Invocation (skill_invocation)

Validates which Copilot skills the agent invoked during a session. Useful for multi-skill orchestration testing — verifying the agent delegates to the right skills, or that a specific skill was not invoked.

- type: skill_invocation
  name: routing_check
  config:
    required_skills:
      - azure-prepare
      - azure-deploy
    mode: in_order
    allow_extra: true

Option	Type	Default	Description
required_skills	list[str]	[]	Skills that must be invoked
forbidden_skills	list[str]	[]	Skills that must not be invoked
mode	string	—	Matching mode for required_skills: exact_match, in_order, or any_order
allow_extra	bool	true	Whether extra skill invocations beyond required_skills are penalized

At least one of required_skills or forbidden_skills must be non-empty. mode is required when required_skills is set and ignored for forbidden-only configs.

When allow_extra: false, extra invocations beyond the required list reduce the score by up to 60%.

For forbidden-only configs, unrelated skill invocations are allowed and the grader scores 1.0 unless a forbidden skill appears. For mixed required and forbidden configs, forbidden skill use fails the grader with a score of 0.0.

- type: skill_invocation
  name: no_prod_deploy
  config:
    forbidden_skills:
      - azure-prod-deploy
    allow_extra: true

Scoring: F1 score (precision × recall) for required skills, with optional penalty for extra invocations. Forbidden-only configs score 1.0 when all forbidden skills are absent and 0.0 when any are present.

---

Tool Constraint (tool_constraint)

Validates which tools an agent should or shouldn’t use. Reads from the session digest to check tool usage.

- type: tool_constraint
  name: guardrails
  config:
    expect_tools:
      - tool: bash
      - tool: edit
    reject_tools:
      - tool: bash
        command_pattern: "rm\\s+-rf"
    allow_only:
      - tool: bash
      - tool: edit
      - tool: view

Option	Type	Default	Description
expect_tools	list	[]	Tool specs that must appear in the session (lower bound). tool is a regex.
reject_tools	list	[]	Tool specs that must not appear (blocklist). tool is a regex.
allow_only	list	null	Allow-list of tool specs (upper bound). Any observed tool call not matching an entry is a violation. Declared entries do not need to be exercised — the agent may use a subset. tool is an exact name (case-insensitive), NOT a regex. An empty list denies all tool use.

At least one option must be configured. Each expect_tools/reject_tools entry counts as one check. For allow_only, each observed tool call counts as one check and each undeclared call as one failure (a session with zero tool calls counts as one vacuous pass).

Scoring: passed_checks / total_checks

Auto-injected for custom agents

When you target a custom agent (.agent.md) with a tools: field in its frontmatter, waza automatically adds an implicit tool_constraint grader with an allow_only list built from those tools. The agent may use any subset of the declared tools but not any tool outside the list. You don’t need to declare it in eval.yaml unless you want to add additional constraints. See the Evaluating Custom Agents guide for examples.

`expect_tools` vs `allow_only`

expect_tools is a lower bound (each listed tool must be used at least once); allow_only is an upper bound (each observed tool must be in the list). They compose — use both together when the agent must call X and may only call from a set of Y.

Example: safety guardrails

- type: tool_constraint
  name: safety_check
  config:
    reject_tools:
      - tool: bash
        command_pattern: "rm\\s+-rf|sudo|kill"

Example: required workflow tools

- type: tool_constraint
  name: workflow_tools
  config:
    expect_tools:
      - tool: bash
      - tool: edit
      - tool: grep

Example: sandbox allow-list

- type: tool_constraint
  name: sandbox
  config:
    allow_only:
      - tool: view
      - tool: grep
      - tool: glob

---

Tool Calls (tool_calls)

Validates which tools an agent called during execution. Supports required tools, forbidden tools, and call-count bounds — all in one grader. Each configured constraint counts as one check; partial credit is awarded.

- type: tool_calls
  name: tool_usage
  config:
    required_tools:
      - bash
      - edit
    forbidden_tools:
      - rm
      - sudo
    min_calls: 2
    max_calls: 20

Option	Type	Default	Description
required_tools	list[str]	[]	Tool names that must appear in the session
forbidden_tools	list[str]	[]	Tool names that must not appear
min_calls	int	0	Minimum total tool calls required (0 = no min)
max_calls	int	0	Maximum total tool calls allowed (0 = no limit)

At least one constraint must be configured. When both min_calls and max_calls are set, min_calls must be ≤ max_calls.

Scoring: passed_checks / total_checks

tool_calls vs. tool_constraint vs. behavior

Use tool_calls for simple required/forbidden tool lists with call-count bounds. Use tool_constraint when you also need turn and token budgets. Use behavior when you need max_duration_ms or a single max_tool_calls cap without tool-name validation.

Example: enforce safe tool usage

- type: tool_calls
  name: safety_check
  config:
    forbidden_tools:
      - rm
      - sudo
    max_calls: 30

Example: verify minimum workflow

- type: tool_calls
  name: workflow_check
  config:
    required_tools:
      - bash
      - edit
    min_calls: 3

Argument matchers (schema 1.1+)

When you need to assert what a tool was called with — not just which tools ran — use the expect: field with structured argument matchers. Each entry names a tool and a map of args: constraints; the grader checks every entry against the recorded tool calls for the run (the same data that is also serialized into tool_events[] in results.json).

- type: tool_calls
  name: edit_correctness
  config:
    expect:
      - tool: edit
        args:
          path: { equals: "src/auth.go" }
          file_text: { contains: "func Login" }
      - tool: bash
        args:
          command: { regex: "^go test " }

Each matcher is a single-key mapping naming exactly one of the kinds below:

Key	Matches when …
equals	the argument is deeply equal to the supplied value
regex	the stringified argument matches the supplied RE2 pattern
contains	the stringified argument contains the supplied substring
range	the numeric argument satisfies the bounds (gte / lte / gt / lt)
json_schema	the argument validates against the supplied JSON Schema (draft-07+)

The range matcher accepts any combination of inclusive (gte, lte) and exclusive (gt, lt) bounds; at least one must be set:

limit: { range: { gte: 1, lte: 5 } }
score: { range: { gt: 0.0 } }

tool_constraint accepts the same args: block on each expect_tools entry, so you can require both which tool ran and how it was called in a single grader:

- type: tool_constraint
  name: guardrails
  config:
    expect_tools:
      - tool: edit
        args:
          path: { regex: "^src/" }
      - tool: bash
    reject_tools:
      - tool: bash
        command_pattern: "^rm -rf"

Argument matchers apply on top of the existing required/forbidden checks — a tool must still be present (or absent) and also satisfy its matchers to count as a pass.

---

Program

Runs any external command to grade the agent output. The agent output is passed via stdin, and the workspace directory is available as the WAZA_WORKSPACE_DIR environment variable. Exit code 0 means pass (score 1.0); non-zero means fail (score 0.0).

- type: program
  name: lint_check
  config:
    command: "python3"
    args: ["scripts/grade.py"]
    timeout: 60

Option	Type	Default	Description
command	string	(required)	Program to execute
args	list[str]	[]	Arguments passed to the program
timeout	int	30	Max execution time in seconds

Example: shell script grader

- type: program
  name: build_test
  config:
    command: "bash"
    args: ["-c", "cd $WAZA_WORKSPACE_DIR && npm test"]
    timeout: 120

Example: custom Python grader

#!/usr/bin/env python3
"""scripts/grade.py — reads agent output from stdin, exits 0 or 1."""
import sys, json, os

output = sys.stdin.read()
workspace = os.environ.get("WAZA_WORKSPACE_DIR", "")

# Check that a required file was created
if os.path.exists(os.path.join(workspace, "result.json")):
    print("✓ result.json created")
    sys.exit(0)
else:
    print("✗ result.json missing")
    sys.exit(1)

---

Trigger

Heuristic grader for validating whether a prompt should activate a skill.

- type: trigger
  name: deploy_trigger
  config:
    skill_path: skills/azure-deploy/SKILL.md
    mode: positive
    threshold: 0.6

Option	Type	Description
skill_path	string	(required) Path to the SKILL.md file or the skill directory containing it.
mode	string	Either positive or negative
threshold	number	Score threshold between 0.0 and 1.0. Default is 0.6.

For more information, see the triggers grader documentation

Using graders in eval YAML

Global graders

Defined at the top level of eval.yaml, applied to every task:

graders:
  - type: text
    name: no_errors
    config:
      regex_not_match:
        - "(?i)fatal error|crashed|exception occurred"

  - type: code
    name: has_output
    config:
      assertions:
        - "len(output) > 10"

tasks:
  - task_files: ["tasks/*.yaml"]

Per-task graders (by reference)

Define graders globally, then reference them by name in individual tasks:

graders:
  - type: text
    name: format_check
    config:
      regex_match: ["^[A-Z]"]

  - type: code
    name: length_check
    config:
      assertions:
        - "len(output) > 100"

tasks:
  - id: task-001
    inputs:
      prompt: "Explain this code"
    expected:
      graders:
        - format_check
        - length_check

Per-task graders (inline)

Define graders directly inside a task:

tasks:
  - id: task-001
    inputs:
      prompt: "Create a REST API"
    expected:
      graders:
        - type: file
          name: api_files
          config:
            must_exist:
              - "src/api.py"
              - "requirements.txt"
        - type: diff
          name: api_content
          config:
            expected_files:
              - path: "src/api.py"
                contains:
                  - "+from fastapi import FastAPI"

---

Weighted Scoring

By default every grader counts equally toward the composite score. Add a weight field to shift importance:

graders:
  - type: text
    name: critical_check
    weight: 3.0 # Counts 3×
    config:
      regex_match: ["deployed"]

  - type: text
    name: nice_to_have
    weight: 0.5 # Counts 0.5×
    config:
      contains: [summary]

  - type: code
    name: basic_length
    # weight omitted → defaults to 1.0
    config:
      assertions:
        - "len(output) > 50"

Option	Type	Default	Description
weight	float	1.0	Relative importance of this grader in the composite score

Formula: (score₁ × weight₁ + score₂ × weight₂ + …) / (weight₁ + weight₂ + …)

With the config above and scores of 1.0, 0.0, and 1.0, the composite score is (1.0×3 + 0.0×0.5 + 1.0×1) / (3+0.5+1) = 0.89.

When to use weights

Weight critical checks (correctness, security) higher and cosmetic checks (style, formatting) lower. A task still passes only when all graders pass — weights affect the composite score, not the pass/fail verdict.

---

Combining graders

You can stack multiple graders on a single task. All graders run independently and each produces its own score. A task passes when all graders pass.

graders:
  # 1. Output mentions key concepts
  - type: text
    name: concepts
    config:
      contains: [authentication, JWT, middleware]

  # 2. No error patterns
  - type: text
    name: no_errors
    config:
      regex_not_match: ["(?i)error|exception"]

  # 3. Required files exist
  - type: file
    name: deliverables
    config:
      must_exist: ["src/auth.ts", "src/middleware.ts"]

  # 4. Agent was efficient
  - type: behavior
    name: efficiency
    config:
      max_tool_calls: 20
      max_tokens: 40000

  # 5. LLM judge confirms quality
  - type: prompt
    name: quality
    config:
      prompt: |
        Review the implementation for security best practices.
        Call set_waza_grade_pass if secure, set_waza_grade_fail if not.
      model: gpt-4o-mini

---

Real-world examples

Skill trigger accuracy

Verify a skill activates on the right prompts and stays silent on the wrong ones:

tasks:
  - id: should-trigger
    inputs:
      prompt: "Deploy my app to Azure"
    expected:
      graders:
        - type: text
          name: azure_response
          config:
            contains: [azure, deploy, resource]

  - id: should-not-trigger
    inputs:
      prompt: "What's the weather today?"
    expected:
      graders:
        - type: text
          name: no_azure
          config:
            not_contains: [azure, deploy, bicep]

Code editing task

Test that the agent correctly modifies source files:

graders:
  - type: file
    name: files_created
    config:
      must_exist:
        - "src/utils.ts"
        - "tests/utils.test.ts"

  - type: diff
    name: correct_edits
    config:
      expected_files:
        - path: "src/utils.ts"
          contains:
            - "+export function formatDate"
            - "+export function parseConfig"
        - path: "tests/utils.test.ts"
          contains:
            - "+describe('formatDate'"

  - type: program
    name: tests_pass
    config:
      command: bash
      args: ["-c", "cd $WAZA_WORKSPACE_DIR && npm test --silent"]
      timeout: 60

Multi-skill orchestration

Verify the agent invokes the right skills in the right order:

graders:
  - type: skill_invocation
    name: correct_workflow
    config:
      required_skills:
        - brainstorming
        - azure-prepare
        - azure-deploy
      mode: in_order
      allow_extra: false

  - type: action_sequence
    name: tool_usage
    config:
      matching_mode: in_order_match
      expected_actions:
        - bash
        - create
        - edit
        - bash

---

Best practices

1. Start simple — Begin with keyword or regex graders, then add stricter graders as you identify failure modes.
2. Layer your checks — Combine output graders (regex, keyword) with workspace graders (file, diff) and behavior graders for comprehensive coverage.
3. Use descriptive names — checks_auth_flow beats grader1. Names appear in the dashboard and CLI output.
4. Use prompt for subjective quality — When you can’t express the check as a pattern or assertion, let an LLM judge it.
5. Set behavior budgets — Use the behavior grader to catch runaway agents that burn too many tokens or tool calls.
6. Test graders in isolation — Run a single task with waza run eval.yaml --task my-task -v to verify graders before running the full suite.
7. Use program as an escape hatch — When you need full programmatic control, write a script in any language and use the program grader.

---

Next steps

• Writing Eval Specs — Task and fixture configuration
• Web Dashboard — Visualize grader results
• CLI Reference — All commands and flags