CLI Commands

Complete reference for all waza CLI commands and their options.

Version boundary (October 20, 2026): published standalone v0.39.0 includes scenario flags, preflight, selected-file evidence/regrade, assurance profiles, controlled policy commands and serve --lab. Install the release or run waza update; a source build is optional. Explicit opt-in and platform restrictions remain; see the feature/platform matrix. The azd extension is versioned separately; verify its own help rather than assuming prefix parity.

Installation

# macOS/Linux or Windows Git Bash/MSYS/Cygwin
curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
waza --version

For native Windows PowerShell:

irm https://raw.githubusercontent.com/microsoft/waza/main/install.ps1 | iex
waza --version

waza update

Update waza to the latest release.

waza update

The command prompts for confirmation, then downloads and runs the official installer for the current OS. It uses the Bash installer on macOS/Linux and the PowerShell installer on native Windows. Use --yes to skip the prompt in scripted environments:

waza update --yes

The native Windows update path schedules the final binary replacement after the running waza.exe exits, avoiding Windows executable file locks. In WSL, waza update uses the Linux Bash installer and updates the WSL installation.

waza preflight

Inspect an explicit eval file offline, without an agent/model, grader execution, subprocesses, mock servers, live services, dependency downloads, or update checks:

waza preflight eval.yaml
waza preflight eval.yaml --format json > preflight.json
waza preflight eval.yaml --context-dir ./fixtures --strict

Flag	Default	Meaning
--format	human	human or json inventory/diagnostics
--context-dir	eval directory/fixtures	Context directory; relative values resolve from cwd
--strict	false	Exit 1 for unresolved/unsupported prerequisites, not only invalid configuration

Invalid configuration exits 1; unresolved/unsupported warn and exit 0 unless --strict opts into 1. Usage/output errors use exit 2. Existing command exits are unchanged. verified means static inspection, not a satisfied requirement or runtime enforcement. The separately typed report is not results.json. Ref-only eval graders with a missing pinned cache remain unresolved until their preset/fallback can supply the effective name. Explicit duplicate names remain invalid; task/checkpoint graders still require inline declarations. See Offline Eval Preflight for descriptive requirement references, capability/dependency plans, and offline limitations.

waza run

Run an evaluation benchmark.

waza run [eval.yaml | skill-name | agent-name]

Arguments

Argument	Description
[eval.yaml]	Path to evaluation spec file
[skill-name]	Skill name (auto-detects eval.yaml)
[agent-name]	Custom agent name (auto-detects eval.yaml for .agent.md files)
(none)	Auto-detect using workspace detection

Custom agents: Waza discovers both SKILL.md and .agent.md files. You can target a custom agent by name just like a skill. See Evaluating Custom Agents for details.

APM-managed skills: Waza detects compiled APM skills at skills/<name>/.apm/skills/<name>/SKILL.md, so waza run <name> and no-argument workspace detection work without symlinking SKILL.md into the source skill directory. If both skills/<name>/SKILL.md and the APM compiled output exist for the same skill, the top-level SKILL.md takes precedence.

Flags

Finite MCP/CLI fault sequences are configured in scenario eval YAML, not through a run flag. They do not add an automatic retry policy or certify real CLI/MCP implementations. See Dependency Recovery.

Flag	Short	Type	Default	Description
--context-dir	-c	string	./fixtures	Fixtures directory path
--output	-o	string		Save results JSON to file
--output-dir	-d	string		Directory for structured output; each run creates a UTC-timestamped subdirectory of the specified directory. Mutually exclusive with --output.
--verbose	-v	bool	false	Detailed progress output
--parallel		bool	false	Run tasks concurrently
--workers	-w	int	auto	Number of concurrent workers
--trials		int	config.trials_per_task	Run each task N times for flakiness detection (omit to use config.trials_per_task; when provided, value must be >= 1)
--task	-t	string		Filter tasks by name (repeatable)
--tags		string		Filter tasks by tags (repeatable)
--model	-m	string		Override model (repeatable)
--judge-model		string		Model for LLM-as-judge graders (overrides execution model)
--cache		bool	false	Enable result caching
--cache-dir		string	.waza-cache	Cache directory path
--format	-f	string	default	Output format: default, github-comment
--reporter		string[]		Output reporters: json, junit:<path> (repeatable)
--timeout		int	300	Task timeout in seconds
--baseline		bool	false	A/B testing mode — runs each task twice (without skill = baseline, with skill = normal) and computes improvement scores
--update-snapshots		bool	false	Update or create diff grader snapshot files to match current workspace output
--discover		string		Auto skill discovery — walks directory tree for SKILL.md + eval.yaml (root/tests/evals)
--strict		bool	false	Fail if any SKILL.md lacks eval coverage (use with --discover)
--suggest		bool	false	Generate a Copilot suggestion report based on test outcomes
--recommend		bool	false	Generate heuristic recommendation after multi-model run
--session-log		bool	false	Enable session event logging (NDJSON)
--session-dir		string	.	Directory for session log files
--no-summary		bool	false	Skip combined summary.json for multi-skill runs
--interpret		bool	false	Print optional interpretation with per-trial scoped requirement observations/evidence metadata when present; legacy no-metadata text is unchanged and malformed evidence is withheld.
--skip-graders		bool	false	Skip grading (execution only); grade later with waza grade
--no-skills		bool	false	Disable all skill loading for the evaluation
--transcript-dir		string		Save per-task transcript JSON files
--no-cache		bool	false	Explicitly disable result caching
--keep-workspace		bool	false	Preserve temp workspaces after execution for debugging
--auto-file-issue		bool	false	Auto-file or update a GitHub issue for failing runs (requires gh and GITHUB_REPOSITORY)
--otel-exporter		string		Export OpenTelemetry traces using otlp, stdout, or file. Off by default. See OpenTelemetry Tracing.
--otel-endpoint		string		OTLP endpoint (host:port or URL); only used with --otel-exporter=otlp
--otel-headers		string		Comma-separated key=value OTLP headers (e.g. for auth)
--otel-file		string		File path for span JSON when --otel-exporter=file
--otel-include-payloads		bool	false	Include prompt/tool-arg/tool-result/completion content in spans (default: redacted to sha256+length)
--snapshot		string		Capture redacted per-trial observations and portable evidence to the given directory; not a complete workspace. See Snapshot & Replay.
--snapshot-file		string[]		Repeatable reviewed agent-visible UTF-8 regular-file allow-list; requires --snapshot. Linux/macOS only.
--snapshot-evaluator-only		string[]		Repeatable evaluator-only file/directory exclusions; requires --snapshot and --snapshot-file. Metadata inventory capped at 256 inspections/depth 32 before selected content; limit/inspection/resource failure prevents capture. Never capture credentials or evaluator-only files.
--snapshot-max-file-bytes		int64	1048576	Positive input/sanitized byte limit per file (1 MiB).
--snapshot-max-total-bytes		int64	10485760	Positive aggregate input/sanitized byte limit (10 MiB).
--snapshot-env-allow		string[]		Allow-list of environment variable name patterns to embed in snapshots (default-deny; supports WAZA_* wildcards).
--redact		string		Custom snapshot redaction policy; replaces defaults unless extend: true.

Examples

# Run all tasks
waza run eval.yaml -v

# Run specific skill
waza run code-explainer

# Specify fixtures directory
waza run eval.yaml -c ./fixtures -v

# Save results
waza run eval.yaml -o results.json

# Filter to specific tasks
waza run eval.yaml --task "basic*" --task "edge*"

# Multiple models (parallel)
waza run eval.yaml --model gpt-4o --model claude-sonnet-4.6

# Use a different judge model for LLM-as-judge graders
waza run eval.yaml --model gpt-4o --judge-model claude-opus-4.6

# Run a Copilot SDK eval with top-level mcp_mocks fixtures and no live MCP service
waza run eval.yaml --session-log

# Parallel execution with 8 workers
waza run eval.yaml --parallel --workers 8

# With caching
waza run eval.yaml --cache --cache-dir .waza-cache

# Generate JUnit XML for CI test reporting
waza run eval.yaml --reporter junit:results.xml

# A/B testing: baseline vs skill performance
waza run eval.yaml --baseline -o results.json
# Output includes improvement breakdown (quality, tokens, turns, time, completion)

# Auto-update diff grader snapshots
waza run eval.yaml --update-snapshots

# Auto skill discovery
waza run --discover ./skills/

# Auto discovery with strict mode (fail if any SKILL.md lacks eval coverage)
waza run --discover --strict ./skills/

# Skip grading, then grade separately
waza run eval.yaml --skip-graders -o results.json
waza grade eval.yaml --results results.json

# Session event logging
waza run eval.yaml --session-log --session-dir ./logs

# Multi-model comparison with recommendation
waza run eval.yaml --model gpt-4o --model claude-sonnet-4.6 --recommend

# Keep temp workspaces for debugging fixture issues
waza run eval.yaml --keep-workspace -v

# Auto-file or update a triage issue on failures
GITHUB_REPOSITORY=microsoft/waza waza run eval.yaml --auto-file-issue

waza init

Initialize a new waza project.

waza init [directory]

Arguments

Argument	Description
[directory]	Project directory (default: current dir)

Flags

Flag	Description
--no-skill	Skip first skill creation prompt

Creates

project-root/
├── skills/
├── evals/
├── .github/workflows/eval.yml
├── .gitignore
└── README.md

Examples

waza init my-project
waza init my-project --no-skill

waza get

Resolve remote grader refs and update waza.lock.

waza get [eval.yaml | ref]

When given an eval file, waza get resolves each graders[].ref, downloads the module into ~/.waza/cache/{host}/{org}/{repo}/{sha}/, and writes a lockfile next to the eval. The lock pins the original ref to a commit SHA, source URL, and sha256: content digest.

waza get eval.yaml
waza get github.com/waza-evals/fact#factuality@v1.0.0

waza run is strict: evals with remote grader refs require a valid waza.lock and local cache entry. If the lock is missing, cache is unavailable, or the digest mismatches, the run fails instead of resolving mutable refs implicitly.

waza new skill

Create a new skill. In interactive mode, the wizard collects spec-aligned metadata: name, description, trigger phrases, and anti-trigger phrases.

waza new skill [skill-name]

Modes

Project mode (inside a skills/ directory):

cd my-skills-repo
waza new skill code-explainer
# Creates skills/code-explainer/SKILL.md + evals/code-explainer/

Standalone mode (no skills/ directory):

waza new skill my-skill
# Creates my-skill/ with all files

Flags

Flag	Description
--template	Template pack (coming soon)

Examples

# Interactive wizard
waza new skill code-analyzer

# Non-interactive (CI/CD)
waza new skill code-analyzer << EOF
Code Analyzer
Analyzes code for patterns and issues
code, analysis
EOF

waza new eval

Scaffold an eval suite for an existing skill.

For target-free workflows, use waza new eval <name> --scenario. Scenario scaffolds do not read or create SKILL.md and explicitly select copilot-sdk, even if project defaults use a mock engine.

waza new eval

Generate an eval scaffold from an existing SKILL.md.

waza new eval <skill-name>

Creates:

• evals/<skill-name>/<files.evalFile>
• evals/<skill-name>/tasks/positive-trigger-1<files.taskFileSuffix>
• evals/<skill-name>/tasks/positive-trigger-2<files.taskFileSuffix>
• evals/<skill-name>/tasks/negative-trigger-1<files.taskFileSuffix>

Flags

Flag	Description
--output	Custom path for the generated eval file; task YAMLs are created in a sibling tasks/ directory next to this file.
--scenario	Scaffold a workflow without a target skill/custom agent; eval schema 2.0.
--template	Scenario template: repository (default), cli, or mcp; requires --scenario.

The generated eval filename, task glob, and task filename suffix come from .waza.yaml files settings when configured. Scenario mode creates one workflow<files.taskFileSuffix> task under tasks/ when selected by files.taskGlob, otherwise in its matching literal directory, and a fixtures/ directory beside the eval file; the repository template includes fixtures/inventory.txt. It rejects blank names and refuses to overwrite any generated file. The three trigger tasks above belong to the default skill flow. The repository scenario has three file graders: report outcome, no private.txt, and exact final inventory.txt bytes. CLI/MCP scenarios retain two graders. This is a final-state check, not proof that no transient edits occurred. In YAML and JSON, a present scenario key requires a nonblank string and exact schema 2.0; null or other types are invalid. Omit the key for a legacy eval. Repository mode emits fixture-local .gitattributes to preserve inventory LF bytes during Git checkout, or a safe root rule if the configured glob selects local metadata. If neither location is safe, generation fails before writing; narrow the task-directory glob or choose a different eval filename. Metadata files are also protected against overwrite. Older working copies are not rewritten automatically. Wildcard-directory globs that select the default tasks/ path remain supported. Globs that cannot select that task or a safe literal alternative, or that select generated non-task files, are rejected before writing. For example, fixtures/*.yaml is supported, but fixtures/* selects .gitkeep and must be narrowed. A failed creation or write rolls back files created by the attempt, including a partial output. File ownership handles remain open through success or rollback, preventing an unlinked inode from being recycled before the ownership check. Observed replacements are preserved; cleanup and handle-close failures are reported instead of silently deleting unowned output. Empty directories may remain, so this is not a whole-directory transaction. Scenario mode also rejects symlinked parents in the selected output path, including its root and nested task/fixture directories. Choose a physical output path instead. Relative paths use the retained working directory; absolute paths are checked from their filesystem root. The literal macOS /var and /tmp aliases are accepted only when their targets exactly match the expected /private locations. Admitted directory handles are retained, so replacing a parent path cannot redirect subsequent file creation. Moving an admitted directory can move its outputs; rollback removes only this attempt’s owned files, not unrelated content or whole directories. This is not a filesystem sandbox. This parent policy applies to scenario generation, not legacy skill/eval scaffolding.

Scenario templates honor configured task discovery, including literal custom directories such as cases/*.task.yaml. An incompatible glob/suffix, unsafe destination, or glob selecting generated non-task artifacts fails scaffolding instead of producing an undiscoverable task.

Examples

# Generate eval scaffold using skills/my-skill/SKILL.md
waza new eval my-skill

# Write eval.yaml to a custom location
waza new eval my-skill --output evals/custom/my-skill-eval.yaml

# Real repository workflow; no skill required
waza new eval inventory --scenario --template repository
waza run evals/inventory/eval.yaml

# For offline harness checks, set config.executor: mock in a copy.
# Mock output is not agent-quality evidence.

waza new task

Generate task YAML files from a recorded prompt execution.

waza new task from-prompt

Run your prompt through Copilot and generate a task file with inferred validators from the recorded session.

waza new task from-prompt <prompt> <task-path>

Arguments

Argument	Description
<prompt>	Prompt to execute during recording
<task-path>	Output path for the generated task YAML

Flags

Flag	Type	Default	Description
--model	string	claude-sonnet-4.5	Copilot model to use for recording
--testname	string	auto-generated-test	Test name and test ID for generated task
--tags	string[]		Tags to add to generated task
--timeout	duration	5m0s	Max time to allow prompt completion
--overwrite	bool	false	Overwrite output file if it already exists
--root	string	.	Directory used to discover skills

Examples

# Generate a task from a recorded prompt run
waza new task from-prompt "Refactor this function for readability" evals/code-explainer/tasks/refactor-readability.yaml

# Add metadata and allow file replacement
waza new task from-prompt "Explain this diff and risks" evals/code-explainer/tasks/diff-analysis.yaml \
  --testname diff-analysis \
  --tags recorded,regression \
  --overwrite

waza check

Validate skill compliance and readiness.

waza check [skill-name | skill-path]

Arguments

Argument	Description
[skill-name]	Skill name (e.g., code-explainer)
[skill-path]	Relative or absolute path to a skill directory or SKILL.md file
(none)	Check all skills in workspace

Explicit paths discover evals relative to the target skill, not the current directory. Discovery honors that workspace’s .waza.yaml paths.evals and files.evalFile settings (with eval.yaml as the legacy filename fallback). Without a config file, it searches up to 10 directory levels for evals/<skill-name>/eval.yaml. Separated evals take priority over <skill-dir>/evals/eval.yaml, then <skill-dir>/eval.yaml.

Flags

Flag	Description
--verbose	Detailed compliance report
--format	Output format: text (default), json

Output

🔍 Skill Readiness Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Skill: code-explainer

📋 Compliance Score: High
   ✅ Excellent! Your skill meets all requirements.

📊 Token Budget: 420 / 500 tokens
   ✅ Within budget.

🧪 Evaluation Suite: Found
   ✅ eval.yaml detected.

💡 Advisory Checks
   ✅ scope-reduction: Capability scope: 3 signal(s) detected

✅ Your skill is ready for submission!

Advisory: Scope Reduction Warning

The scope-reduction advisory check detects when a SKILL.md may have lost workflow coverage due to token-limit compression. It parses three concrete signals:

Signal	What it counts
USE FOR items	Comma-separated phrases after USE FOR:
Level-2 headings	## headings in the body (distinct capability sections)
Numbered procedures	Sequences starting with 1. (distinct workflows)

The check takes the maximum of these three indicators. If the count is below the minimum threshold (default: 2), a warning is emitted suggesting possible scope loss. This is advisory only — it does not block readiness.

Examples

waza check code-explainer
waza check ./skills/code-explainer
waza check --verbose

waza compare

Compare evaluation results across models.

waza compare [results-1.json] [results-2.json] ...

Arguments

Argument	Description
[results-N.json]	Result files to compare (2+ required)

Flags

Flag	Description
--format	Output format: table (default), json
--release-policy	Explicit independent fixed-design release policy; requires --collection-dir instead of historical positional results.
--collection-dir	New paired collection sidecars. Invalid, incomplete, rejected or inconclusive selected policy exits 1.
--assurance-contract	Explicit independent offline assurance profile; requires --release-policy, --collection-dir and current per-arm eval/reference inputs. Empty selection fails.
--baseline-eval, --candidate-eval	Current per-arm eval sources required for selected assurance assessment.
--baseline-context-dir, --candidate-context-dir	Optional cwd-relative per-arm context overrides.
--baseline-references, --candidate-references	Current evaluator-only finite authored reference labels required for selected assurance.
--baseline-review, --candidate-review	Optional separately supplied current review files.
--baseline-accept-review-source, --candidate-accept-review-source	Explicit acceptance of each supplied current review source; not authenticated human review.

Selected policy assessment does not change default descriptive comparisons. Policy metadata in ordinary results is not enforcement, and older executables reject the new flags. compare-plan and compare-collect default to offline mock collection; explicitly pass --collector native-copilot to both to select the bounded Copilot SDK profile. It may incur provider usage and is limited to fixed named models, single turns, no skills, deny-all client tools and native text/string checks. It does not establish server isolation, authenticated assurance or provider/runtime versions. Native durable creation supports Linux/macOS only; Windows reader checks do not certify writing. Do not adopt historical outcomes as fresh evidence. See controlled comparisons.

Existing legacy JSON limitation: when task sets differ, missing-task score and pass-rate entries use non-finite values internally, so compare --format json fails to encode the report. Table output still displays those entries as n/a. This predates controlled-policy selection; no missing-task JSON fix or change to legacy task-set semantics is included here.

Examples

waza compare gpt4.json sonnet.json
waza compare gpt4.json sonnet.json opus.json
waza compare results-*.json --format json

Tool-use metrics (schema 1.1+)

When the input files include the tool_events[] array (results.json schema 1.1 or later), waza compare prints an additional TOOL USE section with aggregate per-file metrics:

Metric	Meaning
total_calls	Total number of tool calls across every task and run
tasks_with_tools	Tasks where at least one tool call was recorded
avg_calls_per_task	Mean calls per task
success_rate	Fraction of tool calls that returned success: true
selection_accuracy	Fraction of tasks where the tool_calls grader passed (excludes tasks without a tool_calls grader)
call_count_histogram	Distribution of per-task call counts in buckets 0, 1, 2, 3+

The section is suppressed when none of the compared files contain tool data, so legacy 1.0 results remain unchanged.

waza compare-plan

Bind a mandatory fixed design and explicit requirements to inspected sources before collecting outcomes. Collection defaults to offline mock. Explicit --collector native-copilot selects the bounded Copilot SDK profile; static metadata is not assurance.

Flag	Description
--collector	Collection profile: mock (default, offline) or native-copilot (explicit, may incur provider usage).
--baseline-eval, --candidate-eval	Required source eval YAML files.
--baseline-context-dir, --candidate-context-dir	Cwd-relative context override; default is each eval directory’s fixtures.
--design	Required policy-1.0 design member JSON, with empty allocation; seed generated once.
--requirements	Required explicit identity/runtime/assurance/billing member JSON.
--allow-change	Comma-separated engine,model,reasoning_effort; exact actual per-task differences only.
-o, --output	Required new private policy file; no overwrite.

Invalid or unsupported planning exits 1; nothing is collected. Planned trial ordinals must match effective config.trials_per_task. The policy’s metadata alone never enforces a gate.

waza compare-assurance-plan

Create a new independently versioned offline assurance contract before BEGIN, without changing frozen base policy/journal/decision 1.0 semantics. The selected base policy must explicitly require assurance.

Flag	Description
--release-policy	Required exact base 1.0 policy with assurance required.
--baseline-eval, --candidate-eval	Required current per-arm eval YAML.
--baseline-context-dir, --candidate-context-dir	Optional context overrides with the existing cwd semantics.
--baseline-references, --candidate-references	Required current evaluator-only reference-label JSON with finite authored outputs.
--baseline-review, --candidate-review	Optional separately supplied current review declarations.
--baseline-accept-review-source, --candidate-accept-review-source	Explicit acceptance of each supplied current review source; supplying a file alone is not acceptance.
-o, --output	Required new exclusive contract file; no overwrite.

No existing contract is an input to planning. Collection/assessment must select the resulting --assurance-contract explicitly with --release-policy and current per-arm inputs; assessment additionally needs --collection-dir. Missing/empty required flags fail; empty contract selection never falls back. Unreviewed examples remain nonpassing. Review-source acceptance is not authenticated human review; test-only reviewed declarations are synthetic.

Only mock/no-skills/native-text finite authored outputs are supported. Fresh exact offline1.0 reports for both arms, retained actual-response deterministic regrade and every base non-assurance condition are required for an outer pass. No live SDK assurance, calibration, paid1.1/billing support or automatic paid validation. Stored report pass labels are never adopted. The base API/dashboard intentionally remain nonpassing for required assurance. See the independent profile lifecycle.

waza compare-collect

Collect fresh paired attempts under an explicit frozen policy. The default mock collector is offline. Explicitly select --collector native-copilot for the fixed-model, single-turn Copilot SDK profile, which may incur provider usage. Sources are rechecked before BEGIN and every attempt; private frozen requests avoid mutable fixture rereads. Native durable creation is Linux/macOS only.

Flag	Description
--collector	Collection profile: mock (default, offline) or native-copilot (explicit, may incur provider usage).
--baseline-eval, --candidate-eval	Required exact source eval YAML files.
--baseline-context-dir, --candidate-context-dir	Same context semantics used during planning.
--release-policy	Required precollection policy.
--collection-dir	Required new private directory; no overwrite, adoption or resume.
--assurance-contract	Optional explicit independent offline contract; requires --release-policy and current per-arm references. Empty selection fails.
--baseline-references, --candidate-references	Required current evaluator-only labels when assurance is selected.
--baseline-review, --candidate-review	Optional separately supplied current review files for selected assurance.
--baseline-accept-review-source, --candidate-accept-review-source	Separate explicit acceptance of each supplied review source.

Source drift, unsupported capability, cancellation or persistence errors exit 1. Successful collection persists evidence, not a release pass. Assess it separately with compare or gate --release-policy --collection-dir. The offline example intentionally remains inconclusive; no default live/paid execution or adaptive stopping is enabled. The native profile supports no skills, deny-all client tools and native text/string checks only; hooks, MCP, repositories, cache, concurrency, resume, checkpoints, multiturn and other graders are unsupported. Usage limits use final observations and best-effort prestart checks, so in-flight overshoot is possible; they are not a hard spending cap. Runtime version, provider currency and ambiguous output presence remain unavailable. Operationally incomplete native collection exits nonzero even if core publication completed; inspect native-stop.json for the error, budget or currentness cause. The API collection item’s error may coexist with attempts. Selected assurance additionally persists assurance-contract.json, assurance.ndjson actual-response rows and assurance-ledger.json. Sources, evaluator evidence and review inputs are rechecked after verification and before final publication. A captured empty response has an explicit available value: ""; unavailable output has a reason and no value. Neither collection nor stored grader assurance reports are adopted as an outer release pass.

waza replay

Check a task snapshot’s offline consistency or compare two snapshots. Snapshots are produced by waza run --snapshot <dir> and contain the prompt, fixture digests, ordered tool events, environment allow-list, and redacted grader outcomes. Hashes are not preserved file contents. Replay does not reconstruct a workspace, re-run graders, verify external state, or reproduce an agent execution.

waza replay <snapshot.json> [flags]

Arguments

Argument	Description
<snapshot.json>	Path to a snapshot produced by waza run --snapshot <dir>

Flags

Flag	Type	Default	Description
--mode	string	model-replay	Offline consistency check. live is unsupported and returns exit code 2.
--bisect	string		Path to a second snapshot to compare against the primary. Reports the first divergent turn.
--json	bool	false	Emit machine-readable JSON to stdout instead of a human summary.
--strict	bool	true	In model-replay mode, also re-check final status and grader outcome consistency.

Modes

• model-replay — Re-checks the snapshot’s grader outcomes and tool-event tape for internal consistency (monotonic sequence, validation/score agreement). Fast, fully offline, ideal for CI.
• live — Unsupported; returns exit code 2. No live replay behavior or delivery date is promised.

Exit codes

Code	Meaning
0	Snapshots match / replay succeeded
1	Divergence detected (kind/value rendered to stderr or --json output)
2	Snapshot load, parse, schema-version or I/O error; JSON output error; unsupported live mode or unknown mode

Examples

# Capture snapshots during a run
waza run eval.yaml --snapshot ./snapshots/

# Re-verify a single task's snapshot is internally consistent
waza replay ./snapshots/my-task-run1.json

# Diff two snapshots and find the first divergent turn (CI-friendly JSON)
waza replay ./snapshots/run-a.json --bisect ./snapshots/run-b.json --json

waza gate

Compare a current results file against a baseline and enforce regression policy. Designed for CI: emits stable exit codes and supports GitHub Actions annotations.

waza gate --baseline baseline.json --current results.json [flags]

Flags

Flag	Default	Description
--baseline	(required)	Baseline results JSON (the “known good” run).
--current	(required)	Current results JSON to evaluate.
--max-regression-pct	0	Maximum allowed drop in success rate (percentage points). 0 means no regression tolerated.
--golden-must-pass	true	If true, any failing task marked golden: true causes a golden failure (exit 2).
--on-new-tasks	allow	Policy when current has tasks not in baseline: allow, warn, fail.
--on-removed-tasks	warn	Policy when baseline has tasks not in current: allow, warn, fail.
--format	human	Output format: human, json, markdown, github-actions.
--release-policy	(off)	Explicit independently versioned fixed-design policy assessment; selected nonpass exits 1.
--collection-dir	(unset)	New paired collection, required with --release-policy; no historical adoption or resume.
--assurance-contract	(off)	Explicit independent offline profile; requires --release-policy, --collection-dir and current per-arm inputs. Empty selection fails.
--baseline-eval, --candidate-eval	(unset)	Required current eval sources for selected assurance.
--baseline-context-dir, --candidate-context-dir	(unset)	Optional per-arm context overrides.
--baseline-references, --candidate-references	(unset)	Required current evaluator-only finite authored reference labels for selected assurance.
--baseline-review, --candidate-review	(unset)	Optional separately supplied current review declarations.
--baseline-accept-review-source, --candidate-accept-review-source	(unset)	Explicit per-arm acceptance of each supplied current review source, not authenticated human review.

The default exit matrix below is unchanged. Explicit policy mode cannot be combined with legacy baseline/current/threshold/golden/task-set flags. It emits separate compatibility, completeness, assurance, golden, billing, statistical and operational states. Missing required golden evidence, unavailable final billing, missing assurance and an inconclusive comparison are not the same reason. Static preflight declarations never count as observed assurance.

An independent contract permits only an outer waza.release-assured-decision 1.0 pass when both fresh exact offline1.0 reports, retained actual-response deterministic regrade and every base non-assurance condition pass. The embedded base decision and API/dashboard remain nonpassing for required assurance. Assessment rechecks current sources/evidence/review; stored report pass labels are not adopted. Use --format json for the machine-readable outer contract or the default human output. Selected assurance does not promise legacy markdown/GitHub Actions layouts. Its nonpass exit is 1, not the legacy exit matrix below. No live SDK assurance, calibration, paid1.1/billing support or automatic paid validation is selected; old executables reject new command/unknown flags.

Exit codes

Code	Meaning
0	Pass — within thresholds and policies.
1	Regression — success rate dropped beyond --max-regression-pct, or a task-set policy with fail triggered.
2	Golden failure — at least one task marked golden: true did not pass in --current. Takes precedence over regression.
3	Config error — invalid flags, missing files, or unparseable JSON.

Golden tasks

Mark a task in your eval YAML as a golden task to enforce that it must always pass:

tasks:
  - id: critical-flow
    golden: true
    prompt: "..."
    grader:
      type: exact_match
      expected: "OK"

golden is propagated into results.json so waza gate can read it without re-reading the YAML. Detection is conservative: a task is treated as golden if either the baseline or the current run marks it golden.

Examples

# Basic CI gate: no regression allowed, golden must pass
waza gate --baseline baseline.json --current results.json

# Allow up to 5 percentage-point drop, fail on new tasks
waza gate --baseline baseline.json --current results.json \
  --max-regression-pct 5 --on-new-tasks fail

# Emit GitHub Actions annotations + step summary
waza gate --baseline baseline.json --current results.json \
  --format github-actions

# Markdown report (e.g. for posting as a PR comment)
waza gate --baseline baseline.json --current results.json \
  --format markdown > gate-report.md

See the CI/CD guide for full GitHub Actions and Azure DevOps snippets.

waza adversarial

Run adversarial packs with an explicit engine. Mock validates harness wiring only; real skill evaluation requires --skill, --engine copilot-sdk, authentication/model access and possible usage costs. See the Adversarial harness guide.

waza adversarial [flags]

Every adversarial task is marked golden: true, so unsafe outcomes also flip waza gate to exit 2. The dedicated command additionally enforces an on_unsafe_outcome policy and prints a focused safety summary.

Flags

Flag	Default	Description
--packs	(all)	Comma-separated pack names. Defaults to every built-in pack.
--list-packs	false	Print the pack catalog (name, task count, description) and exit.
--spec	(none)	Inherit adversarial:, graders, and metrics from an eval.yaml.
--on-unsafe-outcome	fail	fail returns exit 2 on any unsafe outcome; warn returns 0.
--engine	(inferred)	mock or copilot-sdk. Defaults to copilot-sdk, or mock when --skill is unset.
--skill	(none)	Skill name to evaluate. Required for --engine copilot-sdk.
--model	claude-sonnet-4-20250514	Model id forwarded to the engine.
--output	(none)	Write the full results.json to a file.
--workers	0	Concurrent task workers (0 = sequential).
--parallel	false	Enable parallel task execution.
--keep-artifacts	false	Keep the temp dir holding extracted packs and the synthesized eval.yaml.
--artifacts-dir	(temp)	Override the artifacts root. Implies --keep-artifacts.
-v, --verbose	false	Verbose progress output.

Exit codes

Code	Meaning
0	All adversarial tasks were refused safely (or --on-unsafe-outcome warn).
2	At least one unsafe outcome was observed, and policy is fail. Matches waza gate exit 2 so a single CI step gates both signals.
3	Configuration error — unknown pack, malformed spec, etc.

Examples

# List built-in packs
waza adversarial --list-packs

# Run every pack against a real skill
waza adversarial --skill ./skills/code-review --model gpt-4o

# Run a single pack and tolerate failures (non-blocking CI smoke)
waza adversarial --packs prompt-injection --on-unsafe-outcome warn

# Read pack selection + policy from eval.yaml
waza adversarial --spec eval.yaml --output adversarial.json

waza spec verify

Verify that eval.yaml covers the executable contract in SKILL.md.

waza spec verify [skill-path] [eval.yaml]

The command parses the skill description, USE FOR triggers, DO NOT USE FOR triggers, and parameter blocks into deterministic requirement IDs with source spans. It then loads the eval task files and reports which task IDs exercise each requirement.

Arguments

Argument	Description
[skill-path]	Path to SKILL.md or a skill directory
[eval.yaml]	Path to the eval spec

Flags

Flag	Description
--skill	Path to SKILL.md or a skill directory
--eval	Path to eval.yaml
--format	Output format: human (default), json, github-actions
--warn	Report uncovered requirements and exit 0 (default); set false to suppress GitHub Actions warning annotations
--fail	Exit 1 when uncovered requirements are greater than or equal to --threshold
--threshold	Uncovered requirement count threshold for --fail (default: 1)
--semantic	Opt in to LLM-assisted semantic matching after deterministic matching
--judge-model	Judge model for --semantic (defaults to config.judge_model, then config.model)

Examples

# Human-readable report
waza spec verify skills/pr-summarizer evals/pr-summarizer/eval.yaml

# Machine-readable report
waza spec verify --skill skills/pr-summarizer --eval evals/pr-summarizer/eval.yaml --format json

# GitHub Actions annotations and non-zero exit on any uncovered requirement
waza spec verify skills/pr-summarizer evals/pr-summarizer/eval.yaml \
  --fail \
  --format github-actions

waza migrate

Check or migrate a public schema artifact.

waza migrate <file>

Legacy eval/task/result defaults remain 1.4. Compatible legacy YAML evals use the eval.yaml/eval.yml filename convention; JSON results are recognized by their artifact shape, including custom .json filenames.

This legacy input/reader policy is separate from new result emission: current evaluations and regrades write results 1.7. Task YAML has no independent version field. Historical result versions and missing reliability metadata are preserved when reading/re-saving; migration does not upgrade them. Compatibility reports select the current version by artifact type: eval 1.4 or result writer 1.7. Missing/1.4 result versions are explicitly labeled as the legacy reader default alongside the current writer version, not rewritten.

A present scenario header requires a nonblank native string and exact schema 2.0, including in canonical eval.yaml/eval.yml files. Supported scenario evals are validated without rewriting them. Custom .yaml/.yml filenames, such as a configured files.evalFile: suite.yml or a custom --output, are recognized only when the scenario header is present. Task/unrelated YAML and custom-named legacy YAML are not treated as evals.

This does not extend migration recognition to scenario .json, extensionless or metadata filenames; JSON dispatch remains results-only even though the scaffold can emit YAML to other filenames. Use a .yaml/.yml filename when checking a scenario with migrate. Invalid present headers, unsupported versions and invalid scenario bodies fail without changing the file. Future breaking changes will require explicit migration steps here.

Arguments

Argument	Description
<file>	Path to a schema artifact such as eval.yaml or results.json

Examples

waza migrate eval.yaml
waza migrate results.json
waza migrate evals/inventory/suite.yml

waza coverage

Generate an eval coverage grid for discovered skills and custom agents.

waza coverage [root]

Arguments

Argument	Description
[root]	Root directory to scan (default: current directory)

Flags

Flag	Description
-f, --format	Output format: text (default), markdown, json
--path	Additional directories to scan for skills/agents/evals (repeatable)

Coverage Levels

• Full: Skill or agent has an eval.yaml/eval.yml with tasks (via tasks: or tasks_from:) and at least 2 distinct grader types.
• Partial: Skill or agent has an eval.yaml/eval.yml but fewer than 2 grader types or no tasks.
• Missing: No eval.yaml/eval.yml found for the skill or agent.

Note: Coverage now reports both SKILL.md (skills) and .agent.md (custom agents) files. The reported coverage percentage reflects only fully covered items (Fully Covered / Total Skills + Agents).

Examples

waza coverage
waza coverage --format markdown
waza coverage --format json
waza coverage --path custom-evals --path agents

waza suggest

Generate suggested eval artifacts from a skill’s SKILL.md using an LLM.

waza suggest <skill-path>

Flags

Flag	Description
--model	Model to use for suggestions (default: project default model)
--dry-run	Print suggestions to stdout (default)
--apply	Write suggested files to disk
--force	Allow --apply to overwrite existing eval/task/fixture files (requires --apply)
--count <n>	Generate exactly N tasks (default: at least 3 diverse + 1 negative)
--focus <category>	Steer generation toward one of triggers, negative-triggers, edge-fixtures, do-not-use-for, parameters
--output-dir	Output directory (default: <skill-path>/evals)
--format	Output format: yaml (default), json

Each suggested task entry carries a confidence value in [0, 1] and a rationale string citing the SKILL.md span it was derived from. Both appear in dry-run / JSON output, but are not written into the task YAML (the task schema rejects unknown fields).

Before writing files, --apply derives a missing or blank string name from the task id (for example, summarize-document becomes Summarize Document). Explicit nonblank names are preserved. Null and other invalid name values still fail task schema validation; dry-run output is unchanged.

--apply is merge-safe by default:

• An existing eval.yaml is preserved — new task files are picked up by its existing tasks: glob pattern.
• New task files refuse to overwrite existing task files (by path or by id) and print a diff between the existing and suggested task.
• New fixtures refuse to overwrite existing fixtures.
• Pass --force to overwrite existing eval / task / fixture files.

Examples

# Preview suggestions
waza suggest skills/code-explainer --dry-run

# Generate exactly 5 negative-trigger tasks and merge them into the existing suite
waza suggest skills/code-explainer --focus negative-triggers --count 5 --apply

# Write eval/task/fixture files
waza suggest skills/code-explainer --apply

# Overwrite a previously generated suite
waza suggest skills/code-explainer --apply --force

# JSON output
waza suggest skills/code-explainer --format json

Worked example: focused negative-trigger coverage

Use --focus negative-triggers when a skill’s DO NOT USE FOR section is underspecified by existing evals:

waza suggest skills/code-explainer --focus negative-triggers --count 2 --dry-run

The dry-run output keeps generation metadata outside the task YAML so the proposed case still validates against the task schema:

tasks:
  - path: tasks/negative-trigger-deploy.yaml
    confidence: 0.88
    rationale: "matches SKILL.md DO NOT USE FOR: deployment"
    content: |
      id: negative-trigger-deploy
      name: Does not trigger for deployment help
      inputs:
        prompt: "Deploy this app to production and configure rollout rings."
      expected:
        should_trigger: false

After review, merge the proposals:

waza suggest skills/code-explainer --focus negative-triggers --count 2 --apply

If a generated task would replace an existing task path or id, --apply stops and prints a diff. Re-run with --force only after confirming the suggested task should replace the curated one.

waza tokens

Token budget management.

waza tokens count

Count tokens in skill files.

waza tokens count [path]

waza tokens count skills/code-explainer/SKILL.md
waza tokens count skills/

waza tokens check

Check token usage against budget.

waza tokens check [skill-name]

waza tokens check code-explainer
# Output:
# code-explainer: 420 / 500 tokens (84%)
# ✅ Within budget

Token limits are resolved in priority order: .waza.yaml tokens.limits → .token-limits.json (deprecated; migrate to .waza.yaml) → built-in defaults. See the Token Limits guide for configuration details, pattern syntax, and migration instructions.

Configure per-file limits in .waza.yaml:

tokens:
  limits:
    defaults:
      "*.md": 2000
      "skills/**/SKILL.md": 5000
    overrides:
      "skills/complex-skill/SKILL.md": 7500

waza tokens compare

Compare markdown token counts between git refs. Supports general-purpose file-level comparison and skill-aware comparison with CI gating.

waza tokens compare [refs...]

# Compare HEAD to working tree (default)
waza tokens compare

# Compare a specific ref to working tree
waza tokens compare main

# Skill-aware comparison with CI threshold
waza tokens compare main --skills --threshold 10

# JSON output with strict absolute limits
waza tokens compare main --skills --threshold 10 --strict --format json

Flags: --format table|json, --show-unchanged, --strict, --skills, --threshold <percent>

Without --skills, compares all markdown files. With --skills, restricts comparison to SKILL.md files under configured skill roots (skills/, .github/skills/, APM .apm/skills/ outputs, and paths.skills from .waza.yaml). In skills mode the default base ref is origin/main (falling back to main).

--threshold sets a percentage-change gate for CI — newly added files are exempt from threshold checks (no baseline to compare) but still subject to absolute limit checks when --strict is set.

waza tokens profile

Structural analysis of SKILL.md files — reports token count, section count, code block count, and workflow step detection.

waza tokens profile [skill-name | path]

Flags: --format text|json, --tokenizer bpe|estimate

Example:

📊 my-skill: 1,722 tokens (detailed ✓), 8 sections, 4 code blocks
   ⚠️  no workflow steps detected

Warnings: no workflow steps, >2,500 tokens, fewer than 3 sections.

waza tokens suggest

Get optimization suggestions.

waza tokens suggest [skill-name]

Analyzes SKILL.md and suggests:

• Sections to shorten
• Removable content
• Restructuring opportunities

waza results

Manage evaluation results from cloud storage or local storage.

waza results list

List evaluation runs from configured cloud storage.

waza results list
waza results list --limit 20
waza results list --format json

Flags

Flag	Description
--limit <n>	Maximum results to display (default: 10)
--format	Output format: table or json (default: table)

waza results compare

Compare two evaluation runs side by side.

waza results compare run-id-1 run-id-2
waza results compare run-id-1 run-id-2 --format json

Flags

Flag	Description
--format	Output format: table or json (default: table)

waza cache

Manage evaluation result cache.

waza cache clear [--cache-dir=.waza-cache]

Subcommands

waza cache clear

Clear all cached evaluation results.

The cache stores test outcomes to speed up repeated evaluations with the same inputs. Cached results are keyed by spec configuration, task definition, model, and fixture file contents.

waza cache clear
waza cache clear --cache-dir /path/to/cache

Flags

Flag	Description
--cache-dir	Cache directory to clear (default: .waza-cache)

Examples

# Clear default cache
waza cache clear

# Clear custom cache directory
waza cache clear --cache-dir .my-cache

---

waza registry

Discover shared registry artifacts and add remote grader presets to an eval.

Registry discovery uses configured index sources. The default public source is https://github.com/waza-evals; project .waza.yaml files can override sources with:

registries:
  - name: public
    url: https://github.com/waza-evals
  - name: company
    url: https://github.com/myorg/waza-registry
    priority: 10

waza registry search

Search configured registry indexes. The current implementation returns bundled sample metadata while live registry index integration is pending.

waza registry search factual
waza registry search factual --kind grader
waza registry search factual --registry company
waza registry search factual --format json

Flags

Flag	Description
--kind	Filter by artifact kind: grader, eval, or dataset
--registry	Search only the named registry source
--format	Output format: table or json (default: table)

waza registry add

Append a remote grader preset reference to eval.yaml, resolve it with the shared remote grader resolver, and update waza.lock with the pinned commit SHA and sha256: content digest.

waza registry add github.com/waza-evals/fact#factuality@v1.0.0
waza registry add github.com/waza-evals/fact#factuality@v1.0.0 --eval eval.yaml
waza registry add github.com/waza-evals/fact#factuality@v1.0.0 --name factuality_strict --set config.threshold=0.9

Remote program graders require --allow-exec or interactive confirmation before waza.lock marks the ref as trusted.

Flags

Flag	Description
--eval	Eval file to update (default: eval.yaml)
--name	Local alias for the grader
--set	Add a local override as key=value; repeatable
--allow-exec	Allow remote program graders without interactive confirmation

---

waza dev

Improve skill compliance iteratively.

waza dev [skill-name | skill-path]

Flags

Flag	Description
--target	Target level: low, medium, high
--max-iterations	Max improvement loops (default: 5)
--auto	Auto-apply without prompting
--fast	Skip integration tests

Workflow

waza dev code-explainer --target high --auto

Iteratively:

1. Scores current compliance
2. Identifies issues
3. Suggests improvements
4. Applies changes
5. Re-scores
6. Repeats until target reached

waza assure

Challenge scoped native file/output checks against evaluator-only references without running a task agent.

waza assure eval.yaml --references labels.json
waza assure eval.yaml --references labels.json \
  --review current-review.json --accept-review-source selected-source \
  --output assurance.json

Flag	Meaning
--references	Required strict versioned label JSON; relative snapshots are rooted at this document’s directory
--review	Separate current-source supplied review declaration
--accept-review-source	Explicit evaluator acceptance of that source; not human authentication
-o, --output	Write standalone waza.grader-assurance report
--calibrate	Currently unavailable: report not assessed with zero paid calls and null usage/credits

Exit 1 unless the strict finite-corpus assessment passes. Missing evidence, unknown completeness, unreviewed labels, operational failures and critical false acceptance cannot pass. Preserved subsets cannot prove missing-path absence. Executable provenance observes the on-disk file opened at os.Executable(), not mapped executing bytes, RAM or authenticity. Path replacement may change the digest or make acquisition fail; unavailable provenance cannot pass. The portable digest algorithm and binding domain are unchanged. No assurance cache is reused. Existing grade, run and golden-task semantics are unchanged. See grader challenges.

The separate internal evaluator API assurance.Calibrate has no default engine factory. Its version-1.1 report claims require separate explicit consumer selection; the offline CLI above continues to emit offline version 1.0. See the evaluator boundary for selection, notice, supported rubric modes and missing/operational evidence.

waza assure calibrate

This distinctly selected command can incur paid provider usage. It is not enabled by the existing assure --calibrate flag, and bundled labels remain unreviewed.

waza assure calibrate eval.yaml --references reviewed-labels.json \
  --review current-review.json --accept-review-source selected-source \
  --rubric-root evaluator-rubrics --accept-paid-calls \
  --timeout 10m --output calibrated-assurance.json

Flag	Meaning
--references	Required evaluator-only labels and rooted evidence
--review	Required separate current supplied review
--accept-review-source	Required explicit source acceptance, not human authentication
--rubric-root	Required root for original local rubric reads
--accept-paid-calls	Default false; explicit acknowledgment for this invocation
--timeout	Positive cooperative assessment duration, default 10m; not a spend cap
-o, --output	Unix only: atomically replace requested standalone report; also write JSON to stdout

On other platforms, including Windows, an explicit --output exits 2 before preflight, producer, paid notice or engine creation. Omit the flag and capture stdout instead; default stdout remains available on every platform.

Without payment acknowledgment the admitted notice refuses execution with an empty ledger/null billing. No interactive prompt, generic --yes, model override or automatic review acceptance is provided. Notice failure blocks construction; ambient custom-provider redirects are rejected. Reviewed model/protocol and actual/maximum independent-job counts print to stderr before any owned engine; these counts do not bound provider follow-ups, retries, credits or money.

Only a clean strict pass exits 0. Ordinary nonpass/refusal exits 1; invalid, operational, canceled, notice-I/O, output or root-close failure exits 2. A nonnil partial report is output even when the API also returns an error. Cleanup uses cooperative deadlines, not forced join. Version-1.1 inspection is explicit and does not certify server isolation, current evidence or human identity.

waza grade

Re-grade previous evaluation results without re-executing the agent.

File-based graders require the actual preserved agent workspace via --workspace. Snapshot hashes and tool events cannot reconstruct those files. Prompt graders may contact the configured judge model.

Final prompt graders with config.evidence_context instead verify the persisted judge_evidence bundle and create fresh isolated judge sessions, never resuming saved actor/judge sessions or fetching snapshot_path. Evidence-only grading does not inspect --workspace. Do not mix it with legacy prompt graders or --evidence-snapshot; newly selected files must already be captured.

waza grade examples/evidence-judge/eval.yaml --results results.json --task report --output regraded.json

This can incur new model charges. Original execution billing is unchanged; new sessions appear in regrade_usage. Ordinary failed judgments retain grade’s zero exit with passed: false / status: failed. Operational evidence/judge errors return nonzero, status: error and no overall_score; requested output retains admitted-source run diagnostics, and output-write errors propagate. See frozen judging configuration and prerequisites.

The narrow --evidence-snapshot mode verifies captured file contents instead. It requires --task, exactly one assessed matching run, original manifest SHA-256/full-origin verification, complete unredacted required files and text digests, and no explicit --workspace. Cached source eval IDs can differ while task/run/attempt identity must match. No snapshot_path is fetched implicitly. Only file graders with explicit required must_exist / content_patterns paths are supported; wildcard/absence checks, no-explicit-file-check graders, and all non-file graders and output-based expectations are rejected before materialization. Only required files are privately materialized; checks cover that captured subset, not original-workspace absence. must_not_match checks only its selected file’s complete verified text. Older workspace grading remains unchanged.

Regrading retains source_manifest_sha256 and invalidates original result/validation/grader-configuration/checkpoint links. Original checkpoints do not observe new declarations. Inspection adds no automatic paid calls. See capture and regrading limits.

waza grade <eval.yaml>

Flags

Flag	Short	Type	Default	Description
--results		string		Required. Path to waza run output JSON
--task		string		Grade a specific task ID only
--workspace		string	.	Agent workspace directory for file-based graders
--evidence-snapshot		string		Explicit reviewed snapshot for verified file-only regrading on Linux/macOS; requires --task and is incompatible with explicit --workspace
--judge-model		string		Model for prompt graders (overrides execution model)
--output	-o	string		Write full EvaluationOutcome JSON
--verbose	-v	bool	false	Verbose output

Examples

# Grade all tasks from a previous run
waza grade eval.yaml --results results.json

# Grade a specific task
waza grade eval.yaml --results results.json --task "basic-function"

# Use a different judge model
waza grade eval.yaml --results results.json --judge-model claude-opus-4.6

# Save graded results for comparison
waza grade eval.yaml --results results.json -o graded.json

# File-only regrading of one assessed run from reviewed captured contents
waza grade eval.yaml --results results.json --task report-task \
  --evidence-snapshot ./reviewed-snapshot.json -o regraded.json

waza session

Manage and inspect session event logs.

waza session list

List session event logs in a directory.

waza session list [--dir <path>]

Flag	Type	Default	Description
--dir	string	.	Directory to search for session logs

waza session view

Render a session timeline from an NDJSON event log.

waza session view <session-file>

waza serve

Start the interactive dashboard.

For JSON-RPC, eval.list recursively lists canonical eval.yaml/eval.yml files plus native-loadable scenario .yaml/.yml files with configured/custom names. Classification and native loading determine eligibility, not full authoring-schema or task validation. MCP waza_eval_list uses the same handler. Custom absent or classification/loading-invalid scenario, task, unrelated and legacy YAML remains unlisted; use eval.get/eval.validate for diagnostics. Canonical invalid-file fallback and caller-root traversal are unchanged. No depth/vendor exclusions or filesystem containment are added; HTTP dashboard result listing is unchanged.

waza serve

Flags

Flag	Description
--port	Port (default: 3000)
--lab	Open the experimental real-results dashboard at /#/lab; classic dashboard remains at /#/
--results-dir	Directory to scan for saved result artifacts (default: .)
--no-browser	Print the dashboard URL without opening a browser
--tcp	TCP address for JSON-RPC (e.g., :9000)
--stdio	Use stdin/stdout for piping

Examples

waza serve                    # http://localhost:3000
waza serve --port 8080        # http://localhost:8080
waza serve --lab --results-dir ./results  # http://localhost:3000/#/lab
waza serve --tcp :9000        # JSON-RPC TCP server

--lab and --tcp are mutually exclusive. The experimental lab refreshes saved results, exposes actual task evidence, and exports reports. Its activity stream replays artifact-derived events, not live worker state. Unsupported readiness/coverage checks and revision metadata remain unavailable. The synthetic demo is separate at /#/lab/demo, never an API-error fallback.

AI Credit usage

Copilot-backed runs query the SDK’s typed session.usage.getMetrics after completion for authoritative final AI Credits (totalNanoAiu ÷ 1e9), falling back to shutdown metrics if the RPC is unavailable. Waza does not compute credits from its token rate table.

Field	Description
summary.usage.ai_credits	Final AI Credits for the run
summary.usage.model_metrics.<model>.ai_credits	Final AI Credits attributed to a model
summary.usage.model_metrics.<model>.{input,output,cache_read,cache_write}_tokens	Token-category diagnostics

The CLI usage summary prints AI Credits (and a per-model column) at nano-unit precision, and the classic dashboard shows AI Credits plus a Usage by model table. Totals include task, trigger, and prompt-grader sessions initiated by waza, not account-wide Copilot usage. A run or model aggregate with missing contributing metrics is unavailable, never a partial subtotal. Main currently uses github.com/github/copilot-sdk/go v1.0.21; published standalone v0.38.10 uses v1.0.15. Older result artifacts show — in the dashboard (n/a in the CLI) rather than an estimate.

Graders

Waza supports multiple grader types for comprehensive evaluation. See the complete Grader Reference for detailed documentation.

Built-in Graders

Grader	Purpose
code	Python/JavaScript assertion-based validation
regex	Pattern matching in output
file	File existence and content validation
diff	Workspace file comparison with snapshots
behavior	Agent behavior constraints (tool calls, tokens, duration)
action_sequence	Tool call sequence validation with F1 scoring
skill_invocation	Skill orchestration sequence validation
prompt	LLM-as-judge evaluation with rubrics
tool_constraint	Validate tool usage constraints (e.g., required/forbidden tools, argument patterns)
trigger	Prompt trigger accuracy — detects whether a prompt should activate a skill

tool_constraint Grader

Validate agent tool usage constraints during evaluation.

graders:
  - type: tool_constraint
    name: check_tools
    config:
      expect_tools:
        - tool: "bash"                         # Required tool call
          command_pattern: "azd\\s+up"         # Optional regex on the command argument
        - tool: "skill"
          skill_pattern: "my-skill"            # Optional regex on the skill argument
        - tool: "edit"
          path_pattern: "\\.go$"               # Optional regex on the path argument
      reject_tools:
        - tool: "bash"                         # Prohibited when args match this pattern
          command_pattern: "rm\\s+-rf"         # Optional regex on the command argument
        - tool: "create_file"                  # Always prohibited

All config fields are optional. Omitted fields skip that constraint.

prompt Grader with Pairwise Mode

Use the prompt grader for LLM-as-judge evaluation. In pairwise mode, compare two approaches side-by-side to reduce position bias.

graders:
  - type: prompt
    name: code_quality_judge
    config:
      mode: pairwise                        # Enable pairwise comparison (requires --baseline flag)
      rubric: "Compare these solutions for code quality and correctness"
      max_tokens: 500

Requirements:

• Pairwise mode requires the --baseline flag on waza run
• Baseline execution must complete before pairwise comparison runs
• Each task is evaluated twice: once without the skill (baseline) and once with it (treatment)

Example:

waza run eval.yaml --baseline -o results.json
# Output includes pairwise judge scores comparing baseline vs treatment approaches

waza quality

Evaluate skill content quality using an LLM-as-Judge. Scores the skill across five dimensions: clarity, completeness, trigger precision, scope coverage, and anti-patterns.

waza quality <skill-path>

Flags

Flag	Type	Default	Description
--model	string	project default	Model to use as judge
--format	string	table	Output format: table or json
--rubric	string		Path to custom rubric file (reserved for future use)

Output

Displays a table with dimension scores (1-5), visual bars, and feedback:

DIMENSION          SCORE  FEEDBACK
──────────────────────────────────────────────────────────────────────
clarity            ████░  Instructions are clear and well-structured.
completeness       ███░░  Missing some edge case documentation.
trigger_precision  █████  USE FOR and DO NOT USE FOR are precise.
scope_coverage     ████░  Scope is well-defined with clear boundaries.
anti_patterns      ███░░  Some steps could be more specific.
──────────────────────────────────────────────────────────────────────
Overall: 3.8/5.0

A solid skill with good clarity and triggers.

Authentication

Requires Copilot authentication when using the default provider. If not authenticated, you will see:

Error: not authenticated — run "copilot login" first

Examples

# Table output (default)
waza quality skills/code-explainer

# JSON output for CI pipelines
waza quality skills/code-explainer --format json

# Use a specific judge model
waza quality skills/code-explainer --model gpt-4o

waza models

List models available for evaluation via the Copilot SDK.

waza models [flags]

Flags

Flag	Type	Default	Description
--json	bool	false	Output as JSON

Output

Displays a table with model ID, name, vision support, and context window size.

MODEL ID          NAME              VISION    CONTEXT WINDOW
──────────────────────────────────────────────────────────────
claude-sonnet-4   Claude Sonnet 4   no        200k
gpt-4o            GPT-4o            yes       128k

2 models available

Authentication

Requires Copilot authentication. Custom provider configuration only applies when creating or resuming Copilot SDK sessions. If not authenticated, you will see:

Error: not authenticated — run "copilot login" first

Examples

# List all models in table format
waza models

# List models as JSON (for scripting)
waza models --json

Exit Codes

Code	Meaning
0	Success
1	One or more tasks failed
2	Configuration or runtime error

Global Flags

Flag	Description
--help	Show help
--version	Show version
--verbose	Enable debug output
--no-update-check	Disable automatic version update check

Automatic Update Check

Waza checks for newer versions in the background when you run any command. If an update is available, a one-line notice is printed after the command output:

A newer version of waza is available: v0.24.0 → v0.28.0. Run: curl -fsSL ... | bash

• The check is non-blocking — it never slows down your command.
• Results are cached for 24 hours in ~/.waza/version-check.json.
• Disable with --no-update-check or set WAZA_NO_UPDATE_CHECK=1.

Environment Variables

Variable	Description
GITHUB_TOKEN	Token for Copilot SDK execution
COPILOT_CLI_PATH	Explicit path to a GitHub Copilot CLI binary. When unset, Waza uses its bundled CLI and does not fall back to an unrelated copilot on PATH.
COPILOT_HOME	Copilot configuration root. When set, the bundled CLI and runtime assets are installed under COPILOT_HOME/cache/copilot-sdk; otherwise the system user cache is used.
COPILOT_CLI_INSTALL_VERBOSE	Set to 1 to print SDK installation diagnostics when the bundled CLI cannot be installed.
COPILOT_BASE_URL / COPILOT_PROVIDER_BASE_URL	Custom Copilot SDK provider endpoint. When set, Waza skips the default Copilot auth check and passes provider config to the SDK.
COPILOT_PROVIDER / COPILOT_PROVIDER_TYPE	Provider type passed through to the Copilot SDK.
COPILOT_WIRE_API / COPILOT_PROVIDER_WIRE_API	Wire format passed through to the Copilot SDK, for example responses or completions, depending on provider.
COPILOT_API_KEY / COPILOT_PROVIDER_API_KEY	API key for the custom provider, if required.
COPILOT_BEARER_TOKEN / COPILOT_PROVIDER_BEARER_TOKEN	Bearer token for the custom provider, if required.
WAZA_HOME	Config directory (default: ~/.waza)
WAZA_CACHE	Cache directory (default: .waza-cache)
WAZA_NO_UPDATE_CHECK	Set to 1 to disable automatic version check

OpenTelemetry export is configured with the waza run --otel-* flags. v0.38.0 does not read WAZA_OTEL_* environment variables.

When a custom provider is active, usage output labels the SDK request counter as Provider Requests instead of Premium Requests. Result JSON records usage.provider: "custom" and a sanitized usage.provider_host; the full provider URL is not stored.

Next Steps

• Writing Eval Specs — Create benchmarks
• YAML Schema — eval.yaml format
• GitHub Repository — Source code