Quick Start

Start with Concepts if suite/task/trial or scenario is new to you. This walkthrough is offline harness validation, not proof of agent quality.

1. Select the right build

Published standalone v0.39.0 (October 20, 2026) includes the scenario/preflight/evidence journey below. Install it from Releases or run waza update; no source build is required. If you prefer to build from source, use Go 1.26+, Git LFS, Make, Node.js and npm:

git clone https://github.com/microsoft/waza.git
cd waza
git lfs install
git lfs pull
make build
./waza --version
export PATH="$PWD:$PATH"

make build installs locked dashboard dependencies and generates real embedded assets before Go compilation; bare go build/go install in a fresh checkout is not this procedure. See Releases for stable installers, native Windows binaries and separate azd extension availability. The shell examples use Bash; on Windows use a suitable development shell. Selected-file publication and strict durable collection have narrower platform support than ordinary execution.

2. Scaffold a project and scenario

Run outside the source checkout, in a new directory:

waza init workflow-lab --no-skill
cd workflow-lab
waza new eval inventory --scenario --template repository
waza preflight evals/inventory/eval.yaml

Preflight makes no agent/model calls and executes no graders or subprocesses. Warnings about unresolved runtime readiness are expected; exit zero means configuration inventory readiness, not execution quality. --strict deliberately makes unresolved prerequisites fail.

The scenario has evals/inventory/eval.yaml, task YAML under evals/inventory/tasks/, and sanitized fixtures under evals/inventory/fixtures/. For existing skills, the supported alternative is:

waza new skill my-skill

That creates skills/my-skill/SKILL.md and evals/my-skill/{eval.yaml,tasks/,fixtures/} — not skill.yaml or nested skills/my-skill/evals/. init in a TTY can prompt to create a skill; --no-skill skips it. There is no init --interactive or init --path.

3. Run and inspect offline evidence

Edit evals/inventory/eval.yaml: change config.executor: copilot-sdk to config.executor: mock. Executor selection is configuration, not a run --engine flag. Leave all graders intact.

waza run evals/inventory/eval.yaml \
  --snapshot ./snapshots --output ./results.json

Expect exit 1: the mock does not write the required report.txt. This is a useful negative control, not a broken installation. The mock echoes prompt/task metadata and does not solve the workflow. Step 4 supplies a positive wiring control. Inspect results.json: tasks[].runs[] contains individual observations, validations, source evidence and snapshot_path when capture succeeded. Do not infer missing state from a passing summary.

With jq installed, inspect the recorded path and check its consistency:

snapshot=$(jq -r '.tasks[0].runs[0].snapshot_path' results.json)
waza replay "$snapshot" --json
waza serve --results-dir .

Replay checks recorded events/scores; it does not rerun the agent or graders. Review snapshots for secrets before sharing; capture is local and opt-in. See Snapshot & Replay for redaction, Windows publication limits, selected-file capture and regrade restrictions.

4. Prove a predicate can reject bad output

Keep the wiring control in its own suite so it cannot enter inventory’s tasks/*.yaml selection:

mkdir -p evals/docs-smoke/tasks

Create evals/docs-smoke/tasks/docs-probe.yaml:

id: docs-probe
name: Offline response wiring
inputs:
  prompt: "Reply with the word inventory."

Create evals/docs-smoke/eval.yaml. Task entries are file paths, not inline task objects; native text predicates use regex_match, not pattern:

schemaVersion: "2.0"
name: docs-smoke
scenario: Offline response wiring
config:
  executor: mock
  trials_per_task: 1
  timeout_seconds: 30
  model: mock
metrics:
  - name: task_completion
    weight: 1
    threshold: 1
graders:
  - name: response-wiring
    type: text
    config:
      regex_match: ["^Mock response for:"]
tasks:
  - tasks/docs-probe.yaml

waza preflight evals/docs-smoke/eval.yaml
waza run evals/docs-smoke/eval.yaml --output smoke.json

Expect exit 0. Change only the regex to ["^IMPOSSIBLE_DOCS_SENTINEL$"] and run again: expect exit 1 and a missing-pattern failure. An impossible pattern: field can be inert in legacy decoding; never use it as a text assertion. This negative control is essential: rendering YAML is not testing the check. Do not add this reply-only task to evals/inventory/tasks/: inventory’s native workflow graders require a report. Its original task set must stay unchanged when you later restore the real executor.

Next: choose actual agent execution

Getting Started explains supported authentication, permissions, model access and possible costs before real SDK commands. Skills and custom agents remain supported; CLI and MCP scenarios do not require a skill. Then learn the 13 native graders, project defaults and CI regression gates.