Getting Started

A workflow-first path from offline harness validation to real agent evaluation. Read Concepts and complete the credential-free Quick Start first.

Availability: standalone stable is v0.39.0 (October 20, 2026). Scenario authoring, preflight and selected evidence are included in this release. Install the published binary or run waza update; source builds are optional. Explicit opt-in and platform limits still apply. See the version/platform matrix.

Installation

Choose one of three methods:

1. Binary Install (Recommended)

The fastest way to get started on macOS, Linux, or Windows Bash environments such as Git Bash, MSYS2, or Cygwin:

curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash

The script auto-detects the environment where Bash is running. If you run this from PowerShell and bash resolves to WSL, it installs the Linux binary inside WSL. For native Windows, download waza-windows-amd64.exe or waza-windows-arm64.exe from the newest stable vX.Y.Z CLI release on the Releases page, rename it to waza.exe, and place it in a directory on your PATH.

This downloads the latest standalone waza CLI release, verifies the checksum, and installs the waza binary to:

/usr/local/bin/waza (if writable), or
~/bin/waza (if /usr/local/bin is not writable)

Verify installation:

waza --version

2. Install from Source

Requires Go 1.26 or later, Git LFS, Make, Node.js, and npm. The source build generates the dashboard assets before embedding them in the binary:

git clone https://github.com/microsoft/waza.git
cd waza
git lfs install
git lfs pull
make build
./waza --version

3. Azure Developer CLI Extension

If you use Azure Developer CLI (azd):

# Add the waza extension registry
azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json

# Install the extension
azd ext install microsoft.azd.waza

# Verify
azd waza --help

Extension artifacts are versioned separately from standalone main. The registry lists through 0.38.9; published 0.38.10 is a separate extension artifact. No upcoming command parity is claimed. Check azd waza --version and azd waza <command> --help before using:

azd waza init my-project
azd waza run eval.yaml
azd waza serve

Authenticate before real execution

The offline Quick Start needs no credentials. For copilot-sdk, you need GitHub Copilot access, permission to use the selected model and runtime permissions suitable for the task. Install a supported standalone GitHub Copilot CLI using its official installation instructions, then authenticate that installation:

copilot --version
copilot login

Waza’s embedded runtime is extracted to its private cache; it does not automatically install a copilot executable on PATH. Waza checks SDK authentication status and supports the standalone CLI login route. COPILOT_CLI_PATH explicitly selects another supported CLI binary; extraction failures do not silently fall back to PATH. Custom-provider setups can instead configure COPILOT_BASE_URL/COPILOT_PROVIDER_BASE_URL and their corresponding provider credentials. Consult CLI runtime options before selecting one; never paste credentials into task YAML.

Usage and permissions: real runs and prompt judges may incur model/provider charges or premium-request credits. Use an authorized test repository and review tool access first. --workers, cooperative timeouts and token-content limits are not a provider spending cap. Inspect finalized credit/currency attribution in results; missing or provisional usage is not zero cost.

Run a real workflow

Continue in the project created by Quick Start and reuse its inventory scenario. Only scaffold when that file does not already exist:

if [ ! -f evals/inventory/eval.yaml ]; then
  waza new eval inventory --scenario --template repository
fi

Before running, edit evals/inventory/eval.yaml and restore config.executor: copilot-sdk if Quick Start changed it to mock. Keep the separate wiring-control task under evals/docs-smoke/, not inventory’s tasks. After completing the authentication, model-access, permissions and cost prerequisites above:

waza run evals/inventory/eval.yaml --output agent-results.json

The restored config.executor: copilot-sdk runs a real agent against sanitized fixtures. CLI and MCP templates are also available: CLI uses installed Git; MCP uses a harness-only mocked dependency, not a production service. config.executor: mock validates the harness only. Scenario evals require explicit schema 2.0; tasks/results remain 1.x. See Writing Eval Specs for optional skill/agent context.

Keep using skills and custom agents

The same runtime prerequisites apply to skills and .agent.md custom agents. The following skill path uses an explicit mock first; switch to copilot-sdk only after reviewing the authentication and usage section above.

Step 1: Initialize a Project

Create a new directory and initialize a waza project:

waza init my-eval-suite --no-skill
cd my-eval-suite
waza new skill code-explainer

This creates the project and skill without a duplicate interactive skill prompt:

my-eval-suite/
├── skills/
│   └── code-explainer/
│       └── SKILL.md
├── evals/
│   └── code-explainer/
│       ├── eval.yaml
│       ├── tasks/
│       │   ├── basic-usage.yaml
│       │   └── edge-cases.yaml
│       └── fixtures/
│           └── sample.py
├── .github/workflows/
│   └── eval.yml
└── .gitignore

Skip the prompt with --no-skill:

waza init --no-skill

Step 2: Create a New Skill

If you didn’t create one during init, add a new skill:

waza new skill another-skill

Use a different name only if you need another skill; continue with code-explainer below.

Step 3: Define Your Skill

Edit skills/code-explainer/SKILL.md:

---
name: code-explainer
description: |
  USE FOR: Explaining code, analyzing code patterns, refactoring suggestions
  DO NOT USE FOR: Running code, generating boilerplate
---

# Code Explainer

## Overview

Helps developers understand existing code by breaking down logic and identifying patterns.

## Usage

**Triggers:**
- "Explain this Python function"
- "What does this code do?"

Step 4: Write Evaluation Tasks

Edit evals/code-explainer/tasks/basic-usage.yaml:

id: basic-usage-001
name: Basic Usage - Python Function
description: Test that the skill explains a simple Python function correctly.
tags:
  - basic
  - happy-path
inputs:
  prompt: "Read sample.py and explain how the function works, what each parameter means, and how its return value is computed. Include any edge cases."
  files:
    - path: sample.py
expected:
  output_contains:
    - "function"
    - "parameter"
    - "return"
  behavior:
    max_tool_calls: 5

Step 5: Add Test Fixtures

Create test files in evals/code-explainer/fixtures/sample.py:

def fibonacci(n):
    """Calculate the nth Fibonacci number."""
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

inputs.files copies each listed file from the suite’s fixtures/ directory into the fresh workspace for that task. File contents are not automatically added to the prompt, so name the file and ask the agent to read it, as in the task above.

Step 6: Configure Your Evaluation

Edit evals/code-explainer/eval.yaml:

name: code-explainer-eval
description: Evaluation suite for code-explainer skill
skill: code-explainer
version: "1.0"

config:
  trials_per_task: 1
  timeout_seconds: 300
  parallel: false
  executor: mock
  model: claude-sonnet-4.6
metrics:
  - name: task_completion
    weight: 1
    threshold: 1

graders:
  - type: text
    name: explains_concepts
    config:
      regex_match: ["(?i)(function|variable|parameter|return)"]

tasks:
  - "tasks/basic-usage.yaml"

Step 7: Run Evaluations

# Run all evaluations
waza run

# Run one skill's evaluations
waza run code-explainer

# Verbose output
waza run code-explainer -v

# Save results
waza run code-explainer -o results.json

Expect exit 0 and 1/1 selected tasks passed: basic-usage-001. The explanation predicate applies only to the task edited above. Other scaffolded tasks remain on disk but are not selected by this suite; give them their own appropriate checks before including them. This mock pass checks response/grader wiring, not whether an agent can explain code.

Step 8: Check Skill Readiness

Check static skill compliance (not production readiness or safety):

waza check code-explainer

Output:

🔍 Skill Readiness Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Skill: code-explainer

📋 Compliance Score: High
   ✅ Excellent! Your skill meets all compliance requirements.

✅ Your skill is ready for submission!

Step 9: View Results in Dashboard

Start the interactive web dashboard:

waza serve

Opens http://localhost:3000. View runs, detailed results, comparisons, and trends.

Next Steps

Writing Eval Specs — Master YAML structure and task definitions
Validators & Graders — Reference for all grader types
Web Dashboard — Explore interactive results
CLI Reference — All commands and flags
GitHub Repository — Source code, examples, issues

Troubleshooting

Install script fails or binary not found

If the install script exits with a download error or 404, check the Releases page to confirm a binary for your platform is available in the newest stable vX.Y.Z CLI release. You can also install manually:

# Example: macOS ARM64 (Apple Silicon) — replace <version> with the latest tag (e.g. v0.34.0)
curl -fSL -o waza https://github.com/microsoft/waza/releases/download/<version>/waza-darwin-arm64
chmod +x waza
sudo mv waza /usr/local/bin/waza

Replace <version> and the platform suffix (darwin-arm64, linux-amd64, etc.) with the values that match your system.

“skill not found”

Make sure you’re in a project with skills/ directory or a standalone skill with SKILL.md.

“eval.yaml not found”

Check that:

File is at evals/{skill-name}/eval.yaml (project mode)
Or at {skill}/evals/eval.yaml (standalone)

“No tasks in eval.yaml”

Ensure your eval.yaml has:

tasks:
  - "tasks/*.yaml"

And that you have .yaml files in the tasks/ directory.

Support

Issues: github.com/microsoft/waza/issues
GitHub: github.com/microsoft/waza