Getting Started

A complete walkthrough for installing waza and running your first evaluation.

Custom Agents

Waza can also evaluate VS Code custom agents defined in .agent.md files. This guide focuses on SKILL.md-based skills. For custom agents, see Evaluating Custom Agents after completing this walkthrough.

Installation

Choose one of three methods:

1. Binary Install (Recommended)

The fastest way to get started on macOS, Linux, or Windows Bash environments such as Git Bash, MSYS2, or Cygwin:

curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash

The script auto-detects the environment where Bash is running. If you run this from PowerShell and bash resolves to WSL, it installs the Linux binary inside WSL. For native Windows, download waza-windows-amd64.exe or waza-windows-arm64.exe from the newest stable vX.Y.Z CLI release on the Releases page, rename it to waza.exe, and place it in a directory on your PATH.

This downloads the latest standalone waza CLI release, verifies the checksum, and installs the waza binary to:

• /usr/local/bin/waza (if writable), or
• ~/bin/waza (if /usr/local/bin is not writable)

Verify installation:

waza --version

2. Install from Source

Requires Go 1.26 or later and Git LFS:

git clone https://github.com/microsoft/waza.git
cd waza
git lfs install
git lfs pull
go build -o waza ./cmd/waza
./waza --version

3. Azure Developer CLI Extension

If you use Azure Developer CLI (azd):

# Add the waza extension registry
azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json

# Install the extension
azd ext install microsoft.azd.waza

# Verify
azd waza --help

All waza commands are available under azd waza:

azd waza init my-project
azd waza run eval.yaml
azd waza serve

Automatic Update Notifications

Waza checks for new versions in the background. If an update is available, you’ll see a notice after command output. Disable with --no-update-check or WAZA_NO_UPDATE_CHECK=1.

Quick Start

Get a complete evaluation suite running in 5 minutes.

Step 1: Initialize a Project

Create a new directory and initialize a waza project:

mkdir my-eval-suite
cd my-eval-suite
waza init

You’ll be prompted to create your first skill. Enter a name like code-explainer:

my-eval-suite/
├── skills/
│   └── code-explainer/
│       └── SKILL.md
├── evals/
│   └── code-explainer/
│       ├── eval.yaml
│       ├── tasks/
│       │   ├── basic-usage.yaml
│       │   └── edge-cases.yaml
│       └── fixtures/
│           └── sample.py
├── .github/workflows/
│   └── eval.yml
└── .gitignore

Skip the prompt with --no-skill:

waza init --no-skill

Step 2: Create a New Skill

If you didn’t create one during init, add a new skill:

waza new skill code-analyzer

The interactive wizard collects metadata (name, description, use cases).

Step 3: Define Your Skill

Edit skills/code-explainer/SKILL.md:

---
name: code-explainer
description: |
  USE FOR: Explaining code, analyzing code patterns, refactoring suggestions
  DO NOT USE FOR: Running code, generating boilerplate
---

# Code Explainer

## Overview

Helps developers understand existing code by breaking down logic and identifying patterns.

## Usage

**Triggers:**
- "Explain this Python function"
- "What does this code do?"

Step 4: Write Evaluation Tasks

Edit evals/code-explainer/tasks/basic-usage.yaml:

id: basic-usage-001
name: Basic Usage - Python Function
description: Test that the skill explains a simple Python function correctly.
tags:
  - basic
  - happy-path
inputs:
  prompt: "Read sample.py and explain the function it contains."
  files:
    - path: sample.py
expected:
  output_contains:
    - "function"
    - "parameter"
    - "return"
  behavior:
    max_tool_calls: 5

Step 5: Add Test Fixtures

Create test files in evals/code-explainer/fixtures/sample.py:

def fibonacci(n):
    """Calculate the nth Fibonacci number."""
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

inputs.files copies each listed file from the suite’s fixtures/ directory into the fresh workspace for that task. File contents are not automatically added to the prompt, so name the file and ask the agent to read it, as in the task above.

Step 6: Configure Your Evaluation

Edit evals/code-explainer/eval.yaml:

name: code-explainer-eval
description: Evaluation suite for code-explainer skill
skill: code-explainer
version: "1.0"

config:
  trials_per_task: 1
  timeout_seconds: 300
  parallel: false
  executor: mock
  model: claude-sonnet-4.6

graders:
  - type: text
    name: explains_concepts
    config:
      pattern: "(?i)(function|variable|parameter|return)"

tasks:
  - "tasks/*.yaml"

Step 7: Run Evaluations

# Run all evaluations
waza run

# Run one skill's evaluations
waza run code-explainer

# Verbose output
waza run code-explainer -v

# Save results
waza run code-explainer -o results.json

Example output:

Running evaluations for code-explainer...
  ✓ basic-usage-001 passed
  ✓ edge-case-001 passed

Results: 2/2 tasks passed ✓

Step 8: Check Skill Readiness

Validate your skill is production-ready:

waza check code-explainer

Output:

🔍 Skill Readiness Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Skill: code-explainer

📋 Compliance Score: High
   ✅ Excellent! Your skill meets all compliance requirements.

✅ Your skill is ready for submission!

Step 9: View Results in Dashboard

Start the interactive web dashboard:

waza serve

Opens http://localhost:3000. View runs, detailed results, comparisons, and trends.

Next Steps

• Writing Eval Specs — Master YAML structure and task definitions
• Validators & Graders — Reference for all grader types
• Web Dashboard — Explore interactive results
• CLI Reference — All commands and flags
• GitHub Repository — Source code, examples, issues

Troubleshooting

Install script fails or binary not found

If the install script exits with a download error or 404, check the Releases page to confirm a binary for your platform is available in the newest stable vX.Y.Z CLI release. You can also install manually:

# Example: macOS ARM64 (Apple Silicon) — replace <version> with the latest tag (e.g. v0.34.0)
curl -fSL -o waza https://github.com/microsoft/waza/releases/download/<version>/waza-darwin-arm64
chmod +x waza
sudo mv waza /usr/local/bin/waza

Replace <version> and the platform suffix (darwin-arm64, linux-amd64, etc.) with the values that match your system.

“skill not found”

Make sure you’re in a project with skills/ directory or a standalone skill with SKILL.md.

“eval.yaml not found”

Check that:

• File is at evals/{skill-name}/eval.yaml (project mode)
• Or at {skill}/evals/eval.yaml (standalone)

“No tasks in eval.yaml”

Ensure your eval.yaml has:

tasks:
  - "tasks/*.yaml"

And that you have .yaml files in the tasks/ directory.

Support

• Issues: github.com/microsoft/waza/issues
• GitHub: github.com/microsoft/waza