waza

The way of technique — measure, refine, master 🥋

Waza is a Go CLI for evaluating agent workflows, skills and custom agents. Define repository, CLI or MCP scenarios in YAML, run graded trials and inspect the recorded evidence. Existing skill suites remain supported.

Perfect for skill authors, platform teams, and developers building AI-powered applications.

What can waza do?

Define Workflows

Author scenario suites for repository, CLI and MCP tasks, optionally with skill or custom-agent context.

Run Benchmarks

Execute evaluations against different AI models with fixture isolation and pluggable validators.

Compare Results

Cross-model comparison to measure skill effectiveness and track improvements.

Validate Quality

Use 13 native grader types to inspect task output, files, tool observations and behavior.

View Metrics

Interactive web dashboard with live results, trends, detailed analysis, and exports.

Integrate with CI/CD

Automated evaluation runs on pull requests with GitHub Actions integration.

Quick Start

Start credential-free with the offline Quick Start. After installing v0.39.0 or building from source as described there:

waza init my-eval-suite --no-skill
cd my-eval-suite
waza new eval inventory --scenario --template repository
waza preflight evals/inventory/eval.yaml
# Continue in Quick Start: explicitly select mock in YAML before any run.

Quick Start Guide

Concepts and workflow choices

Full Getting Started

A mock pass proves harness wiring, not agent quality. Before real SDK execution, read the authentication, permissions and possible-cost prerequisites in Getting Started. Preflight is inventory, not execution or release assurance.

Key Features

Structured YAML Benchmarks

Define evaluation specs, tasks, and validation rules in simple YAML:

After scaffolding code-explainer, use the Getting Started explanation task and sample.py fixture. This suite selects only tasks/basic-usage.yaml, not the unrelated scaffolded edge-case/negative tasks. With the mock executor, expect exit 0 and 1/1 selected tasks passed (basic-usage-001): a wiring check, not agent quality. The code grader requires Python.

name: code-explainer-eval
description: Test agent's ability to explain code
skill: code-explainer

config:
  executor: mock
  trials_per_task: 1
  model: claude-sonnet-4.6
  timeout_seconds: 300
metrics:
  - name: task_completion
    weight: 1
    threshold: 1

graders:
  - type: text
    name: checks_logic
    weight: 2.0
    config:
      regex_match: ["(?i)(function|logic|parameter)"]
  - type: code
    name: has_output
    config:
      assertions:
        - "len(output) > 100"

tasks:
  - "tasks/basic-usage.yaml"

Pluggable Validators

13 native grader kinds (regex and keyword checks are part of text):

Code — Python assertions
Text — Substring and regex matching
File — Output file checking
Diff — Diff comparison
JSON Schema — JSON structure validation
Prompt — LLM-powered evaluation
Behavior — Agent behavior validation
Action Sequence — Tool call sequence validation
Skill Invocation — Skill invocation validation
Program — Program execution validation
Trigger — Skill activation checks
Tool Constraint — Required/forbidden tool constraints
Tool Calls — Tool events and argument checks

Fixture Isolation

Each task gets a fresh temp workspace with fixtures copied in. Original fixtures never modified.

Cross-Model Comparison

Test skills against multiple AI models:

waza run eval.yaml --model gpt-4o -o gpt4.json
waza run eval.yaml --model claude-sonnet-4.6 -o sonnet.json
waza compare gpt4.json sonnet.json

Web Dashboard

Serve interactive results:

waza serve
# opens http://localhost:3000 — view runs, comparisons, trends, and live updates

CI/CD Ready

Integrated GitHub Actions support for automated evaluation runs.

Questions? Open an issue or check the GitHub repository.