About

What is Waza?

Waza (技 - Japanese for “skill/technique”) is a Go CLI for evaluating agent workflows, skills and custom agents. Repository, CLI and MCP scenarios make workflow behavior first-class without requiring a SKILL.md.

Start with Concepts and the offline Quick Start, then configure an authenticated model-backed run. Published v0.39.0 includes these workflows; see release availability and platform limits.

The Problem

Evaluating an agent workflow requires more than checking its final response: teams need repeatable tasks, meaningful graders, and evidence they can inspect across runs. Skills and custom agents can be part of that workflow, but are not required.

The Solution

Waza supports a workflow from an offline harness check to agent-backed evaluation:

Phase	Capability
Scaffold	Create a suite for a repository, CLI, MCP, skill or custom-agent scenario
Preflight	Check configuration and dependencies without running an agent
Test	Run mock trials or explicitly configured agent-backed evaluations
Evaluate	Grade results, inspect recorded evidence and compare runs

Mock results check harness wiring, not agent quality. Real agent runs require appropriate authentication, permissions and model access, and may incur costs.

Architecture

Waza is built in Go for:

• Bundled core runtime; selected graders and dependencies can need interpreters or services
• Fast execution
• Cross-platform compatibility (Linux, macOS, Windows)

Components

• cmd/waza/: CLI entrypoint and commands.
• internal/: execution engines, models, orchestration, graders and other packages.
• web/: dashboard (React and Tailwind).
• examples/: example evaluations and scenarios.

Design Principles

1. Fixture isolation — Task workspaces protect original fixtures; isolation is not a security sandbox.
2. Pluggable graders — 13 native grader kinds check different aspects of a run.
3. Cross-model evaluation — Compare configured agent runs across supported models and providers.
4. Offline-first checks — Use preflight and mock execution before opting into real runs.
5. Inspectable evidence — Review recorded results and events in the CLI or dashboard; unavailable evidence remains explicit.

Key Features

Structured Benchmarks

Define test cases in YAML:

name: code-explainer-eval
tasks:
  - "tasks/*.yaml"

graders:
  - type: text
    config:
      regex_match: ["function", "parameter"]

13 Native Grader Kinds

• Code — Python assertions
• Text — Text matching
• File — Output file checking
• Diff — Diff comparison
• JSON Schema — JSON structure validation
• Prompt — LLM-powered evaluation
• Behavior — Agent behavior validation
• Action Sequence — Tool call sequence validation
• Skill Invocation — Skill invocation validation
• Program — Program execution validation
• Trigger — Skill activation checks
• Tool Constraint — Required/forbidden tool constraints
• Tool Calls — Tool events and argument checks

Multi-Model Comparison

waza run eval.yaml --model gpt-4o -o gpt4.json
waza run eval.yaml --model claude-sonnet-4.6 -o sonnet.json
waza compare gpt4.json sonnet.json

Interactive Dashboard

waza serve

Explore results, trends, and comparisons in a web interface.

CI/CD Ready

Pre-configured GitHub Actions workflow:

waza init my-project
# Creates .github/workflows/eval.yml

Contributing

We welcome contributions! Here’s how to get involved:

Report Issues

Found a bug or have a feature request?

1. Check existing issues
2. Open a new issue with clear reproduction steps
3. Include error logs and environment details

Contribute Code

1. Fork the repository
2. Create a branch for your feature: git checkout -b feature/my-feature
3. Make changes following the code style
4. Write tests for new functionality
5. Run linter and tests:

   make lint
   make test

6. Commit with clear messages: feat: Add my feature
7. Push to your fork and open a pull request

Development Setup

# Clone repository
git clone https://github.com/microsoft/waza.git
cd waza

# Set up Go environment
go version  # Requires 1.26+
git lfs install
git lfs pull

# Build (Make, Node.js 22.23.4+ on 22.x or 24+, npm 9.6.5+; generates dashboard assets)
make build

# Test
make test

# Lint
make lint

# Run
./waza --help

Code Style

• Follow Go idioms (Effective Go)
• Use gofmt for formatting
• Include unit tests for new code
• Document public functions

Adding Validators

To add a native grader:

1. Implement Grader in internal/graders/.
2. Add the parameter type in internal/models/grader_params.go and the kind/schema entry.
3. Add the matching Create() factory case and tests.
4. Update the README, canonical grader catalog and configuration examples.

Documentation

• Update relevant docs when changing behavior
• Add examples for new features
• Keep README.md and GUIDE.md in sync

Community

Discussions

Have questions or ideas? Join the conversation:

• GitHub Discussions — Ask questions, share ideas
• GitHub Issues — Report bugs, request features

Support

• Documentation: docs/
• Examples: examples/

Roadmap

See the active feature tracker #657 and delivery tracker #866. The original platform roadmap #8 is closed and retained for historical context.

License

MIT License

Author

Shayne Boyer (@spboyer)

Maintained by the waza team and community contributors.

---

Questions? Open an issue or start a discussion.

---

Inspiration

The waterfall timeline visualization in the waza dashboard was inspired by the .NET Aspire distributed application dashboard, which provides a similar trace/span view for distributed systems observability.