CLI Commands

Complete reference for all waza CLI commands and their options.

Installation

curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
waza --version

waza run

Run an evaluation benchmark.

waza run [eval.yaml | skill-name]

Arguments

Argument	Description
[eval.yaml]	Path to evaluation spec file
[skill-name]	Skill name (auto-detects eval.yaml)
(none)	Auto-detect using workspace detection

Flags

Flag	Short	Type	Default	Description
--context-dir	-c	string	./fixtures	Fixtures directory path
--output	-o	string		Save results JSON to file
--output-dir	-d	string		Directory for structured output; each run creates a UTC-timestamped subdirectory of the specified directory. Mutually exclusive with --output.
--verbose	-v	bool	false	Detailed progress output
--parallel		bool	false	Run tasks concurrently
--workers	-w	int	4	Number of concurrent workers
--trials		int	config.trials_per_task	Run each task N times for flakiness detection (omit to use config.trials_per_task; when provided, value must be >= 1)
--task	-t	string		Filter tasks by name (repeatable)
--tags		string		Filter tasks by tags (repeatable)
--model	-m	string		Override model (repeatable)
--judge-model		string		Model for LLM-as-judge graders (overrides execution model)
--cache		bool	false	Enable result caching
--cache-dir		string	.waza-cache	Cache directory path
--format	-f	string	default	Output format: default, github-comment
--reporter		string[]		Output reporters: json, junit:<path> (repeatable)
--timeout		int	300	Task timeout in seconds
--baseline		bool	false	A/B testing mode — runs each task twice (without skill = baseline, with skill = normal) and computes improvement scores
--update-snapshots		bool	false	Update or create diff grader snapshot files to match current workspace output
--discover		string		Auto skill discovery — walks directory tree for SKILL.md + eval.yaml (root/tests/evals)
--strict		bool	false	Fail if any SKILL.md lacks eval coverage (use with --discover)
--suggest		bool	false	Generate a Copilot suggestion report based on test outcomes
--recommend		bool	false	Generate heuristic recommendation after multi-model run
--session-log		bool	false	Enable session event logging (NDJSON)
--session-dir		string	.	Directory for session log files
--no-summary		bool	false	Skip combined summary.json for multi-skill runs
--skip-graders		bool	false	Skip grading (execution only); grade later with waza grade
--no-skills		bool	false	Disable all skill loading for the evaluation
--transcript-dir		string		Save per-task transcript JSON files
--no-cache		bool	false	Explicitly disable result caching
--keep-workspace		bool	false	Preserve temp workspaces after execution for debugging

Examples

# Run all tasks
waza run eval.yaml -v

# Run specific skill
waza run code-explainer

# Specify fixtures directory
waza run eval.yaml -c ./fixtures -v

# Save results
waza run eval.yaml -o results.json

# Filter to specific tasks
waza run eval.yaml --task "basic*" --task "edge*"

# Multiple models (parallel)
waza run eval.yaml --model gpt-4o --model claude-sonnet-4.6

# Use a different judge model for LLM-as-judge graders
waza run eval.yaml --model gpt-4o --judge-model claude-opus-4.6

# Parallel execution with 8 workers
waza run eval.yaml --parallel --workers 8

# With caching
waza run eval.yaml --cache --cache-dir .waza-cache

# Generate JUnit XML for CI test reporting
waza run eval.yaml --reporter junit:results.xml

# A/B testing: baseline vs skill performance
waza run eval.yaml --baseline -o results.json
# Output includes improvement breakdown (quality, tokens, turns, time, completion)

# Auto-update diff grader snapshots
waza run eval.yaml --update-snapshots

# Auto skill discovery
waza run --discover ./skills/

# Auto discovery with strict mode (fail if any SKILL.md lacks eval coverage)
waza run --discover --strict ./skills/

# Skip grading, then grade separately
waza run eval.yaml --skip-graders -o results.json
waza grade eval.yaml --results results.json

# Session event logging
waza run eval.yaml --session-log --session-dir ./logs

# Multi-model comparison with recommendation
waza run eval.yaml --model gpt-4o --model claude-sonnet-4.6 --recommend

# Keep temp workspaces for debugging fixture issues
waza run eval.yaml --keep-workspace -v

waza init

Initialize a new waza project.

waza init [directory]

Arguments

Argument	Description
[directory]	Project directory (default: current dir)

Flags

Flag	Description
--no-skill	Skip first skill creation prompt

Creates

project-root/
├── skills/
├── evals/
├── .github/workflows/eval.yml
├── .gitignore
└── README.md

Examples

waza init my-project
waza init my-project --no-skill

waza new skill

Create a new skill.

waza new skill [skill-name]

Modes

Project mode (inside a skills/ directory):

cd my-skills-repo
waza new skill code-explainer
# Creates skills/code-explainer/SKILL.md + evals/code-explainer/

Standalone mode (no skills/ directory):

waza new skill my-skill
# Creates my-skill/ with all files

Flags

Flag	Description
--template	Template pack (coming soon)

Examples

# Interactive wizard
waza new skill code-analyzer

# Non-interactive (CI/CD)
waza new skill code-analyzer << EOF
Code Analyzer
Analyzes code for patterns and issues
code, analysis
EOF

waza new eval

Scaffold an eval suite for an existing skill.

waza new eval

Generate an eval scaffold from an existing SKILL.md.

waza new eval <skill-name>

Creates:

• evals/<skill-name>/eval.yaml
• evals/<skill-name>/tasks/positive-trigger-1.yaml
• evals/<skill-name>/tasks/positive-trigger-2.yaml
• evals/<skill-name>/tasks/negative-trigger-1.yaml

Flags

Flag	Description
--output	Custom path for generated eval.yaml; task YAMLs are created in a sibling tasks/ directory next to this file.

Examples

# Generate eval scaffold using skills/my-skill/SKILL.md
waza new eval my-skill

# Write eval.yaml to a custom location
waza new eval my-skill --output evals/custom/my-skill-eval.yaml

waza new task

Generate task YAML files from a recorded prompt execution.

waza new task from-prompt

Run your prompt through Copilot and generate a task file with inferred validators from the recorded session.

waza new task from-prompt <prompt> <task-path>

Arguments

Argument	Description
<prompt>	Prompt to execute during recording
<task-path>	Output path for the generated task YAML

Flags

Flag	Type	Default	Description
--model	string	claude-sonnet-4.5	Copilot model to use for recording
--testname	string	auto-generated-test	Test name and test ID for generated task
--tags	string[]		Tags to add to generated task
--timeout	duration	5m0s	Max time to allow prompt completion
--overwrite	bool	false	Overwrite output file if it already exists
--root	string	.	Directory used to discover skills

Examples

# Generate a task from a recorded prompt run
waza new task from-prompt "Refactor this function for readability" evals/code-explainer/tasks/refactor-readability.yaml

# Add metadata and allow file replacement
waza new task from-prompt "Explain this diff and risks" evals/code-explainer/tasks/diff-analysis.yaml \
  --testname diff-analysis \
  --tags recorded,regression \
  --overwrite

waza check

Validate skill compliance and readiness.

waza check [skill-name | skill-path]

Arguments

Argument	Description
[skill-name]	Skill name (e.g., code-explainer)
[skill-path]	Path to skill directory
(none)	Check all skills in workspace

Flags

Flag	Description
--verbose	Detailed compliance report
--format	Output format: text (default), json

Output

🔍 Skill Readiness Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Skill: code-explainer

📋 Compliance Score: High
   ✅ Excellent! Your skill meets all requirements.

📊 Token Budget: 420 / 500 tokens
   ✅ Within budget.

🧪 Evaluation Suite: Found
   ✅ eval.yaml detected.

✅ Your skill is ready for submission!

Examples

waza check code-explainer
waza check ./skills/code-explainer
waza check --verbose

waza compare

Compare evaluation results across models.

waza compare [results-1.json] [results-2.json] ...

Arguments

Argument	Description
[results-N.json]	Result files to compare (2+ required)

Flags

Flag	Description
--format	Output format: table (default), json

Examples

waza compare gpt4.json sonnet.json
waza compare gpt4.json sonnet.json opus.json
waza compare results-*.json --format json

waza coverage

Generate an eval coverage grid for discovered skills.

waza coverage [root]

Arguments

Argument	Description
[root]	Root directory to scan (default: current directory)

Flags

Flag	Description
-f, --format	Output format: text (default), markdown, json
--path	Additional directories to scan for skills/evals (repeatable)

Coverage Levels

• Full: Skill has an eval.yaml/eval.yml with tasks (via tasks: or tasks_from:) and at least 2 distinct grader types.
• Partial: Skill has an eval.yaml/eval.yml but fewer than 2 grader types or no tasks.
• Missing: No eval.yaml/eval.yml found for the skill.

Note: The reported coverage percentage reflects only fully covered skills (Fully Covered / Total Skills).

Examples

waza coverage
waza coverage --format markdown
waza coverage --format json
waza coverage --path custom-evals --path plugins

waza suggest

Generate suggested eval artifacts from a skill’s SKILL.md using an LLM.

waza suggest <skill-path>

Flags

Flag	Description
--model	Model to use for suggestions (default: project default model)
--dry-run	Print suggestions to stdout (default)
--apply	Write suggested files to disk
--output-dir	Output directory (default: <skill-path>/evals)
--format	Output format: yaml (default), json

Examples

# Preview suggestions
waza suggest skills/code-explainer --dry-run

# Write eval/task/fixture files
waza suggest skills/code-explainer --apply

# JSON output
waza suggest skills/code-explainer --format json

waza tokens

Token budget management.

waza tokens count

Count tokens in skill files.

waza tokens count [path]

waza tokens count skills/code-explainer/SKILL.md
waza tokens count skills/

waza tokens check

Check token usage against budget.

waza tokens check [skill-name]

waza tokens check code-explainer
# Output:
# code-explainer: 420 / 500 tokens (84%)
# ✅ Within budget

Token limits are resolved in priority order: .waza.yaml tokens.limits → .token-limits.json (deprecated; migrate to .waza.yaml) → built-in defaults. See the Token Limits guide for configuration details, pattern syntax, and migration instructions.

Configure per-file limits in .waza.yaml:

tokens:
  limits:
    defaults:
      "*.md": 2000
      "skills/**/SKILL.md": 5000
    overrides:
      "skills/complex-skill/SKILL.md": 7500

waza tokens compare

Compare markdown token counts between git refs. Supports general-purpose file-level comparison and skill-aware comparison with CI gating.

waza tokens compare [refs...]

# Compare HEAD to working tree (default)
waza tokens compare

# Compare a specific ref to working tree
waza tokens compare main

# Skill-aware comparison with CI threshold
waza tokens compare main --skills --threshold 10

# JSON output with strict absolute limits
waza tokens compare main --skills --threshold 10 --strict --format json

Flags: --format table|json, --show-unchanged, --strict, --skills, --threshold <percent>

Without --skills, compares all markdown files. With --skills, restricts comparison to SKILL.md files under configured skill roots (skills/, .github/skills/, and paths.skills from .waza.yaml). In skills mode the default base ref is origin/main (falling back to main).

--threshold sets a percentage-change gate for CI — newly added files are exempt from threshold checks (no baseline to compare) but still subject to absolute limit checks when --strict is set.

waza tokens profile

Structural analysis of SKILL.md files — reports token count, section count, code block count, and workflow step detection.

waza tokens profile [skill-name | path]

Flags: --format text|json, --tokenizer bpe|estimate

Example:

📊 my-skill: 1,722 tokens (detailed ✓), 8 sections, 4 code blocks
   ⚠️  no workflow steps detected

Warnings: no workflow steps, >2,500 tokens, fewer than 3 sections.

waza tokens suggest

Get optimization suggestions.

waza tokens suggest [skill-name]

Analyzes SKILL.md and suggests:

• Sections to shorten
• Removable content
• Restructuring opportunities

waza results

Manage evaluation results from cloud storage or local storage.

waza results list

List evaluation runs from configured cloud storage.

waza results list
waza results list --limit 20
waza results list --format json

Flags

Flag	Description
--limit <n>	Maximum results to display (default: 10)
--format	Output format: table or json (default: table)

waza results compare

Compare two evaluation runs side by side.

waza results compare run-id-1 run-id-2
waza results compare run-id-1 run-id-2 --format json

Flags

Flag	Description
--format	Output format: table or json (default: table)

waza cache

Manage evaluation result cache.

waza cache clear [--cache-dir=.waza-cache]

Subcommands

waza cache clear

Clear all cached evaluation results.

The cache stores test outcomes to speed up repeated evaluations with the same inputs. Cached results are keyed by spec configuration, task definition, model, and fixture file contents.

waza cache clear
waza cache clear --cache-dir /path/to/cache

Flags

Flag	Description
--cache-dir	Cache directory to clear (default: .waza-cache)

Examples

# Clear default cache
waza cache clear

# Clear custom cache directory
waza cache clear --cache-dir .my-cache

---

waza dev

Improve skill compliance iteratively.

waza dev [skill-name | skill-path]

Flags

Flag	Description
--target	Target level: low, medium, high
--max-iterations	Max improvement loops (default: 5)
--auto	Auto-apply without prompting
--fast	Skip integration tests

Workflow

waza dev code-explainer --target high --auto

Iteratively:

1. Scores current compliance
2. Identifies issues
3. Suggests improvements
4. Applies changes
5. Re-scores
6. Repeats until target reached

waza grade

Re-grade previous evaluation results without re-executing the agent.

waza grade <eval.yaml>

Flags

Flag	Short	Type	Default	Description
--results		string		Required. Path to waza run output JSON
--task		string		Grade a specific task ID only
--workspace		string	.	Agent workspace directory for file-based graders
--judge-model		string		Model for prompt graders (overrides execution model)
--output	-o	string		Write full EvaluationOutcome JSON
--verbose	-v	bool	false	Verbose output

Examples

# Grade all tasks from a previous run
waza grade eval.yaml --results results.json

# Grade a specific task
waza grade eval.yaml --results results.json --task "basic-function"

# Use a different judge model
waza grade eval.yaml --results results.json --judge-model claude-opus-4.6

# Save graded results for comparison
waza grade eval.yaml --results results.json -o graded.json

waza session

Manage and inspect session event logs.

waza session list

List session event logs in a directory.

waza session list [--dir <path>]

Flag	Type	Default	Description
--dir	string	.	Directory to search for session logs

waza session view

Render a session timeline from an NDJSON event log.

waza session view <session-file>

waza serve

Start the interactive dashboard.

waza serve

Flags

Flag	Description
--port	Port (default: 3000)
--tcp	TCP address for JSON-RPC (e.g., :9000)
--stdio	Use stdin/stdout for piping

Examples

waza serve                    # http://localhost:3000
waza serve --port 8080        # http://localhost:8080
waza serve --tcp :9000        # JSON-RPC TCP server

Graders

Waza supports multiple grader types for comprehensive evaluation. See the complete Grader Reference for detailed documentation.

Built-in Graders

Grader	Purpose
code	Python/JavaScript assertion-based validation
regex	Pattern matching in output
file	File existence and content validation
diff	Workspace file comparison with snapshots
behavior	Agent behavior constraints (tool calls, tokens, duration)
action_sequence	Tool call sequence validation with F1 scoring
skill_invocation	Skill orchestration sequence validation
prompt	LLM-as-judge evaluation with rubrics
tool_constraint	Validate tool usage constraints (e.g., required/forbidden tools, argument patterns)
trigger	Prompt trigger accuracy — detects whether a prompt should activate a skill

tool_constraint Grader

Validate agent tool usage constraints during evaluation.

graders:
  - type: tool_constraint
    name: check_tools
    config:
      expect_tools:
        - tool: "bash"                         # Required tool call
          command_pattern: "azd\\s+up"         # Optional regex on the command argument
        - tool: "skill"
          skill_pattern: "my-skill"            # Optional regex on the skill argument
        - tool: "edit"
          path_pattern: "\\.go$"               # Optional regex on the path argument
      reject_tools:
        - tool: "bash"                         # Prohibited when args match this pattern
          command_pattern: "rm\\s+-rf"         # Optional regex on the command argument
        - tool: "create_file"                  # Always prohibited

All config fields are optional. Omitted fields skip that constraint.

prompt Grader with Pairwise Mode

Use the prompt grader for LLM-as-judge evaluation. In pairwise mode, compare two approaches side-by-side to reduce position bias.

graders:
  - type: prompt
    name: code_quality_judge
    config:
      mode: pairwise                        # Enable pairwise comparison (requires --baseline flag)
      rubric: "Compare these solutions for code quality and correctness"
      max_tokens: 500

Requirements:

• Pairwise mode requires the --baseline flag on waza run
• Baseline execution must complete before pairwise comparison runs
• Each task is evaluated twice: once without the skill (baseline) and once with it (treatment)

Example:

waza run eval.yaml --baseline -o results.json
# Output includes pairwise judge scores comparing baseline vs treatment approaches

waza models

List models available for evaluation via the Copilot SDK.

waza models [flags]

Flags

Flag	Type	Default	Description
--json	bool	false	Output as JSON

Output

Displays a table with model ID, name, vision support, and context window size.

MODEL ID          NAME              VISION    CONTEXT WINDOW
──────────────────────────────────────────────────────────────
claude-sonnet-4   Claude Sonnet 4   no        200k
gpt-4o            GPT-4o            yes       128k

2 models available

Authentication

Requires Copilot authentication. If not authenticated, you will see:

Error: not authenticated — run "copilot login" first

Examples

# List all models in table format
waza models

# List models as JSON (for scripting)
waza models --json

Exit Codes

Code	Meaning
0	Success
1	One or more tasks failed
2	Configuration or runtime error

Global Flags

Flag	Description
--help	Show help
--version	Show version
--verbose	Enable debug output
--no-update-check	Disable automatic version update check

Automatic Update Check

Waza checks for newer versions in the background when you run any command. If an update is available, a one-line notice is printed after the command output:

A newer version of waza is available: v0.24.0 → v0.28.0. Run: curl -fsSL ... | bash

• The check is non-blocking — it never slows down your command.
• Results are cached for 24 hours in ~/.waza/version-check.json.
• Disable with --no-update-check or set WAZA_NO_UPDATE_CHECK=1.

Environment Variables

Variable	Description
GITHUB_TOKEN	Token for Copilot SDK execution
WAZA_HOME	Config directory (default: ~/.waza)
WAZA_CACHE	Cache directory (default: .waza-cache)
WAZA_NO_UPDATE_CHECK	Set to 1 to disable automatic version check

Next Steps

• Writing Eval Specs — Create benchmarks
• YAML Schema — eval.yaml format
• GitHub Repository — Source code