CI/CD Integration

Automate waza evaluations in your CI/CD pipeline to catch regressions early and validate agent behavior across models.

Explicit fixed-design release assessment

Ordinary gate defaults and exits stay unchanged. Explicitly select a new protocol collection with:

waza gate --release-policy policy.json --collection-dir new-collection --format github-actions

Selected invalid, rejected, incomplete or inconclusive evidence exits 1. Do not combine this mode with legacy gate policy flags or manufacture receipts from historical result files. Use compare-plan and compare-collect before assessment. They default to the supported offline mock path; a mock run is not agent-quality evidence.

Design sampling clusters, retry policy, weights, independent-execution justification, endpoint, family bound and precision before collection. Retries/tasks inside a cluster are not extra independent samples. Required golden tasks must pass their first attempt in both arms, even when retry recovery is the selected endpoint. Required unavailable billing or assurance cannot strict-pass. See controlled comparisons.

Explicit native Copilot collection

The native SDK collector is never selected by default. Opt in on both planning and collection only when provider usage is intended:

Do not run these native commands as part of offline example checks; collection may incur provider usage.

waza compare-plan --collector native-copilot \
  --baseline-eval baseline/eval.yaml --candidate-eval candidate/eval.yaml \
  --design design.json --requirements requirements.json \
  --output native-policy.json
waza compare-collect --collector native-copilot \
  --baseline-eval baseline/eval.yaml --candidate-eval candidate/eval.yaml \
  --release-policy native-policy.json --collection-dir new-native-collection
waza gate --release-policy native-policy.json \
  --collection-dir new-native-collection --format json

This profile supports fixed named models, one turn, no skills, deny-all client tools and native text/string checks only. It may incur provider usage; token and AI-credit limits are final-observation budgets with best-effort start checks, not a hard spend cap, and asynchronous overshoot is possible. Unknown required prior usage stops later starts. Runtime version, provider currency and authenticated assurance are unavailable. Native durable collection is supported only on Linux/macOS. Client-side tool denial is not server isolation. The bundled mock example is intentionally not runnable with this collector.

Independent offline assurance in CI

Required assurance remains nonpassing in the frozen base 1.0 decision and API/dashboard. To select the independent outer profile, first create a new exclusive contract with compare-assurance-plan --output ... --release-policy ..., current per-arm --baseline-eval / --candidate-eval, and --baseline-references / --candidate-references. Optional context overrides retain their existing resolution. Collect with explicit --assurance-contract plus --release-policy, then assess with those flags, --collection-dir and current per-arm inputs.

# Illustrative authored source paths; not a ready-to-pass bundled fixture.
waza gate --release-policy policy.json \
  --assurance-contract assurance-contract.json \
  --collection-dir new-assured-collection \
  --baseline-eval baseline/eval.yaml --candidate-eval candidate/eval.yaml \
  --baseline-references baseline/references.json \
  --candidate-references candidate/references.json --format json

Use JSON for the outer machine-readable contract or default human output; do not assume selected assurance reproduces legacy CI annotation/markdown layouts. Selected nonpass exits 1. Both fresh exact offline1.0 reports, deterministic regrading of retained actual responses and every base non-assurance condition must pass. A saved report label is not current evidence. Recheck current source/evidence/review after verify and before publication/ assessment; use the same bound executable throughout.

Per-arm --review files and --accept-review-source acceptance are separate current inputs, not authenticated human review. Supply selected inputs at each stage; examples stay unreviewed and test reviews are explicitly synthetic. Only mock/no-skills/native-text/finite-authored-output qualification is supported. No live SDK assurance, calibration, paid1.1/billing support or automatic paid validation. Local hashes establish consistency, not authenticity or globally witnessed precommitment. See the profile limits.

Installation in CI

Via azd Extension (Recommended)

If your CI environment supports the Azure Developer CLI, use the waza azd extension:

# GitHub Actions example
- uses: Azure/setup-azd@v2

- run: |
    azd config set alpha.extensions on
    azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json
    azd ext install microsoft.azd.waza

- run: azd waza run ./evals/my-skill/eval.yaml -v

Via Binary

Download the latest waza binary for your platform:

# Linux / macOS
curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
export PATH="$HOME/bin:$PATH"

For Windows, download waza.exe from the latest GitHub release assets and add its directory to PATH.

Via Source

Requires Go 1.26+, Git LFS, Make, Node.js and npm. Generate the real dashboard assets before Go embedding; go build alone is not a fresh-checkout setup:

git clone https://github.com/microsoft/waza.git
cd waza
git lfs install
git lfs pull
make build

Offline setup gate

Available in v0.39.0: install the published standalone release or run waza update before using preflight or controlled policy commands. Explicit policy selection and platform limits still apply. The azd extension is versioned separately; see availability.

waza preflight eval.yaml --format json > preflight.json

This setup gate makes no model/live-service calls and launches no subprocesses or mock servers. Invalid config exits 1. Credentials, live readiness, interpreter syntax, and external state stay unresolved; default warnings exit 0. Use --strict only when you deliberately want those unresolved/unsupported prerequisites to fail with exit 1. It is not a release-assurance or runtime requirement-enforcement policy and does not change existing run/gate exits. See Offline Eval Preflight for diagnostics and limitations.

GitHub Actions

Offline challenge assessment

Use waza assure eval.yaml --references labels.json as a separate strict file-content challenge gate. It does not change grade, gate, golden-task or evaluation exits. Its standalone report preserves actual observations, scoped requirements, evidence bindings, review eligibility and finite-case coverage. Missing/unknown evidence, unreviewed labels and critical false acceptance fail the gate. Supply a current review source explicitly; do not promote bundled candidate labels to reviewed truth in automation.

The existing assure --calibrate remains unavailable: requesting it makes no paid calls and returns not assessed, with unknown usage/credits remaining null. This offline gate does not certify external state, runtime history or statistical confidence.

Do not select assure calibrate --accept-paid-calls in an offline CI gate: it is a distinct paid-operation acknowledgment, not ordinary assurance or generic noninteractive consent. The explicit command still requires eligible current review-source acceptance and a rubric root; job ceilings do not cap provider credits or money. Deterministic CI uses injected engines and a rejecting runtime guard instead.

The Offline Grader Assurance repository workflow separately tests the evaluator calibration producer through injected deterministic engines. It proves a rejecting native-runtime guard before exporting actual API reports, requires exact frozen offline-reader rejection of calibrated version 1.1 with a passing offline version-1.0 control, and checks the same returned reports in the explicitly selected browser inspector. Fixed test review declarations are synthetic, not authenticated human labels. The job does not configure a production factory, invoke a paid model or certify runtime/server isolation. Full Go and compatibility gates remain independent.

The same guarded workflow exports actual positive/nonpass preserved-native mechanical 1.2 reports and proves that exact frozen 1.0 and opt-in 1.1 browser readers reject them, retaining actual positive controls for both older profiles. It does not rewrite versions or use metadata-only stand-ins. The operation evaluates the current selected grader, not historical grader correspondence, and is not added to default CLI/dashboard dispatch.

Private engine-free qualification controls run separately under that same absolute rejecting runtime guard. Local journal concurrency, sync-failure and receipt/clock tests exercise the accepted local implementation and injected authority seams; they do not authenticate human review, establish global reservation or authorize a native lifecycle. Unsupported backend platforms must remain explicit nonpass, not successful durability fallbacks.

Quick Start ΓÇö Scaffold with waza init

The waza init command generates a ready-to-use workflow:

waza init my-project

This creates .github/workflows/eval.yml with sensible defaults.

Complete Example: Multi-Model Testing on PR

name: Evaluate Skills

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
    paths:
      - 'evals/**'
      - 'skills/**'
      - '.github/workflows/eval.yml'

permissions:
  contents: read
  pull-requests: write

jobs:
  evaluate:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        model: [gpt-4o, claude-sonnet-4.6, claude-opus-4]
      max-parallel: 3
    timeout-minutes: 30

    steps:
      - uses: actions/checkout@v4

      - name: Install waza
        run: |
          curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
          echo "$HOME/bin" >> $GITHUB_PATH

      - name: Run evaluations (${{ matrix.model }})
        run: |
          waza run \
            --model "${{ matrix.model }}" \
            --output "results-${{ matrix.model }}.json" \
            --verbose

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: results-${{ matrix.model }}
          path: results-${{ matrix.model }}.json
          retention-days: 30

      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('results-${{ matrix.model }}.json', 'utf8'));
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `### Eval Results: ${{ matrix.model }}\n\n✅ **Passed:** ${results.summary.succeeded}\n❌ **Failed:** ${results.summary.failed}`
            });

Recurring Regression Loop (Scheduled)

Waza includes a production-ready recurring loop at .github/workflows/weekly-regression-loop.yml. It runs weekly, archives date-stamped results, compares against the latest successful baseline artifact, and upserts a follow-up issue when regressions are detected.

Key properties:

• Runs on schedule and workflow_dispatch
• Always uploads artifacts (results, session logs, regression report)
• Uses deterministic comparability checks (eval_name, model_id, runs_per_test)
• Flags regressions on increased failures/errors, passed→failed flips, and score/success-rate drops beyond thresholds

To run it manually:

gh workflow run weekly-regression-loop.yml --repo microsoft/waza

Regression gates with waza gate

For a focused PR gate, use waza gate. It compares a baseline against the current run and exits with stable codes (0 pass, 1 regression, 2 golden failure, 3 config error), making it easy to wire into any CI system.

Mark business-critical tasks in your eval YAML with golden: true and they must always pass:

tasks:
  - id: critical-flow
    golden: true
    prompt: "..."
    grader:
      type: exact_match
      expected: "OK"

GitHub Actions — annotations + step summary, with the baseline pulled from the default branch:

- name: Download baseline from main
  uses: dawidd6/action-download-artifact@v6
  with:
    workflow: eval.yml
    branch: main
    name: results
    path: baseline/

- name: Run current evaluation
  run: waza run evals/main.yaml --output results.json

- name: Regression gate
  run: |
    waza gate \
      --baseline baseline/results.json \
      --current results.json \
      --max-regression-pct 5 \
      --on-new-tasks allow \
      --on-removed-tasks warn \
      --format github-actions

The github-actions format emits ::error:: / ::warning:: / ::notice:: annotations and appends a markdown summary to $GITHUB_STEP_SUMMARY. Non-zero exit fails the step automatically.

Azure DevOps — exit codes drive task status; task.logissue surfaces inline warnings/errors:

- script: |
    waza gate \
      --baseline $(Pipeline.Workspace)/baseline/results.json \
      --current $(Build.ArtifactStagingDirectory)/results.json \
      --max-regression-pct 5 \
      --format markdown > $(Build.ArtifactStagingDirectory)/gate-report.md
    code=$?
    cat $(Build.ArtifactStagingDirectory)/gate-report.md
    if [ $code -eq 2 ]; then
      echo "##vso[task.logissue type=error]Golden task failure"
    elif [ $code -eq 1 ]; then
      echo "##vso[task.logissue type=error]Regression exceeded threshold"
    elif [ $code -eq 3 ]; then
      echo "##vso[task.logissue type=error]waza gate configuration error"
    fi
    exit $code
  displayName: 'Regression gate'

- publish: $(Build.ArtifactStagingDirectory)/gate-report.md
  artifact: gate-report
  condition: always()

Token Budget Checks

Use waza tokens compare to track token usage across PRs and fail if budgets are exceeded:

- name: Check token budget
  run: |
    waza tokens compare origin/main HEAD \
      --format table \
      --strict

This compares token counts between your main branch and current branch. Exit code is non-zero if any limit is exceeded with --strict.

Spec Coverage Checks

Use waza spec verify to ensure eval tasks cover the SKILL.md description, USE FOR triggers, DO NOT USE FOR triggers, and parameter blocks. --format github-actions emits annotations on uncovered requirements, and --fail exits non-zero when uncovered requirements meet the threshold.

- name: Verify SKILL.md eval coverage
  run: |
    waza spec verify skills/my-skill evals/my-skill/eval.yaml \
      --fail \
      --threshold 1 \
      --format github-actions

Smart Path Filtering

Run evaluations only when relevant files change:

on:
  pull_request:
    branches: [main]
    paths:
      - 'evals/**'           # Eval config changed
      - 'skills/**'          # Skill code changed
      - 'examples/**'        # Examples changed
      - '.github/workflows/eval.yml'  # Workflow itself changed

jobs:
  evaluate:
    runs-on: ubuntu-latest

Caching Fixtures and Results

Cache fixture data to speed up repeated evaluations:

- name: Restore fixture cache
  uses: actions/cache@v4
  with:
    path: .waza-cache
    key: waza-cache-${{ hashFiles('examples/**/fixtures/**') }}
    restore-keys: waza-cache-

- name: Run with cache
  run: |
    waza run \
      --cache \
      --cache-dir .waza-cache \
      --verbose

Azure DevOps Pipelines

Basic Pipeline YAML

trigger:
  branches:
    include:
      - main
      - develop
  paths:
    include:
      - evals/**
      - skills/**
      - azure-pipelines.yml

pr:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: EvaluateSkills
    displayName: Run Waza Evaluations
    timeoutInMinutes: 30
    strategy:
      matrix:
        GPT4:
          MODEL: 'gpt-4o'
        Claude:
          MODEL: 'claude-sonnet-4.6'

    steps:
      - checkout: self
        fetchDepth: 0  # Full history for diffs

      - script: |
          curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
          echo "##vso[task.prependpath]$HOME/bin"
        displayName: 'Install waza'

      - script: |
          waza run \
            --model "$(MODEL)" \
            --output "results-$(MODEL).json" \
            --verbose
        displayName: 'Run evaluations ($(MODEL))'
        env:
          GITHUB_TOKEN: $(System.AccessToken)

      - task: PublishBuildArtifacts@1
        inputs:
          pathToPublish: 'results-$(MODEL).json'
          artifactName: 'eval-results-$(MODEL)'
          publishLocation: 'Container'
        condition: always()

      - script: |
          echo "##vso[task.logissue type=error;]Evaluations failed"
          exit 1
        condition: failed()
        displayName: 'Report failure'

Token Budget in Azure DevOps

- script: |
    waza tokens compare origin/main HEAD \
      --format table \
      --strict
  displayName: 'Check token budget'

GitLab CI

Complete Pipeline

GitLab CI is highly portable and works anywhere your code lives:

stages:
  - evaluate

evaluate:
  stage: evaluate
  image: ubuntu:latest
  timeout: 30m
  parallel:
    matrix:
      - MODEL: gpt-4o
      - MODEL: claude-sonnet-4.6
      - MODEL: claude-opus-4

  before_script:
    - apt-get update && apt-get install -y curl ca-certificates
    - curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
    - export PATH="$HOME/bin:$PATH"

  script:
    - waza run --model $MODEL --output results-$MODEL.json --verbose

  artifacts:
    paths:
      - results-$MODEL.json
    expire_in: 30 days
    when: always

  only:
    variables:
      - $CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop"
    changes:
      - evals/**
      - skills/**
      - .gitlab-ci.yml

  retry:
    max: 2
    when: script_failure

Best Practices

1. Run on Specific Triggers

Avoid running expensive evaluations on every commit. Use path filters:

on:
  pull_request:
    paths:
      - 'evals/**'      # Only if eval files changed
      - 'skills/**'     # Or if skill code changed

2. Matrix Testing for Multiple Models

Test across models in parallel to catch model-specific issues:

strategy:
  matrix:
    model: [gpt-4o, claude-sonnet-4.6]
  max-parallel: 2  # Limit concurrency to avoid rate limits

3. Artifact Retention

Keep results long enough to analyze trends but not forever:

retention-days: 30  # 30 days for typical workflows

4. Timeout Configuration

Set reasonable task timeouts to avoid hanging CI jobs:

# In eval.yaml
config:
  timeout_seconds: 300  # 5 minutes per task

# In CI workflow
timeout-minutes: 30     # 30 minutes total

5. Fail Fast on Critical Tests

Stop CI early on critical evaluation failures:

# In eval.yaml
config:
  fail_fast: true  # Stop on first failure

6. Token Budget Enforcement

Prevent models from exceeding token budgets in PRs:

- name: Check token budget
  run: waza tokens compare origin/main HEAD --format table --strict

7. Cache Results Intelligently

Cache fixture data across runs, but invalidate on eval config changes:

- uses: actions/cache@v4
  with:
    path: .waza-cache
    key: waza-cache-${{ hashFiles('evals/**/*.yaml') }}

8. Secure Secrets

Never hardcode API keys. Use CI secrets:

GitHub Actions

- name: Run evaluations
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: waza run -v

Azure DevOps

- script: waza run -v
  env:
    OPENAI_API_KEY: $(OPENAI_API_KEY)

GitLab CI

script:
  - waza run -v
variables:
  OPENAI_API_KEY: $OPENAI_API_KEY

9. Safe Auto-Merge Boundaries

If you enable auto-merge for low-risk changes, gate it with explicit safeguards:

• Require a dedicated label (for example, agent-merge)
• Restrict to trusted author associations (owner/member/collaborator)
• Restrict to your protected default branch (main)
• Keep branch protection and required checks enabled

In this repository, those boundaries are enforced in .github/workflows/auto-merge.yml.

Troubleshooting

“waza: command not found”

Ensure the binary is in your PATH after installation:

# GitHub Actions
echo "$HOME/bin" >> $GITHUB_PATH

# Azure DevOps
echo "##vso[task.prependpath]$HOME/bin"

# GitLab CI (in before_script)
export PATH="$HOME/bin:$PATH"

Evaluations Timeout

Increase the timeout in both your CI workflow and eval config:

# CI level: GitHub Actions
timeout-minutes: 45

# Eval config level
config:
  timeout_seconds: 600  # 10 minutes per task

High Token Usage

Use waza tokens check to audit token consumption before running full evaluations:

waza tokens check ./skills/my-skill/

Rate Limiting from API Calls

Reduce parallel workers and add rate-limit awareness:

config:
  parallel: false  # Run tasks sequentially

Or use --workers 1 in your CI command.

Advanced Workflows

Comparing Models with Baseline

Use the baseline flag in your eval spec to establish a baseline, then compare new runs:

baseline-eval.yaml

baseline: true
config:
  model: claude-sonnet-4.6

Then in CI, compare with:

waza compare baseline-results.json candidate-results.json

Multi-Stage Pipeline: PR → Staging → Production

# GitHub Actions example
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  pr-eval:
    name: "Fast eval (PR)"
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
          $HOME/bin/waza run --tags "smoke" -v

  full-eval:
    name: "Full eval (after merge)"
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
          $HOME/bin/waza run -v

Next Steps

• Writing Eval Specs ΓÇö Configure your evaluations
• CLI Reference ΓÇö All flags and commands
• Graders Guide ΓÇö Set up scoring rules
• Examples ΓÇö Real eval setups