CI/CD Integration

Automate waza evaluations in your CI/CD pipeline to catch regressions early and validate agent behavior across models.

Explicit fixed-design release assessment

Ordinary gate defaults and exits stay unchanged. Explicitly select a new protocol collection with:

waza gate --release-policy policy.json --collection-dir new-collection --format github-actions

Selected invalid, rejected, incomplete or inconclusive evidence exits 1. Do not combine this mode with legacy gate policy flags or manufacture receipts from historical result files. Use compare-plan and compare-collect before assessment. They default to the supported offline mock path; a mock run is not agent-quality evidence.

Design sampling clusters, retry policy, weights, independent-execution justification, endpoint, family bound and precision before collection. Retries/tasks inside a cluster are not extra independent samples. Required golden tasks must pass their first attempt in both arms, even when retry recovery is the selected endpoint. Required unavailable billing or assurance cannot strict-pass. See controlled comparisons.

Explicit native Copilot collection

The native SDK collector is never selected by default. Opt in on both planning and collection only when provider usage is intended:

Do not run these native commands as part of offline example checks; collection may incur provider usage.

waza compare-plan --collector native-copilot \
  --baseline-eval baseline/eval.yaml --candidate-eval candidate/eval.yaml \
  --design design.json --requirements requirements.json \
  --output native-policy.json
waza compare-collect --collector native-copilot \
  --baseline-eval baseline/eval.yaml --candidate-eval candidate/eval.yaml \
  --release-policy native-policy.json --collection-dir new-native-collection
waza gate --release-policy native-policy.json \
  --collection-dir new-native-collection --format json

This profile supports fixed named models, one turn, no skills, deny-all client tools and native text/string checks only. It may incur provider usage; token and AI-credit limits are final-observation budgets with best-effort start checks, not a hard spend cap, and asynchronous overshoot is possible. Unknown required prior usage stops later starts. Runtime version, provider currency and authenticated assurance are unavailable. Native durable collection is supported only on Linux/macOS. Client-side tool denial is not server isolation. The bundled mock example is intentionally not runnable with this collector.

Independent offline assurance in CI

Required assurance remains nonpassing in the frozen base 1.0 decision and API/dashboard. To select the independent outer profile, first create a new exclusive contract with compare-assurance-plan --output ... --release-policy ..., current per-arm --baseline-eval / --candidate-eval, and --baseline-references / --candidate-references. Optional context overrides retain their existing resolution. Collect with explicit --assurance-contract plus --release-policy, then assess with those flags, --collection-dir and current per-arm inputs.

# Illustrative authored source paths; not a ready-to-pass bundled fixture.
waza gate --release-policy policy.json \
  --assurance-contract assurance-contract.json \
  --collection-dir new-assured-collection \
  --baseline-eval baseline/eval.yaml --candidate-eval candidate/eval.yaml \
  --baseline-references baseline/references.json \
  --candidate-references candidate/references.json --format json

Use JSON for the outer machine-readable contract or default human output; do not assume selected assurance reproduces legacy CI annotation/markdown layouts. Selected nonpass exits 1. Both fresh exact offline1.0 reports, deterministic regrading of retained actual responses and every base non-assurance condition must pass. A saved report label is not current evidence. Recheck current source/evidence/review after verify and before publication/ assessment; use the same bound executable throughout.

Per-arm --review files and --accept-review-source acceptance are separate current inputs, not authenticated human review. Supply selected inputs at each stage; examples stay unreviewed and test reviews are explicitly synthetic. Only mock/no-skills/native-text/finite-authored-output qualification is supported. No live SDK assurance, calibration, paid1.1/billing support or automatic paid validation. Local hashes establish consistency, not authenticity or globally witnessed precommitment. See the profile limits.

Installation in CI

Via azd Extension (Recommended)

If your CI environment supports the Azure Developer CLI, use the waza azd extension:

# GitHub Actions example
- uses: Azure/setup-azd@v2

- run: |
    azd config set alpha.extensions on
    azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json
    azd ext install microsoft.azd.waza

- run: azd waza run ./evals/my-skill/eval.yaml -v

Via Binary

Download the latest waza binary for your platform:

# Linux / macOS
curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
export PATH="$HOME/bin:$PATH"

For Windows, download waza.exe from the latest GitHub release assets and add its directory to PATH.

Via Source

Requires Go 1.26+, Git LFS, Make, Node.js 22.23.4+ (22.x) or 24+ (recommended), and npm 9.6.5+. Generate the real dashboard assets before Go embedding; go build alone is not a fresh-checkout setup:

git clone https://github.com/microsoft/waza.git
cd waza
git lfs install
git lfs pull
make build

Offline setup gate

Available in v0.39.0: install the published standalone release or run waza update before using preflight or controlled policy commands. Explicit policy selection and platform limits still apply. The azd extension is versioned separately; see availability.

waza preflight eval.yaml --format json > preflight.json

This setup gate makes no model/live-service calls and launches no subprocesses or mock servers. Invalid config exits 1. Credentials, live readiness, interpreter syntax, and external state stay unresolved; default warnings exit 0. Use --strict only when you deliberately want those unresolved/unsupported prerequisites to fail with exit 1. It is not a release-assurance or runtime requirement-enforcement policy and does not change existing run/gate exits. See Offline Eval Preflight for diagnostics and limitations.

GitHub Actions

Repository CI performance evidence

Waza’s existing native Go shards, ordinary Linux/Windows and preview Go jobs, and dashboard Chromium job retain sanitized ci-evidence-* JSON artifacts for 14 days, named by run and attempt (and native shard or ordinary job). They capture the actual checkout SHA separately from the workflow SHA, event, platform, runtime versions, execution policy and observed dependency-cache hits. Missing cache outputs stay null; configured caching is not proof of a hit or miss. No model calls or additional full-suite run are added. Native exhaustive inventory and atomic-coverage verification remain independent required gates. Release remaining-package jobs and their native shards share the guarded capture and 14-day retention. This describes the workflow wiring, not proof of hosted release execution or performance-target attainment.

Go discovery uses the existing shard .all/.selected inventories. Top-level tests, nested tests and packages have separate observed/executed/pass/fail/skip counts. Complete Go admission requires a bounded, unique discovery inventory containing every selected target; a shard may select a proper subset. The root ordinary capture sidecars coverage.txt.all, coverage.txt.selected and coverage.txt.evidence.json are ignored generated outputs so they do not mark a subsequently built release binary’s source dirty. Unrelated files remain visible. top_level_seconds sums only top-level target durations, never their overlapping subtests; it is not wall-clock latency and parallel top-level tests can overlap. Playwright discovery is a separate --list operation, not another test run. Each browser result retains its retry index, so a final pass cannot replace a recorded initial failure. Missing, fractional, duplicate or nonconsecutive retry indices make the evidence incomplete; indices must start at zero. A no-browser producer contract test checks the current installed Playwright reporter using tiny pass/fail/skip controls before the real suite. Go execution and collector process exits are recorded separately from emitted test verdicts. A command failure after passing JSON records is not promoted to success; late failure without a terminal failing package is explicit incomplete evidence. Original command failures still determine the CI exit. Finalization failures cannot replace an earlier producer or collector failure. Until execution exits are finalized, a Go report remains explicitly incomplete. Go test attribute payloads are discarded, not measurements; malformed attributes and unknown actions remain diagnostics. A package pass cannot supply missing per-test terminal verdicts or timings: those captures remain incomplete. Go compiler ImportPath records retain build diagnostics in the console, not the uploaded report; build failures remain explicit incomplete evidence. Browser evidence records the actual discovery and execution step outcomes. Failed discovery still attempts sanitized retention, including any available inventory; missing execution remains unavailable, not zero. A failing browser command with no terminal failure evidence is incomplete, while a fully observed test failure remains complete failure evidence. Baselines require successful recorded discovery and execution outcomes.

The collectors omit stdout, error messages, stack traces and attachment payloads from these new artifacts. Raw Playwright discovery and execution JSON remain outside the uploaded test-results directory; only their sanitized report is added to retained artifacts. Existing console output and ordinary failure artifacts are unchanged. Inputs and individual outputs are bounded to 64 MiB, records to 1 MiB, and target inventories to 100,000. Malformed/truncated input, missing durations, unfinished tests, selection mismatches and cancellations are explicit diagnostics with complete: false, not invented zero timings or success. Once Go report admission reaches its total cap, valid bounded output records continue to the console without adding measurements to the incomplete report. Capture and discovery forward one bounded console record at a time, waiting for write acknowledgment and backpressure drain before reading another record. Output errors or premature closure fail the collector; capture saves available evidence with explicit diagnostics and complete: false. Discovery also forwards ordinary package-output diagnostics, not only compiler build-output events; discovered test names remain in the inventory. Package failures are recorded while the bounded JSON stream drains, so later packages’ diagnostics survive; failed discovery never publishes an inventory. Hard input limits, malformed/truncated records and output errors still reject discovery. collector_seconds records parser wall time (including input wait for streaming Go capture and console backpressure), not incremental instrumentation overhead. Go collector_cpu_seconds observes the collector process’s own user/system CPU through final report preparation. It excludes child-process startup and artifact upload, and is not a claim about added full-pipeline latency.

Finalize a completed run without rerunning tests

Automatic job capture cannot observe its own final completion or upload. After completion, run this read-only command from the instrumented checkout:

node .github/scripts/ci-evidence.cjs actions RUN_ID actions-evidence.json

It reads only microsoft/waza Actions metadata through the existing gh identity. Original attempt failures/cancellations remain visible even if the final run is green. Retained job IDs are counted once across retries. The report separates first-attempt creation-to-start delay, each attempt’s new-job span and retained-job span, longest observed step, summed elapsed runner-minutes by runner OS label, and source-bound artifact bytes. New-job span excludes earlier retained jobs. Job span is the observed critical-path envelope, not a reconstructed dependency DAG. Retry-request queue time is unavailable: the original creation timestamp must not be reused for retries. Job attempts must be positive safe integers no later than the queried attempt; valid jobs retained from earlier attempts remain admissible. Job IDs must be positive safe integers. Nonpositive integer IDs remain readable but cannot supply complete aggregates or eligible baseline/history samples. Invalid or duplicate job IDs, unverified job sources, conflicting retained jobs, or incomplete job pagination withhold the affected attempt’s span metrics. Elapsed runner-minutes and artifact bytes are not billed cost. Malformed step rows or invalid step numbers, names, or conclusions produce explicit incomplete diagnostics, including when a job was retained from an earlier attempt. Unavailable step durations remain null rather than zero; valid neighboring observations and independent job timings remain readable. Such runs cannot contribute eligible baseline/history samples. An observed, complete empty artifact inventory records zero bytes. An inventory that was not queried, failed, or remained incomplete records unavailable bytes, not zero; unrelated job diagnostics do not erase an observed artifact total. Artifact IDs must be unique positive safe integers; byte sizes and their sum must be nonnegative safe integers. Overlapping pages, invalid metadata, or changing pagination totals withhold the aggregate and baseline eligibility; readable test observations remain available as ineligible evidence.

Collection is capped at 10 attempts and 10 pages of 100 records per jobs/artifacts endpoint. An API failure, cap, missing timestamp or unverified source produces explicit incomplete evidence; the completed-run collector writes the available report and exits nonzero. Expired artifacts can retain metadata bytes but their test content is unavailable. Invalid attempt-response bodies produce attempt-api-invalid, retain unavailable start-time evidence, and do not prevent collection of jobs, artifacts, later attempts or other runs. Thrown attempt requests retain the separate attempt-api-failed diagnostic. Readable observations remain ineligible when their run metadata is incomplete. Upload steps use always() where supported; cancellation, timeout, setup failure or runner loss can still prevent capture/upload. No cancellation-upload guarantee is made. No new workflow, token grant or publication bypass is installed.

Opt-in observed cold starts

Existing Go and dashboard workflows declare an optional boolean cold_observation, defaulting to false; reusable native/dashboard calls accept the same optional input. Ordinary runs keep their existing cache semantics. The Go dispatch declaration is inactive until separately reviewed default-branch activation. A declaration on a feature branch alone does not authorize or enable dispatch. Reusable dashboard capture retains the caller’s actual workflow/event; called PR validation is not an independent non-PR cold cohort.

This first opt-in implementation supports ephemeral hosted Linux runners only: remaining Go tests, all four native shards, and dashboard validation. Windows continues ordinary caching and reporting; there is no Windows cold-start claim. Opt-in disables setup-go restore/save, setup-node’s explicit npm cache, and setup-node v7 automatic package-manager caching. Disabling actions alone does not manufacture a cache miss.

Before dependencies, the rejecting runtime guard, vet or discovery, the producer exclusively creates a private invocation root under RUNNER_TEMP and empty GOMODCACHE/GOCACHE stores, plus an npm store where applicable. It probes the actual Go/npm paths under the candidate environment before publishing GITHUB_ENV. Ambient stores are neither inspected nor reset. Native shards report npm as not-applicable. Only complete bound observations produce new isolated-store false values; unavailable observations remain null, diagnosed and ineligible. Existing absent/null/recorded-false reader behavior is preserved.

The optional schema-1 cache_observation records mode: "isolated-empty", phase: "before-dependencies", Linux scope, disabled restore/save, all applicable stores’ initial entry counts and actual-path matches, and repository/checkout/workflow/run/attempt/job binding. Actual runner ImageOS/ImageVersion and toolchain identity split cohorts when they change. Private absolute paths and invocation names are not public evidence or cohort keys.

Initialization precedes warming within the job; these are initial cold-store observations, not wholly cold test-only durations. Populated private stores and partial initialization residue remain for ephemeral host teardown, with a normalized RUNNER_TEMP locator on failure. State/environment write, close, ownership and path failures fail explicitly. This is not a rooted race-free, transactional, durable or cancellation-cleanup guarantee.

Release and insider validation capture their existing remaining-package command once, using the shared nonexecuting discovery and explicit -count=1 execution. They retain sanitized attempt/job artifacts without changing native selection, race/atomic coverage, rejecting-runtime checks or publication prerequisites. Release capture does not opt release jobs into cold mode. A first genuinely eligible public ingestion and three distinct comparable observations remain separate prerequisites; neither opt-in configuration nor fixture controls prove baseline/history acceptance or timing attainment.

Non-blocking trends and reviewed timing history

Finalize up to 10 distinct completed run IDs, without downloading ordinary logs or transcript artifacts:

node .github/scripts/ci-evidence.cjs trends RUN_ID_1,RUN_ID_2,RUN_ID_3 trends.json
node .github/scripts/ci-evidence.cjs history RUN_ID_1,RUN_ID_2,RUN_ID_3 history-proposal.json

These commands read only the named sanitized artifacts from microsoft/waza, verify the GitHub artifact SHA-256 digest and exact single JSON member in memory (no archive extraction), and bind run/attempt/workflow source before admitting any observation. Local JSON declarations cannot establish this binding. Declared event and workflow names must match the authenticated run metadata. Missing/expired artifacts, missing server digests and source mismatches are unavailable, not eligible samples. The collection is bounded to 100 reports and 128 MiB aggregate compressed/expanded input; individual expanded files remain bounded to 64 MiB. Expanded size is reserved before decompression or JSON parsing and remains charged when either fails. Deflate output is limited to its declared expanded size; dishonest size declarations are rejected, not decoded against the larger per-file cap. Complete Go artifacts must supply bounded, typed target rows consumed by history. Complete browser artifacts also require bounded result rows with recorded durations, consecutive retry indices, matching discovered/selected inventories, consistent project/retry policy, and recomputed target/result counts. Malformed reports are rejected individually without discarding other available observations. Unavailable schema 1 partial targets remain unavailable, not fabricated empty arrays. Standard pull-request checkout uses GitHub’s synthetic merge commit, whereas the Actions run API can report the PR source head. These are distinct identities. When that binding cannot be established, authenticated artifact observations are retained with checkout_binding: "unavailable-pr-merge" and the diagnostic pull-request-merge-checkout-binding-unavailable; they are not invalidated or silently assigned the source-head SHA. They cannot supply eligible baseline or history samples. This initial collector does not verify historical synthetic merge refs. Cohorts separate kind, checkout SHA, workflow, event/ref, OS/architecture, exact runtimes, selection, execution policy and cache observations. Duplicate run/attempt/job observations do not add samples. Eligibility requires complete successful first attempts, disabled test-result caching and retries, and observed misses for dependency caches. Unknown cache/runtime/source evidence is ineligible. Both Go and browser samples require a recognized Go-version response. Invalid new probe responses are discarded with diagnostics; historical schema 1 runtime values remain unchanged but cannot qualify when missing or unrecognized. Three distinct comparable runs, not three retries/tests within one run, are necessary before assessing any timing target. The initial summary always reports attainment: "not-assessed"; it installs no hard-fail timing budget.

Regression budget policy

trends also publishes budget_policy (schema 1). It evaluates two separate budgets, which never change the exit code:

Budget	Sample	Cohort	Tolerance
Latency (cohorts[].latency_budget)	Eligible test execution seconds (Go top_level_seconds, Playwright runner_seconds)	Evidence cohort above	20%
Runner consumption (runner_consumption[])	runner_elapsed_minutes_by_os for one OS label	Workflow, event, checkout SHA, OS label and the exact set of report cohorts in the run	20%

The candidate is the eligible sample with the highest run ID. The baseline is the median of every earlier eligible sample in the same cohort. A budget reports regressed when the candidate exceeds baseline × (1 + tolerance), and within otherwise. It reports insufficient-samples until at least three baseline samples exist, so a decision needs four comparable runs.

Runner-consumption samples come only from complete, successful, single-attempt non-PR runs with a verified 40-hex head SHA. The run must have succeeded or skipped every job and produced at least one verified evidence report. Runner minutes are elapsed job time, not billed cost.

Promoting a budget to a required check is a separate reviewed change. That change must keep these baseline, cohort and tolerance rules. Artifact names alone do not authenticate an upload attempt. Runs whose API records more than one attempt retain available observations but cannot supply eligible baseline/history samples, even if an artifact claims attempt 1.

history-proposal.json writer schema 2 retains individual Go top-level measurements with exact run/attempt/checkout provenance and verified report-content SHA-256. Each observation’s cohort_id joins to one content-addressed entry in cohorts; the complete selection is stored once per cohort, not once per target. The writer rejects unsupported proposal versions, changed cohort content and missing or source-mismatched references before writing. There is no proposal import or automatic planner TSV update. Go and browser artifact readers remain schema 1; unsupported artifact versions are rejected, not reinterpreted. Browser capture observes go version and the checkout at the CLI boundary, without invoking subprocesses in the browser JSON parser. Browser baselines require both observed Go and Playwright versions; missing or invalid Go versions remain explicitly ineligible, including in retained schema-1 artifacts. Review the complete source-bound artifacts and final Actions report before applying a same-cohort refresh to .github/scripts/go_test_shard_durations.tsv in a normal reviewed change. The canonical planner still uses authoritative discovery, longest-first assignment and the median historical duration for unknown targets (1 second only with empty history). History never admits or removes tests, and this instrumentation does not establish #825/#822 timing attainment. Locally authored reports and unverified/expired artifact content must not be promoted to execution evidence.

Offline challenge assessment

Use waza assure eval.yaml --references labels.json as a separate strict file-content challenge gate. It does not change grade, gate, golden-task or evaluation exits. Its standalone report preserves actual observations, scoped requirements, evidence bindings, review eligibility and finite-case coverage. Missing/unknown evidence, unreviewed labels and critical false acceptance fail the gate. Supply a current review source explicitly; do not promote bundled candidate labels to reviewed truth in automation.

The existing assure --calibrate remains unavailable: requesting it makes no paid calls and returns not assessed, with unknown usage/credits remaining null. This offline gate does not certify external state, runtime history or statistical confidence.

Do not select assure calibrate --accept-paid-calls in an offline CI gate: it is a distinct paid-operation acknowledgment, not ordinary assurance or generic noninteractive consent. The explicit command still requires eligible current review-source acceptance and a rubric root; job ceilings do not cap provider credits or money. Deterministic CI uses injected engines and a rejecting runtime guard instead.

The Offline Grader Assurance repository workflow separately tests the evaluator calibration producer through injected deterministic engines. It proves a rejecting native-runtime guard before exporting actual API reports, requires exact frozen offline-reader rejection of calibrated version 1.1 with a passing offline version-1.0 control, and checks the same returned reports in the explicitly selected browser inspector. Fixed test review declarations are synthetic, not authenticated human labels. The job does not configure a production factory, invoke a paid model or certify runtime/server isolation. Full Go and compatibility gates remain independent.

The same guarded workflow exports actual positive/nonpass preserved-native mechanical 1.2 reports and proves that exact frozen 1.0 and opt-in 1.1 browser readers reject them, retaining actual positive controls for both older profiles. It does not rewrite versions or use metadata-only stand-ins. The operation evaluates the current selected grader, not historical grader correspondence, and is not added to default CLI/dashboard dispatch.

Private engine-free qualification controls run separately under that same absolute rejecting runtime guard. Local journal concurrency, sync-failure and receipt/clock tests exercise the accepted local implementation and injected authority seams; they do not authenticate human review, establish global reservation or authorize a native lifecycle. Unsupported backend platforms must remain explicit nonpass, not successful durability fallbacks.

Quick Start — Scaffold with waza init

The waza init command generates a ready-to-use workflow:

waza init my-project

This creates .github/workflows/eval.yml with sensible defaults.

Complete Example: Multi-Model Testing on PR

name: Evaluate Skills

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
    paths:
      - 'evals/**'
      - 'skills/**'
      - '.github/workflows/eval.yml'

permissions:
  contents: read
  pull-requests: write

jobs:
  evaluate:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        model: [gpt-4o, claude-sonnet-4.6, claude-opus-4]
      max-parallel: 3
    timeout-minutes: 30

    steps:
      - uses: actions/checkout@v4

      - name: Install waza
        run: |
          curl -fsSL \
            https://raw.githubusercontent.com/microsoft/waza/v0.41.0/.github/scripts/install-waza-release.sh \
            -o "$RUNNER_TEMP/install-waza-release.sh"
          bash "$RUNNER_TEMP/install-waza-release.sh" 0.41.0

      - name: Run evaluations (${{ matrix.model }})
        id: run-eval
        continue-on-error: true
        run: |
          rm -f -- "results-${{ matrix.model }}.json"
          waza run \
            --model "${{ matrix.model }}" \
            --output "results-${{ matrix.model }}.json" \
            --verbose

      - name: Validate result contract
        id: validate
        if: always()
        run: |
          curl -fsSL \
            https://raw.githubusercontent.com/microsoft/waza/v0.41.0/.github/scripts/validate-waza-result.sh \
            -o "$RUNNER_TEMP/validate-waza-result.sh"
          run_exit=1
          if [ "${{ steps.run-eval.outcome }}" = success ]; then run_exit=0; fi
          bash "$RUNNER_TEMP/validate-waza-result.sh" \
            "results-${{ matrix.model }}.json" 0 "$run_exit"

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: results-${{ matrix.model }}
          path: results-${{ matrix.model }}.json
          retention-days: 30
          if-no-files-found: error

      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('results-${{ matrix.model }}.json', 'utf8'));
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `### Eval Results: ${{ matrix.model }}\n\n✅ **Passed:** ${results.summary.succeeded}\n❌ **Failed:** ${results.summary.failed}`
            });

      - name: Enforce evaluation status
        if: always()
        env:
          RUN_OUTCOME: ${{ steps.run-eval.outcome }}
          VALIDATION_OUTCOME: ${{ steps.validate.outcome }}
          EVALUATION_STATUS: ${{ steps.validate.outputs.status }}
        run: |
          if [ "$RUN_OUTCOME" != "success" ] || [ "$VALIDATION_OUTCOME" != "success" ] || [ "$EVALUATION_STATUS" != "success" ]; then
            exit 1
          fi

Removing the old result before execution prevents a stale file from satisfying the artifact step. Treat a missing or incomplete result as a workflow failure, including when the CLI itself fails; upload the fresh artifact with if: always() before re-emitting the evaluation failure.

Workflow success requires all configured trials and every recorded final and checkpoint grader to pass, without grader evaluation errors or failed/stopped checkpoints. Contradictory task/run success statuses remain diagnostic evidence, not successful workflow outputs.

The shared validator is shipped with v0.41.0; publish that release before using these cross-repository examples. Repository automatic checks build the checked-out Go source with real dashboard assets, use the mock code-explainer suite and a rejecting Copilot runtime, and do not download a mutable release. An omitted context override leaves fixture resolution to the eval spec; an explicit override is forwarded unchanged (including nonexistent paths, which the CLI must report rather than silently ignore).

The repository’s waza-eval.yml uploads the selected JSON separately, including explicit hidden output paths. It stages only the exact regular result file in a fresh directory, preserving the basename, so custom paths are not upload patterns. Missing files, symlink results, or staging failures fail the workflow rather than uploading stale paths. Optional non-hidden transcripts use the <artifact-name>-transcripts artifact, so transcripts cannot substitute for the result and hidden transcript files are not newly exposed. An empty or hidden-only optional transcript selection is ignored; missing required JSON evidence still fails the workflow.

Recurring Regression Loop (Scheduled)

Waza includes a production-ready recurring loop at .github/workflows/weekly-regression-loop.yml. It runs weekly, archives date-stamped results, compares against the latest successful baseline artifact, and upserts a follow-up issue when regressions are detected.

Key properties:

• Runs on schedule and workflow_dispatch
• Always uploads artifacts (results, session logs, regression report)
• Uses deterministic comparability checks (eval_name, model_id, runs_per_test)
• Flags regressions on increased failures/errors, passed→failed flips, and score/success-rate drops beyond thresholds

To run it manually:

gh workflow run weekly-regression-loop.yml --repo microsoft/waza

Regression gates with waza gate

For a focused PR gate, use waza gate. It compares a baseline against the current run and exits with stable codes (0 pass, 1 regression, 2 golden failure, 3 config error), making it easy to wire into any CI system.

Mark business-critical tasks in your eval YAML with golden: true and they must always pass:

tasks:
  - id: critical-flow
    golden: true
    prompt: "..."
    grader:
      type: exact_match
      expected: "OK"

GitHub Actions — annotations + step summary, with the baseline pulled from the default branch:

- name: Download baseline from main
  uses: dawidd6/action-download-artifact@v6
  with:
    workflow: eval.yml
    branch: main
    name: results
    path: baseline/

- name: Run current evaluation
  run: waza run evals/main.yaml --output results.json

- name: Regression gate
  run: |
    waza gate \
      --baseline baseline/results.json \
      --current results.json \
      --max-regression-pct 5 \
      --on-new-tasks allow \
      --on-removed-tasks warn \
      --format github-actions

The github-actions format emits ::error:: / ::warning:: / ::notice:: annotations and appends a markdown summary to $GITHUB_STEP_SUMMARY. Non-zero exit fails the step automatically.

Azure DevOps — exit codes drive task status; task.logissue surfaces inline warnings/errors:

- script: |
    waza gate \
      --baseline $(Pipeline.Workspace)/baseline/results.json \
      --current $(Build.ArtifactStagingDirectory)/results.json \
      --max-regression-pct 5 \
      --format markdown > $(Build.ArtifactStagingDirectory)/gate-report.md
    code=$?
    cat $(Build.ArtifactStagingDirectory)/gate-report.md
    if [ $code -eq 2 ]; then
      echo "##vso[task.logissue type=error]Golden task failure"
    elif [ $code -eq 1 ]; then
      echo "##vso[task.logissue type=error]Regression exceeded threshold"
    elif [ $code -eq 3 ]; then
      echo "##vso[task.logissue type=error]waza gate configuration error"
    fi
    exit $code
  displayName: 'Regression gate'

- publish: $(Build.ArtifactStagingDirectory)/gate-report.md
  artifact: gate-report
  condition: always()

Token Budget Checks

Use waza tokens compare to track token usage across PRs and fail if budgets are exceeded:

- name: Check token budget
  run: |
    waza tokens compare origin/main HEAD \
      --format table \
      --strict

This compares token counts between your main branch and current branch. Exit code is non-zero if any limit is exceeded with --strict.

Spec Coverage Checks

Use waza spec verify to ensure eval tasks cover the SKILL.md description, USE FOR triggers, DO NOT USE FOR triggers, and parameter blocks. --format github-actions emits annotations on uncovered requirements, and --fail exits non-zero when uncovered requirements meet the threshold.

- name: Verify SKILL.md eval coverage
  run: |
    waza spec verify skills/my-skill evals/my-skill/eval.yaml \
      --fail \
      --threshold 1 \
      --format github-actions

Smart Path Filtering

Run evaluations only when relevant files change:

on:
  pull_request:
    branches: [main]
    paths:
      - 'evals/**'           # Eval config changed
      - 'skills/**'          # Skill code changed
      - 'examples/**'        # Examples changed
      - '.github/workflows/eval.yml'  # Workflow itself changed

jobs:
  evaluate:
    runs-on: ubuntu-latest

Caching Fixtures and Results

Cache fixture data to speed up repeated evaluations:

- name: Restore fixture cache
  uses: actions/cache@v4
  with:
    path: .waza-cache
    key: waza-cache-${{ hashFiles('examples/**/fixtures/**') }}
    restore-keys: waza-cache-

- name: Run with cache
  run: |
    waza run \
      --cache \
      --cache-dir .waza-cache \
      --verbose

Azure DevOps Pipelines

Basic Pipeline YAML

trigger:
  branches:
    include:
      - main
      - develop
  paths:
    include:
      - evals/**
      - skills/**
      - azure-pipelines.yml

pr:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: EvaluateSkills
    displayName: Run Waza Evaluations
    timeoutInMinutes: 30
    strategy:
      matrix:
        GPT4:
          MODEL: 'gpt-4o'
        Claude:
          MODEL: 'claude-sonnet-4.6'

    steps:
      - checkout: self
        fetchDepth: 0  # Full history for diffs

      - script: |
          curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
          echo "##vso[task.prependpath]$HOME/bin"
        displayName: 'Install waza'

      - script: |
          waza run \
            --model "$(MODEL)" \
            --output "results-$(MODEL).json" \
            --verbose
        displayName: 'Run evaluations ($(MODEL))'
        env:
          GITHUB_TOKEN: $(System.AccessToken)

      - task: PublishBuildArtifacts@1
        inputs:
          pathToPublish: 'results-$(MODEL).json'
          artifactName: 'eval-results-$(MODEL)'
          publishLocation: 'Container'
        condition: always()

      - script: |
          echo "##vso[task.logissue type=error;]Evaluations failed"
          exit 1
        condition: failed()
        displayName: 'Report failure'

Token Budget in Azure DevOps

- script: |
    waza tokens compare origin/main HEAD \
      --format table \
      --strict
  displayName: 'Check token budget'

GitLab CI

Complete Pipeline

GitLab CI is highly portable and works anywhere your code lives:

stages:
  - evaluate

evaluate:
  stage: evaluate
  image: ubuntu:latest
  timeout: 30m
  parallel:
    matrix:
      - MODEL: gpt-4o
      - MODEL: claude-sonnet-4.6
      - MODEL: claude-opus-4

  before_script:
    - apt-get update && apt-get install -y curl ca-certificates
    - curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
    - export PATH="$HOME/bin:$PATH"

  script:
    - waza run --model $MODEL --output results-$MODEL.json --verbose

  artifacts:
    paths:
      - results-$MODEL.json
    expire_in: 30 days
    when: always

  only:
    variables:
      - $CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop"
    changes:
      - evals/**
      - skills/**
      - .gitlab-ci.yml

  retry:
    max: 2
    when: script_failure

Best Practices

1. Run on Specific Triggers

Avoid running expensive evaluations on every commit. Use path filters:

on:
  pull_request:
    paths:
      - 'evals/**'      # Only if eval files changed
      - 'skills/**'     # Or if skill code changed

2. Matrix Testing for Multiple Models

Test across models in parallel to catch model-specific issues:

strategy:
  matrix:
    model: [gpt-4o, claude-sonnet-4.6]
  max-parallel: 2  # Limit concurrency to avoid rate limits

3. Artifact Retention

Keep results long enough to analyze trends but not forever:

retention-days: 30  # 30 days for typical workflows

4. Timeout Configuration

Set reasonable task timeouts to avoid hanging CI jobs:

# In eval.yaml
config:
  timeout_seconds: 300  # 5 minutes per task

# In CI workflow
timeout-minutes: 30     # 30 minutes total

5. Fail Fast on Critical Tests

Stop CI early on critical evaluation failures:

# In eval.yaml
config:
  fail_fast: true  # Stop on first failure

6. Token Budget Enforcement

Prevent models from exceeding token budgets in PRs:

- name: Check token budget
  run: waza tokens compare origin/main HEAD --format table --strict

7. Cache Results Intelligently

Cache fixture data across runs, but invalidate on eval config changes:

- uses: actions/cache@v4
  with:
    path: .waza-cache
    key: waza-cache-${{ hashFiles('evals/**/*.yaml') }}

8. Secure Secrets

Never hardcode API keys. Use CI secrets:

GitHub Actions

- name: Run evaluations
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: waza run -v

Azure DevOps

- script: waza run -v
  env:
    OPENAI_API_KEY: $(OPENAI_API_KEY)

GitLab CI

script:
  - waza run -v
variables:
  OPENAI_API_KEY: $OPENAI_API_KEY

9. Safe Auto-Merge Boundaries

If you enable auto-merge for low-risk changes, gate it with explicit safeguards:

• Require a dedicated label (for example, agent-merge)
• Restrict to trusted author associations (owner/member/collaborator)
• Restrict to your protected default branch (main)
• Keep branch protection and required checks enabled

In this repository, those boundaries are enforced in .github/workflows/auto-merge.yml.

Troubleshooting

“waza: command not found”

Ensure the binary is in your PATH after installation:

# GitHub Actions
echo "$HOME/bin" >> $GITHUB_PATH

# Azure DevOps
echo "##vso[task.prependpath]$HOME/bin"

# GitLab CI (in before_script)
export PATH="$HOME/bin:$PATH"

Evaluations Timeout

Increase the timeout in both your CI workflow and eval config:

# CI level: GitHub Actions
timeout-minutes: 45

# Eval config level
config:
  timeout_seconds: 600  # 10 minutes per task

High Token Usage

Use waza tokens check to audit token consumption before running full evaluations:

waza tokens check ./skills/my-skill/

Rate Limiting from API Calls

Reduce parallel workers and add rate-limit awareness:

config:
  parallel: false  # Run tasks sequentially

Or use --workers 1 in your CI command.

Advanced Workflows

Comparing Models with Baseline

Use the baseline flag in your eval spec to establish a baseline, then compare new runs:

baseline-eval.yaml

baseline: true
config:
  model: claude-sonnet-4.6

Then in CI, compare with:

waza compare baseline-results.json candidate-results.json

Multi-Stage Pipeline: PR → Staging → Production

# GitHub Actions example
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  pr-eval:
    name: "Fast eval (PR)"
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
          $HOME/bin/waza run --tags "smoke" -v

  full-eval:
    name: "Full eval (after merge)"
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
          $HOME/bin/waza run -v

Next Steps

• Writing Eval Specs — Configure your evaluations
• CLI Reference — All flags and commands
• Graders Guide — Set up scoring rules
• Examples — Real eval setups