Statistical Fields

When you run evaluations with multiple trials (trials_per_task > 1), waza quantifies how far the results can be trusted. This page explains the statistical fields in your results JSON.

---

Separate fixed-design release decisions

Legacy statistical fields below are descriptive and unchanged. They are not evidence that an explicitly selected release policy passed. Policy decisions use independently versioned sidecars and emit estimate, lower, upper, half_width, effective_clusters, full planned trial/attempt accounting and separate decision dimensions.

The fixed-suite repeated-execution contrast assumes independent bounded cluster differences in [-1,1]; retries and tasks within a cluster are not iid samples. The two-sided weighted Hoeffding radius is sqrt(2*log(2*M/alpha)*sum(w*w)). One selected contrast is assessed; M is an external conservative family bound, not a verified family inventory. Insufficient declared precision is reported separately and is not a power calculation. Unjustified independence, missing/partial/invalid evidence and inconclusive intervals cannot strict-pass. The explicitly selected native Copilot collector supplies attempts to this same fixed-design method; it does not change the estimand, cluster assumptions or endpoint selection.

Equal-cluster planning uses an exact rational logarithm enclosure with a bounded remainder, not a nearest-rounded radius comparison. An ambiguous boundary may conservatively require an extra cluster; subnormal alpha and integer family sizes retain their exact values. Validate the actual frozen weights separately before collection: a certified count is not a power or release-pass promise.

Canonical declared coefficients undergo representation-only mass admission before exact normalization; omitted clusters are never reweighted away. Golden first-attempt failures and operational errors cannot be averaged into a statistical success. Read the controlled comparison limitations before using selected policy decisions.

Independent offline assurance never changes the frozen base statistical decision, sample denominator, first-attempt golden veto or legacy fields. Only explicit outer assessment with --assurance-contract can pass required assurance: both arms need fresh exact offline1.0 finite-authored-output reports, retained actual-response deterministic regrade and all base non-assurance conditions. A grader qualification pass cannot repair insufficient precision, unjustified independence, operational failure or a statistically inconclusive comparison. Base API/dashboard required-assurance decisions remain nonpassing.

Finite supplied-corpus agreement is not confidence calibration or live-agent reliability. Stored report labels are not fresh qualification; current sources, evidence and separately supplied review acceptance are rechecked. Examples remain unreviewed; synthetic test reviews are not authenticated human review. No live SDK assurance, calibration, paid1.1/billing support or automatic paid validation is selected. See the independent profile.

When Statistical Fields Appear

• Per-task fields (tasks[].stats.pass_rate_ci, tasks[].stats.bootstrap_ci) appear when a task ran more than one trial.
• The suite success-rate interval (summary.statistics.success_rate_ci) appears when trials_per_task > 1 and at least one task outcome exists. Its denominator includes every task outcome, including failures before trial execution; missing trial statistics do not remove known task outcomes from this interval.
• The suite bootstrap weighted-score interval requires at least two tasks with recorded scores from executed trials. It is omitted when those scores are unavailable.
• The pass^k curve requires recorded trial counts for every task. Missing trial statistics omit the curve rather than inventing estimates.

Single-trial runs skip statistical analysis.

---

Pass-Rate Confidence Interval (pass_rate_ci)

tasks[].stats.pass_rate_ci is a 95% Wilson score interval on the task’s pass rate across its trials. Wilson stays inside [0, 1] and keeps its coverage at the 3–10 trials typical of agent evals, where normal-approximation and bootstrap intervals come out too narrow (Bowyer et al., 2025).

"stats": {
  "pass_rate": 0.8,
  "passed_runs": 4,
  "total_runs": 5,
  "pass_rate_ci": { "lower": 0.3755, "upper": 0.9638, "mean": 0.8, "confidence_level": 0.95 }
}

Interpretation: 4 of 5 trials passed, but the task’s true pass rate could plausibly be anywhere from about 38% to 96%. A wide interval means you need more trials before trusting the pass rate.

summary.statistics.success_rate_ci is the same interval for the suite’s success_rate, counting each task as one sample.

The dashboard’s Trial Pass Rate column shows each recorded task estimate and Wilson interval with available passed/total trial counts. Recorded reliability shows the suite interval and the stored pass^k curve, independently of the bootstrap weighted-score interval. The API maps these to tasks[].passRateCI, tasks[].trials, statistics.successRateCI, and statistics.passHatK on run detail. Confidence levels and estimates come from the artifact; readers do not recompute them. Historical absence is omitted from the API and shown as Not assessed, never zero or inferred success.

waza run prints both:

Success CI95:   [6.1%, 79.2%]
...
      pass_rate=80.0% [CI95 38-96%]  avg=0.85  ...

---

Reliability (pass_hat_k)

summary.statistics.pass_hat_k is the pass^k curve from τ-bench (Yao et al., 2024). Element k−1 is the probability that k fresh trials of a task all pass, averaged over tasks. For a task with c passes in n trials, the unbiased estimate is C(c, k) / C(n, k). The curve runs up to the fewest trials any task ran.

pass^1 is the average per-trial pass rate. The drop from pass^1 to pass^n shows how much of that rate comes from tasks that pass only some of the time, which is what users of a skill experience as unreliability.

"statistics": {
  "pass_hat_k": [0.7333, 0.5667, 0.4667, 0.4, 0.3333]
}

Reliability:    pass^1=73%  pass^2=57%  pass^3=47%  pass^4=40%  pass^5=33%

Interpretation: a single trial passes 73% of the time, but five trials of the same task all pass only 33% of the time.

Note

With max_attempts > 1, a trial that passes on a retry counts as passed, so these intervals and pass^k describe reliability with retries.

---

Score Confidence Interval (bootstrap_ci)

tasks[].stats.bootstrap_ci is a percentile bootstrap interval (10,000 resamples) over the task’s weighted trial scores, and summary.statistics.bootstrap_ci is the same over per-task average scores. The dashboard shows the per-task interval next to each task’s weighted score.

"bootstrap_ci": { "lower": 0.72, "upper": 0.95, "mean": 0.85, "confidence_level": 0.95, "num_bootstraps": 10000 }

With only a few trials a bootstrap interval is too narrow, so prefer pass_rate_ci when judging pass/fail reliability.

---

Significance (is_significant, no longer populated)

Earlier versions set is_significant to true when a task’s bootstrap_ci excluded zero. That interval is on the task’s own score, which can’t be negative, so the check only asked whether the score was above zero. It compared nothing, so neither the flag nor the dashboard’s “significant” badge indicated statistical significance. It is no longer populated, and older results files that contain it still load.

Significance applies to a difference, such as with-skill minus baseline. The intervals on this page describe each run on its own, so comparing two runs’ intervals cannot establish whether a change is real: overlapping intervals can still hide a significant difference, and runs over the same tasks need a paired comparison of per-task results.

---

Normalized Gain (normalized_gain)

Normalized Gain measures improvement as a percentage of the maximum possible gain, useful for comparing progress across different score ranges.

Formula

Normalized Gain = (Actual Score - Baseline Score) / (1.0 - Baseline Score)

Structure

{
  "task": "code-quality",
  "baseline_score": 0.60,
  "improved_score": 0.75,
  "normalized_gain": 0.375
}

Interpretation

Scenario	Baseline	Improved	Gain	Normalized Gain
Low starting point	0.40	0.60	0.20	0.33 (33%)
High starting point	0.80	0.90	0.10	0.50 (50%)

In the second example, a 0.10 improvement from 0.80 is more impressive (50% of remaining room) than a 0.20 improvement from 0.40 (33% of remaining room).

When to Use

• Comparing tasks with different baseline scores
• Measuring effectiveness of an improvement across multiple tasks
• Aggregating progress across heterogeneous benchmarks

---

Using These Fields for CI/CD Gating

- name: Run evaluations with 10 trials
  run: |
    rm -f results.json
    status=0
    waza run eval.yaml --trials 10 -o results.json || status=$?
    if [ ! -s results.json ]; then
      echo "Evaluation produced no result artifact (exit $status); refusing to gate stale results." >&2
      exit 1
    fi
    # The next step assesses only this invocation's result artifact.

- name: Gate on the success-rate interval
  run: |
    python3 << 'EOF'
    import json, sys
    with open('results.json') as f:
        results = json.load(f)

    ci = (results['summary'].get('statistics') or {}).get('success_rate_ci')
    # Fail only when even the optimistic end of the interval misses the target.
    if ci and ci['upper'] < 0.8:
        sys.exit(f"❌ success rate CI95 [{ci['lower']:.0%}, {ci['upper']:.0%}] is below the 80% target")

    # Flag tasks whose pass rate is still too uncertain to trust.
    for task in results['tasks']:
        task_ci = (task.get('stats') or {}).get('pass_rate_ci')
        if task_ci and task_ci['upper'] - task_ci['lower'] > 0.5:
            print(f"⚠️  {task['display_name']}: pass-rate CI95 wider than 50 points; add trials")
    EOF

The run step removes any earlier artifact and fails if this invocation does not produce a nonempty result. A failed evaluation may still produce a valid result for the interval gate; a command failure without results cannot reuse stale data. Recorded one-element pass_hat_k curves are valid pass^1 evidence in the CLI, API, and dashboard; they do not imply any unrecorded higher-k estimates.

Gate Strategies

Strategy	When	Check
Interval vs target	Avoid failing on noise	success_rate_ci.upper below target fails
Interval width	Too few trials to judge	pass_rate_ci width above your tolerance
Reliability	Users need consistent behavior	pass_hat_k[k-1] below target for your k
Normalized gain	Cross-task comparison	Gain > 0.25 (25% of room)

---

Minimal Trials Recommendations

Complexity	Min Trials	Typical	Rationale
Simple, deterministic	3	5	Low variability
Standard LLM tasks	5	10	Moderate variability
Complex multi-step tasks	10	20	High variability
Critical production gates	20	30	Stringent confidence

---

---

Example Results JSON

Three tasks with 5 trials each; the tasks passed 5, 4, and 2 of their trials (fields abridged):

{
  "summary": {
    "total_tests": 3,
    "succeeded": 1,
    "failed": 2,
    "success_rate": 0.3333,
    "statistics": {
      "bootstrap_ci": { "lower": 0.52, "upper": 0.93, "mean": 0.74, "confidence_level": 0.95, "num_bootstraps": 10000 },
      "success_rate_ci": { "lower": 0.0615, "upper": 0.7923, "mean": 0.3333, "confidence_level": 0.95 },
      "pass_hat_k": [0.7333, 0.5667, 0.4667, 0.4, 0.3333]
    }
  },
  "tasks": [
    {
      "display_name": "deploy-to-azure",
      "status": "failed",
      "stats": {
        "pass_rate": 0.8,
        "passed_runs": 4,
        "total_runs": 5,
        "flaky": true,
        "pass_rate_ci": { "lower": 0.3755, "upper": 0.9638, "mean": 0.8, "confidence_level": 0.95 },
        "bootstrap_ci": { "lower": 0.71, "upper": 0.97, "mean": 0.86, "confidence_level": 0.95, "num_bootstraps": 10000 }
      }
    }
  ]
}

A task’s status is passed only when every non-skipped trial passed. So when every task ran n trials and none were skipped, success_rate equals pass^n, the last element of pass_hat_k.

---

Next steps

• CI/CD Integration — Gate deployments on these fields
• Graders Guide — Understanding grader weighting
• CLI Reference — All command-line flags