Foundry evaluations and release quality gates

Chapter 6 of 6

Confirm the result#

Intended path#

Run the gate with the approved aggregate as both inputs. It must return PASS with separate final-answer, tool-process, and safety summaries.

python .\scripts\release-gate.py `
  --policy .\artifacts\eval\thresholds.yaml `
  --spec .\artifacts\eval\evaluation-spec.json `
  --dataset .\artifacts\eval\data\golden-v1.jsonl `
  --baseline-result (Join-Path $releaseStore "approved-baseline.json") `
  --candidate-result (Join-Path $releaseStore "approved-baseline.json") `
  --evaluated-target approved `
  --expect pass

python ./scripts/release-gate.py \
  --policy ./artifacts/eval/thresholds.yaml \
  --spec ./artifacts/eval/evaluation-spec.json \
  --dataset ./artifacts/eval/data/golden-v1.jsonl \
  --baseline-result "$release_store/approved-baseline.json" \
  --candidate-result "$release_store/approved-baseline.json" \
  --evaluated-target approved \
  --expect pass

Blocked or failure path#

python .\scripts\test_release_gate.py --mode blocked-tool-process

python ./scripts/test_release_gate.py --mode blocked-tool-process

The test must return BLOCK for tool-call accuracy and success while final-answer quality and safety remain passing.

Delivery-owner checkpoint#

The release owner reviews both outcomes in the approved release platform. A blocked candidate stays unpinned. After a successful checkpoint, the owner may enable the gate for the controlled promotion workflow. The gate still cannot promote a version.

After implementation#

Keep the golden data, evaluation specification, thresholds, release policy, scripts, and disable-and-restore runbook in operation. The quality owner maintains data and thresholds. The safety owner maintains safety coverage and the protected-material boundary. The tool owner owns Function Tool compatibility. The cost owner approves consumption. The release owner controls gate state and the stable selector.

The observability and cost controls compare current aggregate signals with the approved release baseline. It does not reuse row-level evaluation content as production telemetry. When the quality owner classifies a change as material drift, create a fixed candidate version and rerun this gate.

Use artifacts/operations/disable-and-restore.md to disable the gate, keep or restore the approved version at 100%, remove delivery integration after dependency review, and cancel unnecessary evaluations. Do not delete the Foundry project, agent, model, tool path, logs, or customer data as a shortcut. The scripts never change the stable endpoint or delete cloud state.