Implementation | Session 09: Foundry evaluations and release quality gates

Chapter 5 of 6

Implement#

1. Set the runtime context#

Complete the shared execution-environment setup in the root README. Sign in to the approved Azure subscription, install the Python dependencies, and set the external release-store path.

PowerShell

python -m pip install -r .\scripts\requirements.txt
$approvedSubscriptionId = $env:AZURE_SUBSCRIPTION_ID
$env:FOUNDRY_RESOURCE_ID = $env:APPROVED_FOUNDRY_RESOURCE_ID
$env:FOUNDRY_PROJECT_ENDPOINT = $env:APPROVED_FOUNDRY_PROJECT_ENDPOINT
$env:FOUNDRY_MODEL_NAME = $env:APPROVED_EVALUATION_MODEL
$releaseStore = $env:APPROVED_RELEASE_STORE

Bash

python -m pip install -r ./scripts/requirements.txt
approved_subscription_id="${AZURE_SUBSCRIPTION_ID:?Set AZURE_SUBSCRIPTION_ID.}"
export FOUNDRY_RESOURCE_ID="${APPROVED_FOUNDRY_RESOURCE_ID:?Set APPROVED_FOUNDRY_RESOURCE_ID.}"
export FOUNDRY_PROJECT_ENDPOINT="${APPROVED_FOUNDRY_PROJECT_ENDPOINT:?Set APPROVED_FOUNDRY_PROJECT_ENDPOINT.}"
export FOUNDRY_MODEL_NAME="${APPROVED_EVALUATION_MODEL:?Set APPROVED_EVALUATION_MODEL.}"
release_store="${APPROVED_RELEASE_STORE:?Set APPROVED_RELEASE_STORE outside this repository.}"

2. Run the approved baseline#

PowerShell

.\scripts\preflight.ps1 -ApprovedSubscriptionId $approvedSubscriptionId -Phase Baseline
python .\scripts\run-evaluation.py `
  --spec .\artifacts\eval\evaluation-spec.json `
  --target approved `
  --output (Join-Path $releaseStore "approved-baseline.json")

Bash

./scripts/preflight.sh --approved-subscription-id "$approved_subscription_id" --phase baseline
python ./scripts/run-evaluation.py \
  --spec ./artifacts/eval/evaluation-spec.json \
  --target approved \
  --output "$release_store/approved-baseline.json"

Preflight resolves the exact target and rejects invalid scope, support, identity, dataset, policy, dependency, and budget inputs. The Evals API has no dry run. Inspect row-level detail in Foundry, then let the control owners activate thresholds within the approved floor and baseline bounds.

3. Run and gate the candidate#

PowerShell

.\scripts\preflight.ps1 -ApprovedSubscriptionId $approvedSubscriptionId -Phase Candidate
python .\scripts\run-evaluation.py `
  --spec .\artifacts\eval\evaluation-spec.json `
  --target candidate `
  --output (Join-Path $releaseStore "candidate.json")
python .\scripts\release-gate.py `
  --policy .\artifacts\eval\thresholds.yaml `
  --spec .\artifacts\eval\evaluation-spec.json `
  --dataset .\artifacts\eval\data\golden-v1.jsonl `
  --baseline-result (Join-Path $releaseStore "approved-baseline.json") `
  --candidate-result (Join-Path $releaseStore "candidate.json") `
  --evaluated-target candidate `
  --expect pass

Bash

./scripts/preflight.sh --approved-subscription-id "$approved_subscription_id" --phase candidate
python ./scripts/run-evaluation.py \
  --spec ./artifacts/eval/evaluation-spec.json \
  --target candidate \
  --output "$release_store/candidate.json"
python ./scripts/release-gate.py \
  --policy ./artifacts/eval/thresholds.yaml \
  --spec ./artifacts/eval/evaluation-spec.json \
  --dataset ./artifacts/eval/data/golden-v1.jsonl \
  --baseline-result "$release_store/approved-baseline.json" \
  --candidate-result "$release_store/candidate.json" \
  --evaluated-target candidate \
  --expect pass

PASS makes the fixed candidate eligible for the approved promotion path. BLOCK leaves the approved version pinned. Remediate the failed layer and evaluate a new fixed candidate.