Implementation | Optional module: Continuous evaluation and Foundry platform alerts

Chapter 5 of 6

Implement#

1. Complete the retained decisions#

Resolve the monitoring parameters and the continuous evaluation decision record in the approved private configuration path:

artifacts/monitoring/foundry-platform-monitoring.bicepparam
artifacts/evaluation/continuous-evaluation-decision.json

Set the deployment coordinates:

PowerShell

$approvedSubscriptionId = $env:AZURE_SUBSCRIPTION_ID
$resourceGroupName = $env:FOUNDRY_RESOURCE_GROUP
$targetScope = "<approved Foundry project scope alias>"
$deploymentLocation = $env:AZURE_DEPLOYMENT_LOCATION

Bash

approved_subscription_id="${AZURE_SUBSCRIPTION_ID}"
resource_group_name="${FOUNDRY_RESOURCE_GROUP}"
target_scope="<approved Foundry project scope alias>"
deployment_location="${AZURE_DEPLOYMENT_LOCATION}"

2. Run preflight and review the preview#

PowerShell

.\scripts\preflight.ps1 `
  -ApprovedSubscriptionId $approvedSubscriptionId `
  -ResourceGroupName $resourceGroupName `
  -TargetScope $targetScope `
  -DeploymentLocation $deploymentLocation

Bash

./scripts/preflight.sh \
  --approved-subscription-id "$approved_subscription_id" \
  --resource-group-name "$resource_group_name" \
  --target-scope "$target_scope" \
  --deployment-location "$deployment_location"

Preflight reads the artifacts, rejects unresolved decisions, validates syntax, checks the target scope, and runs a resource-group what-if preview. It reaches Azure only after every sentinel is resolved.

3. Deploy the monitoring controls#

Deploy only after the observability owner approves the what-if preview.

PowerShell

az deployment group create `
  --subscription $approvedSubscriptionId `
  --resource-group $resourceGroupName `
  --template-file .\artifacts\monitoring\foundry-platform-monitoring.bicep `
  --parameters .\artifacts\monitoring\foundry-platform-monitoring.bicepparam `
  --only-show-errors

Bash

az deployment group create \
  --subscription "$approved_subscription_id" \
  --resource-group "$resource_group_name" \
  --template-file ./artifacts/monitoring/foundry-platform-monitoring.bicep \
  --parameters ./artifacts/monitoring/foundry-platform-monitoring.bicepparam \
  --only-show-errors

4. Create or update the continuous evaluation rule#

Install the runtime dependencies in the approved operator environment. Do not install packages into this repository.

PowerShell

python -m pip install -r .\artifacts\evaluation\requirements.txt
$env:AZURE_AI_PROJECT_ENDPOINT = "<resolved approved project endpoint>"
python .\artifacts\evaluation\continuous-evaluation-rule.py `
  --decision-file .\artifacts\evaluation\continuous-evaluation-decision.json

Bash

python -m pip install -r ./artifacts/evaluation/requirements.txt
export AZURE_AI_PROJECT_ENDPOINT="<resolved approved project endpoint>"
python ./artifacts/evaluation/continuous-evaluation-rule.py \
  --decision-file ./artifacts/evaluation/continuous-evaluation-decision.json

The script prints the evaluation ID and rule ID. Keep those IDs in the approved operations record, not in this repository.

5. Check telemetry after approved traffic#

Run the KQL file against the approved workspace after normal traffic or an approved synthetic call. The queries check diagnostic rows, token metrics, throttling shape, and latency percentiles.

For evaluation runs, list recent runs with the evaluation ID printed by the setup step:

PowerShell

python .\artifacts\evaluation\continuous-evaluation-rule.py `
  --list-runs `
  --evaluation-id "<resolved evaluation id>" `
  --limit 10

Bash

python ./artifacts/evaluation/continuous-evaluation-rule.py \
  --list-runs \
  --evaluation-id "<resolved evaluation id>" \
  --limit 10