Observability, cost, and operational controls

Chapter 6 of 6

Confirm the result#

Run the paired smoke check from the controlled GitHub promotion workflow. The normal and handled failure routes must be different HTTPS endpoints. The scripts resolve the live Application Insights workspace before either request and reject a result path outside RUNNER_TEMP.

$env:session12_SMOKE_URL = $env:APPROVED_SYNTHETIC_SMOKE_URL
$env:session12_SMOKE_FAILURE_URL = $env:APPROVED_SYNTHETIC_FAILURE_URL
$env:session12_AI_RESOURCE_ID = $env:APPROVED_APPLICATION_INSIGHTS_RESOURCE_ID
$env:session12_LOG_ANALYTICS_WORKSPACE_ID = $env:APPROVED_LOG_ANALYTICS_WORKSPACE_ID
$env:session12_SMOKE_BEARER_TOKEN = $env:APPROVED_SYNTHETIC_SMOKE_TOKEN
$env:session12_SMOKE_TIMEOUT_SECONDS = "180"
$env:session12_SMOKE_RETRY_SECONDS = "15"
$resultPath = Join-Path $env:RUNNER_TEMP "session11-smoke.json"

.\scripts\smoke.ps1 `
  -Mode Pipeline `
  -Environment nonproduction `
  -CommitSha $env:RELEASE_COMMIT_SHA `
  -ResultPath $resultPath

export session12_SMOKE_URL="${APPROVED_SYNTHETIC_SMOKE_URL}"
export session12_SMOKE_FAILURE_URL="${APPROVED_SYNTHETIC_FAILURE_URL}"
export session12_AI_RESOURCE_ID="${APPROVED_APPLICATION_INSIGHTS_RESOURCE_ID}"
export session12_LOG_ANALYTICS_WORKSPACE_ID="${APPROVED_LOG_ANALYTICS_WORKSPACE_ID}"
export session12_SMOKE_BEARER_TOKEN="${APPROVED_SYNTHETIC_SMOKE_TOKEN}"
export session12_SMOKE_TIMEOUT_SECONDS="180"
export session12_SMOKE_RETRY_SECONDS="15"
result_path="${RUNNER_TEMP}/session11-smoke.json"

./scripts/smoke.sh \
  --mode pipeline \
  --environment nonproduction \
  --commit-sha "${RELEASE_COMMIT_SHA}" \
  --result-path "$result_path"

The scripts discard response bodies and keep only safe correlation IDs. They poll Application Insights until the normal and failure traces are complete and three consecutive queries have the same counts and latest TimeGenerated value. The default wait is 180 seconds with a 15-second retry.

The result must show:

• status: passed, the exact release commit SHA on both request records, and the live workspace binding;
• successful model and tool results for the normal route;
• a failed tool dependency and independent successful model result for the failure route;
• distinct safe correlation IDs, stable ingestion, and telemetryPollTimedOut: false;
• the expected request, error, latency, model, token, agent, tool, and quality fields; and
• no probe marker or prohibited payload property in requests, dependencies, events, traces, or exceptions.

Stop on a missing hop, matching correlation IDs, commit mismatch, unstable ingestion, workspace mismatch, sensitive content, or a tool failure without an independent model result. Do not weaken redaction to make a trace complete.

After implementation#

Run the control against the approved nonproduction service. The controlled promotion workflow calls smoke.ps1 or smoke.sh with the fixed pipeline mode, environment, commit SHA, runtime inputs, and runner-temporary result path.

Restore through the owning change paths:

1. Route the application to the last approved governed agent version if instrumentation causes a fault.
2. Restore the previous APIM gateway policy without removing authentication, safety, routing, quota, token-limit, rate-limit, or backend controls.
3. Disable only the noisy observability alert rules while correcting their queries or thresholds.
4. Remove only resources listed in the approved observability-control what-if and tagged implementationSession=11-observability-cost-operations.
5. Delete the exact observability budget only after the cost owner confirms that no workflow uses it.
6. Keep records required by an active incident, legal hold, or retention obligation.

Do not disable telemetry, Defender, or SOC routing to silence a real signal.