Risk-State Bootstrap and Recovery

Risk evidence is authorization state, not test preparation. Missing evidence never means low risk. Tests and deployment verification must not reset production risk or call Graph confirmSafe.

Authority and Persistence

CA_RISK_PROVIDER selects the ingress gateway's provider as well as the Python and portal provider:

Provider	Authority	Restart behavior
entra (default)	Explicit Graph riskyAgents/{object-id} rating	Hydrate required callers from Graph; the portal monitoring cache is not authority
sidecar	Explicit timestamped records in Azure Blob Storage	Read the same central records after restart; never seed from process memory

Both modes persist Security Portal updates in the environment's dedicated strisk<resource-token> storage account, container agent-risk-evidence. Ingress startup requires durable storage in both modes, including Entra mode: an acknowledged SOC elevation must not disappear when the process restarts. Memory-only risk stores are confined to local fixtures. The account disables shared-key and anonymous access. Budget Backend's managed identity receives container-scoped Storage Blob Data Contributor; the portal's policy-store identity has no role on this account. The authenticated deployment operator also receives container-scoped access when provisioning sidecar evidence.

Each exact SPIFFE ID maps to sha256(spiffe_id).json. A record contains spiffe_id, risk_level, observed_at, and expires_at. Cross-identity, malformed, future-dated, expired, or unavailable records cannot authorize traffic. An absent record is unknown. In Entra mode only, an absent SOC record means no additional signal; it cannot replace a missing Entra rating.

Entra evidence may be none, low, medium, or high. Manual management updates accept only low, medium, or high. Explicit Entra none can be persisted during provisioning; it is never inferred from a missing Graph record. SOC signals can raise but cannot lower the effective Entra rating. Expired or unreadable existing SOC evidence denies rather than silently lowering risk.

Freshness and Conflicts

• Management updates are stamped by the gateway and expire after 24 hours by default. RISK_EVIDENCE_LIFETIME can shorten this duration, never exceed 24h. Deployment resolves the explicit environment value, then the saved azd value, then 24h; validates and persists it and injects the same duration into the gateway and first-deploy bootstrap. Use Go duration syntax (for example 30m or 1h30m), with microsecond precision or coarser. Standalone bootstrap accepts --lifetime or RISK_EVIDENCE_LIFETIME. Cross-cloud onboarding reads the actual gateway duration from authenticated /admin/health, not the operator's local default; absent or invalid duration fails before writes or admission. Renew evidence through the authenticated risk API; do not renew it merely because a process restarted. Deployment preserves existing records and fails on expired records rather than quietly clearing or extending them.
• A governed call revalidates its authenticated caller's durable record from Blob. Cold start, a changed policy or admitted caller set, and explicit readiness probes hydrate all required enabled callers. Successful hydration completion is remembered separately from risk ratings, avoiding redundant full-store reads on every request. A failed readiness probe or caller-risk lookup invalidates that completion. CA policy freshness is checked even after hydration. Remote readiness reads run outside the hydration-state mutex: a slow unrelated caller cannot stall a warm request's own fresh risk lookup. Before publishing completion, a scan revalidates the policy, admitted caller set, CA levels/freshness/observation, and invalidation generation. An intervening failed caller lookup or probe prevents stale completion from being restored. A read/write operation has a five-second deadline. Provider errors cannot use a cached safe rating. Full readiness scans use at most four simultaneous identity reads and a twelve-second total deadline, inherited from the requester. Every required enabled identity must finish successfully before readiness/completion is published. Cancellation invalidates completion even if a provider returns success after cancellation. Budget Backend waits fifteen seconds for readiness, the admin control plane twenty, and the deployment verifier thirty-five. Process /health remains independent and reachable for recovery. The four full-scan slots are shared process-wide across overlapping readiness, sidecar snapshots, and combined Entra snapshots—not four additional reads per request. Waiting for a slot consumes the request's existing total budget; cancelled queued reads do no provider/cache work. Slots are released on success, error, or cancellation. Authenticated actual-caller reads remain outside this scan-only queue, so repeated readiness probes cannot queue warm governed traffic behind unrelated scans. This is not a distributed tenant-wide provider limit.
• Management risk snapshots (both sidecar and combined Entra) use the same four-worker fanout with an eight-second total deadline; Budget Backend's risk-read bridges wait ten seconds. Sidecar snapshots return an explicit 503 on read failure/cancellation, never a partial or cached-safe map. Combined Entra snapshots preserve explicit unknown for unavailable evidence and fail the whole snapshot on total-deadline/cancellation. Exact source attribution and per-caller authority remain unchanged.
• Writes use Blob ETags (If-Match, or If-None-Match: * for a new record). Persist before acknowledging. Concurrent conflicts return HTTP 409; provider failures return 503. Re-read authoritative state before retrying a conflict. Sequential authenticated updates are newer explicit assertions, not merges. The Go writer checks the exact serialized JSON size before upload against the same inclusive 4096-byte bound as the reader. Oversized updates are neither uploaded nor acknowledged and cannot poison subsequent reads/recovery.
• Entra runtime risk_cache_seconds retains its existing default of 90 seconds; zero forces a lookup every call. Failed required refreshes invalidate evidence.
• Graph CA policies must be observed successfully. A failed refresh retains the last policy for less than three sync intervals (180 seconds by default); the successful-fetch timestamp is not extended. At the boundary the policy becomes unavailable, including a previously empty policy list.

First Deployment and Upgrades

Use ./deploy.sh --new or ./deploy.sh --skip-provision. Do not use --portal-only for this upgrade: agents, sidecar authentication, managed identity access, and authorization readiness all change.

For sidecar mode, deployment provisions missing report, approval, and management identity records before rolling sidecars. It reads explicit ratings from Entra using the configured provisioner. Existing evidence, including high risk, is never overwritten. Missing/unlicensed/unreachable Graph data stops deployment. This is onboarding, not a post-deploy test reset.

For genuinely new agents without Entra evidence, the operator can supply an approved JSON array via RISK_BOOTSTRAP_FILE:

[
  {
    "spiffe_id": "spiffe://aim.microsoft.com/ests/bp/<blueprint-oid>/aid/<agent-oid>",
    "risk_level": "low",
    "observed_at": "<actual-observation-time-in-RFC3339>",
    "expires_at": "<actual-expiry-time-in-RFC3339>"
  }
]

Use a real assessment and current timestamps, not the placeholders above. Approval is an operator trust decision; the file is not a default-safe template. Both timestamps must use RFC3339 with uppercase T, seconds, optional decimal fractional seconds (up to nine digits), and uppercase Z or a ±HH:MM timezone offset. Other ISO formats (including space separators or compact dates) are rejected before upload because the sidecar cannot hydrate them. Approved timestamps are preserved exactly; bootstrap does not replace or renew assessment times. Freshness and lifetime comparisons retain nanosecond precision, including Go-generated nine-digit fractions; Python microsecond truncation cannot accept an overlong assessment. Approved extra fields are preserved, but the complete record must serialize to at most 4096 UTF-8 bytes using the writer's JSON encoding (including its Unicode escaping). Every pending record is size-checked before the first upload; an oversized note cannot poison the durable recovery path. The maximum lifetime is the configured gateway duration (at most 24 hours). Existing and approved evidence exceeding a shortened duration is rejected, never clamped or renewed. All pending records are validated before the first conditional write. A partially completed upload is safe to retry: existing records are preserved, and conflicting creates fail rather than overwrite.

An upgrade with --skip-provision provisions only the missing dedicated risk storage resource and access roles before configuring full Container App YAML. Azure deployment permissions must permit resource creation and role assignments. Storage role propagation is retried within bounded deadlines; failure is fatal. Backend access uses the same risk-store-access.bicep module in full provisioning and skip-provision deployment, retaining its deterministic role-assignment name across upgrade and reprovisioning. Operator access remains separately scoped. Bootstrap rejects redirects for both token and Graph requests; neither client credentials nor Graph bearer tokens are forwarded to redirect destinations.

Cross-cloud onboarding

add-google-agent.sh and add-github-agent.sh stage the new caller through onboard_federated_agent.py using the HTTPS admin-control-plane URL from the current azd environment. The protocol is: preserve existing caller policy, establish authoritative evidence, install an exact caller policy, then add the identity to the mTLS allow list. New budget-reader policies retain required JWT validation and the Budget.Read role; existing disabled callers cannot be re-enabled by onboarding. Graph tag synchronization remains a separate requirement, not an onboarding exemption.

In sidecar mode, the authenticated operator must have container-scoped Blob access and supplies either an explicit Graph rating or approved timestamped evidence through RISK_BOOTSTRAP_FILE, as on first deployment. In Entra mode, an explicit Graph rating remains mandatory; an approved local assessment cannot replace missing Entra evidence. Both scripts hard-fail on incomplete onboarding instead of reporting a skipped admission as success.

The management admission API also validates every newly added exact identity before changing the allow list. Missing policy, disabled state, unavailable CA policy, or required missing/stale/unavailable evidence returns HTTP 409 caller_onboarding_incomplete and leaves the existing allow list unchanged. Removal of an existing identity does not require evidence for that identity. Policy replacement is validated against the current admitted caller set before publication, including management evidence when enabling enforcement. Removing an admitted caller's policy, disabling the recovery identity, or enabling enforcement with unavailable required evidence returns HTTP 409 policy_authorization_state_unavailable and retains the prior policy and load timestamp. Disabled business callers need no evidence. Policy replacement and admission mutations are serialized so admission cannot interleave between validation and publication. Remove transport admission before deleting a caller policy. No broad management bypass is introduced. Policy replacement and mTLS admission have an eight-second total validation deadline and use the request context for Blob, Graph token/risk requests, and contested Graph identity/token locks. Cancellation is checked under the policy-store lock before publication (or under the allowlist publication lock for admission), so a failed requester cannot leave a later acknowledged preference rollback divergent from the active gateway policy. Budget Backend's policy and mTLS-admission bridges wait ten seconds, within the portal's fifteen-second and control-plane's twenty-second budgets. Provider reads retain their own shorter bounds where applicable. Cancelled or over-budget replacement returns a failure without changing the prior policy or its load timestamp. Cancelled admission likewise leaves the previous allowlist intact, including when validation succeeded just before cancellation or the request only removes existing IDs. The policy/admission serialization lock remains shared; no guard or disabled-identity control is bypassed. Both portal presets retain the unchanged policies of currently admitted callers, including federated callers, while installing the new preset, then reduce transport admission. Retained but no-longer-admitted policies cannot confer transport access. If admission update fails, the existing policy rollback remains valid. The pre-publication gateway guard is never bypassed. Staging policy and evidence is not an atomic transaction, but failed staging never admits the caller; retry preserves existing evidence and policies. Retain installed external caller stanzas in your deployment policy configuration: runtime policy updates are not a replacement for deployed policy configuration.

Cross-cloud runtime setup keeps Blueprint identifiers distinct. setup-gce-agent.sh requires BLUEPRINT_OBJECT_ID (or --blueprint-object-id) for the SPIFFE namespace and BLUEPRINT_APP_ID (or --blueprint-app-id) for the OAuth audience. Deployment supplies them from ENTRA_BLUEPRINT_OBJECT_ID and ENTRA_OAUTH2_AUDIENCE, respectively; the object ID is never inferred from the OAuth client ID. Example:

./scripts/setup-gce-agent.sh \
  --blueprint-object-id "<blueprint-object-id>" \
  --blueprint-app-id "<blueprint-oauth-client-id>" \
  --google-agent-id "<agent-object-id>" \
  # Supply the remaining required tenant, backend, SPIRE and ACR options.

GitHub runner SPIRE registration and admission use the same Blueprint object ID, while setup-github-runner.sh receives the OAuth client ID as BLUEPRINT_APP_ID for its generated workflow environment.

Readiness and Recovery

Sidecar /health remains process liveness. Its authenticated /ready reports 200 only when policy and risk evidence for admitted, enabled callers have hydrated; unavailable evidence returns 503. A known high-risk caller does not make evidence readiness fail: the gateway is ready to deny it.

Budget Backend exposes /healthz/ready for authorization readiness monitoring and authenticated /mgmt/ready through the management tunnel. Container App routing uses process readiness (/health), not authorization readiness: withdrawing an unhydrated revision from routing would also cut off authenticated recovery. The gateway's authorization gate, not platform routing, blocks ordinary traffic until evidence is available. The portal's /healthz/ready queries /admin/ready, not process liveness. Required callers follow the current mTLS allow list, including identities added at runtime. Governed traffic cannot pass while required evidence remains unhydrated. An unmatched admitted caller cannot obtain risk-governed access through default_action: allow, and unmatched risk writes remain denied. This risk readiness contract does not replace JWT, tag, or other authorization checks; absent Graph tags still deny and require normal attribute synchronization.

Only the exact configured SPIFFE_PREFIX_ADMIN_CONTROL_PLANE workload may use the risk-recovery exception for PUT /mgmt/agent-risk and GET /mgmt/ready. Policy names, prefixes, descendants, and caller-supplied headers do not establish recovery authority. Recovery bypasses only the caller's risk check and hydration gate. It does not bypass:

• mTLS trust and allow-list admission;
• shared-key management authentication at both backend and sidecar;
• caller-policy/RBAC rules or required JWT checks;
• the disabled-agent kill switch, target tags, or policy availability.

To establish or renew approved evidence, use the existing authenticated PUT /admin/agent-risk path with {"spiffe_id": "...", "risk_level": "..."}. The management caller can recover its own expired evidence without first being treated as safe. Other management methods and routes remain governed. A disabled management identity cannot use this path. Storage corruption and permissions failures require an authorized storage/operator repair; no default-safe recovery exists.

Portal Settings labels enforcement for the selected provider. Durable-sidecar enforcement can be re-enabled after its authoritative-evidence preflight; the Entra cache control remains separate and disabled for sidecar mode. Risk snapshots enumerate deduplicated exact admitted workloads plus exact policy identities, not namespace prefixes. Raw local snapshots explicitly report source: "sidecar"; combined Entra snapshots report source: "entra". Independent portal Entra monitoring settings never determine gateway attribution. The portal selects /admin/entra-risk for configured Entra mode even when gateway enforcement is off, retaining the combined Graph/SOC snapshot. Sidecar mode uses /admin/agent-risk; a raw local snapshot is never relabeled as Entra evidence. The independent portal monitoring switch controls its separate Graph display, not the gateway snapshot's provider or endpoint.

Deployment's default verifier is the read-only scripts/verify_risk_readiness.py. Unhydrated state, denied recovery identity, or an unreachable management path is a hard failure, not a warning. --no-verify is an explicit opt-out and does not initialize evidence. The legacy scripts/test_agents.py no longer resets baseline risk or runs risk mutations by default; scoped mutating exercises belong to the approved live harness. The verifier rejects every redirect without forwarding the management key. Its final check runs after optional Google/GitHub onboarding and before an optional local portal launch, covering the final admitted caller set rather than only the base deployment.

Evidence Boundaries

The local protocol matrix includes cold start, none/low/medium/high restart, missing, malformed, cross-identity, future-dated and expired evidence, provider and write outages, ETag conflicts, unauthorized recovery, disabled recovery, and unrelated RBAC/JWT/tag checks. It exercises the actual Azure Blob SDK adapter against an ephemeral Blob HTTP boundary, including real adapter writes and fresh-store reloads. This is not proof of Azure managed identity role propagation or Container App/SPIRE recovery. Admission regressions also exercise the actual management HTTP API and validate that failed onboarding leaves management policy installation and rollback reachable, rejected policy removal/enforcement transitions retain the previous state, and unmatched default-allow callers cannot bypass hydration. Script tests execute the gateway YAML generator and compare its lifetime with bootstrap arguments, and use the gateway's advertised lifetime during federated provisioning. They also cover evidence-before-admission ordering, preserved policies, provider failures, and final verification ordering; a Chromium API regression covers source attribution with independent Entra monitoring enabled.

Run the local matrix via tests/run.py and the repeated gate via tests/check.py. Live rollout/restart testing requires explicit environment and mutation authorization; never restart a shared environment or clear tenant risk as a test fixture.