Skip to main content

Trident ACL Agent

trident-acl-agent is an on-node daemon that drives Trident A/B updates from a Kubernetes control plane, using node annotations instead of a direct API call as the trigger. It is the on-node half of Azure Container Linux (ACL)'s update mechanism. Any Kubernetes control-plane component (a custom controller, an operator, or a script driven by kubectl patch) can orchestrate updates across a fleet of nodes by writing to the annotation contract described below. It supplies the image source on a Nebraska server and honors the agent's per-node protocol; the agent speaks Omaha to Nebraska for image distribution.

Deployment​

trident-acl-agent ships in the trident-acl RPM subpackage (built alongside, and Requires: the same version of, the main trident package). Installing it:

$ tdnf install trident-acl

lays down the /usr/bin/trident-acl-agent binary and its trident-acl-agent.service unit (packaging/systemd/trident-acl-agent.service) under the systemd unit directory, along with the package's %license-installed LICENSE/NOTICE files. Installing the package does not by itself enable or start the service — a deployment decides when that happens, e.g. by running systemctl enable --now trident-acl-agent.service on the node, or by baking that enablement into the image build (as this repo's own tests/images/trident-vm-testimage/base/updateimg-acl-agent.yaml does via Image Customizer's services: enable list).

The shipped unit invokes the agent via ExecStart and has no Environment= lines, so every deployment-specific choice — which annotation prefix to watch, where to read the current version from, which Kubernetes API server to talk to, and so on — is supplied the same way any other systemd service is configured: standard Environment=/EnvironmentFile= constructs, most commonly a drop-in applied on top of the packaged unit. See Configuration below for the full list of variables and Setting env vars via a systemd drop-in for how to apply them without editing the packaged unit.

The annotation contract​

An orchestrator (a Kubernetes controller with RBAC permission to PATCH the target Node object) triggers an update by writing a JSON payload to a request annotation on the Node. The agent watches that annotation, drives the requested operation against tridentd, and writes its progress and result via annotations on the same Node.

Three annotation keys make up the contract, all sharing one configurable prefix (acl.microsoft.com by default — see Configuration below):

AnnotationWritten byPurpose
<prefix>/update-requestOrchestratorRequests stage, finalize, or rollback for this node.
<prefix>/update-statusAgentReports the status of the requested operation.
<prefix>/update-commit-statusAgentReports the status of the implicit post-reboot commit that follows a finalize or rollback.

A request annotation looks like:

{
  "schemaVersion": "1.0",
  "nodeUpdateId":  "550e8400-e29b-41d4-a716-446655440000",
  "operationId":   "c9d6f0a2-3b41-4e8d-9f27-1a5b6c7d8e90",
  "operation":     "stage",
  "targetVersion": "202606.29.0",
  "server":        "https://nebraska.example.com/v1/update",
  "appId":         "11111111-2222-3333-4444-555555555555",
  "track":         "pin-202606.29.0"
}
  • nodeUpdateId identifies one node's update sequence and is held constant across stage → finalize → commit.
  • operationId identifies this specific step; the agent uses it to decide whether to start new work, resume in-flight work, or re-emit a cached terminal status as a no-op on a duplicate PATCH.
  • targetVersion is the image release version to update to. Required for stage/finalize; omitted for rollback, whose target (the previous partition) is implicit.
  • server, appId, and track name the Omaha instance that serves the target image and receives progress events for it. They are required on stage/finalize requests, with no static fallback — a request missing them is rejected with InvalidRequest rather than falling back to a built-in endpoint, so a node can never update from a source the orchestrator did not explicitly choose. server must be an https URL — a plain-http (or any other scheme) value is rejected outright, since Nebraska's response drives a root-privileged tridentd fetch and must not be readable/tamperable by a network-position attacker. Likewise, the package/image URL Nebraska's response resolves to is only accepted if it is https or oci; http and file are rejected, closing off an SSRF vector where a compromised/spoofed Nebraska response could otherwise direct tridentd to fetch or open an arbitrary local file or unencrypted network resource.

operation maps to Trident invocations as follows:

operationTrident invocationEffect
stagetrident update --allowed-operations=stageQueries the server/appId/track Omaha endpoint with the node's current version, requires the offered version to match targetVersion, then streams the resulting image to the inactive partition. No reboot.
finalizetrident update --allowed-operations=finalize (gRPC UpdateFinalize, caller-handled reboot)Arms boot for the staged target, writes a terminal finalize status, then triggers the reboot.
rollbacktrident rollback --ab (gRPC RollbackStage/RollbackFinalize, caller-handled reboot)Swaps back to the previous partition, mirroring finalize on the return path. Only the last update can be undone this way.

A fourth phase, commit, runs implicitly after the post-finalize/ rollback reboot: the agent runs trident commit on the new partition and writes a commit status without needing a separate annotation request. The orchestrator watches <prefix>/update-commit-status as the terminal signal that the reboot half of the update succeeded.

stage end-to-end:

finalize / rollback, spanning the reboot:

A status annotation (<prefix>/update-status or <prefix>/update-commit-status) looks like:

{
  "schemaVersion":   "1.0",
  "nodeUpdateId":    "550e8400-e29b-41d4-a716-446655440000",
  "operationId":     "c9d6f0a2-3b41-4e8d-9f27-1a5b6c7d8e90",
  "operation":       "stage",
  "code":            "Success",
  "message":         "staged update to 202606.29.0",
  "fromVersion":     "202606.15.0",
  "toVersion":       "202606.29.0",
  "tridentVersion":  "0.22.0",
  "startedUtc":      "2026-06-29T12:00:00Z",
  "lastUpdatedUtc":  "2026-06-29T12:03:41Z",
  "finishedUtc":     "2026-06-29T12:03:41Z"
}
  • operation is stage, finalize, rollback, or commit (commit only ever appears on <prefix>/update-commit-status, never on <prefix>/update-status).
  • code is the outcome — see the table below.
  • message is a short, human-readable explanation of code, useful for logs/alerts; treat its exact wording as informational, not something to match on (it may include error detail that varies run to run).
  • fromVersion/toVersion are the versions the operation moved between (toVersion is absent for rollback, whose target is implicit).
  • tridentError is present only when the failure originated from a Trident call that returned an error response (a remote error); it can appear on both OperationFailed and TargetBootFailed (the reboot landed but the post-reboot commit call itself failed remotely) - every other failure mode (timeouts, connection failures, agent-generated errors like InvalidRequest) carries no tridentError. It has kind/subkind (Trident's own error classification, e.g. SERVICING_ERROR/ab-update-reboot-check, or unknown/unknown when Trident reported failure without a structured error payload) and an optional location (path/line in Trident's source), letting an orchestrator key off structured fields instead of parsing message.
  • tridentVersion is the agent's own build version (AGENT_VERSION), stamped on every status write. tridentd and trident-acl-agent are built and released together from a single RPM spec (packaging/rpm/trident.spec), so this value doubles as the tridentd version for that install.
  • startedUtc/lastUpdatedUtc/finishedUtc bound the operation: lastUpdatedUtc refreshes on a heartbeat cadence while code is InProgress (see below); finishedUtc is absent until code reaches a terminal value.

code is one of:

codeTerminal?Meaning
InProgressNoThe operation is running. lastUpdatedUtc refreshes on a heartbeat cadence; a terminal code always follows.
SuccessYesThe operation completed as requested. For commit, this means the reboot landed on the target partition and it was promoted.
AlreadyAtTargetYesstage/finalize was requested for the version the node is already running (per TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY); treated as a no-op success.
NotStagedYesfinalize was requested for a nodeUpdateId with no prior successful stage. Issue a stage first.
OperationFailedYesThe operation failed for a reason other than a boot/rollback outcome (e.g. the Omaha server has no update for the requested version, or the underlying tridentd call returned an error). See message for detail.
TargetBootFailedYesThe post-reboot commit found the node had rolled back to its previous partition instead of booting the target — Trident's own health checks rejected the new boot. The node is back on fromVersion; the orchestrator should treat this as a failed update, not retry the same nodeUpdateId blindly.
AgentInternalErrorYesA failure in the agent itself rather than in Trident or the requested operation (e.g. it triggered a reboot but the reboot call failed, or it lost track of an in-flight commit). Distinct from OperationFailed so an orchestrator can decide to treat these differently (e.g. retry vs. escalate).
InvalidRequestYesThe request annotation itself was rejected before any action was taken — malformed JSON, a schema/version mismatch, a missing required field (server/appId/track/targetVersion), a finalize whose targetVersion doesn't match what was staged, or a second finalize/rollback submitted while one is already pending its post-reboot commit. No Trident operation runs.

See the request/status schema types in crates/trident-acl-agent/src/annotations/protocol.rs for the validation rules and the formal JSON Schemas pinned by the protocol conformance tests.

Pre/post-reboot state and the watchdog​

Because finalize/rollback spans a reboot, the agent persists a small state file (TRIDENT_ACL_AGENT_ORCHESTRATION_STATE_PATH) recording that a commit is pending and a marker for "a boot happened after this point". On restart, the agent checks this state to resume the post-reboot commit step rather than re-running finalize from scratch.

While a stage, finalize, or rollback Trident call is in flight, the agent refreshes the InProgress status's lastUpdatedUtc on a heartbeat cadence (TRIDENT_ACL_AGENT_ORCHESTRATION_HEARTBEAT_INTERVAL), so an external watchdog can distinguish a working agent from a stuck one and reprovision a node that never reports a terminal commit status within its SLA.

Nebraska affinity cookies​

Cookie handling is opt-in via TRIDENT_ACL_AGENT_ORCHESTRATION_COOKIE_ENABLED=true. When disabled (the default), the agent neither sends nor adopts cookies and leaves any existing cookie file untouched.

When enabled, each initial staging update check starts without a cookie and clears the previous stored value. The cookie returned by that check is persisted separately from state.json and reused for progress, failure, finalize, and post-reboot completion requests. Later responses do not replace it. If the initial response sets no cookie, subsequent requests remain cookie-less. A restarted staging attempt starts fresh, even with unchanged request IDs; resuming finalize or completion retains the stored cookie.

The agent reads the file for each request. Cookies are saved only from successful HTTP responses with readable bodies, before Omaha XML validation. The file uses owner-only permissions and atomic replacement; cookie values are never logged. Cookie I/O failures warn without failing the update, so affinity is best-effort if storage fails.

Configuration​

At startup, the agent logs its effective AgentConfig at INFO after applying environment overrides and defaults, including for --validate-connection. Network URL credentials, paths, query strings, and fragments are redacted. Kubeconfig, state, and cookie file paths are logged, but their contents are not read for logging.

There is no config file. Every setting is an environment variable prefixed TRIDENT_ACL_AGENT_, systemd-style: set it in the unit's own Environment= lines, via a drop-in override, or by any other means that sets the process's environment before it starts.

A variable that is unset, or set to the empty string, falls back to its default. A variable set to a malformed value (a bad URL, a bad duration, an invalid boolean) causes the agent to fail to start with an error naming the offending variable.

VariableDefaultDescription
TRIDENT_ACL_AGENT_COSI_SHA384_SOURCElabelsWhich source resolves a package's COSI metadata SHA-384: labels reads the cosi.metadata.sha384 label value as hex from the Nebraska manifest's <labels> child element; hash reads the base64-encoded legacy hash attribute instead (for Nebraska deployments with no label support). Both must decode to 48 bytes. No automatic fallback between the two.
TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIXacl.microsoft.comThe annotation-key prefix for the request/status/commit-status annotations (e.g. the acl.microsoft.com in acl.microsoft.com/update-request). Any orchestrator can pick its own namespace here so its annotations don't collide with another controller's.
TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH/etc/os-releaseThe file the agent reads to determine the node's currently running version. Any file works, as long as it follows the os-release format (KEY=VALUE lines, optionally single- or double-quoted, blank lines and # comments ignored) — see below.
TRIDENT_ACL_AGENT_CURRENT_VERSION_KEYVERSION_IDThe key the agent looks up in TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH (/etc/os-release by default) to determine the node's currently running version, used to compare against a request's targetVersion (e.g. to short-circuit to AlreadyAtTarget). VERSION_ID is the standard os-release field most images already stamp; a deployment that instead carries an ACL-specific IMAGE_VERSION field can point this variable at that key instead — see below.
TRIDENT_ACL_AGENT_CURRENT_VERSION_FALLBACKalwaysControls what happens when the configured version file is unreadable, the configured key is missing, or its value is empty. always reports 0.0.0 as the node's current version. error fails the operation instead of using a placeholder version. Any other value is used verbatim as the current version, with no format validation.
TRIDENT_ACL_AGENT_KUBERNETES_API_SERVERunsetExplicit override for the Kubernetes API server URL. When unset, the server embedded in the kubeconfig is used as-is.
TRIDENT_ACL_AGENT_KUBERNETES_KUBECONFIG/var/lib/kubelet/kubeconfigPath to the kubeconfig used to reach the Kubernetes API server and authenticate as this node.
TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAMEderived from TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAME_SOURCEThe Node object this agent watches/patches.
TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAME_SOURCEhostnameHow node_name is derived when TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAME is unset. hostname uses the node's own hostname, lowercased — matches AKS's current (but not guaranteed) convention of registering Nodes under the lowercased hostname. kubelet-cert instead reads the Subject CN (system:node:<name>) from kubelet's own client certificate (path set by TRIDENT_ACL_AGENT_KUBERNETES_KUBELET_CERT_PATH) — the identity kubelet itself authenticates to the API server with, so it's guaranteed to match the real Node name. There is deliberately no fallback to the hostname behavior if the cert can't be read or parsed — resolving to the wrong Node is worse than waiting. Instead, the long-running agent treats this the same way it treats its own Node object not existing yet: it retries indefinitely with capped exponential backoff (same policy used while waiting for the Node to reappear), since an unreadable cert at startup is typically just kubelet's own TLS bootstrapping not having finished yet. trident-acl-agent --validate-connection kubernetes, by contrast, makes one fail-fast attempt and reports failure immediately — appropriate for a one-shot diagnostic. Set TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAME explicitly to override either source.
TRIDENT_ACL_AGENT_KUBERNETES_KUBELET_CERT_PATH/var/lib/kubelet/pki/kubelet-client-current.pemPath to kubelet's own client certificate, consulted when TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAME_SOURCE=kubelet-cert. Override when kubelet's cert lives somewhere other than the standard path (e.g. a non-default kubelet --cert-dir). Ignored when node_name_source is hostname or TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAME is set.
TRIDENT_ACL_AGENT_TRIDENT_SOCKETunix:///run/trident/trident.sockThe gRPC Unix socket URI used to reach tridentd.
TRIDENT_ACL_AGENT_ORCHESTRATION_STATE_PATH/var/lib/trident-acl-agent/state.jsonPath to the agent's persistent state file bridging the pre-reboot and post-reboot halves of finalize/rollback across the reboot.
TRIDENT_ACL_AGENT_ORCHESTRATION_COOKIE_ENABLEDfalseEnable Nebraska affinity cookie transmission and persistence. When disabled, existing cookie files are left untouched. See Nebraska affinity cookies.
TRIDENT_ACL_AGENT_ORCHESTRATION_COOKIE_PATH/var/lib/trident-acl-agent/nebraska-cookiePath to the persisted Nebraska affinity cookie, separate from the agent's state file.
TRIDENT_ACL_AGENT_ORCHESTRATION_STAGE_TIMEOUT20mTimeout for each Trident update-stage or rollback-stage call (a humantime duration, e.g. 20m, 1h), not the entire staging workflow.
TRIDENT_ACL_AGENT_ORCHESTRATION_FINALIZE_TIMEOUT10mTimeout for each Trident update-finalize, rollback-finalize, or post-reboot commit call.
TRIDENT_ACL_AGENT_ORCHESTRATION_HEARTBEAT_INTERVAL60sRefresh cadence for the InProgress status heartbeat.
TRIDENT_ACL_AGENT_ORCHESTRATION_NODE_GONE_MAX_WAITunset (wait forever)How long the agent will keep waiting for its own Node object to reappear after a 404 before giving up and exiting (a humantime duration, e.g. 30m, 1h). Unset means the agent never gives up on its own — see below.
TRIDENT_ACL_AGENT_KUBERNETES_VALIDATE_NODE_UUIDfalseWhen true, every Node read verifies status.nodeInfo.systemUUID against this machine's own /sys/class/dmi/id/product_uuid, treating a mismatch the same as the Node not existing (NodeGone) — see below.

Kubernetes API server connectivity (both the startup/recovery Node read and the long-lived watch loop) is retried indefinitely by default; there is no configurable attempt limit for the watch loop. The startup/recovery Node read uses capped exponential backoff with full jitter between attempts (2-second initial interval, doubling up to a 30-second cap, actual delay randomized over [0, 2x the current interval]); the watch loop's reconnect delay is governed entirely by kube::runtime::watcher's own default_backoff() instead, which uses the same shape of backoff. A quiet Node has no reliable signal that distinguishes an isolated, otherwise-healthy reconnect blip from a genuinely ongoing outage, so retrying forever is the simple, correct default.

This also covers the Node object disappearing out from under the agent entirely (e.g. deleted and later recreated by the orchestrator, or by node replacement): a 404 reading or patching the Node (NodeGone) is not treated as fatal. The agent logs that the Node no longer exists and parks, polling with the same capped-backoff policy, until the Node reappears — local state (a pending post-reboot commit, state.json) is untouched by this, so nothing is lost while it waits. By default this waits forever, on the theory that a transient 404 and a genuinely-deleted-forever Node look identical from the agent's point of view and it's cheaper to keep polling than to guess wrong and exit. An operator that wants the old exit-on-Node-gone behavior back (e.g. to let an external supervisor/alert fire instead) can opt into a bound via TRIDENT_ACL_AGENT_ORCHESTRATION_NODE_GONE_MAX_WAIT.

Node identity verification​

When TRIDENT_ACL_AGENT_KUBERNETES_VALIDATE_NODE_UUID is set to true, every Node read - the startup/recovery GET, --validate-connection, and each Node delivered by the long-lived watch stream - additionally checks that the fetched Node's status.nodeInfo.systemUUID matches this machine's own /sys/class/dmi/id/product_uuid. kubelet populates systemUUID by reading that same file, so on a healthy, correctly-identified node the two values always match; a mismatch means the Node object fetched by name doesn't actually describe this machine (e.g. a stale or recycled Node name) and is treated identically to the Node not existing (NodeGone), including the same retry/backoff and TRIDENT_ACL_AGENT_ORCHESTRATION_NODE_GONE_MAX_WAIT handling described above. An empty systemUUID on the Node, or an unreadable local product_uuid file, skips the check (logged at warn) rather than treating it as a mismatch, since either case only means the UUID couldn't be determined, not that the Node describes a different machine. The check is disabled (false) by default because, while kubelet is known to source systemUUID from this same file in general, this hasn't been confirmed as a hard guarantee across every environment this agent runs in — opt in once that's been validated for a given deployment.

Setting env vars via a systemd drop-in​

The agent ships as trident-acl-agent.service, with an ExecStart command but no Environment= lines of its own. Any setting is overridden with a drop-in file, without editing the packaged unit:

$ sudo systemctl edit trident-acl-agent.service

This opens /etc/systemd/system/trident-acl-agent.service.d/override.conf in an editor. For example, to point the agent at a custom annotation namespace:

[Service]
Environment=TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIX=acl.contoso.com

With the prefix above, the orchestrator now reads/writes acl.contoso.com/update-request, acl.contoso.com/update-status, and acl.contoso.com/update-commit-status instead of the acl.microsoft.com/* defaults. Reload and restart to apply:

$ sudo systemctl daemon-reload
$ sudo systemctl restart trident-acl-agent.service

systemctl cat trident-acl-agent.service shows the merged unit (packaged unit plus drop-in), useful for confirming the override took effect.

Configuring the on-disk version​

The agent determines the node's current version by reading a key out of TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH (/etc/os-release by default), defaulting to the key VERSION_ID — the standard os-release field most distributions already stamp. A deployment that keeps its version stamp under a different key, a different file entirely, or both, can point the agent there instead, as long as that file follows the os-release key-value schema (KEY=VALUE lines, optionally quoted, blank lines and # comments ignored):

[Service]
Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH=/etc/my-app-release
Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY=BUILD_VERSION

With this set, the agent reads BUILD_VERSION from /etc/my-app-release (e.g. BUILD_VERSION=202606.29.0) as the node's current version, and compares it against a request's targetVersion the same way it would for VERSION_ID//etc/os-release — including short-circuiting to AlreadyAtTarget when they already match.

If the configured file is unreadable, the key is missing, or its value is empty, the agent consults TRIDENT_ACL_AGENT_CURRENT_VERSION_FALLBACK (always by default): always reports 0.0.0 as the current version; error fails the operation instead of guessing; any other value is used verbatim as the current version, unvalidated.

Diagnostics​

trident-acl-agent --validate-connection <kubernetes|tridentd|nebraska> checks connectivity to a single dependency using the current environment and exits immediately — useful for a systemd ExecStartPre check or manual on-node troubleshooting without running the full orchestrator loop.

The Nebraska probe honors the cookie enable/path settings. When enabled, it reuses an existing persisted cookie without clearing or replacing it; a probe cannot establish affinity. A standalone invocation does not inherit a systemd service's environment, so supply those settings explicitly when checking cookie reuse.

--validate-connection nebraska is the one place TRIDENT_ACL_AGENT_NEBRASKA_ENDPOINT, TRIDENT_ACL_AGENT_NEBRASKA_APP_ID, and TRIDENT_ACL_AGENT_NEBRASKA_TRACK are used: it sends a non-mutating Omaha probe (a bare <app> with no <updatecheck/> or <ping/>) against the configured endpoint/app id/track and reports whether the Omaha server is reachable, without registering the instance or granting/consuming an update. They default to deliberately invalid values (https://nebraska.example.invalid/v1/update, an all-zero UUID, and unspecified, respectively) so this check fails loudly unless a deployment sets them. Since these variables otherwise play no role in the annotation-driven flow, there's no reason to add them to the service's persistent environment (e.g. via a drop-in) — set them just for this one-off invocation instead:

$ sudo TRIDENT_ACL_AGENT_NEBRASKA_ENDPOINT=https://updates.contoso.com/v1/update \
    TRIDENT_ACL_AGENT_NEBRASKA_APP_ID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee \
    TRIDENT_ACL_AGENT_NEBRASKA_TRACK=stable \
    trident-acl-agent --validate-connection nebraska