Trident ACL Agent

trident-acl-agent is an on-node daemon that drives Trident A/B updates from a Kubernetes control plane, using node annotations instead of a direct API call as the trigger. It is the on-node half of Azure Container Linux (ACL)'s update mechanism. Any Kubernetes control-plane component (a custom controller, an operator, or a script driven by kubectl patch) can orchestrate updates across a fleet of nodes by writing to the annotation contract described below, provided it is willing to speak the Omaha protocol for image distribution and honors the agent's per-node protocol.

Deployment

trident-acl-agent ships in the trident-acl RPM subpackage (built alongside, and Requires: the same version of, the main trident package). Installing it:

$ tdnf install trident-acl

lays down the /usr/bin/trident-acl-agent binary and its trident-acl-agent.service unit (packaging/systemd/trident-acl-agent.service) under the systemd unit directory, along with the package's %license-installed LICENSE/NOTICE files. Installing the package does not by itself enable or start the service — a deployment decides when that happens, e.g. by running systemctl enable --now trident-acl-agent.service on the node, or by baking that enablement into the image build (as this repo's own updateimg-acl-agent.yaml test image does via Image Customizer's services: enable list).

The shipped unit carries no Environment= lines beyond ExecStart, so every deployment-specific choice — which annotation prefix to watch, where to read the current version from, which Kubernetes API server to talk to, and so on — is supplied the same way any other systemd service is configured: standard Environment=/EnvironmentFile= constructs, most commonly a drop-in applied on top of the packaged unit. See Configuration below for the full list of variables and Setting env vars via a systemd drop-in for how to apply them without editing the packaged unit.

The annotation contract

An orchestrator (a Kubernetes controller with RBAC permission to PATCH the target Node object) triggers an update by writing a JSON payload to a request annotation on the Node. The agent watches that annotation, drives the requested operation against tridentd, and writes its progress and result via annotations on the same Node.

Three annotation keys make up the contract, all sharing one configurable prefix (acl.microsoft.com by default — see Configuration below):

Annotation	Written by	Purpose
<prefix>/update-request	Orchestrator	Requests stage, finalize, or rollback for this node.
<prefix>/update-status	Agent	Reports the status of the requested operation.
<prefix>/update-commit-status	Agent	Reports the status of the implicit post-reboot commit that follows a finalize or rollback.

A request annotation looks like:

{
  "schemaVersion": "1.0",
  "nodeUpdateId":  "550e8400-e29b-41d4-a716-446655440000",
  "operationId":   "c9d6f0a2-3b41-4e8d-9f27-1a5b6c7d8e90",
  "operation":     "stage",
  "targetVersion": "202606.29.0",
  "server":        "https://nebraska.example.com/v1/update",
  "appId":         "11111111-2222-3333-4444-555555555555",
  "track":         "pin-202606.29.0"
}

• nodeUpdateId identifies one node's update sequence and is held constant across stage → finalize → commit.
• operationId identifies this specific step; the agent uses it to decide whether to start new work, resume in-flight work, or re-emit a cached terminal status as a no-op on a duplicate PATCH.
• targetVersion is the image release version to update to. Required for stage/finalize; omitted for rollback, whose target (the previous partition) is implicit.
• server, appId, and track name the Omaha instance that serves the target image and receives progress events for it. They are required on stage/finalize requests, with no static fallback — a request missing them is rejected with InvalidRequest rather than falling back to a built-in endpoint, so a node can never update from a source the orchestrator did not explicitly choose.

operation maps to Trident invocations as follows:

operation	Trident invocation	Effect
stage	trident update --allowed-operations=stage	Queries the server/appId/track Omaha endpoint for targetVersion, then streams the resulting image to the inactive partition. No reboot.
finalize	trident update --allowed-operations=finalize (gRPC UpdateFinalize, caller-handled reboot)	Arms boot for the staged target, writes a terminal finalize status, then triggers the reboot.
rollback	trident rollback --ab (gRPC RollbackStage/RollbackFinalize, caller-handled reboot)	Swaps back to the previous partition, mirroring finalize on the return path. Only the last update can be undone this way.

A fourth phase, commit, runs implicitly after the post-finalize/ rollback reboot: the agent runs trident commit on the new partition and writes a commit status without needing a separate annotation request. The orchestrator watches <prefix>/update-commit-status as the terminal signal that the reboot half of the update succeeded.

stage end-to-end:

finalize / rollback, spanning the reboot:

A status annotation (<prefix>/update-status or <prefix>/update-commit-status) looks like:

{
  "schemaVersion":   "1.0",
  "nodeUpdateId":    "550e8400-e29b-41d4-a716-446655440000",
  "operationId":     "c9d6f0a2-3b41-4e8d-9f27-1a5b6c7d8e90",
  "operation":       "stage",
  "code":            "Success",
  "message":         "staged update to 202606.29.0",
  "fromVersion":     "202606.15.0",
  "toVersion":       "202606.29.0",
  "tridentVersion":  "0.22.0",
  "startedUtc":      "2026-06-29T12:00:00Z",
  "lastUpdatedUtc":  "2026-06-29T12:03:41Z",
  "finishedUtc":     "2026-06-29T12:03:41Z"
}

• operation is stage, finalize, rollback, or commit (commit only ever appears on <prefix>/update-commit-status, never on <prefix>/update-status).
• code is the outcome — see the table below.
• message is a short, human-readable explanation of code, useful for logs/alerts; treat its exact wording as informational, not something to match on (it may include error detail that varies run to run).
• fromVersion/toVersion are the versions the operation moved between (toVersion is absent for rollback, whose target is implicit).
• tridentError is present only when the failure originated from a Trident call that returned an error response (a remote error); it can appear on both OperationFailed and TargetBootFailed (the reboot landed but the post-reboot commit call itself failed remotely) - every other failure mode (timeouts, connection failures, agent-generated errors like InvalidRequest) carries no tridentError. It has kind/subkind (Trident's own error classification, e.g. SERVICING_ERROR/ab-update-reboot-check, or unknown/unknown when Trident reported failure without a structured error payload) and an optional location (path/line in Trident's source), letting an orchestrator key off structured fields instead of parsing message.
• tridentVersion is the agent's own build version (AGENT_VERSION), stamped on every status write. tridentd and trident-acl-agent are built and released together from a single RPM spec (packaging/rpm/trident.spec), so this value doubles as the tridentd version for that install.
• startedUtc/lastUpdatedUtc/finishedUtc bound the operation: lastUpdatedUtc refreshes on a heartbeat cadence while code is InProgress (see below); finishedUtc is absent until code reaches a terminal value.

code is one of:

code	Terminal?	Meaning
InProgress	No	The operation is running. lastUpdatedUtc refreshes on a heartbeat cadence; a terminal code always follows.
Success	Yes	The operation completed as requested. For commit, this means the reboot landed on the target partition and it was promoted.
AlreadyAtTarget	Yes	stage/finalize was requested for the version the node is already running (per TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY); treated as a no-op success.
NotStaged	Yes	finalize was requested for a nodeUpdateId with no prior successful stage. Issue a stage first.
OperationFailed	Yes	The operation failed for a reason other than a boot/rollback outcome (e.g. the Omaha server has no update for the requested version, or the underlying tridentd call returned an error). See message for detail.
TargetBootFailed	Yes	The post-reboot commit found the node had rolled back to its previous partition instead of booting the target — Trident's own health checks rejected the new boot. The node is back on fromVersion; the orchestrator should treat this as a failed update, not retry the same nodeUpdateId blindly.
AgentInternalError	Yes	A failure in the agent itself rather than in Trident or the requested operation (e.g. it triggered a reboot but the reboot call failed, or it lost track of an in-flight commit). Distinct from OperationFailed so an orchestrator can decide to treat these differently (e.g. retry vs. escalate).
InvalidRequest	Yes	The request annotation itself was rejected before any action was taken — malformed JSON, a schema/version mismatch, a missing required field (server/appId/track/targetVersion), a finalize whose targetVersion doesn't match what was staged, or a second finalize/rollback submitted while one is already pending its post-reboot commit. No Trident operation runs.

See the request/status schema types in crates/trident-acl-agent/src/annotations/protocol.rs for the full contract, including the formal JSON Schema both sides validate against.

Pre/post-reboot state and the watchdog

Because finalize/rollback spans a reboot, the agent persists a small state file (TRIDENT_ACL_AGENT_ORCHESTRATION_STATE_PATH) recording that a commit is pending and a marker for "a boot happened after this point". On restart, the agent checks this state to resume the post-reboot commit step rather than re-running finalize from scratch.

While an operation is in flight, the agent refreshes the InProgress status's lastUpdatedUtc on a heartbeat cadence (TRIDENT_ACL_AGENT_ORCHESTRATION_HEARTBEAT_INTERVAL), so an external watchdog can distinguish a working agent from a stuck one and reprovision a node that never reports a terminal commit status within its SLA.

Configuration

There is no config file. Every setting is an environment variable prefixed TRIDENT_ACL_AGENT_, systemd-style: set it in the unit's own Environment= lines, via a drop-in override, or by any other means that sets the process's environment before it starts.

A variable that is unset, or set to the empty string, falls back to its default. A variable set to a malformed value (a bad URL, a bad duration, an unrecognized mode) causes the agent to fail to start with an error naming the offending variable.

Variable	Default	Description
TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIX	acl.microsoft.com	The annotation-key prefix for the request/status/commit-status annotations (e.g. the acl.microsoft.com in acl.microsoft.com/update-request). Any orchestrator can pick its own namespace here so its annotations don't collide with another controller's.
TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH	/etc/os-release	The file the agent reads to determine the node's currently running version. Any file works, as long as it follows the os-release format (KEY=VALUE lines, optionally single- or double-quoted, blank lines and # comments ignored) — see below.
TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY	VERSION_ID	The key the agent looks up in TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH (/etc/os-release by default) to determine the node's currently running version, used to compare against a request's targetVersion (e.g. to short-circuit to AlreadyAtTarget). VERSION_ID is the standard os-release field most images already stamp; a deployment that instead carries an ACL-specific IMAGE_VERSION field can point this variable at that key instead — see below.
TRIDENT_ACL_AGENT_CURRENT_VERSION_FALLBACK	always	Controls what happens when TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY isn't present at TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH (e.g. a dev/test host, or an image that hasn't started stamping that key yet). always reports 0.0.0 as the node's current version — a sentinel that can never collide with a real release version and cause a false AlreadyAtTarget. error fails the operation instead of using a placeholder version. Any other value is used verbatim as the current version, with no format validation.
TRIDENT_ACL_AGENT_KUBERNETES_API_SERVER	unset	Explicit override for the Kubernetes API server URL. When unset, the server embedded in the kubeconfig is used as-is.
TRIDENT_ACL_AGENT_KUBERNETES_KUBECONFIG	/var/lib/kubelet/kubeconfig	Path to the kubeconfig used to reach the Kubernetes API server and authenticate as this node.
TRIDENT_ACL_AGENT_KUBERNETES_NODE_NAME	The node's own hostname, lowercased	The Node object this agent watches/patches.
TRIDENT_ACL_AGENT_TRIDENT_SOCKET	unix:///run/trident/trident.sock	The gRPC Unix socket URI used to reach tridentd.
TRIDENT_ACL_AGENT_ORCHESTRATION_STATE_PATH	/var/lib/trident-acl-agent/state.json	Path to the agent's persistent state file bridging the pre-reboot and post-reboot halves of finalize/rollback across the reboot.
TRIDENT_ACL_AGENT_ORCHESTRATION_STAGE_TIMEOUT	20m	How long a stage is allowed to run (a humantime duration, e.g. 20m, 1h) before it's considered failed.
TRIDENT_ACL_AGENT_ORCHESTRATION_FINALIZE_TIMEOUT	10m	How long a finalize is allowed to run before it's considered failed.
TRIDENT_ACL_AGENT_ORCHESTRATION_HEARTBEAT_INTERVAL	60s	Refresh cadence for the InProgress status heartbeat.

Kubernetes API server connectivity (both the startup/recovery Node read and the long-lived watch loop) is retried indefinitely; there is no configurable attempt limit for either path. The startup/recovery Node read uses capped exponential backoff with full jitter between attempts (2-second initial interval, doubling up to a 30-second cap, actual delay randomized over [0, 2x the current interval]); the watch loop's reconnect delay is governed entirely by kube::runtime::watcher's own default_backoff() instead, which uses the same shape of backoff. A quiet Node has no reliable signal that distinguishes an isolated, otherwise-healthy reconnect blip from a genuinely ongoing outage, so retrying forever is the simple, correct default.

Setting env vars via a systemd drop-in

The agent ships as trident-acl-agent.service, with no Environment= lines of its own beyond ExecStart. Any setting is overridden with a drop-in file, without editing the packaged unit:

$ sudo systemctl edit trident-acl-agent.service

This opens /etc/systemd/system/trident-acl-agent.service.d/override.conf in an editor. For example, to point the agent at a custom annotation namespace:

[Service]
Environment=TRIDENT_ACL_AGENT_KUBERNETES_ANNOTATION_PREFIX=acl.contoso.com

With the prefix above, the orchestrator now reads/writes acl.contoso.com/update-request, acl.contoso.com/update-status, and acl.contoso.com/update-commit-status instead of the acl.microsoft.com/* defaults. Reload and restart to apply:

$ sudo systemctl daemon-reload
$ sudo systemctl restart trident-acl-agent.service

systemctl cat trident-acl-agent.service shows the merged unit (packaged unit plus drop-in), useful for confirming the override took effect.

Configuring the on-disk version

The agent determines the node's current version by reading a key out of TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH (/etc/os-release by default), defaulting to the key VERSION_ID — the standard os-release field most distributions already stamp. A deployment that keeps its version stamp under a different key, a different file entirely, or both, can point the agent there instead, as long as that file follows the os-release key-value schema (KEY=VALUE lines, optionally quoted, blank lines and # comments ignored):

[Service]
Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH=/etc/my-app-release
Environment=TRIDENT_ACL_AGENT_CURRENT_VERSION_KEY=BUILD_VERSION

With this set, the agent reads BUILD_VERSION from /etc/my-app-release (e.g. BUILD_VERSION=202606.29.0) as the node's current version, and compares it against a request's targetVersion the same way it would for VERSION_ID//etc/os-release — including short-circuiting to AlreadyAtTarget when they already match.

If the configured key is absent from the configured file (for example, on a dev/test host with a minimal os-release), the agent consults TRIDENT_ACL_AGENT_CURRENT_VERSION_FALLBACK (always by default): always reports 0.0.0 as the current version — a sentinel that can never accidentally match a real requested version; error fails the operation instead of guessing; any other value is used verbatim as the current version, unvalidated.

Diagnostics

trident-acl-agent --validate-connection <kubernetes|tridentd|nebraska> checks connectivity to a single dependency using the current environment and exits immediately — useful for a systemd ExecStartPre check or manual on-node troubleshooting without running the full orchestrator loop.

--validate-connection nebraska is the one place TRIDENT_ACL_AGENT_NEBRASKA_ENDPOINT, TRIDENT_ACL_AGENT_NEBRASKA_APP_ID, and TRIDENT_ACL_AGENT_NEBRASKA_TRACK are used: it issues a real update-check query against the configured endpoint/app id/track and reports whether the Omaha server is reachable. They default to deliberately invalid values (https://nebraska.example.invalid/v1/update, an all-zero UUID, and unspecified, respectively) so this check fails loudly unless a deployment sets them. Since these variables otherwise play no role in the annotation-driven flow, there's no reason to add them to the service's persistent environment (e.g. via a drop-in) — set them just for this one-off invocation instead:

$ sudo TRIDENT_ACL_AGENT_NEBRASKA_ENDPOINT=https://updates.contoso.com/v1/update \
    TRIDENT_ACL_AGENT_NEBRASKA_APP_ID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee \
    TRIDENT_ACL_AGENT_NEBRASKA_TRACK=stable \
    trident-acl-agent --validate-connection nebraska