Trident ACL Agent Tests

storm-trident run aclagent is the single supported validation entrypoint for the annotation-driven trident-acl-agent protocol. Unlike Servicing Tests, which drive Trident's own stage/finalize gRPC calls directly, this scenario validates trident-acl-agent itself: it deploys a VM, starts fake in-process test doubles for the Kubernetes API server and the Nebraska/Omaha update server, patches update-request annotations onto the fake Node, and lets the real trident-acl-agent binary running inside the VM drive a full A/B update (and, separately, a rollback) against those fakes.

There is intentionally no fake tridentd — the scenario talks to the real tridentd and real trident-acl-agent running inside the VM.

What It Validates

• trident-acl-agent watching its own Kubernetes Node object for RP-authored acl.azure.com/update-request annotation changes (via kube::runtime::watcher())
• Reading the target OS image version, Nebraska server/appId/track from the request annotation and triggering a real Trident stage + finalize A/B update (or rollback + rollback finalize) through the normal gRPC path
• Writing back progress/result via the acl.azure.com/update-status and acl.azure.com/update-commit-status annotations as the update progresses
• Resuming correctly after a real reboot (see Reboot Choice)

VM Image Contents

The VM image used by this scenario must already contain:

• tridentd.socket installed and enabled (starts tridentd.service on demand)
• trident-acl-agent package installed, but trident-acl-agent.service left disabled in the base image — it must not start before the fake kubeconfig exists (the harness delivers it at runtime and enables/starts the service itself; the update image, by contrast, bakes enablement in directly, since there is no test harness left to run systemctl enable --now after a real A/B update boots into it)
• the same SSH user/key setup expected by the servicing scenario

The harness never writes /etc/trident/trident-acl-agent.conf at all — nebraska.endpoint/nebraska.app_id/nebraska.track are supplied per-request via the update-request annotation's server/appId/track fields instead, and kubernetes.node_name defaults to the node's own real hostname (which the image's hostname is set to match). What the harness does deliver, once at runtime after deploy-vm, is a fake kubeconfig at /var/lib/kubelet/kubeconfig pointing at the fake apiserver. That path lives on its own dedicated ext4 partition (not part of the A/B-swapped /usr/root volume pair in this usr-verity image layout), so it persists across a finalize reboot without needing to be re-delivered.

run-ab-update and run-rollback differ in what happens after their finalize reboot, since they land on different roots:

• run-ab-update reboots into the update image, where trident-acl-agent.service is baked in enabled by default (see VM Image Contents) and a trident-acl-agent-cert-install oneshot service re-installs the Nebraska TLS cert into that root's trust store before the agent starts. It does not re-run the harness's prepare/restart step after this reboot — the agent comes up on its own against the same fake-apiserver instance run-ab-update already started, and re-running prepare/restart here would just interrupt the post-reboot commit it's expected to report.
• run-rollback reboots back onto the base image, where trident-acl-agent.service is disabled by default (per VM Image Contents) and run-rollback starts its own separate fake-apiserver instance. The harness therefore does re-run its short prepare/restart step after this reboot, to re-enable/restart the agent and reconnect it to that fresh fake-apiserver instance — not to re-create the kubeconfig, which already persisted across the reboot.

Nebraska/Image Server TLS

trident-acl-agent requires https for the Nebraska server URL and only accepts https/oci for the package/image URL Nebraska's response resolves to (http and file are rejected) — see Trident ACL Agent. The fake Nebraska proxy and fake image server this scenario starts must therefore serve real TLS, not plain HTTP.

run-ab-update generates one ephemeral, self-signed TLS certificate per run (in memory, in the Go test process — nothing is ever written to the repo) and serves both fakes with it. Both fakes also cap their TLS version at 1.2 (tls.Config.MaxVersion) — this build's OpenSSL has an interop bug with Go's TLS 1.3 post-handshake NewSessionTicket message, surfacing as a bad record MAC error on the very first request to a freshly-started fake server. The cap is a pragmatic workaround; it only affects these test-only fakes, not tridentd/trident-acl-agent's real TLS clients.

The cert is installed into the VM's system trust store in two steps, both required:

• System trust bundle (update-ca-trust extract), which regenerates the PEM bundle files under /etc/pki/ca-trust/extracted/ — what tools like curl/openssl s_client consult by default.
• System trust directory (a copy under /etc/pki/tls/certs/, followed by openssl rehash there to create the hash-named symlink OpenSSL's directory-based lookup requires). This step is required in addition to update-ca-trust: this image's statically-linked reqwest/OpenSSL stack — used by both tridentd's image downloader and trident-acl-agent's Nebraska client — defaults to directory-based lookup (/usr/lib/ssl/certs, symlinked to /etc/pki/tls/certs on Azure Linux), which update-ca-trust never populates. Skipping this step reproduces as an unknown certificate authority TLS alert from tridentd's image download and from this harness's own --validate-connection nebraska check, even though curl/openssl s_client would report the same cert as trusted. This is a platform/build quirk of this test image, not a bug in trident/trident-acl-agent.

trident-acl-agent has no CA-override config of its own — a private CA is expected to be trusted via the node's system trust store, installed at image-build time, not via a runtime agent config knob — so neither install step above survives finalize's A/B root swap (each root has its own /etc, and /etc is itself an overlay backed by the per-slot trident-overlay-a/-b partitions). prepareVmForAclAgent also drops the cert onto the persistent, non-A/B /var/lib/trident-acl-agent partition, and the update image bakes in a trident-acl-agent-cert-install.service oneshot unit (Before=trident-acl-agent.service, ConditionPathExists= that persistent-partition file) that redoes the two steps above into whichever root just booted, before trident-acl-agent.service starts — no SSH round-trip from this harness is needed after the reboot. The base image has no need for this unit: run-rollback reboots back into it, but never queries Nebraska (see below), so there is nothing to reprovision.

run-rollback never queries Nebraska at all (see its own scenario description above), so it starts no TLS-serving fakes and installs no cert.

This is why the harness needs no static Nebraska endpoint/CA configuration of its own: endpoint, app_id, and track are all supplied per-request via the update-request annotation, as described in VM Image Contents.

Prerequisites

• Linux host with root access
• libvirt and QEMU installed and configured
• Docker (for building images with Image Customizer)
• Go 1.25+ (for building Go tools; matches tools/go.mod's go 1.25.0)
• Rust (latest stable, for building Trident and trident-acl-agent)

See Dependencies for full build dependency details.

Building Dependencies

1. Build Trident, trident-acl-agent, and RPMs

Always build through make, not a raw cargo build, when the RPM tarball needs to reflect a source change — make injects the dev version string (TRIDENT_VERSION) that the RPM spec's %check step verifies against. A plain cargo build skips that and produces an RPM build failure.

make target/release/trident target/release/trident-acl-agent
make bin/trident-rpms.tar.gz

2. Build Go Tools

make bin/storm-trident

3. Generate SSH Keys

make artifacts/id_rsa

note

The VM images below bake in the public key from artifacts/id_rsa.pub (via the files/id_rsa.pub Makefile rule), not ~/.ssh/id_rsa.pub. Always pass --ssh-private-key-path artifacts/id_rsa when running the scenario locally — using your personal ~/.ssh/id_rsa doesn't fail fast, it just hangs/retries during check-deployment's SSH auth.

4. Download the qemu_guest Base Image

Same base image as the servicing tests — see Servicing Tests, step 4 for details.

5. Build the Base and Update VM Images

The scenario needs two images built from the current source:

# Base image: trident-acl-agent installed but disabled
make artifacts/trident-vm-acl-agent-testimage.qcow2

# Update image: what the agent updates the VM to
make artifacts/trident-vm-acl-agent-update-testimage.cosi

Rebuild after any trident-acl-agent change

Both image targets embed the RPM built in step 1. If you only rebuild the Rust binary and re-run the scenario without rebuilding these images, you are still testing the old binary baked into the existing qcow2/cosi files — the failure (or fix) you're trying to observe silently won't reproduce. Clear stale artifacts first if you're not sure they're current:

rm -f artifacts/trident-vm-acl-agent-testimage.qcow2 \
      artifacts/trident-vm-acl-agent-update-testimage.cosi

Running the ACL Agent Scenario

The scenario requires root access for VM creation via virt-install:

sudo bin/storm-trident run aclagent \
    --artifacts-dir ./artifacts \
    --output-path /tmp/aclagent-output \
    --ssh-private-key-path ./artifacts/id_rsa \
    --postgres-image postgres:16-alpine \
    --image-pattern '^trident-vm-acl-agent-testimage\.qcow2$' \
    --cosi-pattern '^trident-vm-acl-agent-update-testimage\.cosi$' \
    --verbose

--image-pattern scopes the QEMU image lookup to this scenario's own base image. The flag's shared default (^trident-vm-.*-testimage\.qcow2$) matches every trident-vm-*-testimage.qcow2 in the artifacts directory; FindFile errors out if more than one file matches, so a local artifacts/ directory holding another storm scenario's test image would otherwise make deploy-vm fail. Pass it explicitly here rather than relying on the default.

--cosi-pattern does the same for the update .cosi image used when --image-path is not set: the default (.*\.cosi$) matches any .cosi file anywhere under --artifacts-dir, so a shared artifacts directory holding another scenario's update image would make FindFile fail with a "multiple files found" error (or, worse, silently pick the wrong single match). Scoping it to this scenario's own update image name avoids both.

--postgres-image already defaults to the public postgres:16-alpine image from Docker Hub, so it only needs to be passed explicitly here to make that default visible; a plain internet connection is all this scenario (and its go test ./tools/storm/aclagent/proxies/... unit tests) need to pull it. Trident's own pipelines override this to an internal ACR mirror instead, since the pools they run on restrict egress to an allow-list that excludes Docker Hub - see postgresImage in .pipelines/templates/stages/testing_acl_agent/trident-acl-agent-test.yml.

Test Cases

The scenario runs these test cases in order:

1. deploy-vm — Copies the base qcow2 image and creates a QEMU VM
2. check-deployment — Verifies the VM booted and is accessible via SSH
3. run-node-resilience — Proves trident-acl-agent survives its own Node object disappearing (see NodeGone/await_node_recreation) instead of exiting, across the two ways that can actually happen against a real apiserver: phase 1 restarts trident-acl-agent.service while the fake Node is "deleted" (NodeStore.DeleteNode), forcing the agent's startup Node read into the 404; phase 2 patches in a stage request and deletes the Node immediately after, so the already-running agent's own in-flight status PATCH (no restart at all) hits the 404 instead. Both phases assert the service's systemd MainPID stays unchanged throughout (i.e. it never crashes or restarts) and that it resumes once NodeStore.RestoreNode brings the Node back — all before any fake Nebraska/image-server mocks or real update/rollback traffic is involved
4. run-ab-update — Starts the fake apiserver and fake Nebraska/Omaha endpoints in-process (the latter over HTTPS — see Nebraska/Image Server TLS), delivers a fake kubeconfig and restarts trident-acl-agent.service, patches the acl.azure.com/update-request annotation, and waits for trident-acl-agent to drive a real Trident A/B update to completion (including a real reboot)
5. run-rollback — Exercises the rollback annotation end-to-end against tridentd's RollbackService gRPC API, followed by a real reboot and post-reboot commit; must run after run-ab-update in the same VM lifetime, since it rolls back to the volume active before that update
6. collect-logs — Fetches trident-acl-agent and Trident logs from the VM via SSH; also runs automatically (with a journalctl dump for trident-acl-agent.service) if an update/rollback/resilience step times out waiting for the service to become active, to make crash-loops self-diagnosing
7. cleanup-vm — Destroys the QEMU VM

Flags

Flag	Description	Default
--artifacts-dir	Directory containing VM images	.
--output-path	Output directory for logs	./output
--platform	qemu or azure	qemu
--ssh-private-key-path	Path to SSH private key	~/.ssh/id_rsa
--api-server-port	Port for the fake Kubernetes API server	18080
--nebraska-port	Port for the fake Nebraska/Omaha server	18081
--host-endpoint-ip	Host IP the VM reaches the fake endpoints at	192.168.122.1
--image-path	Real .cosi update image to serve during staging	none - falls back to --cosi-pattern search under --artifacts-dir
--image-pattern	Regex to find the QEMU base image under --artifacts-dir	^trident-vm-.*-testimage\.qcow2$
--cosi-pattern	Regex to find the update .cosi image under --artifacts-dir when --image-path is not set	.*\.cosi$
--postgres-image	Container image for the ephemeral Postgres instance backing the fake Nebraska endpoint	postgres:16-alpine
--verbose	Enable verbose logging	false
--test-case-to-run	Run a specific test case only	all

Reboot Choice

This scenario uses a real VM reboot rather than a shim: trident-acl-agent issues a genuine systemctl reboot on finalize, and the scenario polls SSH until it goes unreachable (confirming the reboot actually happened) and then reachable again (confirming the VM came back up), exercising the agent's real post-reboot resume logic end to end. This is slower than a shim-based approach, but it validates the real reboot path instead of a simulation of it.

Debugging Failures

If trident-acl-agent.service gets stuck reporting activating and the test times out, that almost always means a crash-restart loop, not a slow start — the unit has no explicit Type=, so Type=simple (the implicit default) is used, and systemd marks such units active immediately on fork/exec with no readiness signal. A persistent activating state for the full wait window can only mean Restart=on-failure (RestartSec=5) is cycling the service.

run-ab-update's wait-for-active check captures journalctl -u trident-acl-agent.service --no-pager -n 200 on timeout and includes it in the test failure, so the actual crash reason (e.g. a panic, a fatal error from the Kubernetes client, or a config problem) should be visible directly in the CI log or local output without a separate log-collection step.

The fake Kubernetes API server (tools/storm/aclagent/proxies/apiserver.go) is a minimal, hand-rolled HTTP handler — it does not implement the full Kubernetes API surface. If trident-acl-agent is changed to make a new kind of API call (a different verb, a new field selector, list pagination, etc.), the fake apiserver's routing may need a corresponding update or the call will simply 404 and (since trident-acl-agent treats such client errors as fatal) crash-loop the service. For example, migrating node-watching from polling to kube::runtime::watcher() introduced an initial LIST call to the collection endpoint (GET /api/v1/nodes?fieldSelector=...) that the fake server didn't originally route, only the singular /api/v1/nodes/<name> path — surfacing as exactly this crash-loop symptom until the collection route was added.