nacelle documentation

nacelle is an experimental Tokio-based Rust library for streaming application handlers across TCP and HTTP transports.

This book is the narrative documentation site. It follows the same broad delivery model as the Rust Book: chapter-oriented Markdown, search, keyboard navigation, and a local/offline build. Its content is organized like Django's documentation:

• Tutorials take you through a working path.
• Topic guides explain concepts and design choices.
• How-to guides solve specific operational tasks.
• Reference pages document exact behavior and APIs.

Rust API reference is still generated separately with cargo doc.

Start here

If you are new to nacelle, read:

1. Getting started
2. Architecture
3. Configure production limits

If you are validating performance, read:

1. Run the stress harness
2. Compare performance profiles
3. Performance model

Internal readiness plans and assessments live under docs/internal and are not part of this public book.

Getting started

This tutorial gets a minimal TCP service running with the repository's example length-delimited protocol.

Add nacelle

In this workspace, the umbrella crate is nacelle. The unpublished reference protocol package is an example consumer of its TCP and codec APIs:

#![allow(unused)]
fn main() {
use nacelle::core::pipeline::handler_fn;
use nacelle::core::{NacelleError, NacelleTelemetry};
use nacelle::tcp::{TcpRequestContext, TcpResponse, TcpServer};
use nacelle::NacelleApp;
use nacelle_reference_protocol::LengthDelimitedProtocol;
}

Build a handler

TCP handlers receive a protocol-specific TcpRequestContext and must complete it through its typed responder.

#![allow(unused)]
fn main() {
let handler = handler_fn(
    |mut context: TcpRequestContext<LengthDelimitedProtocol>| async move {
    while let Some(chunk) = context.request_mut().body.next_chunk().await {
        let _ = chunk?;
    }

    context.respond(TcpResponse::bytes("ok")).await
});
}

Start the app

#![allow(unused)]
fn main() {
let addr = "127.0.0.1:8080".parse().map_err(NacelleError::protocol)?;
let server = TcpServer::<LengthDelimitedProtocol>::builder()
    .protocol(LengthDelimitedProtocol)
    .handler(handler)
    .build()?;

NacelleApp::with_telemetry(NacelleTelemetry::default())
    .with_ctrl_c_shutdown()
    .tcp("echo", addr, server)
    .run()
    .await?;
Ok::<(), NacelleError>(())
}

Next steps

• Run cargo run -p nacelle-examples --bin app_core to see one app core served through multiple protocol adapters.
• Read the architecture guide to understand the request path.
• Read runtime limits and backpressure before raising connection counts.
• Use Run the stress harness to validate a local build.

Run the stress harness

The stress harness has two binaries:

• nacelle-stress-server, from nacelle-stress-server
• nacelle-stress-test, from nacelle-stress-test

Run the convenience script:

./examples/run-stress-test.sh

The script reads root config.toml by default. Pass --config to select a repeatable benchmark profile. If the effective tls_self_signed value is true, it passes --tls-insecure to the client so the server and client speak the same transport.

For a plain TCP baseline, use examples/nacelle-stress-server/configs/tcp.toml.

For full details, see the how-to guide:

Run the server:

cargo run --release --package nacelle-stress-server -- --config examples/nacelle-stress-server/configs/tcp.toml

Run a bounded client smoke test:

cargo run --release --package nacelle-stress-test -- --connections 32 --pipeline 16 --duration-secs 15

The examples/run-stress-test.sh and examples/run-stress-test.ps1 helpers accept --config/-Config and pass --tls-insecure to the stress client only when the effective tls_self_signed value is true.

The stress client enables its Rustls support by default so --tls-insecure works with the local self-signed server. For a Rustls-free plain TCP build, run both stress binaries with --no-default-features and use examples/nacelle-stress-server/configs/tcp.toml.

Repeatable profiles:

• examples/nacelle-stress-server/configs/tcp.toml: plain TCP baseline.
• examples/nacelle-stress-server/configs/tcp-low-memory.toml: plain TCP with mimalloc low-memory behavior and experimental runtime memory accounting.
• examples/nacelle-stress-server/configs/tcp-tls.toml: TCP wrapped in self-signed TLS.

Linux example:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

PowerShell example:

.\examples\run-stress-test.ps1 -Config examples/nacelle-stress-server/configs/tcp.toml -ServerThreads 48 -Connections 256 -Pipeline 8 -DurationSecs 30 -PayloadBytes 256

The stress server installs a metrics-util recorder and prints a compact metrics snapshot every 5 seconds. It enables request started/completed counters plus request/response byte counters by default. The generic telemetry API groups those switches under request_metrics; the stress server exposes byte accounting as byte_metrics = true. Use --no-byte-metrics for a lower-overhead recorder run. Use --no-default-features with the plain TCP config for a Rustls-free, system-allocator baseline; metrics collection remains active.

The launch helpers enable experimental-memory automatically when an effective config contains max_memory_bytes. For a direct low-memory server run, pass --features experimental-memory explicitly.

The Tokio stress server default build includes tls-self-signed support. The checked-in root config.toml enables tls_self_signed = true, so the local stress client should use --tls-insecure with that default config. Use --no-default-features with examples/nacelle-stress-server/configs/tcp.toml when you need a Rustls-free plain TCP baseline.

CI-friendly scenarios should stay short and deterministic:

• baseline echo throughput
• max connection cap
• max request cap
• slow reader
• slow writer
• graceful shutdown under load

Heavy RPS and soak tests should run manually or nightly on dedicated Linux hosts.

Serve TCP with self-signed TLS

Use tls-self-signed for local load tests and auto-deploy flows that need a certificate immediately. It implies the rustls provider.

#![allow(unused)]
fn main() {
use nacelle::core::pipeline::handler_fn;
use nacelle::core::NacelleError;
use nacelle::rustls::NacelleTlsConfig;
use nacelle::tcp::{TcpRequestContext, TcpResponse, TcpServer};
use nacelle_reference_protocol::LengthDelimitedProtocol;

let generated = NacelleTlsConfig::self_signed(["localhost", "127.0.0.1"])?;
let server = TcpServer::<LengthDelimitedProtocol>::builder()
    .protocol(LengthDelimitedProtocol)
    .handler(handler_fn(
        |mut context: TcpRequestContext<LengthDelimitedProtocol>| async move {
        let mut response = Vec::new();
        while let Some(chunk) = context.request_mut().body.next_chunk().await {
            response.extend_from_slice(&chunk?);
        }
        context.respond(TcpResponse::bytes(response)).await
    }))
    .build()?;

server
    .serve_tcp_tls("127.0.0.1:8443".parse()?, generated.tls_config)
    .await?;
Ok::<(), NacelleError>(())
}

Self-signed certificates are for local and automated test flows. Public edge deployments should use managed certificate material and a documented rotation process.

For OpenSSL-backed TCP TLS, enable openssl and use NacelleOpenSslConfig::from_pem_files(...) with serve_tcp_openssl(...). Use openssl-vendored only when the build machine has the tooling needed to compile OpenSSL from source. The openssl feature enables provider-neutral tls without selecting Rustls.

How the documentation is organized

This book uses four kinds of documentation:

• Tutorials are guided paths. They assume little context and aim for a working result.
• Topic guides explain how nacelle works and why it is shaped the way it is.
• How-to guides are recipes for specific tasks.
• Reference pages are precise descriptions of behavior, APIs, and protocol contracts.

Use tutorials when you are new, topic guides when you need a model of the system, how-to guides when you have a concrete job, and reference pages when you need exact details.

Architecture

Nacelle is organized as a small core plus protocol-specific transport crates.

Crate Layout

• nacelle-core: shared handler, request/response body, limits, lifecycle, telemetry, and provider-neutral TLS metadata.
• nacelle-openssl: OpenSSL configuration reload and negotiated metadata extraction.
• nacelle-rustls: Rustls configuration reload, certificate parsing, SNI policy, and negotiated metadata extraction.
• nacelle-tcp: TCP/Unix socket server, protocol trait, connection loop, and listener runtime.
• nacelle-http: Hyper HTTP/1 server, HTTP request policy, and HTTP TLS listener integration.
• nacelle: convenience crate with core, codec, tcp, http, and runtime capability namespaces.
• examples/nacelle-reference-protocol: unpublished length-delimited protocol fixture used by examples, tests, benchmarks, and stress tools.
• examples/nacelle-examples: unpublished runnable examples and benchmarks.

The reference protocol intentionally stays out of nacelle-core and nacelle-tcp. It demonstrates the public protocol and codec contracts without becoming part of the published library API.

App Core And Protocol Adapters

Nacelle is organized so application behavior lives behind statically dispatched handler boundaries. TCP handlers receive TcpRequestContext<P> and complete requests with the response type associated with P. HTTP handlers receive HttpRequestContext<State> and complete through HttpResponse.

TCP Protocol implementations are adapters: they decode a wire format into request metadata and encode responses back into frames. Swapping protocols should not require rewriting the app core. The app-first serving path wires concrete typed servers together with NacelleApp::new().tcp(...).http(...).run(). The app owns shared runtime state, telemetry, shutdown, and supervision. nacelle::runtime::NacelleHost remains available for services that need manual listener control.

Provider-neutral TLS identity and per-connection metadata live in nacelle-core. Concrete configuration, certificate handling, reload policy, and negotiated metadata extraction live in nacelle-rustls and nacelle-openssl. Transport crates retain listener lifecycle and async I/O adaptation so provider crates do not depend back on TCP or HTTP. The nacelle facade preserves the rustls, openssl, and tls-self-signed feature names and exposes provider namespaces.

Request Flow

listener
  -> connection limit
  -> connection task
  -> protocol/HTTP decode
  -> request limit
  -> handler
  -> response body encode/stream

TCP and Unix socket listeners use the nacelle-tcp Protocol trait and its associated request, response, and connection-state types to decode request heads and encode bounded response frames. HTTP uses nacelle-http with Hyper HTTP/1 and a transport-owned typed request/response pipeline.

Connection metadata carries the transport, a stable connection id, listener label, peer and local addresses, local Unix socket path, effective peer IP, and TLS metadata. The typed TCP pipeline presents this immutable metadata to handlers as ConnectionInfo through RequestContext.

The pipeline model for connection-local application state is ConnectionContext<State>. TCP protocols construct Protocol::ConnectionState once per accepted connection. TcpServer and LocalTcpServer expose shared state as Arc<P::ConnectionState>. SerialTcpServer and LocalSerialTcpServer instead lend &mut ConnectionContext<P::ConnectionState> to exactly one awaited handler at a time, avoiding an async mutex for state confined to one serial connection loop. No dynamic extension map participates in either request path.

The shared multi-thread Tokio runtime remains the default. Experimental thread-per-core execution is explicit and currently supports TCP, HTTP, Rustls TCP/HTTPS, required OpenSSL TCP, and optional plaintext/OpenSSL TCP on Linux. Each selected worker owns a current-thread Tokio runtime, LocalSet, reuse-port listener, protocol, and LocalHandler pipeline. Accepted streams, handshakes, and connection tasks remain on the accepting worker. Unsupported platforms fail configuration; Nacelle does not silently switch runtime topology.

Serial mutable-state listeners support plain TCP, required OpenSSL, optional OpenSSL detection, and Unix sockets in the shared runtime. Worker-local serial listeners support plain TCP, required OpenSSL, and optional OpenSSL detection. Rustls serial and worker-local Unix socket serial variants are not exposed.

ThreadPerCoreConfig::with_max_threads(...) caps the selected worker set after automatic or explicit selection and before any worker thread is created. It does not configure the caller-owned shared Tokio runtime.

Thread-per-core resource accounting is selected statically at startup. Global mode shares all existing counters. Worker mode partitions finite connection, request, streaming, and per-peer capacities in configured worker order while retaining a single shared FIFO hard memory ceiling when experimental-memory is enabled. Worker factories execute once per worker. Process-wide client pools, backend limits, and other external resource budgets must be shared explicitly when they must not scale with worker count.

HTTP-specific edge policy remains in nacelle-http: Host, method, URI/header shape checks, per-peer request rate limits, access logging, and security header injection. TCP keeps protocol semantics in the protocol implementation and shared lifecycle/limit enforcement in core.

Runtime State

NacelleRuntimeState owns shared budgets and counters. Connection, request, and streaming-task limits are non-blocking atomic bounded counters. With the non-default experimental-memory feature, memory uses a checked allocation guard that releases on drop.

This keeps the common request path allocation-light while still enforcing bounded defaults.

Bodies

NacelleBody has three internal shapes:

• empty/single chunk for fast small responses
• buffered chunks for decoded TCP bodies already in memory
• streaming channel for request/response bodies that move asynchronously

With experimental-memory, TCP streaming request bodies reserve their declared length by default. Set TcpStreamingBodyMemoryPolicy::LiveChunks to reserve each chunk before it is detached from read-ahead or allocated for a socket read. The charge then follows the chunk through the body channel and any application-owned Bytes clones. HTTP request bodies reserve Content-Length when Hyper exposes a bounded size hint. TCP protocols can override Protocol::max_request_body_bytes(...) to choose a phase-aware body limit from the decoded head, immutable connection metadata, and concrete connection state immediately after head decoding and before body-specific allocation or additional body reads.

Shutdown

Listeners own a JoinSet of accepted connection tasks. Shutdown proceeds in stages:

1. signal shutdown
2. stop accepting
3. drain active connection tasks
4. abort remaining tasks after the drain deadline
5. emit shutdown telemetry

Task tracking is at the connection boundary, not the per-request hot path.

Observability

Telemetry is deliberately low-cardinality. Reasons are static strings such as connections, request_body_bytes, or http_body_read.

Nacelle emits through the backend-neutral metrics facade and does not own an exporter. With experimental-memory, runtime state and shared memory budgets cache a memory gauge handle and update it at existing acquire/release transitions. NacelleTelemetry owns lifecycle, request, phase, error, and byte metrics for all transports. Transports that can provide extra low-cardinality detail attach a NacelleMetricsContext with listener, protocol, transport, and TLS labels. Applications must install their recorder before constructing these values so cached handles bind to it; without a recorder the handles are no-ops.

NacelleTelemetryConfig separates connection, request, runtime, error, and TCP phase-duration metric domains. All domains except phase durations are enabled by default for compatibility. Request sub-switches live under request_metrics: started/completed counters and byte counters are on by default; in-flight counters and duration histograms are disabled by default. with_metrics(false) gates every domain without replacing those individual settings, so re-enabling metrics restores the configured domain policy. Telemetry observers are independent and continue receiving events while metrics are disabled.

TCP phase histograms additionally require the non-default phase-timing Cargo feature. Enable them deliberately with NacelleTelemetry::default() builder methods on the server or app when you need diagnostic detail and can afford the extra timers and metric writes. Without phase-timing, TCP phase timer storage and Instant calls are not compiled. Core/HTTP request paths do not start a request timer unless duration metrics or HTTP access logging are enabled.

Runtime limits and backpressure

Runtime limits are enforced through NacelleRuntimeState. They are intended to make overload predictable rather than perfectly invisible.

Key budgets include:

• active connections
• in-flight requests
• streaming body tasks
• optional per-peer connections
• experimental runtime memory budget allocations
• request and response body size
• core handler timeout
• TCP read, write, final shutdown, and idle timeouts through NacelleTcpLimits
• HTTP header, body, write, keep-alive, and connection-age limits through NacelleHttpLimits
• TLS handshake timeouts through the TLS config types

The important production habit is to size limits together. A high connection count with large read and response buffers is a memory budget decision, not just a concurrency decision.

For configuration details:

Start from NacelleLimits::default() and tune shared resource budgets for the deployment. Use NacelleTcpLimits for TCP socket timeouts and NacelleHttpLimits for HTTP edge timeouts and keep-alive behavior. Active connections, in-flight requests, streaming tasks, body sizes, handler timeouts, and transport timeouts are bounded by default. Runtime memory budgeting is compiled only with the non-default experimental-memory feature. Without that feature, memory fields, allocation APIs, transport accounting, ownership tracking, waiters, and the memory gauge are absent.

Recommended presets:

• Internal service: keep defaults, set body limits to the largest expected payload, and run behind process supervision.
• Internet-facing behind proxy: cap connections and requests to the container budget, keep 30 second transport timeouts, and let the proxy own coarse traffic filtering or certificate automation when desired.
• Proxy-aware HTTP: configure NacelleHttpPolicy::with_trusted_proxy_ips(...) only with known proxy addresses before allowing Forwarded or X-Forwarded-For to affect per-peer request limits or request metadata.
• Direct HTTPS listener: enable http,tls, load certificate/key material through NacelleTlsConfig, configure an SNI allowlist with from_pem_with_allowed_server_names or from_der_with_allowed_server_names, set a short TLS handshake timeout, configure max_connections_per_peer and max_connection_opens_per_peer_per_second, enable HTTP access logs, and attach NacelleHttpPolicy with Host, method, URI, header, security-header, and per-peer request-rate limits.
• Direct TCP Rustls listener: enable tcp,tls, load certificate/key material through NacelleTlsConfig, register it with NacelleApp::tcp_tls(...), and keep protocol-level authentication/authorization in the application protocol.
• Direct TCP OpenSSL listener: enable tcp,openssl, load certificate/key material through NacelleOpenSslConfig, register it with NacelleApp::tcp_openssl(...), and configure the SslAcceptor yourself when you need OpenSSL-specific policy.
• Local load-test/autodeploy HTTPS: enable tls-self-signed and call NacelleTlsConfig::self_signed(...); do not treat generated certificates as a public trust or rotation strategy.
• High concurrency: reduce TCP buffer capacities before raising max_connections, and tune NacelleTcpLimits separately from shared resource budgets.

Experimental memory budget:

connection_budget =
  max_connections * (read_buffer_capacity + response_buffer_capacity)
body_budget =
  concurrent_buffered_or_streaming_bodies * max_request_body_bytes
total_budget =
  connection_budget + body_budget + handler/backend/runtime headroom

Enable experimental-memory, then set NacelleLimits::with_max_memory_bytes(...) to activate enforcement. With the feature enabled, its default max_memory_bytes is usize::MAX, so accounting does not reject allocations until an explicit finite limit is configured. Nacelle allocates from that budget for connection buffers and buffered or streaming request bodies. The limiter accounts for Nacelle-managed allocations, not total process RSS, so keep process or container memory limits in place. Request body allocations wait in FIFO order when the budget is full. The default wait limit is NacelleLimits::memory_allocation_timeout == Some(5s), and can be tuned with with_memory_allocation_timeout(...) or disabled with without_memory_allocation_timeout(). A timed-out waiter returns NacelleError::Timeout("memory_allocation").

The memory budget is an accounting guard, not a buffer allocator: it grants a NacelleMemoryAllocation that tracks bytes the transport or application intends to hold elsewhere, and releases those bytes when the guard is dropped. When Nacelle associates an allocation with NacelleBody, chunks extracted from the body retain the allocation through their underlying Bytes ownership. Dropping or consuming the body does not release the charge while an extracted chunk or any clone of that chunk remains live. Applications can allocate from the same budget through NacelleRuntimeState::memory_budget(). Use try_allocate(...) for immediate admission, allocate(...) for FIFO waiting, or allocate_with_timeout_and_shutdown(...) when app work should stop waiting during shutdown.

TCP processes requests sequentially per connection. request_body_channel_capacity controls the queued streaming chunks between the socket reader and handler. HTTP uses Hyper's internal buffers plus Nacelle's body queue, so leave extra headroom when enabling large request bodies.

TCP streaming bodies use TcpStreamingBodyMemoryPolicy::DeclaredLength by default, preserving whole-body admission before handler dispatch. The opt-in LiveChunks policy admits bodies larger than the currently available memory budget when each chunk fits: it charges chunks being read, queued, or retained by the handler and releases each charge after the final Bytes clone drops. request_body_chunk_size sets the allocation granularity. A handler that keeps earlier chunks while awaiting the rest of a body can exhaust its own memory budget and reach memory_allocation_timeout; use declared-length accounting or enough body headroom for handlers that aggregate complete payloads.

For TCP protocols, NacelleLimits::max_request_body_bytes is the default body limit. Override Protocol::max_request_body_bytes(request, connection, state, default_limit) to choose a per-request limit from the decoded head, immutable connection metadata, and concrete connection state before body-specific allocation or additional body reads. There is no dynamically typed connection extension.

Thread-per-core server factories execute once per configured worker. Nacelle's global or partitioned runtime counters do not partition external client pools or backend resources automatically; pass explicitly shared resources into worker factories when process-wide budgets must remain global.

Cap Nacelle-owned worker threads after any worker-selection strategy with ThreadPerCoreConfig::with_max_threads(...). The effective capped worker count must also be used when constructing ThreadPerCoreLimits::worker(...). Shared runtime threads belong to the caller's Tokio runtime and are configured there.

Dangerous configurations:

• unbounded connections with large per-connection buffers
• large body limits without a process/container memory limit
• disabled timeouts on internet-facing listeners
• direct internet-facing HTTP without Host/header/method/URI policy
• direct internet-facing TLS without an SNI allowlist
• direct internet-facing listeners without per-peer connection caps
• direct internet-facing listeners without per-peer connection-open rate caps
• direct internet-facing HTTP without per-peer request caps and access logs
• trusting forwarded peer headers without an explicit trusted proxy list
• generated self-signed certificates used as a long-lived public-edge certificate strategy
• high keep-alive connection counts without proxy-level idle limits

TLS certificate rotation:

#![allow(unused)]
fn main() {
let tls = NacelleTlsConfig::from_pem_files("cert.pem", "key.pem")?;
tls.reload_from_pem_files("next-cert.pem", "next-key.pem")?;
}

Reloads affect new TLS handshakes. Existing connections continue with the configuration negotiated when they connected.

Operations model

Deployment Shape

Recommended internet-facing shape:

client -> proxy/load balancer/TLS -> Nacelle service

The proxy should own TLS, coarse connection filtering, and external idle timeouts. Nacelle owns application limits, protocol handling, body limits, and graceful shutdown.

Startup

Use explicit limits and print the effective config for stress or benchmark services. For production services, record:

• process version and git SHA
• configured limits
• listener addresses
• feature flags
• allocator settings

Thread-per-core mode is experimental and Linux-only. Select workers explicitly, record logical CPU ids and affinity settings, and treat any bind, affinity, or worker initialization failure as a whole-runtime startup failure. TCP, HTTP, Rustls TCP/HTTPS, required OpenSSL TCP, and optional plaintext/OpenSSL TCP have worker-local stacks. Performance qualification remains under implementation. Use ThreadPerCoreConfig::with_max_threads(...) to cap any selected worker set; configure the caller-owned Tokio builder separately for shared-runtime thread limits.

Shutdown

Use NacelleApp::with_ctrl_c_shutdown() for the standard signal path, or pass a shared NacelleShutdown through NacelleApp::with_shutdown(...). Configure the drain deadline with with_shutdown_drain_timeout(...). Advanced manual hosts can use nacelle::runtime::NacelleHost::shutdown_and_wait_timeout(...). Short deadlines protect deploy velocity but can abort in-flight work.

Expected shutdown telemetry:

• shutdown requested
• listener stopped accepting
• drain started
• drain completed or timed out
• active connections aborted

Metrics To Watch

• nacelle.connections.active
• nacelle.requests.active
• nacelle.streaming_tasks.active
• nacelle.memory.used_bytes
• nacelle.connections.accepted
• nacelle.connections.closed
• nacelle.connections.in_flight
• nacelle.requests.started
• nacelle.requests.completed
• nacelle.rejections
• nacelle.timeouts
• nacelle.requests.failed
• nacelle.request.bytes
• nacelle.response.bytes

Alerts should focus on sustained saturation, rising rejections, timeout spikes, and memory approaching the configured budget.

Benchmarking

Nacelle emits metrics through the metrics facade according to NacelleTelemetryConfig. Connection, runtime, and error domains are on by default. Request metrics are grouped under request_metrics: started, completed, and byte_counts are on by default, while in_flight and duration_ms are opt-in. TCP phase histograms require the non-default phase-timing Cargo feature and explicit runtime activation.

Use NacelleTelemetry::default().with_metrics(false) to suppress all Nacelle metric domains while retaining any application recorder. This global gate does not erase individual domain settings, and telemetry observers remain active. Use with_connection_metrics, with_request_metrics, with_runtime_metrics, with_error_metrics, and with_phase_duration_metrics for independent policy. A shared NacelleRuntimeState has one runtime-metric policy; configure servers sharing that state consistently before serving traffic.

The stress server installs a debugging recorder and prints a compact console snapshot every 5 seconds. Production applications should install their chosen recorder before constructing Nacelle runtime state, telemetry, or servers. If no recorder is installed, facade handles are no-ops.

Request duration metrics remain opt-in through NacelleTelemetryConfig. With the default config, core/HTTP request paths avoid request timer work unless HTTP access logging is enabled.

Compile and activate TCP phase timing only for a diagnostic build:

[dependencies]
nacelle = { version = "0.3", features = ["phase-timing"] }

#![allow(unused)]
fn main() {
let telemetry = NacelleTelemetry::default()
	.with_phase_duration_metrics(true);
}

The nacelle.phase.duration_ms histogram uses a low-cardinality phase label:

These are operation histograms, not a per-request trace. Do not add their percentiles to infer round-trip latency: pipelining can decode several requests from one read, streaming overlaps body reads with the handler, and response coalescing can write several completed requests in one batch. Use nacelle.request.duration_ms for server request processing and client-side latency for actual round-trip time.

The server cannot measure TCP handshake duration because the kernel completes it before accept() returns. Connection accepted, active, and closed metrics remain available; TLS handshake timing is not currently emitted as a phase.

Canonical metric names are resource-first. Instrument type is documented here rather than embedded in the metric name:

Run microbenchmarks before and after hot-path changes:

cargo bench -p nacelle-examples --features "bench tcp experimental-memory"

Performance model

nacelle's high-throughput TCP path is sensitive to small per-request costs. When comparing runs, keep these variables fixed:

• commit
• Linux kernel and CPU governor
• allocator configuration
• server threads
• connection count
• pipeline depth
• payload size
• TLS versus plain TCP
• stress client version

Use the performance how-to for repeatable command lines.

Do not use a repository-wide RPS number as a baseline. Compare commits on the same host with the same kernel, governor, allocator, features, transport/TLS mode, configuration, workload, and client revision.

Suggested local benchmark:

cargo bench -p nacelle-examples --features "bench tcp experimental-memory"

The runtime_limits benchmark group covers connection/request permit acquire/drop and memory allocation overhead. Watch it closely after changes to NacelleRuntimeState.

Successful-path ownership and dispatch

The default successful TCP and HTTP pipelines use concrete protocol, handler, responder, body, and telemetry observer types. Nacelle does not box handler futures or dynamically dispatch those contracts.

Retained costs are scoped by ownership:

• TCP connection buffers and decoder state are created per connection and reused across requests. Oversized response frames and optional input-buffer rotation can replace those buffers deliberately.
• A non-empty HTTP request uses one bounded Tokio channel for the request-body bridge; an exact zero-length body skips the channel and producer work, while the small scoped future still completes immediately.
• HTTP response bodies remain concrete StreamBody values rather than boxed body trait objects.
• Request/handler/read/write timeouts use concrete futures. The HTTP response write deadline retains a boxed Sleep only after connection-level backpressure because Hyper requires its I/O wrapper to remain Unpin.
• With experimental-memory, memory-wait queue allocation and its boxed timeout occur only under contention; the available-capacity path is atomic and allocation-free.
• Enabled per-peer request and connection-open rate limits use fixed-capacity, lock-free tables. Admission probes a bounded number of atomic slots and rejects a newly observed peer when the configured table is full; it does not take a per-request mutex or sweep every tracked peer.
• Type-erased protocol/handler errors are constructed only on error paths.
• TCP response coalescing is opt-in. It queues only complete bounded frames, retains overflow memory guards until flush, and restores socket backpressure at thresholds, streaming waits, and socket-read boundaries. Overflow grows geometrically through an old-plus-replacement memory-accounted transaction.
• With experimental-memory, TCP streaming body accounting reserves the declared body length by default. TcpStreamingBodyMemoryPolicy::LiveChunks is opt-in and performs one shared memory-budget acquire/release per chunk so accounting follows queued chunks and application-owned Bytes clones.
• App listener installers and worker thread closures erase startup-only closure types; they are not involved in request dispatch.
• Optional tracing, Hyper, Tokio, TLS providers, allocators, and metrics recorders retain their own external indirection.

TCP computes effective telemetry modes once per connection and constructs one NacelleMetricsContext containing cached facade handles. Without an installed recorder, metric writes through those handles are no-ops. Connection/request permits update cached runtime gauges at their existing state transitions. Memory accounting does the same only with experimental-memory. Diagnostic TCP phase timers are compiled only with the non-default phase-timing feature and remain runtime-disabled until explicitly enabled on NacelleTelemetry.

Enable the buffer-rotation feature for long-lived TCP connections that may occasionally receive large requests. Once an oversized cumulative input buffer is empty, Nacelle replaces it with a buffer sized to read_buffer_capacity. Leave the feature disabled when retaining peak buffer capacity is preferable to allocating again after traffic spikes.

Suggested RPS comparison:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

The stress server installs a metrics-util debugging recorder and prints a compact console snapshot every 5 seconds. Request metrics are grouped under the generic telemetry request_metrics config; started/completed counters and byte counters are on by default, while in-flight and duration metrics remain opt-in. Request duration metrics are opt-in as well, which avoids request Instant work on core/HTTP paths unless duration metrics or HTTP access logs are enabled. Use --no-byte-metrics when comparing the cost of byte accounting. Use the telemetry_paths benchmark for a no-recorder facade baseline; the stress server intentionally installs its console recorder in every feature set.

The checked-in root config.toml enables self-signed TCP TLS for local stress runs. For the plain TCP throughput baseline, use examples/nacelle-stress-server/configs/tcp.toml. Compare TLS and non-TLS runs separately:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-low-memory.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-tls.toml

The examples/run-stress-test.sh and examples/run-stress-test.ps1 helpers apply root config.toml first, then the selected profile, and choose the matching client mode automatically. They also enable experimental-memory when either effective config contains max_memory_bytes.

Guardrails:

• keep shutdown task tracking at the connection/listener boundary
• avoid per-request locks in the TCP hot path
• keep telemetry observers optional; default operation uses NoopObserver
• preserve single-chunk body fast paths
• tune TCP buffer sizes for the connection count instead of relying on large defaults

Configure production limits

Start from NacelleLimits::default() and tune shared resource budgets for the deployment. Use NacelleTcpLimits for TCP socket timeouts and NacelleHttpLimits for HTTP edge timeouts and keep-alive behavior. Active connections, in-flight requests, streaming tasks, body sizes, handler timeouts, and transport timeouts are bounded by default. Runtime memory accounting is experimental and not compiled by default. Enable experimental-memory and set max_memory_bytes only after measuring the limiter for your service.

Recommended presets:

• Internal service: keep defaults, set body limits to the largest expected payload, and run behind process supervision.
• Internet-facing behind proxy: cap connections and requests to the container budget, keep 30 second transport timeouts, and let the proxy own coarse traffic filtering or certificate automation when desired.
• Proxy-aware HTTP: configure NacelleHttpPolicy::with_trusted_proxy_ips(...) only with known proxy addresses before allowing Forwarded or X-Forwarded-For to affect per-peer request limits or request metadata.
• Direct HTTPS listener: enable http,tls, load certificate/key material through NacelleTlsConfig, configure an SNI allowlist with from_pem_with_allowed_server_names or from_der_with_allowed_server_names, set a short TLS handshake timeout, configure max_connections_per_peer and max_connection_opens_per_peer_per_second, enable HTTP access logs, and attach NacelleHttpPolicy with Host, method, URI, header, security-header, and per-peer request-rate limits.
• Direct TCP Rustls listener: enable tcp,tls, load certificate/key material through NacelleTlsConfig, register it with NacelleApp::tcp_tls(...), and keep protocol-level authentication/authorization in the application protocol.
• Direct TCP OpenSSL listener: enable tcp,openssl, load certificate/key material through NacelleOpenSslConfig, register it with NacelleApp::tcp_openssl(...), and configure the SslAcceptor yourself when you need OpenSSL-specific policy.
• Optional TCP OpenSSL listener: enable tcp,openssl and use serve_tcp_optional_openssl(...) or the matching host/app builder method when one listener must accept both plain and TLS clients; keep NacelleTlsDetectionOptions::timeout short enough to avoid tying up idle accepted connections.
• IPv4 plus IPv6 TCP bind: use the NacelleApp::*_dual_stack(...) helpers to register separate IPv4 and IPv6 listeners while forcing the IPv6 listener to v6-only mode.
• Unix socket listener: enable tcp on Unix and call NacelleApp::unix_socket(...); use NacelleUnixSocketOptions only when this process owns stale-path cleanup or socket-file permissions.
• Local load-test/autodeploy HTTPS: enable tls-self-signed and call NacelleTlsConfig::self_signed(...); do not treat generated certificates as a public trust or rotation strategy.
• High concurrency: reduce TCP buffer capacities before raising max_connections, and tune NacelleTcpLimits separately from shared resource budgets.

Experimental memory budget:

connection_budget =
  max_connections * (read_buffer_capacity + response_buffer_capacity)
body_budget =
  concurrent_buffered_or_streaming_bodies * max_request_body_bytes
total_budget =
  connection_budget + body_budget + handler/backend/runtime headroom

The APIs in this section require the non-default experimental-memory feature. Set NacelleLimits::with_max_memory_bytes(...) when you want Nacelle to enforce the calculated budget. Without the feature, Nacelle still enforces connection/request/body limits and transport-owned timeouts but leaves memory governance to the application, runtime, process supervisor, or container. When the memory budget is full, request body allocations wait in FIFO order and time out after NacelleLimits::memory_allocation_timeout (5s by default). Tune this with with_memory_allocation_timeout(...), or call NacelleRuntimeState::memory_budget() when application code needs to allocate from the same budget as the transports.

TCP processes requests sequentially per connection. request_body_channel_capacity controls the queued streaming chunks between the socket reader and handler. HTTP uses Hyper's internal buffers plus Nacelle's body queue, so leave extra headroom when enabling large request bodies.

For TCP protocols, NacelleLimits::max_request_body_bytes is the default body limit. Override Protocol::max_request_body_bytes(request, connection, state, default_limit) when the decoded request head, immutable connection metadata, or concrete Protocol::ConnectionState should choose a stricter phase-specific cap. The hook runs before body-specific allocation or additional body reads; normal decoder read-ahead may already have placed bytes in the connection buffer.

Use NacelleTcpOptions for accepted TCP stream behavior. Defaults preserve the existing behavior: TCP_NODELAY enabled and TCP keepalive disabled. Enable keepalive deliberately per deployment target because OS defaults and supported fields vary. NacelleTcpBindOptions adds listener bind controls such as IPv6-only mode for APIs that need explicit family behavior.

NacelleTcpConfig::response_write_policy defaults to ResponseWritePolicy::Immediate. Select CoalesceBuffered or FlushAtBytes(n) only for measured workloads with already-buffered request bursts. Coalescing preserves complete-frame order and flushes before waiting for more socket input; streaming responses flush before awaiting the next body chunk. Larger thresholds can delay earlier responses until a threshold or batch boundary and apply the write timeout to the complete queued batch. Growth above response_buffer_capacity is memory-accounted transactionally with experimental-memory: the budget must temporarily cover both the current batch allocation and its complete replacement. Size the base buffer near a measured batch size when using larger thresholds.

Use NacelleTcpLimits for TCP socket read, socket write, final writer shutdown, and idle timeouts. Set shutdown_timeout independently when finalization needs a shorter deadline than ordinary response delivery. Use NacelleHttpLimits on HyperServer for HTTP header read, request body read, response write, keep-alive, and max connection age behavior.

Dangerous configurations:

• unbounded connections with large per-connection buffers
• large body limits without a process/container memory limit
• disabled timeouts on internet-facing listeners
• direct internet-facing HTTP without Host/header/method/URI policy
• direct internet-facing TLS without an SNI allowlist
• direct internet-facing listeners without per-peer connection caps
• direct internet-facing listeners without per-peer connection-open rate caps
• direct internet-facing HTTP without per-peer request caps and access logs
• trusting forwarded peer headers without an explicit trusted proxy list
• generated self-signed certificates used as a long-lived public-edge certificate strategy
• high keep-alive connection counts without proxy-level idle limits
• long TLS detection timeouts on optional TLS listeners
• Unix stale-path cleanup for a socket path not exclusively owned by this process

TLS certificate rotation:

#![allow(unused)]
fn main() {
let tls = NacelleTlsConfig::from_pem_files("cert.pem", "key.pem")?;
tls.reload_from_pem_files("next-cert.pem", "next-key.pem")?;
}

Reloads affect new TLS handshakes. Existing connections continue with the configuration negotiated when they connected.

Run stress tests

Run the server:

cargo run --release --package nacelle-stress-server -- --config examples/nacelle-stress-server/configs/tcp.toml

Run a bounded client smoke test:

cargo run --release --package nacelle-stress-test -- --connections 32 --pipeline 16 --duration-secs 15

The examples/run-stress-test.sh and examples/run-stress-test.ps1 helpers accept --config/-Config and pass --tls-insecure to the stress client only when the effective tls_self_signed value is true.

The stress client enables its Rustls support by default so --tls-insecure works with the local self-signed server. For a Rustls-free plain TCP build, run both stress binaries with --no-default-features and use examples/nacelle-stress-server/configs/tcp.toml.

Repeatable profiles:

• examples/nacelle-stress-server/configs/tcp.toml: plain TCP baseline.
• examples/nacelle-stress-server/configs/tcp-low-memory.toml: plain TCP with mimalloc low-memory behavior and experimental runtime memory accounting.
• examples/nacelle-stress-server/configs/tcp-tls.toml: TCP wrapped in self-signed TLS.

Linux example:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

PowerShell example:

.\examples\run-stress-test.ps1 -Config examples/nacelle-stress-server/configs/tcp.toml -ServerThreads 48 -Connections 256 -Pipeline 8 -DurationSecs 30 -PayloadBytes 256

The stress server installs a metrics-util recorder and prints a compact metrics snapshot every 5 seconds. It enables request started/completed counters plus request/response byte counters by default. The generic telemetry API groups those switches under request_metrics; the stress server exposes byte accounting as byte_metrics = true. Use --no-byte-metrics for a lower-overhead recorder run. Use --no-default-features with the plain TCP config for a system-allocator, Rustls-free diagnostic; metrics collection remains active. Add --features mimalloc-allocator when the baseline must keep mimalloc while disabling TLS.

The launch helpers enable experimental-memory automatically when an effective config contains max_memory_bytes. For a direct low-memory server run, pass --features experimental-memory explicitly.

TCP phase timing is excluded from the default build. Compile and activate it only for a diagnostic run:

cargo run --release -p nacelle-stress-server --features phase-timing -- \
  --config examples/nacelle-stress-server/configs/tcp.toml \
  --phase-duration-metrics

The five-second metrics snapshot then prints count, mean, minimum, and maximum for each observed phase. Use an external metrics backend for percentiles and longer retention. The phase timers and histogram writes affect the measured hot path, so do not compare this run directly with a phase-timing-free throughput baseline.

The Tokio stress server default build includes tls-self-signed support. The checked-in root config.toml enables tls_self_signed = true, so the local stress client should use --tls-insecure with that default config. Use --no-default-features with examples/nacelle-stress-server/configs/tcp.toml when you need a Rustls-free plain TCP baseline.

CI-friendly scenarios should stay short and deterministic:

• baseline echo throughput
• max connection cap
• max request cap
• slow reader
• slow writer
• graceful shutdown under load

Heavy RPS and soak tests should run manually or nightly on dedicated Linux hosts.

For response-delivery comparisons, the stress server accepts --response-write-mode immediate (the default) or --response-write-mode coalesce-buffered. The same setting can be placed in a server config as response_write_mode. Coalescing drains complete requests already present in the socket read buffer and flushes before awaiting more input, so it does not leave a single response waiting for a later request. It is intended for measured pipelined workloads; keep immediate delivery for latency-first or unmeasured workloads.

The Linux profiling helper records this setting and also exposes shared versus serial handler dispatch for controlled diagnostics:

./scripts/profile-linux.sh \
  --tool baseline \
  --handler-mode shared \
  --response-write-mode coalesce-buffered \
  --pipeline 8 \
  --runs 3

Add --feature-set default for mimalloc. To measure the self-signed Rustls config, also pass --config examples/nacelle-stress-server/configs/tcp-tls.toml and --tls-insecure. The latter disables certificate verification and is only for the local generated certificate. The stress client flushes each populated request window before reading responses so buffered TLS records cannot strand deeply pipelined workers.

Compare performance profiles

Use separate profiles for each transport mode:

• plain TCP
• TCP with low-memory allocator behavior
• TCP with TLS
• HTTP

Do not compare TLS and non-TLS runs as if they measure the same path. Likewise, do not compare two runs if the stress client version changed.

Recommended plain TCP baseline config:

examples/nacelle-stress-server/configs/tcp.toml

Then run:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

Record the tested commit, kernel, CPU model/topology, CPU governor, Rust toolchain, allocator, features, configuration, connection count, pipeline depth, payload, TLS provider, and client revision with every result. A throughput number without that context is not a comparable baseline.

Suggested local benchmark:

cargo bench -p nacelle-examples --features "bench tcp experimental-memory"

For the codec/TCP integration specifically, run both Criterion targets:

cargo bench -p nacelle-codec --bench framed_comparison --all-features
cargo bench -p nacelle-examples --bench critical_paths --features "bench tcp experimental-memory"

The telemetry group can be run independently:

cargo bench -p nacelle-examples --bench critical_paths --features "bench tcp experimental-memory" -- telemetry --noplot

The TCP crate also measures one complete connection plus one request through the metrics facade, with independent phase-timing coverage:

cargo bench -p nacelle-tcp --bench telemetry_paths --all-features -- --noplot

Response delivery policies have a separate end-to-end benchmark:

cargo bench -p nacelle-tcp --bench response_delivery -- --noplot

Compare a tag or commit

The PowerShell comparison scripts keep data under target/performance-comparisons, resolve tags to immutable commit hashes, and use detached temporary worktrees so the current checkout is not modified. Each revision receives an isolated Cargo target directory; only Criterion baseline data is copied into the candidate target before comparison, preventing build artifacts from being reused across worktrees.

Capture all Criterion suites for the current commit, a release tag, or another commit:

./scripts/capture-performance-baseline.ps1
./scripts/capture-performance-baseline.ps1 -Reference v0.3.0
./scripts/capture-performance-baseline.ps1 -Reference 00747f3

With no parameters, HEAD is captured in a detached worktree. Uncommitted working-tree changes are intentionally excluded from that baseline.

Compare the current working tree with a captured baseline:

./scripts/compare-performance.ps1 -BaselineReference v0.3.0

Compare two committed revisions without checking either one out:

./scripts/compare-performance.ps1 `
	-BaselineReference v0.3.0 `
	-CandidateReference 0123456789abcdef

Use -Suite to limit both capture and comparison to one or more matching suites: codec, critical-paths, telemetry, or response-delivery. For example:

./scripts/capture-performance-baseline.ps1 `
	-Reference v0.3.0 `
	-Suite critical-paths,telemetry
./scripts/compare-performance.ps1 `
	-BaselineReference v0.3.0 `
	-Suite critical-paths,telemetry

The default capture runs every suite available at the selected commit. The critical-paths suite follows its historical move from nacelle to nacelle-examples. The telemetry and response-delivery suites are omitted for commits that predate those benchmark targets. When comparison omits -Suite, it uses exactly the suites recorded by the baseline, so newer-only targets do not invalidate an older baseline. Within a shared suite, Criterion compares matching benchmark IDs and measures newer IDs without a delta when no baseline exists for them.

Each capture records the resolved commit, Rust toolchain, operating system, architecture, CPU information, selected suites, and full benchmark log. Each comparison records the same candidate metadata and Criterion's percentage and confidence-interval output. Use -Force to replace an existing baseline for the same resolved commit.

Run native host workloads

On a dedicated Linux host, the native workload harness detects physical cores, SMT siblings, sockets, NUMA nodes, memory, and CPU governors. It reserves one physical core per NUMA node for the operating system and uses one logical CPU per remaining physical core. Review the generated plan without building or running workloads:

./scripts/run-native-performance.ps1 -PlanOnly

Run the default capacity matrix with a warm-up before every measured sample:

./scripts/run-native-performance.ps1 `
	-Runs 3 `
	-WarmupSecs 15 `
	-DurationSecs 30

The harness defaults to three 30-second measured runs per workload. The default capacity matrix uses 50 persistent connections, pipeline depth 1, worker counts of 1, 2, 4, 8, 16, 32, and 36, and response bodies of 0, 1, 10, and 100 KiB. Worker counts larger than the isolated server core set detected on the host are omitted. Requests use a zero-byte body while retaining the protocol's fixed frame overhead; sample banners report request and response body sizes separately.

Run plain TCP and TLS as separate result sets:

./scripts/run-native-performance.ps1 `
	-Config examples/nacelle-stress-server/configs/tcp.toml `
	-OutputDirectory target/native-performance/plain
./scripts/run-native-performance.ps1 `
	-Config examples/nacelle-stress-server/configs/tcp-tls.toml `
	-OutputDirectory target/native-performance/tls

The earlier diagnostic profiles remain available explicitly. For example:

./scripts/run-native-performance.ps1 -Profile min-rtt,pooled

Use -Profile all to run both the capacity matrix and the diagnostic minimum-RTT, pooled, pool-saturation, and pipelined-throughput profiles. Override the capacity dimensions with -CapacityWorkerCounts or -CapacityResponseKiB.

The harness builds native release binaries once, starts a fresh server for each sample, applies process and memory binding with numactl, and writes the host snapshot, exact workload plan, raw server/client logs, per-run parsed results, and median summary JSON under target/native-performance. It warns when it detects virtualization or a non-performance CPU governor and refuses to reuse an occupied bind address. Use -SkipBuild only when the release binaries already match the current source.

The codec target measures length-delimited encoding, compares direct decoding with MessageReader::decode_buffered, measures incomplete-header calls, and separates the no-op buffer-rotation check from replacing an empty 256 KiB buffer. The TCP target measures per-connection decoder construction and compares direct reference-protocol decoding with the buffered 64-request head/body drain used by the connection loop. Treat the rotation replacement result as allocation cost, not per-request overhead.

The runtime_limits benchmark group covers connection/request permit acquire/drop and memory allocation overhead. Watch it closely after changes to NacelleRuntimeState.

Profile Linux CPU and allocations

Use the Linux profiling helper for repeatable plain-TCP diagnostics:

./scripts/profile-linux.sh --tool baseline
./scripts/profile-linux.sh --tool perf
./scripts/profile-linux.sh --tool heaptrack

The helper builds the profiling Cargo profile with optimization, debug information, and frame pointers. Its default --feature-set minimal uses --no-default-features, which disables TLS and mimalloc. The stress server's downstream console metrics recorder remains active. The system allocator is required because Heaptrack cannot intercept calls made directly to mimalloc. Treat this as a diagnostic profile, not as a matched comparison with the default mimalloc build.

Use --feature-set default to profile the default mimalloc and Rustls-capable binaries. Self-signed Rustls workloads also require a TLS config and the explicit local-test trust flag:

./scripts/profile-linux.sh \
	--tool perf \
	--feature-set default \
	--config examples/nacelle-stress-server/configs/tcp-tls.toml \
	--tls-insecure

The helper rejects Heaptrack with the default feature set because direct mimalloc calls are invisible to Heaptrack. It also rejects --tls-insecure with the minimal feature set because that client omits Rustls.

Pass disjoint CPU lists after reviewing the native harness plan:

pwsh -NoProfile -File ./scripts/run-native-performance.ps1 -PlanOnly
./scripts/profile-linux.sh \
	--tool perf \
	--server-cpus 2,4,6,8,10,12,14,16 \
	--client-cpus 3,5,7,9,11,13,15,17,19,21,23

For user-space profiling of processes owned by the current user, Linux must permit performance counters. A value of 2 keeps kernel profiling disabled:

sudo sysctl -w kernel.perf_event_paranoid=2

Each run writes metadata, raw logs, and text reports under target/linux-profiles. The perf mode warms the server before attaching and records cycles:u at 999 Hz with frame-pointer call graphs. Heaptrack launches the server directly so allocator interception survives process startup, then applies any requested CPU affinity to all server threads.

The helper can compare handler ownership and response delivery without changing their safe defaults:

./scripts/profile-linux.sh --tool perf --handler-mode serial
./scripts/profile-linux.sh \
	--tool baseline \
	--response-write-mode coalesce-buffered \
	--pipeline 8 \
	--runs 3

July 2026 Linux response-delivery diagnostic

The following results are local confidence checks from an Intel Xeon Silver 4214 host with 24 physical cores, Linux 6.8, and Rust 1.95.0. The profiling build used the system allocator, no TLS, and the former metrics configuration, eight pinned server workers, 256 persistent connections, 256-byte request bodies, 64-byte response bodies, and all timeout defaults enabled. Server and client CPU sets were disjoint. Each matrix cell used a five-second warm-up and three measured ten-second runs; pipeline depths 1 and 8 also received ABBA interleaved checks.

All clients completed without reported failures. The pipeline-8 ABBA check measured 764,854-774,894 req/s for immediate delivery and 1,072,662-1,075,516 req/s for coalesced delivery. Matched perf captures lost no samples; coalescing reduced write_all_tracked_with_timeout self share from 4.33% to 1.39%, ResponseDelivery::write_pending from 4.14% to 1.68%, and TCP write polling from 1.59% to 0.46%. The connection loop always flushes after it drains the requests already decoded from the current read buffer and before it awaits another socket read.

These loopback saturation results support coalescing as an explicit option for highly pipelined workloads, not as a universal default. Pipeline-1 performance showed no repeatable benefit, and target-network latency, response sizes, backpressure, TLS, and telemetry require separate measurements.

Default-feature plain TCP and Rustls diagnostic

A follow-up on the same host used the then-default stress-server feature set: mimalloc, the former OpenTelemetry recorder with byte metrics, and self-signed Rustls support. The workload, CPU sets, timeout defaults, warm-up, sample duration, and three-run cell size matched the preceding matrix. Plain TCP and Rustls were measured as separate transport profiles.

All 36 measured client runs completed without reported failures; all cells had at most 1.1% min-to-max spread. At pipeline depth 8, matched perf captures lost no samples. Plain coalescing reduced tracked write-loop self share from 2.10% to 0.47%, pending-write self share from 1.78% to 0.42%, send from 0.62% to 0.12%, and TCP write polling from 0.53% to 0.09%. The equivalent Rustls pair was throughput-neutral: tracked write-loop self share remained approximately 1.01%, while encryption, TLS buffering, and the former metrics aggregation remained material costs.

Deep-pipeline Rustls testing also exposed two stress-path correctness gaps. The client now flushes buffered TLS requests before reading responses, and the TCP connection driver flushes the underlying transport before another request read and performs a shutdown-timeout-bounded shutdown. This delivers terminal TLS records promptly and emits close_notify. Final pipeline-32 Rustls samples all completed in approximately 10.02 seconds instead of waiting for the 30-second read timeout.

These results support coalescing for measured plain-TCP pipelined workloads on this host. They do not support enabling it for Rustls or pipeline-1 workloads.

Historical disabled-policy specialization

A matched local comparison used the telemetry_paths benchmark at checkpoint 0bce7f0 and after caching the effective TCP telemetry plan once per connection. Both builds used the same WSL2 host/toolchain and all TCP features:

The optimized path did not construct NacelleMetricsContext or metric attribute arrays when metrics are disabled, and request/phase mode checks are cached in a copyable per-connection plan. Connection/request permits remain active because they enforce runtime limits and expose operational state. Memory accounting also remains active when the benchmark includes experimental-memory; disabling telemetry does not disable either safety policy.

On the same local WSL2 host, 64 already-buffered one-byte requests producing 32-byte responses measured. The first two rows use a 2 KiB base response buffer; the threshold rows use a 1 KiB base buffer:

This is a synthetic in-memory writer benchmark and demonstrates dispatch/write amortization plus one transactional buffer-growth case, not network throughput. Keep immediate delivery for latency-first workloads unless a matched workload shows a benefit.

Response coalescing depth and persistent-pool controls

The response_delivery benchmark also provides two policy-matched workload families at pipeline depths 1, 8, and 32:

• same_socket: one persistent in-memory connection and one request window
• pool: eight persistent in-memory connections, each processing eight request windows; the next window becomes readable only after the previous response window is delivered

Both families compare Immediate with CoalesceBuffered on the same server, decoder, handler, response shape, runtime, and transport. Every response is 32 bytes. The benchmark asserts total write count and largest write size on every iteration. Response-buffer capacity is 2 KiB, so these cases remain inside the base connection allocation and do not exercise overflow growth.

Run them with:

cargo bench -p nacelle-tcp --bench response_delivery --all-features

A local confidence run based on commit 9be59a0 with a modified worktree used Rust 1.95.0 on Linux 6.6.87.2 WSL2 and an Intel Xeon Platinum 8370C virtualized topology (one socket, eight visible cores, two threads per core). Times are Criterion 95% confidence intervals from that one host.

Depth 1 remains the latency control: both policies perform one write per window, and their intervals are close. At depths 8 and 32, coalescing reduces one response window to one write and lowers elapsed time in this synthetic adapter workload. These results demonstrate write-path amortization, not socket syscall cost, network throughput, allocator counts, or production latency.

Suggested RPS comparison:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml --server-threads 48 --connections 256 --pipeline 8 --duration-secs 30 --payload-bytes 256

The stress server installs a metrics-util recorder and prints a compact metrics snapshot every 5 seconds. Request metrics are grouped under the generic telemetry request_metrics config; started/completed counters and byte counters are on by default, while in-flight and duration metrics remain opt-in. Request duration metrics are opt-in as well, which avoids request Instant work on core/HTTP paths unless duration metrics or HTTP access logs are enabled. Use --no-byte-metrics when comparing the cost of byte accounting. Use the telemetry_paths Criterion target for a no-recorder facade baseline; every stress-server feature set installs the console recorder.

The checked-in root config.toml enables self-signed TCP TLS for local stress runs. For the plain TCP throughput baseline, use examples/nacelle-stress-server/configs/tcp.toml. Compare TLS and non-TLS runs separately:

./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-low-memory.toml
./examples/run-stress-test.sh --config examples/nacelle-stress-server/configs/tcp-tls.toml

The examples/run-stress-test.sh and examples/run-stress-test.ps1 helpers apply root config.toml first, then the selected profile, and choose the matching client mode automatically. They also enable experimental-memory when either effective config contains max_memory_bytes.

Guardrails:

• keep shutdown task tracking at the connection/listener boundary
• avoid per-request locks in the TCP hot path
• keep telemetry observers optional; the default NoopObserver must remain allocation-free
• preserve single-chunk body fast paths
• tune TCP buffer sizes for the connection count instead of relying on large defaults

Harden HTTP listeners

Nacelle's HTTP transport is Hyper HTTP/1. Configure HTTP timeout and keep-alive behavior through NacelleHttpLimits, shared body-size budgets through NacelleLimits, and request-shape policy through NacelleHttpPolicy.

Defaults:

• NacelleHttpLimits::header_read_timeout: 30 seconds, enforced with Hyper's HTTP/1 header timeout and TokioTimer.
• NacelleHttpLimits::request_body_read_timeout: 30 seconds, enforced while reading body frames.
• NacelleHttpLimits::response_write_timeout: 30 seconds, enforced at Hyper's I/O write boundary.
• NacelleHttpLimits::keep_alive: enabled.
• NacelleHttpLimits::max_connection_age: disabled by default.
• request and response body size limits: 16 MiB each.

NacelleHttpPolicy can reject requests before the handler runs:

• allowed Host headers
• allowed HTTP methods
• maximum URI length
• maximum header count
• maximum aggregate header bytes
• optional per-peer request rate limits through with_max_requests_per_peer_per_second
• bounded lock-free per-peer request-rate state through with_peer_rate_limit_table_capacity (16,384 peers by default when enabled)
• optional trusted proxy forwarded address handling through with_trusted_proxy_ips
• optional security headers through with_security_header(...) or with_default_security_headers()
• optional per-peer connection caps through NacelleLimits::with_max_connections_per_peer
• optional per-peer connection-open rate caps through NacelleLimits::with_max_connection_opens_per_peer_per_second

Rejected requests receive deterministic HTTP responses where the request parser has already accepted the request: 405, 414, 421, 429, or 431. Rejections emit low-cardinality telemetry reasons such as host, method_not_allowed, uri_too_long, header_count, header_bytes, peer_rate, and peer_rate_table_full.

Per-peer request and connection-open rate limiters use fixed-capacity, lock-free tables. They retain active peer entries for 60 seconds and do not allocate, lock, or scan every tracked peer during admission. Size the HTTP table with NacelleHttpPolicy::with_peer_rate_limit_table_capacity(...) and the TCP/connection table with NacelleLimits::with_connection_rate_limit_table_capacity(...). If a table is full or no inactive entry is found within its fixed probe budget, a newly observed peer is rejected. Choose capacity from the deployment's expected active-peer cardinality rather than silently accepting unbounded state.

Enable the rustls feature to terminate HTTP over TLS. NacelleTlsConfig loads PEM certificate/key pairs, accepts explicit Rustls ServerConfig values, supports reloads for future handshakes, and enforces a TLS handshake timeout. Enable tls-self-signed only when local load tests or auto-deploying applications need to generate a self-signed certificate immediately; it implies rustls.

For direct edge HTTPS, build TLS config with NacelleTlsConfig::from_pem_with_allowed_server_names(...) or NacelleTlsConfig::from_der_with_allowed_server_names(...). When an SNI allowlist is configured, clients that omit SNI or send a name outside the list fail during the TLS handshake. HTTP Host policy is enforced after the handshake, so configure the SNI allowlist and NacelleHttpPolicy::with_allowed_hosts(...) with the same service names unless you intentionally need a narrower Host policy.

NacelleTlsConfig is the Rustls config shared with TCP TLS. NacelleTlsProvider reports Rustls for that config and OpenSsl for the TCP OpenSSL backend. HTTP TLS currently uses Rustls.

Enable HyperServer::with_access_log(true) when direct edge deployments need structured request logs. Access events are emitted with target nacelle::access and include transport, method, URI, status, request bytes, elapsed microseconds, and rejection reason.

Forwarded peer identity is disabled by default. Forwarded and X-Forwarded-For are considered only when the immediate socket peer is listed in NacelleHttpPolicy::with_trusted_proxy_ips(...); otherwise rate limits, request metadata, and access logs use the socket peer address.

For internet-facing deployments, a reverse proxy or load balancer can still own coarse traffic filtering and certificate automation. Nacelle now also enforces application-level body, request, connection, per-peer connection/request/connection-open-rate, timeout, TLS handshake, security header, and optional Host/header/method/URI limits in-process.

Slowloris-style clients are closed by NacelleHttpLimits::header_read_timeout. Trickle request bodies are closed by NacelleHttpLimits::request_body_read_timeout. Slow response readers are closed by NacelleHttpLimits::response_write_timeout when socket writes stop making progress.

Run security scans

Run vulnerability and dependency checks before release:

cargo audit
cargo tree -i serde_yaml
cargo tree -i unsafe-libyaml

serde_yaml and unsafe-libyaml should not appear in the dependency tree. If cargo-deny is adopted, add deny.toml with accepted licenses, advisory exceptions, and source policy.

Reference protocol

This document describes the LengthDelimitedProtocol fixture in the unpublished examples/nacelle-reference-protocol workspace package. It is used by examples, tests, benchmarks, and stress tools, but it is not part of Nacelle's published library API. Applications can use it from a repository checkout or implement Protocol with an associated Request type for TCP or Unix domain sockets.

Frame Layout

All integer fields are little-endian.

frame_len counts the fixed fields after itself plus the body. The minimum valid value is 20.

Flags

Request flags are decoded and preserved in FrameRequest, but the built-in server does not currently interpret request flags.

Requests

Each request frame contains one complete request body. The server decodes only the frame head before dispatch, then exposes the body to the handler as a NacelleBody. Small bodies are served from the connection read buffer. Larger bodies are streamed to the handler in configured chunks.

Custom protocols provide a per-connection MessageDecoder through Protocol::decoder. Decoders follow the nacelle-codec progress contract: returning a request consumes at least one head byte, while requesting more input leaves the cumulative buffer unchanged.

A decoder may wait for a fixed header plus a small protocol prefix before classifying a message as DecodedMessage::Request or DecodedMessage::OneWay. If the prefix is incomplete, return Ok(None) and leave every byte untouched. Once classification is possible, consume only the header/prefix and report the unconsumed body length in DecodedRequest. This preserves early body-limit checks and streaming while allowing one-way flags to live immediately after the fixed header.

opcode is request metadata. The application handler decides whether to use it for routing, reject it, or ignore it. If the handler rejects an opcode after draining the body and returns an error, the server encodes that error as a response frame.

Handlers receive TcpRequestContext<P>. Its request contains the associated protocol head under request().head and the bounded body under request_mut().body. Its connection context contains a stable connection id, peer/local addresses, listener label, TLS metadata, and Arc<P::ConnectionState>. The runtime constructs that state once with Protocol::connection_state and shares it across requests on the connection.

State confined to the serial connection loop can use SerialTcpServer or LocalSerialTcpServer. Their handlers implement SerialTcpHandler or LocalSerialTcpHandler and receive SerialTcpRequestContext<'_, P>, which lends exclusive mutable access to directly owned connection state. The loop awaits completion before decoding the next message, so one connection cannot overlap serial handler calls. Serial one-way contexts expose the same mutable state and still provide no response capability.

Responses

Handlers call context.respond(P::Response) and return the resulting typed completion. A protocol can accept its own response type; the reference protocol uses TcpResponse, whose body may be empty, one chunk, or streaming. Returning an HTTP response from a TCP handler does not compile.

Protocols classify decoded messages as DecodedMessage::Request or DecodedMessage::OneWay. Required requests use TcpRequestContext<P> and must respond. One-way messages use TcpOneWayContext<P> with NoResponse, so no respond method exists. Servers supporting one-way messages install a separate concrete handler with TcpServer::<YourProtocol>::builder().one_way_handler(...). Request-only protocols use Infallible as their one-way request type.

The TCP runtime encodes and writes each streaming response chunk before polling the next one, so socket backpressure bounds response production. It stages only one bounded frame at a time and writes an explicit end frame after a streaming body reaches EOF. With experimental-memory, it also accounts staging growth against the runtime memory budget.

ResponseWritePolicy::Immediate writes each completed frame immediately. CoalesceBuffered and FlushAtBytes may queue multiple completed frames from already-decoded requests, preserving order and rolling back only the current frame on encoder failure. The queue drains before another socket read and before awaiting another streaming response chunk. At that boundary, the runtime also flushes the underlying AsyncWrite, which is required for buffered transports such as TLS. When the connection ends cleanly, the runtime performs a shutdown_timeout-bounded writer shutdown so TLS transports can emit close_notify. Terminal error paths make the same bounded shutdown attempt; an earlier connection failure remains the returned error if shutdown also fails. Request telemetry records encoded response bytes when a request completes; a later batch write, transport flush, or shutdown failure is reported as a connection operation error.

The protocol guarantees:

• the first response frame has FRAME_FLAG_START
• the last response frame has FRAME_FLAG_END
• a handler that returns an empty body still emits a start/end response frame
• a handler error emits a start/end/error frame

Responses are written in request-processing order for a single connection. The prototype does not yet provide concurrent per-connection response interleaving.

Error Handling

Malformed frame heads, oversized frames, and EOF before a complete frame cause the connection to fail. Decoder, framing-progress, and incomplete-head failures are offered to Protocol::encode_error without an error context before the connection closes. Once a request head has been decoded, buffered and streaming body-read failures, handler errors, and handler timeouts are offered with that request's error context. Streaming request read failure also cancels the handler future. If encoding or writing an error frame fails, that delivery failure is returned without another delivery attempt. The connection closes after a terminal request failure so unread body bytes cannot be interpreted as another frame. Unknown opcode handling is application policy.

Limits

The server enforces NacelleTcpConfig::max_frame_len against frame_len. Buffer sizes and request-body chunking are configured through NacelleTcpConfig. Runtime budgets, timeouts, and active counters are configured through NacelleLimits / NacelleRuntimeState.

TCP protocols can apply phase-aware request body limits by overriding Protocol::max_request_body_bytes(request, connection, state, default_limit). The TCP runtime calls this after decoding the request head and before buffering or streaming the body. Implementations can use concrete protocol configuration, the decoded head, immutable ConnectionInfo, and the same concrete Protocol::ConnectionState exposed to handlers. Rejection occurs before body-specific allocation or additional body reads, although decoder read-ahead may already have buffered bytes. One-way messages use the equivalent Protocol::max_one_way_body_bytes(request, connection, state, default_limit) hook and the same early-rejection boundary.

TCP request handling is sequential per connection. Pipelined frames can sit in the socket/read buffer, but Nacelle does not run multiple handlers concurrently for one TCP connection. Streaming request bodies use request_body_channel_capacity for backpressure between socket reads and the handler, and declared streaming body bytes are allocated against the memory budget until the streaming request finishes when experimental-memory is enabled.

SharedProtocol marks protocols whose connection state is Send + Sync and is required by the existing Arc-backed shared server. Shared serial servers require state and handler futures to be Send, but not Sync, because each connection owns its state. Worker-local serial servers may use !Send state and futures.

Codec primitives

nacelle-codec provides ordered byte/message I/O over AsyncRead and AsyncWrite transports.

Components

Decoder contract

Implement MessageDecoder for protocol parsing. decode receives the cumulative BytesMut directly. Return a message only after consuming input, and return Ok(None) without consuming bytes when more input is required. MessageReader reports either progress-contract violation explicitly.

The built-in length-delimited decoder returns a split BytesMut without copying. Callers can retain it, freeze it, copy it, pool it, or parse it into a protocol-specific value.

Writer contract

Implement MessageEncoder<M> by appending directly to BytesMut. MessageWriter checkpoints the buffer before each encoder call and rolls back newly appended bytes when encoding fails.

feed only encodes. flush writes all queued bytes and flushes the transport. send performs both operations.

Buffer behavior

MessageReader and MessageWriter each hold one cumulative BytesMut, because framing partial reads and writes requires storage across transport operations. Use with_buffer to supply storage and into_parts to reclaim it.

A decoded message may share its backing allocation with the reader's input buffer. freeze() converts it to Bytes without copying. Use Bytes::copy_from_slice or BytesMut::from when the decoded message needs an independent allocation.

Limits

The length-delimited codecs reject frames larger than their configured maximum. Custom decoders are responsible for rejecting oversized input before the cumulative MessageReader buffer grows indefinitely.

MessageWriter::feed queues encoded bytes. flush, send, and shutdown write queued bytes to the transport.

Long-lived connections

The optional buffer-rotation feature provides RotatingMessageReader. Configure a threshold and replacement capacity to replace an empty input buffer after a large decoded message. The decoded BytesMut remains zero-copy and retains its original allocation until it is dropped. If coalesced input follows the large message, replacement waits until that input has been consumed.

Stability

The 0.3 API is experimental.

HTTP hardening reference

Nacelle's HTTP transport is Hyper HTTP/1. Configure HTTP timeout and keep-alive behavior through NacelleHttpLimits, shared body-size budgets through NacelleLimits, and request-shape policy through NacelleHttpPolicy.

Defaults:

• NacelleHttpLimits::header_read_timeout: 30 seconds, enforced with Hyper's HTTP/1 header timeout and TokioTimer.
• NacelleHttpLimits::request_body_read_timeout: 30 seconds, enforced while reading body frames.
• NacelleHttpLimits::response_write_timeout: 30 seconds, enforced at Hyper's I/O write boundary.
• NacelleHttpLimits::keep_alive: enabled.
• NacelleHttpLimits::max_connection_age: disabled by default.
• request and response body size limits: 16 MiB each.

NacelleHttpPolicy can reject requests before the handler runs:

• allowed Host headers
• allowed HTTP methods
• maximum URI length
• maximum header count
• maximum aggregate header bytes
• optional per-peer request rate limits through with_max_requests_per_peer_per_second
• bounded lock-free per-peer request-rate state through with_peer_rate_limit_table_capacity (16,384 peers by default when enabled)
• optional trusted proxy forwarded address handling through with_trusted_proxy_ips
• optional security headers through with_security_header(...) or with_default_security_headers()
• optional per-peer connection caps through NacelleLimits::with_max_connections_per_peer
• optional per-peer connection-open rate caps through NacelleLimits::with_max_connection_opens_per_peer_per_second

Rejected requests receive deterministic HTTP responses where the request parser has already accepted the request: 405, 414, 421, 429, or 431. Rejections emit low-cardinality telemetry reasons such as host, method_not_allowed, uri_too_long, header_count, header_bytes, peer_rate, and peer_rate_table_full.

Per-peer request and connection-open rate limiters use fixed-capacity, lock-free tables. They retain active peer entries for 60 seconds and do not allocate, lock, or scan every tracked peer during admission. Size the HTTP table with NacelleHttpPolicy::with_peer_rate_limit_table_capacity(...) and the TCP/connection table with NacelleLimits::with_connection_rate_limit_table_capacity(...). If a table is full or no inactive entry is found within its fixed probe budget, a newly observed peer is rejected. Choose capacity from the deployment's expected active-peer cardinality rather than silently accepting unbounded state.

Enable the rustls feature to terminate HTTP over TLS. NacelleTlsConfig loads PEM certificate/key pairs, accepts explicit Rustls ServerConfig values, supports reloads for future handshakes, and enforces a TLS handshake timeout. Enable tls-self-signed only when local load tests or auto-deploying applications need to generate a self-signed certificate immediately; it implies rustls.

For direct edge HTTPS, build TLS config with NacelleTlsConfig::from_pem_with_allowed_server_names(...) or NacelleTlsConfig::from_der_with_allowed_server_names(...). When an SNI allowlist is configured, clients that omit SNI or send a name outside the list fail during the TLS handshake. HTTP Host policy is enforced after the handshake, so configure the SNI allowlist and NacelleHttpPolicy::with_allowed_hosts(...) with the same service names unless you intentionally need a narrower Host policy.

NacelleTlsConfig is the Rustls config shared with TCP TLS. NacelleTlsProvider reports Rustls for that config and OpenSsl for the TCP OpenSSL backend. HTTP TLS currently uses Rustls.

Enable HyperServer::with_access_log(true) when direct edge deployments need structured request logs. Access events are emitted with target nacelle::access and include transport, method, URI, status, request bytes, elapsed microseconds, and rejection reason.

Forwarded peer identity is disabled by default. Forwarded and X-Forwarded-For are considered only when the immediate socket peer is listed in NacelleHttpPolicy::with_trusted_proxy_ips(...); otherwise rate limits, request metadata, and access logs use the socket peer address.

For internet-facing deployments, a reverse proxy or load balancer can still own coarse traffic filtering and certificate automation. Nacelle now also enforces application-level body, request, connection, per-peer connection/request/connection-open-rate, timeout, TLS handshake, security header, and optional Host/header/method/URI limits in-process.

Slowloris-style clients are closed by NacelleHttpLimits::header_read_timeout. Trickle request bodies are closed by NacelleHttpLimits::request_body_read_timeout. Slow response readers are closed by NacelleHttpLimits::response_write_timeout when socket writes stop making progress.

Production configuration reference

Start from NacelleLimits::default() and tune shared resource budgets for the deployment. Use NacelleTcpLimits for TCP socket timeouts and NacelleHttpLimits for HTTP edge timeouts and keep-alive behavior. Active connections, in-flight requests, streaming tasks, body sizes, handler timeouts, and transport timeouts are bounded by default. Runtime memory accounting is experimental and not compiled by default. Enable experimental-memory and set max_memory_bytes only after measuring the limiter for your service.

Recommended presets:

• Internal service: keep defaults, set body limits to the largest expected payload, and run behind process supervision.
• Internet-facing behind proxy: cap connections and requests to the container budget, keep 30 second transport timeouts, and let the proxy own coarse traffic filtering or certificate automation when desired.
• Proxy-aware HTTP: configure NacelleHttpPolicy::with_trusted_proxy_ips(...) only with known proxy addresses before allowing Forwarded or X-Forwarded-For to affect per-peer request limits or request metadata.
• Direct HTTPS listener: enable http,tls, load certificate/key material through NacelleTlsConfig, configure an SNI allowlist with from_pem_with_allowed_server_names or from_der_with_allowed_server_names, set a short TLS handshake timeout, configure max_connections_per_peer and max_connection_opens_per_peer_per_second, enable HTTP access logs, and attach NacelleHttpPolicy with Host, method, URI, header, security-header, and per-peer request-rate limits.
• Direct TCP Rustls listener: enable tcp,tls, load certificate/key material through NacelleTlsConfig, register it with NacelleApp::tcp_tls(...), and keep protocol-level authentication/authorization in the application protocol.
• Direct TCP OpenSSL listener: enable tcp,openssl, load certificate/key material through NacelleOpenSslConfig, register it with NacelleApp::tcp_openssl(...), and configure the SslAcceptor yourself when you need OpenSSL-specific policy.
• Optional TCP OpenSSL listener: enable tcp,openssl, use serve_tcp_optional_openssl or the matching host/app builder method, and keep NacelleTlsDetectionOptions::timeout short enough for your accepted-connection budget.
• IPv4 plus IPv6 TCP bind: use the NacelleApp::*_dual_stack(...) helpers to register separate IPv4 and IPv6 listeners while forcing the IPv6 listener to v6-only mode.
• Unix socket listener: enable tcp on Unix and use NacelleUnixSocketOptions only when this process owns stale-path cleanup or socket-file permissions.
• Local load-test/autodeploy HTTPS: enable tls-self-signed and call NacelleTlsConfig::self_signed(...); do not treat generated certificates as a public trust or rotation strategy.
• High concurrency: reduce TCP buffer capacities before raising max_connections, and tune NacelleTcpLimits separately from shared resource budgets.

Experimental memory budget:

connection_budget =
  max_connections * (read_buffer_capacity + response_buffer_capacity)
body_budget =
  concurrent_buffered_or_streaming_bodies * max_request_body_bytes
total_budget =
  connection_budget + body_budget + handler/backend/runtime headroom

The APIs in this section require the non-default experimental-memory feature. Set NacelleLimits::with_max_memory_bytes(...) when you want Nacelle to enforce the calculated budget. Without the feature, Nacelle still enforces connection/request/body limits and transport-owned timeouts but leaves memory governance to the application, runtime, process supervisor, or container. When the memory budget is full, request body allocations wait in FIFO order and time out after NacelleLimits::memory_allocation_timeout (5s by default). Tune this with with_memory_allocation_timeout(...), or call NacelleRuntimeState::memory_budget() when application code needs to allocate from the same budget as the transports.

TCP processes requests sequentially per connection. request_body_channel_capacity controls the queued streaming chunks between the socket reader and handler. HTTP uses Hyper's internal buffers plus Nacelle's body queue, so leave extra headroom when enabling large request bodies. NacelleTcpConfig::streaming_body_memory_policy defaults to TcpStreamingBodyMemoryPolicy::DeclaredLength, which reserves the declared body length before dispatch. LiveChunks instead charges each streaming chunk until its final application-owned Bytes clone drops. This can admit a body larger than currently available budget, but retained chunks apply backpressure to later reads. Keep request_body_chunk_size within available budget and do not use LiveChunks for full-body aggregation unless the budget can hold that body.

For TCP protocols, NacelleLimits::max_request_body_bytes is the default body limit. Override Protocol::max_request_body_bytes(request, connection, state, default_limit) when the decoded request head, immutable connection metadata, or concrete Protocol::ConnectionState should choose a stricter phase-specific cap. The hook runs before body-specific allocation or additional body reads; normal decoder read-ahead may already have placed bytes in the connection buffer.

NacelleTcpOptions controls accepted TCP stream behavior. Defaults preserve the existing behavior: TCP_NODELAY enabled and TCP keepalive disabled. NacelleTcpBindOptions adds listener bind controls such as IPv6-only mode for APIs that need explicit family behavior.

TCP response delivery defaults to ResponseWritePolicy::Immediate. CoalesceBuffered uses response_buffer_capacity as its threshold, while the configuration builder normalizes FlushAtBytes(0) to FlushAtBytes(1). Only complete frames enter the pending batch; the runtime flushes before another socket read, before awaiting another streaming body chunk, and when the threshold is reached or crossed. Read-boundary flushes include the underlying AsyncWrite, and clean connection teardown performs a writer shutdown bounded by NacelleTcpLimits::shutdown_timeout; terminal error paths make the same bounded attempt. This delivers buffered TLS records and permits a TLS close_notify. Pending capacity above the connection's base response buffer remains charged to runtime memory until flush or failure cleanup when experimental-memory is enabled. In that mode, geometric growth is transactional and requires memory headroom for both the old buffer and its complete replacement; a growth attempt is rejected before encoding when that temporary allocation cannot be charged.

NacelleTcpLimits controls TCP socket read, socket write, final writer shutdown, and idle timeouts. Shutdown uses its own deadline so finalization policy can be tuned independently of ordinary response delivery. NacelleHttpLimits controls HTTP header read, request body read, response write, keep-alive, and max connection age behavior on HyperServer.

Dangerous configurations:

• unbounded connections with large per-connection buffers
• large body limits without a process/container memory limit
• disabled timeouts on internet-facing listeners
• direct internet-facing HTTP without Host/header/method/URI policy
• direct internet-facing TLS without an SNI allowlist
• direct internet-facing listeners without per-peer connection caps
• direct internet-facing listeners without per-peer connection-open rate caps
• direct internet-facing HTTP without per-peer request caps and access logs
• trusting forwarded peer headers without an explicit trusted proxy list
• generated self-signed certificates used as a long-lived public-edge certificate strategy
• high keep-alive connection counts without proxy-level idle limits
• long TLS detection timeouts on optional TLS listeners
• Unix stale-path cleanup for a socket path not exclusively owned by this process

TLS certificate rotation:

#![allow(unused)]
fn main() {
let tls = NacelleTlsConfig::from_pem_files("cert.pem", "key.pem")?;
tls.reload_from_pem_files("next-cert.pem", "next-key.pem")?;
}

Reloads affect new TLS handshakes. Existing connections continue with the configuration negotiated when they connected.

OpenSSL builds need native OpenSSL development files. The openssl-vendored feature can build OpenSSL from source, but that build requires Perl on Windows.

API stability

Nacelle is 0.3.x, so public APIs are still experimental.

Stable enough for prototype integrations:

• nacelle::core::pipeline typed context, responder, and handler contracts
• nacelle::tcp and nacelle::http transport-owned request/response contracts
• nacelle::core::NacelleBody
• nacelle::core::{NacelleLimits, NacelleRuntimeState}
• nacelle::NacelleApp listener registration and NacelleApp::run(...)
• nacelle::prelude::* for common application imports
• nacelle::core::{NacelleTelemetry, NacelleTelemetryConfig}
• nacelle::core::NacelleTelemetryObserver for statically dispatched application telemetry

Experimental:

• transport-specific metadata
• transport listener option structs
• optional OpenSSL TLS detection on shared TCP listeners
• telemetry observer event details
• stress tooling config
• feature combinations involving phase-timing, TLS providers, and error-hints

Application code should use the app-first path: NacelleApp::new().tcp(...).http(...).run().await. The app owns shared runtime state, telemetry, shutdown, and listener supervision. Concrete transport servers retain transport-specific limits and policy. nacelle::runtime::NacelleHost and lower-level server APIs remain available for advanced manual supervision.

Public transport configuration structs may be non-exhaustive so fields can be added without breaking downstream struct literals. Construct NacelleTcpConfig and NacelleTcpLimits with Default, then use their with_* builders or mutate existing public fields. This preserves defaults for settings introduced by later releases.

The former detached NacelleRequest/NacelleResponse handler and Tower adapter were removed. Transport pipelines now remain strongly typed through completion; there is no compatibility adapter.

Before 1.0, minor releases may change defaults or builder methods when production safety requires it. After 1.0, public API changes should follow semver, with migration notes for config/default changes.

Reference protocol migration

The former reference_protocol feature and its facade/prelude exports have moved to the unpublished examples/nacelle-reference-protocol workspace package. Repository examples depend on that package directly. Application code should implement nacelle::tcp::Protocol or maintain its protocol in a separate application crate rather than depending on a protocol implementation from the Nacelle facade.

Rust API reference

Generate the Rust API reference with:

cargo doc --workspace --all-features --no-deps

On Windows:

.\scripts\build-rustdoc.ps1

The generated index is:

target/doc/nacelle/index.html

Start with these public entry points:

• nacelle::prelude::* for common application imports.
• nacelle::core, nacelle::codec, nacelle::tcp, nacelle::http, nacelle::openssl, nacelle::rustls, and nacelle::runtime for capability-oriented imports.
• nacelle::openssl::NacelleOpenSslConfig and nacelle::rustls::NacelleTlsConfig for concrete provider configuration.
• nacelle::advanced::runtime for raw executor and transport listener helpers when app/host composition is not sufficient.
• nacelle::NacelleApp listener registration and NacelleApp::run(...) for the app-first serving path across TCP, Unix sockets, HTTP, and TLS.
• nacelle::core::pipeline::Handler for typed shared-runtime handlers.
• nacelle::tcp::{NacelleTcpConfig, NacelleTcpLimits} for TCP buffering, framing, and timeout policy. These structs are non-exhaustive; construct them with Default and apply with_* builders so future fields retain their defaults.
• nacelle::runtime::{ThreadPerCoreConfig, WorkerSet} and the run_local_*_thread_per_core(...) functions for experimental Linux-only worker-local TCP, HTTP, Rustls, required OpenSSL, and optional OpenSSL execution. This mode requires explicit selection and does not silently fall back to the shared runtime.
• ThreadPerCoreConfig::with_max_threads(...) to cap the worker threads selected by WorkerSet::all(), WorkerSet::first(...), or WorkerSet::explicit(...) while preserving selection order. The shared runtime is caller-owned; configure its Tokio thread count on the runtime builder instead.
• nacelle::runtime::ThreadPerCoreLimits::Global for exact process-wide counters, or ThreadPerCoreLimits::Worker for partitioned worker-local counters. Worker mode enforces one shared hard memory ceiling across all workers when experimental-memory is enabled.
• nacelle::runtime::WorkerContext::offload_blocking(...) for explicit blocking work whose completion is awaited back on the originating local worker.
• nacelle::tcp::Protocol for TCP wire-format adapters.
• nacelle::tcp::{TcpServer, LocalTcpServer} for Arc-backed connection state, or SerialTcpServer / LocalSerialTcpServer for exclusive mutable state lent to one serial handler at a time.
• With experimental-memory, nacelle::tcp::TcpStreamingBodyMemoryPolicy to retain declared-length admission or account only live streaming chunks.
• NacelleApp and NacelleHost serial listener methods for plain TCP, required OpenSSL, optional OpenSSL, and Unix sockets.
• nacelle::runtime::run_local_serial_tcp_thread_per_core(...) and run_local_serial_tcp_openssl_thread_per_core(...) for worker-local serial plain TCP and required OpenSSL. Use run_local_serial_tcp_optional_openssl_thread_per_core(...) when plaintext and OpenSSL must share one worker-local listener. Worker factories run once per worker, so externally bounded pools should be shared deliberately rather than constructed per worker.
• nacelle::core::{NacelleTelemetry, NacelleTelemetryConfig} for metrics and telemetry.
• nacelle::core::NacelleError::hint() with the error-hints feature for optional operator guidance. NacelleError::Display remains stable across feature combinations; applications append hints deliberately where suitable.
• With experimental-memory, nacelle::core::{NacelleMemoryBudget, NacelleMemoryAllocation} and NacelleRuntimeState::memory_budget() for shared application/transport memory budget allocations. Owned allocation guards can release retained capacity with NacelleMemoryAllocation::shrink_to(...).
• nacelle::tcp::TcpServer, nacelle::http::HyperServer, nacelle::runtime::NacelleHost, and nacelle::advanced::runtime when a service needs lower-level listener control.