GPU-Initiated PortChannel with GPUNetIO

By default, PortChannel operations over InfiniBand are pushed by GPU threads into a FIFO and posted to the NIC by the host proxy thread of a ProxyService. With the optional GPUNetIO backend, GPU threads build the RDMA work requests and ring the NIC doorbell themselves through DOCA GPUNetIO (GDAKI), removing the proxy from the data path. Select the backend per endpoint with EndpointConfig::Ib::Mode::Device, then use a Device-typed channel owner and GPU handle. Posting methods allocate their shared-memory WQE scratch internally.

Build

The backend is built automatically for CUDA builds with InfiniBand support (MSCCLPP_USE_IB=ON, the default); no extra CMake option is needed. ROCm builds and builds with MSCCLPP_USE_IB=OFF skip it. At configure time, CMake fetches the GPUNetIO sources (see cmake/GpuNetIo.cmake) and compiles them into libmscclpp. To build without network access, point CMake to a local checkout of the same revision:

cmake -DFETCHCONTENT_SOURCE_DIR_GPUNETIO=/path/to/gpunetio ..

The GPUNetIO license is installed to share/licenses/mscclpp/gpunetio.

Requirements

  • A Hopper or newer NVIDIA GPU and an RDMA NIC with a BlueFlame UAR (DOCA_GPUNETIO_VERBS_NIC_HANDLER_GPU_SM_BF). Device code must target sm_90 or newer.

  • RDMA atomics on both sides for accumulate; a GPUNetIO endpoint cannot connect to an endpoint in HostNoAtomic mode.

  • The local endpoint must be on a GPU (EndpointConfig::device.type == DeviceType::GPU).

Usage

Create the connection with EndpointConfig::Ib::Mode::Device and construct the PortChannel directly from the semaphore and registered memories (PortChannel<PortChannelBackend::Device> is deduced automatically from the constructor; no ProxyService is needed):

mscclpp::EndpointConfig cfg;
cfg.transport = mscclpp::Transport::IB0;
cfg.ib.mode = mscclpp::EndpointConfig::Ib::Mode::Device;
auto connection = communicator->connect(cfg, remoteRank).get();

auto localMem = communicator->registerMemory(buff, bytes, mscclpp::Transport::IB0);
communicator->sendMemory(localMem, remoteRank);
auto remoteMem = communicator->recvMemory(remoteRank).get();

auto semaphore = std::make_shared<mscclpp::PortDevice2DeviceSemaphore>(*communicator, connection);
mscclpp::PortChannel channel(semaphore, remoteMem, localMem);
auto deviceHandle = channel.deviceHandle();

Host and Device owners have distinct DeviceHandle types; mscclpp::deviceHandle(channel) selects the matching GPU view. The canonical types are BasePortChannelDeviceHandle<Backend> and PortChannelDeviceHandle<Backend>; <> selects Host.

Keep the channel and application buffers alive during GPU use. Channel copies share the semaphore owner and counter; creating a separate semaphore creates an independent counter.

Default owners are unassigned, and handle extraction throws InvalidUsage until a valid channel is assigned. Host construction rejects null semaphore owners or proxies. Device construction rejects null or moved-from Port semaphore owners and uninitialized or incompatible memory registrations. Port handle extraction also rejects a moved-from Port semaphore. Raw GPU handles start at zero but require valid state before use; Host registry ID0 is valid.

Exclusive-mode operations automatically use internal shared-memory WQE slots for BlueFlame submission when the transfer fits device_.blueFlameLimit (default: 512 KiB) and the calling thread has a slot:

__global__ void send(mscclpp::PortChannelDeviceHandle<mscclpp::PortChannelBackend::Device> channel, size_t bytes) {
  using Concurrency = mscclpp::PortChannelDeviceHandle<>::Concurrency;
  if (threadIdx.x != 0) return;
  channel.putWithSignal<Concurrency::Exclusive>(0, bytes);
  channel.flush<Concurrency::Exclusive>();
}

// One block exclusively owns this QP.
send<<<1, 32>>>(deviceHandle, bytes);

Handles do not store shared-memory pointers and can be loaded directly from constant or global memory. Operations without an available slot, or transfers above blueFlameLimit, use regular doorbell submission without WQE staging.

In the kernel, put, signal, putWithSignal, putWithSignalAndFlush, flush, accumulate, wait and poll behave as for proxy-backed channels. flush waits for the local completion of all work requests posted on the queue pair.

signal and accumulate use an RDMA atomic fetch-and-add, which allows any number of writers to the same word. Device-mode channels also provide putWithAccumulate, a put followed by an accumulate, posted together; putWithSignal is putWithAccumulate on the semaphore. The responder applies an atomic only after placing the data of earlier RDMA writes on the same queue pair, so observing the updated word guarantees that those writes are visible. A plain RDMA write gives no such guarantee, even with strong ordering, because packets may be placed out of order at the responder.

Only port and gidIndex of EndpointConfig::Ib are used in this mode; queue sizes are chosen by the backend. Small put and putWithSignal payloads are embedded in the work request rather than read separately by the NIC.

Callers must make source data ready before put or putWithSignal, synchronizing with its producers as needed. These operations rely on DOCA’s queue-publication release ordering rather than adding a full system fence for every submission. Keep non-inline source buffers unchanged until their transfers complete.

Queue pair concurrency and BlueFlame configuration

Channel handles expose a nested Concurrency type (mscclpp::PortChannelDeviceHandle<>::Concurrency). The type is defined in BasePortChannelDeviceHandle<> in port_channel_device.hpp and shared by both backends. For device-backed channels it describes concurrent access to the underlying queue pair, not ownership of the PortChannel object. The default is Concurrency::Device, allowing callers across the device. Callers may instead select Concurrency::Block for callers confined to one thread block, or Concurrency::Exclusive when the caller serializes access, as a template argument to posting and flush methods.

Host-backed channels accept the same template argument but only support Concurrency::Device; other values produce a compile-time error. Calls without an explicit template argument continue to use Device for either backend.

device_.blueFlameLimit controls the maximum payload size (in bytes) submitted through BlueFlame (default: kDefaultBlueFlameLimit = 512 * 1024; set to 0 to disable BlueFlame or adjust it to change which transfer sizes use BlueFlame). Payloads above this limit write their WQEs directly to the GPU-resident send queue and ring the regular doorbell, matching NCCL GIN’s submit_db path. Only Concurrency::Exclusive operations submit through BlueFlame. The NIC executes a BlueFlame work request directly only if all earlier ones were already submitted, which concurrent callers cannot ensure without serializing each other, so Block and Device operations always ring the regular doorbell.

Exclusive-mode per-block thread capacity

Device-backend posting methods (put, signal, putWithSignal, putWithSignalAndFlush, accumulate, and putWithAccumulate) accept an optional second template argument MaxExclusiveThreadsPerBlock that configures how many threads per block can concurrently post in Concurrency::Exclusive mode with BlueFlame (each slot uses 64 bytes of shared memory). The default is 32 threads (2 KiB per block); pass a custom count such as channel.put<Concurrency::Exclusive, 8>(...), or 0 to disable BlueFlame without allocating scratch.

The default slot index is the linear thread index, including all three block dimensions. Posting methods accept an optional slotIdx (-1 selects the default); out-of-range indices fall back to regular doorbells. For example:

using Concurrency = mscclpp::PortChannelDeviceHandle<>::Concurrency;
channel.put<Concurrency::Exclusive, 8>(dstOffset, srcOffset, bytes, slotIdx);
channel.signal<Concurrency::Exclusive, 8>(slotIdx);

For put and putWithSignal, specify both source and destination offsets when passing an explicit index, avoiding ambiguity with the existing two-argument overloads. For putWithSignalAndFlush, the index follows maxSpinCount.

Slots are 64 bytes, 16-byte aligned, and shared across all calls with the same MaxExclusiveThreadsPerBlock in a block. Concurrent posting threads must use distinct indices, even when they own different QPs. BlueFlame waits for async source consumption before returning, so one thread can reuse a slot sequentially across QPs without an extra barrier or flush. Prefer one capacity per kernel: different capacities instantiate separate scratch arrays. Scratch is statically allocated for compiled BlueFlame paths, so runtime doorbell fallback does not reduce its footprint. Large capacities must respect the kernel’s shared-memory limits; Block and Device operations do not use this scratch.

Limitations

  • CQ initialization has known upstream error-handling issues when its CUDA copy fails; see NVIDIA-DOCA/gpunetio#23. No local workaround is applied.

  • Host-side Connection operations (write, updateAndSync, accumulate, flush) and Host2DeviceSemaphore::signal are not supported on GPUNetIO connections and throw InvalidUsage.

  • On RoCE devices only port 1 is supported (the pinned DOCA GPUNetIO resolves the destination MAC on port 1).

  • accumulate requires an 8-byte-aligned destination offset.