mscclpp.default_algos

mscclpp.default_algos.allgather_multi_nodes(spec)

Build a hierarchical AllGather across nodes and local GPUs.

Return type:

CollectiveProgram

Parameters:

spec (AlgoSpec)

mscclpp.default_algos.allreduce_multi_nodes(spec, thread_block_group_size)

Implements a multi-node AllReduce using a hierarchical approach: 1. Intra-node allreduce 2. Inter-node exchange (exchange reduced data between nodes) 3. Intra-node allreduce

Return type:

CollectiveProgram

Parameters:

• spec (AlgoSpec)

• thread_block_group_size (int)

mscclpp.default_algos.reducescatter_multi_nodes(spec, thread_block_group_size=1)

Build a hierarchical ReduceScatter across nodes and local GPUs.

Return type:

CollectiveProgram

Parameters:

• spec (AlgoSpec)

• thread_block_group_size (int)

Modules

allgather_multi_nodes(spec)	Build a hierarchical AllGather across nodes and local GPUs.
allreduce_multi_nodes(spec, ...)	Implements a multi-node AllReduce using a hierarchical approach: 1.
reducescatter_multi_nodes(spec[, ...])	Build a hierarchical ReduceScatter across nodes and local GPUs.