mscclpp.language.default_algos

mscclpp.language.default_algos.allreduce_2nodes(spec, thread_block_group_size)

Implements a multi-node AllReduce using a hierarchical approach: 1. Intra-node allreduce 2. Inter-node exchange (exchange reduced data between nodes) 3. Intra-node allreduce

Return type:

CollectiveProgram

Parameters:

• spec (AlgoSpec)

• thread_block_group_size (int)

Modules

allreduce_2nodes(spec, thread_block_group_size)

Implements a multi-node AllReduce using a hierarchical approach: 1.