UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

Ytk-mp4j is a fast, user-friendly, cross-platform, multi-process, multi-thread collective message passing java library which includes gather, scatter, allgather, reduce-scatter, broadcast, reduce, allreduce communications for distributed machine learning.

Summary of call graphs and data structures of NVIDIA Collective Communication Library (NCCL)

Different Implementations of the AllReduce algorithm used in Distributed Deep Learning in c++

Interactive web visualization for understanding collective communication algorithms (as used in NCCL, RCCL, MPI). Learn how AllReduce, Broadcast, Reduce, AllGather and more work step by step.

This repository outlines a comprehensive guide for training a distributed deep learning model.

Sparse Allreduce

Pool Apple Silicon Macs into a distributed ML training cluster. Zero-config discovery, ring AllReduce gradient sync, HMAC-authenticated fleet isolation with TLS encryption. PyTorch + MLX engines.

AllReduce/AllGather scaling in ASTRA-sim across torus vs switch topologies on the analytical + ns-3 backends — latency-bound vs bandwidth-bound over message size and node count. Reproducible Docker/Chakra harness + write-up.

Analytical simulator for collective communication on ring, fat tree, and dragonfly topologies: graph based hop costs, bandwidth efficiency, and closed form algorithm crossover up to 1024 nodes. Reproducible sweeps in Python (networkx), with a compiled LaTeX report.

Distributed training of a VGG-11 network on the CIFAR-10 dataset

MPI collective operations and communication topologies demonstrated across networked compute nodes in GNS3.

Benchmark suite for MPI collectives (Bcast, Alltoall, Allreduce) with from-scratch ring and recursive-doubling allreduce implementations. Fitted Hockney cost models, measured algorithm crossovers, and LaTeX reports generated entirely from the data. C++17, Open MPI, Python analysis pipeline.

Summary of call graphs and data structures of collective communication plugin in NVIDIA TensorRT-LLM

Ring-allreduce collective built from scratch in C++20 over raw MPI point-to-point primitives, benchmarked against Open MPI 5.0.10's MPI_Allreduce across 8B-128MiB and N=2..16. An alpha-beta cost model shows the ring's small-message loss is the algorithm's 2(N-1) step count, not implementation quality: Open MPI's own forced ring is just as slow.

AdaComp: Adaptive Multi-Tier Compression Middleware for MPI Collectives — 4-tier switaching with pipelined computation-communication overlap

Modified Dissemination/Bruck algorithm for commutative reduction operations in MPI