NumPy & SciPy for GPU

Safe rust wrapper around CUDA toolkit

Distributed and decentralized training framework for PyTorch over graph

Efficient Distributed GPU Programming for Exascale, an SC/ISC Tutorial

An open collection of methodologies to help with successful training of large language models.

An open collection of implementation tips, tricks and resources for training large language models

Federated Learning Utilities and Tools for Experimentation

Best practices & guides on how to write distributed pytorch training code

NCCL Fast Socket is a transport layer plugin to improve NCCL collective communication performance on Google Cloud.

A Julia wrapper for the NVIDIA Collective Communications Library.

NCCL Examples from Official NVIDIA NCCL Developer Guide.

Tool to run rccl-tests/nccl-tests based on from an application and gather performance.

Sample examples of how to call collective operation functions on multi-GPU environments. A simple example of using broadcast, reduce, allGather, reduceScatter and sendRecv operations.

Python Distributed Non Negative Matrix Factorization with custom clustering

High performance NCCL plugin for Bagua.

Installation script to install Nvidia driver and CUDA automatically in Ubuntu

N-Ways to Multi-GPU Programming

Blink+: Increase GPU group bandwidth by utilizing across tenant NVLink.

use ncclSend ncclRecv realize ncclSendrecv ncclGather ncclScatter ncclAlltoall

Distributed deep learning framework based on pytorch/numba/nccl and zeromq.