MultiGPU/multinode support?

[jon-chuang]

What is SYCL's abilities to accomodate multigpu (e.g. NVLink) and multinode (e.g. Infiniband GPUDirect) setups?

[AlexeySachkov]

Hi @jon-chuang, sorry for such long delay,

Section 3.5. The SYCL platform model of SYCL 2020 specification says:

A SYCL context is constructed, either directly by the user or implicitly when creating a queue, to hold all the runtime information required by the SYCL runtime and the SYCL backend to operate on a device, or group of devices. When a group of devices can be grouped together on the same context, they have some visibility of each other’s memory objects. The SYCL runtime can assume that memory is visible across all devices in the same context. Not all devices exposed from the same platform can be grouped together in the same context.

Additionally, Section 3.7.1.1. SYCL backend resources managed by the SYCL application says:

Contexts: any SYCL backend resource that is acquired by the user is attached to a context. A context contains a collection of devices that the host can use and manages memory objects that can be shared between the devices. Devices belonging to the same context must be able to access each other’s global memory using some implementation-specific mechanism. A given context can only wrap devices owned by a single platform. A context is exposed to the user with a sycl::context object.

So, SYCL spec says that implementation can provide to you several devices, which could be grouped into the same context, which will guarantee that the memory is shared between them.

I won't be able to list all mechanisms which are specifically designed with multi-device setup in mind, but as an example, from Section 3.8.3. SYCL memory consistency model:

Memory consistency between the host and SYCL device(s), or different SYCL devices in the same context, can be guaranteed through synchronization in the host application as defined in Section 3.9.8. On SYCL devices supporting concurrent atomic accesses to USM allocations and acquire-release or sequentially consistent memory orderings, cross-device memory consistency can be enforced through the use of mem-fence and atomic operations.

For Level Zero backend we have the following document: Considerations for programming to multi-tile and multi-card under Level-Zero backend.

Tagging @Naghasan to see if he can add more about CUDA backend and multi-device setups

[AerialMantis]

I can answer to this on the CUDA side.

For the CUDA backend we have begun implementing support for multi-GPU by introducing an extension in PI which exposes the CUDA peer-to-peer copy capabilities (#4401) and using this we have been able to enable peer-to-peer copies between Nvidia GPUs via PCIe or NVLink.

However as @AlexeySachkov mentioned above the way SYCL provides this is via a shared SYCL context as devices in the same context must be able access each other's global memory. Unfortunately multi-device contexts are currently not supported in the CUDA backend (#4381), but once this is supported we'll be able to enable peer-to-peer copies for Nvidia devices within a context.