Comparison of GPUDirect ( GPU Memory)- NIC access vs Hop Memory GPU-> Host Memory -> NIC

[alokprasad]

Currently perftest supports GPU Direct support where NIC can directly access GPU memory , but it would be good to have comparison it without GPU Direct i.e. GPU Memory -> Copied to Host Memory -> NIC . Can someone give pointer how to make this change.
what i think we need to allocate host memory and copy gpu memory using cuMemcpyDtoH , then this host memory need to be used for MR?

[sshaulnv]

Hi @alokprasad,
do you mean to perform the copies in the datapath?

[alokprasad]

@sshaulnv yes..that would give insight on the improvement achieved by gpudurect

[sshaulnv]

To ensure optimal bandwidth, we generally avoid performing intensive operations within the datapath.
Assuming GPUDirect is unavailable and we need to send a message from GPU memory, we would probably first copy the buffer to host memory before entering the datapath.

[alokprasad]

@sshaulnv i agree thats good solution if we have constant data..but consider a scenario that Host 1 sends GPU data to Host 2 and it recives back Host2 does some processing , we need to do constant copying host mem-gpu mem in data path.

[drossetti]

@alokprasad IMO implementing a staging data path in perftest does not make much sense. It is much easier and meaningful to leverage UCX or a GPU-aware MPI library + OSU MPI benchmark.
Btw a number of papers have done that already.

[alokprasad]

@drossetti I got the point, can you please point me to the papers , hopefully with some github links to checkout the code.