Moving MG functions into unified API + raft::device_resources_snmg as device resource type for MG functions

[viclafargue]

• Moving MG functions into unified API
• Using raft::device_resources_snmg as device resource type for MG functions
• Removing instantiation of raft::comms::build_comms_nccl_only solving [FEA] Remove dependency of build_comms_nccl_only on UCP raft#2465

raft::device_resources_snmg is introduced in this PR : rapidsai/raft#2549

[cjnolet]

One of the main goals with having the "snmg" resources object match the single-gpu object is that we wanted to be able to remove this additional MG. We should now be able to accept device_resources and then check to see if a nccl clique has been set on it (which would imply that it's a multi-gpu resources object and not a single-gpu object). The whole goal with doing this was to consolidate code paths.

[viclafargue]

The NCCL clique is not set as a resource anymore, but we should still be able to implement the dispatching by checking the dynamic type of the device_resources. The real question then is, do we truly want dispatching on both the regular API (cuvs::neighbors::build) and the mg namespace (cuvs::neighbors::mg::build)? It kind of make sense that a user providing a device_resources_snmg instance to the regular API (cuvs::neighbors::build) would want things to be deployed on multiple GPUs. However, the reverse is not necessarily true. A user who explicitly chose the mg namespace, but did not provide the adequate device_resources_snmg would fallback to single GPU, potentially unintentionally. Is this what we want?

I propose that we implement the dispatching mechanism solely on the regular API (cuvs::neighbors::build) in a dedicated follow-up PR? This also allows the MG doc to explicitly avert users that they should use an adequate device_resources_snmg to use the MG API. What do you think?

[cjnolet]

Not sure why CI is still failing. The RAFT PR was merged quite awhile ago but I'm still seeing errors like this:

/home/coder/cuvs/cpp/include/cuvs/neighbors/mg.hpp:24:10: fatal error: raft/core/device_resources_snmg.hpp: No such file or directory
     24 | #include <raft/core/device_resources_snmg.hpp>
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[bdice]

This fell back to old 24.12 RAFT packages for unclear reasons:

    libraft-headers:             24.12.00-cuda12_241211_geaf9cc72_0 rapidsai   
    libraft-headers-only:        24.12.00-cuda12_241211_geaf9cc72_0 rapidsai   

https://github.com/rapidsai/cuvs/actions/runs/12818869804/job/35749392606?pr=454#step:8:521

[cjnolet]

This fell back to old 24.12 RAFT packages for unclear reasons:

Oh fooey! Thanks for noticing that @bdice, I just realized the darn target branch was never updated!

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cjnolet]

I think this is nearly there. Just a couple very minor things at this point.

Note- cuVS is not header-only and we shouldn't be putting constants in include files (that further couples cuVS to our exact build policies and makes it harder to navigate the build).

Also, we should not be introducing ANY dependencies in the public header files. That's why we're using raft::resources as a facade so that we don't couple device_resources or device_resources_snmg to our public APIs. In other words, the user can use those objects if they want for convenience, but none of the other APIs in cuVS (nor in RAFT) use those objects because raft::resource is a generic container for these types of objects. That way the user can couple themselves to the dependencie sbut it doesn't force the dependencies on all other users. Does that make sense?

[cjnolet]

device_resources_snmg is a type of raft::resource. The pattern here is to accept the raft::resources instance and call the free function inside raft::resource::nccl_resource underneath to get the clique out of the resource. This way the APIs don't need to look any different.

[cjnolet]

Please don't couple the standard headers to nccl. This is no good. Instead, please just accept a raft::device_resources here but provide a function in raft::resource::nccl_resource namespace to introspect the resources for a nccl clique (otherwise fail). The snmg build params should be anough of a difference

[cjnolet]

I thought we were removing the mg namespace completely from the public APIs.

[cjnolet]

/ok to test