Add FutureWarning of UCX < 1.11.1 deprecation

[pentschev]

Besides stability, UCX 1.11.1 includes important features such as:

• Automatic InfiniBand<->CUDA device mapping without specifying UCX_NET_DEVICES explicitly;
• InfiniBand allocation registration cache without patching source;
• Endpoint error handling with CUDA IPC;
• etc.

All features above will simplify usage both with and without Dask and RAPIDS. Therefore, we would like to deprecate older versions of UCX, to be removed at a later release.

[pentschev]

I'm not necessarily tied to this idea, but perhaps we should start warning users of UCX < 1.11.1 deprecation? Also it doesn't need to be right now, we could delay it for 21.12 or another RAPIDS release.

[quasiben]

I'm not against deprecating but I think the sentiment you raised around waiting a bit is probably correct. However, I don't know of anyone using UCX-Py who can't upgrade UCX

[pentschev]

I'm not against deprecating but I think the sentiment you raised around waiting a bit is probably correct. However, I don't know of anyone using UCX-Py who can't upgrade UCX

It's not too much on upgrading or not, but fundamentally older UCX (such as 1.9) lacked a lot of functionality that are now implemented in UCX 1.11.1 (e.g., automatic IB device detection, IB allocation registration cache, endpoint error handling with CUDA IPC) that really simplifies things for us by a lot, and removing that code sometime in the near future would be great. Therefore, alerting users would probably be a good first step, also to hear from users who can't potentially upgrade rather sooner than later.

[jakirkham]

This is just adding a warning right? Not actually dropping anything. Think it makes sense to get this in sooner rather than later. It will give us flexibility later on when deciding whether we want to drop or not. The longer we wait to communicate this to users, the longer it takes for us to actually do the deprecation, which may come at a point where things are more painful

[jakirkham]

That said, it's probably worth raising this with key stakeholders first just so they are aware and we can make any changes to the deprecation schedule if needed.

[pentschev]

cc @dantegd @cjnolet for cuML
cc @BradReesWork @rlratzel for cuGraph
cc @randerzander @beckernick @VibhuJawa @ayushdg for Dask workflows
cc @JohnZed generally for RAPIDS
cc @benjha @MattBBaker @jglaser for Summit/ORNL

[benjha]

Hi all,

What do you recommend for a "quick" check ? Build ucx-py 1.11.1 in a stand alone environment and run the test suite ?

[pentschev]

@benjha yes, it would be good to run UCX-Py and Dask-CUDA tests if you can:

# UCX-Py
git clone https://github.com/rapidsai/ucx-py
cd ucx-py
pip install .
pytest --cache-clear -vs ucp/_libs/tests/
UCX_TLS=rc,tcp,cuda_copy,cuda_ipc pytest --cache-clear -vs tests/

# Dask-CUDA
git clone https://github.com/rapidsai/dask-cuda
cd dask-cuda
pip install .
pytest -vs --cache-clear dask_cuda/tests/

EDIT: For the async UCX-Py API I've added UCX_TLS. It turns out that the shm transport causes some of the tests to hang, which is a transport that still requires some work, see #565 .

[pentschev]

I've also checked with DLFW folks offline, and they're already running some RC version of UCX 1.11.1, so dropping compatibility code isn't a problem for them, as that only affects UCX 1.9 and older.

[jakirkham]

How do we feel about merging this?

[pentschev]

Since no objections were raised in over 2 months, I'll plan to merge this later today. We'll discuss a plan to completely drop support for UCX < 1.11.1 early next year.

[pentschev]

rernu tests

[pentschev]

rerun tests

[pentschev]

I'm doing some final testing, will merge this tomorrow or Monday.