[AWS] RAPIDS 24.10 and the sagemaker library can't be installed together

[jameslamb]

Description

AWS distributes a Python client for SageMaker called sagemaker-sdk-python (aws/sagemaker-python-sdk).

That library's latest versions have a pin of cloudpickle==2.2.1

• pyproject.toml: https://github.com/aws/sagemaker-python-sdk/blob/782fbfb27ba6696fda0bfd750ccb601ce511c1e5/pyproject.toml#L36
• conda recipe: https://github.com/conda-forge/sagemaker-feedstock/blob/3877391c93926db2d4b17a0f131e8e13da6cf80e/recipe/meta.yaml#L27

RAPIDS 24.10 depends on distributed==2024.9.0 (code link) which pins to cloudpickle>=3.0.0.

Because of that, it's not possible to install sagemaker together with RAPIDS 24.10+ libraries that depend on Dask.

Reproducible Example

docker run \
    --rm \
    -it rapidsai/base:24.10a-cuda12.5-py3.11 \
    bash

conda install -y -n base \
    'sagemaker>=2.198'

Results in this solver error:

Could not solve for environment specs
The following packages are incompatible
├─ pin-1 is installable and it requires
│  └─ python 3.11.* , which can be installed;
├─ rapids 24.10**  is installable with the potential options
│  ├─ rapids 24.10.00a would require
│  │  └─ dask-cuda 24.10.*  with the potential options
│  │     ├─ dask-cuda 24.10.00a23 would require
│  │     │  └─ rapids-dask-dependency 24.10.*,>=0.0.0a0 , which requires
│  │     │     └─ distributed 2024.9.0 , which requires
│  │     │        └─ cloudpickle >=3.0.0 , which can be installed;
│  │     ├─ dask-cuda 24.10.00a23 would require
│  │     │  └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
│  │     └─ dask-cuda 24.10.00a23 would require
│  │        └─ python >=3.12,<3.13.0a0 , which conflicts with any installable versions previously reported;
│  ├─ rapids 24.10.00a would require
│  │  └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
│  └─ rapids 24.10.00a would require
│     └─ python >=3.12,<3.13.0a0 , which conflicts with any installable versions previously reported;
└─ sagemaker >=2.198  is not installable because it requires
   └─ cloudpickle 2.2.1 , which conflicts with any installable versions previously reported.

Notes

Upstream issue about relaxing that cloudpickle pin: aws/sagemaker-python-sdk#4871