New ILAB-KERNEL - RAPIDS 23.10

[jordancaraballo]

The original instructions from Matt are no longer available given the versioning issues of several packages. This issue will document each one of the procedures done to achieve that. Taking this from Sav's email as the initial attempt:

[sstrong@ilab201 dev]$ module load mamba/0.27.0
[sstrong@ilab201 dev]$ micromamba create -p /panfs/ccds02/app/modules/jupyter/ilab/dev/tensorflow-dev-rapids-v23_10 -c rapidsai -c nvidia -c conda-forge -c legate rapids=23.10 python=3.8 cudatoolkit=11.2 cudnn tensorflow pip xarray dask matplotlib numpy pandas scikit-learn netCDF4 ipympl nodejs gdal rasterio seaborn geoviews cartopy hvplot holoviews rioxarray=0.13.4 mkl jupyter-dash ipysheet=0.6.0 pytest omegaconf black contextily rio-cogeo localtileserver jupyter-server-proxy dask-geopandas ipyleaflet leafmap pooch pythreejs=2.4.1 owslib cunumeric optuna ipykernel=6.17.1 plotnine mizani tiler datatable hydra ipywidgets=8.0.2 datasets -y

The issue with this command is that cudatoolkit=11.2 is not compatible with rapids=23.10. That is why we use the rapids configuration from the installation website https://docs.rapids.ai/install.

The next step is to apply the correct one.

[jordancaraballo]

From the previous command hydra is no longer supported as "hydra", instead you need to use "hydra-core" as the python3 package. In addition, datatable is no longer supported by python 3.10 and has not been updated since 2021. Testing the environment with the following command, which excludes datatable and includes the new hydra-core package:

micromamba create -n tensorflow-dev-rapids-v23_10 -c rapidsai -c conda-forge -c nvidia -c legate rapids=23.10 python=3.10 cuda-version=12.0 cudnn tensorflow pip xarray dask matplotlib numpy pandas scikit-learn netCDF4 ipympl nodejs gdal rasterio seaborn geoviews cartopy hvplot holoviews rioxarray mkl jupyter-dash ipysheet=0.6.0 pytest omegaconf black contextily rio-cogeo localtileserver jupyter-server-proxy dask-geopandas ipyleaflet leafmap pooch pythreejs owslib cunumeric optuna ipykernel plotnine mizani tiler hydra-core ipywidgets datasets -y

[jordancaraballo]

The command above works well for RAPIDS and the other dependencies, requires some regression testing. Tensorflow GPU support needs to be fixed. @ssfinch feel free to install the environment and test.