[BUG] test_read_region_cuda_memleak failing (GPUtil does not support Python 3.12)

[jameslamb]

Describe the bug

Python 3.12 support is rolling out across RAPIDS (rapidsai/build-planning#40), and was added in this project in #773.

After merge, one nightly performance test failed like this:

________________________ test_read_region_cuda_memleak _________________________
[gw6] linux -- Python 3.12.5 /pyenv/versions/3.12.5/bin/python
python/cucim/tests/performance/clara/test_read_region_memory_usage.py:25: in test_read_region_cuda_memleak
    import GPUtil
/pyenv/versions/3.12.5/lib/python3.12/site-packages/GPUtil/__init__.py:33: in <module>
    from .GPUtil import GPU, getGPUs, getAvailable, getAvailability, getFirstAvailable, showUtilization, __version__
/pyenv/versions/3.12.5/lib/python3.12/site-packages/GPUtil/GPUtil.py:34: in <module>
    from distutils import spawn
E   ModuleNotFoundError: No module named 'distutils'

(build link)

Steps/Code to reproduce bug

Run this test:

cucim/python/cucim/tests/performance/clara/test_read_region_memory_usage.py

Line 24 in 1584885

def test_read_region_cuda_memleak(testimg_tiff_stripe_4096x4096_256_jpeg):

Expected behavior

N/A

Additional context

distutils was removed from the Python standard library in Python 3.12: https://peps.python.org/pep-0632/

GPUtil has not been updated since August 2019 (commit history), and it unconditionally runs import distutils (code link).

Options to fix this:

• remove that performance test
• refactor that performance test to not use GPUtil
• modify GPUtil to make it compatible with Python 3.12 (and either contribute that upstream or patch/vendor it here)

[jameslamb]

I was wondering why, in the failing build linked above, the wheel tests for Python 3.12 were failing in this way but the conda tests were not.

Think I found it... setuptools provides distutils, and setuptools was making it into some but not all CI environments here.

docker run \
    --rm \
    -it rapidsai/ci-conda:cuda12.5.1-rockylinux8-py3.12 \
    python --version
# Python 3.12.5

docker run \
    --rm \
    -it rapidsai/ci-conda:cuda12.5.1-rockylinux8-py3.12 \
    python -c "import distutils; print(distutils.__file__)"
# /opt/conda/lib/python3.12/site-packages/setuptools/_distutils/__init__.py

docker run \
    --rm \
    -it rapidsai/citestwheel:cuda12.5.1-ubuntu22.04-py3.12 \
    python --version
# Python 3.12.5

docker run \
    --rm \
    -it rapidsai/citestwheel:cuda12.5.1-ubuntu22.04-py3.12 \
    python -c "import distutils; print(distutils.__file__)"
# ModuleNotFoundError: No module named 'distutils'

So another approach for now could be to take on setuptools as a test-time dependency, which would allow continuing to use GPUtil unconditionally.

But that will also eventually stop working, as setuptools is trying to whittle down how much of distutils it exposes and may eventually remove distutils entirely:

• how to handle numpy.distutils and setuptools interaction pypa/setuptools#2372 (comment)
• https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html