Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] some RAPIDS repos are incompatible with CPM 0.40.2 #680

Open
jameslamb opened this issue Aug 20, 2024 · 5 comments
Open

[BUG] some RAPIDS repos are incompatible with CPM 0.40.2 #680

jameslamb opened this issue Aug 20, 2024 · 5 comments
Assignees
@robertmaynard
Labels
? - Needs Triage Need team to review and classify bug Something isn't working

Comments

@jameslamb
Copy link
Member

jameslamb commented Aug 20, 2024 •
edited
Loading

Describe the bug

When we tried upgrading to CPM 0.40.2 in #678, found that some RAPIDS repos had build failures.

Opening this to track the work of investigating that.

Steps/Code to reproduce bug

On latest cudf 24.10 (rapidsai/cudf@e450baf), from the root of the cudf repo, and with the changes from #678 applied.

docker run \
   --rm \
   -v $(pwd):/usr/local/src \
   -w /usr/local/src \
   -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.11 \
   ci/configure_cpp_static.sh

Those configure tests failed like this:

-- Using  approach to find dependencies
CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:449 (file):
  file STRINGS file
  "/usr/local/src/build_static/_deps/arrow-src/thirdparty/versions.txt"
  cannot be read.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)

And later on:

-- Building xsimd from source
CMake Error at /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject/shared_internal_commands.cmake:1099 (message):
  URL_HASH is set to

    SHA256=

  but must be ALGO=value where ALGO is

    MD5|SHA1|SHA224|SHA256|SHA384|SHA512|SHA3_224|SHA3_256|SHA3_384|SHA3_512

  and value is a hex string.
Call Stack (most recent call first):
  /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject.cmake:3035 (_ep_add_download_command)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2465 (externalproject_add)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:226 (build_xsimd)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:304 (build_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2491 (resolve_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)
full logs (click me) 

==========
== CUDA ==
==========

CUDA Version 12.5.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .


RAPIDS logger » [08/20/24 20:31:11]
┌──────────────────────────────────┐
|    Configure static cpp build    |
└──────────────────────────────────┘

# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`.
cmake>=3.26.4,!=3.30.0
ninja

Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/rapidsai-wheels-nightly/simple, https://pypi.nvidia.com
Collecting cmake!=3.30.0,>=3.26.4 (from -r /tmp/tmp.IfrYPE84N0/requirements.txt (line 3))
  Downloading cmake-3.30.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
Collecting ninja (from -r /tmp/tmp.IfrYPE84N0/requirements.txt (line 4))
  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Downloading cmake-3.30.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.9/26.9 MB 80.0 MB/s eta 0:00:00
Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 57.2 MB/s eta 0:00:00
Installing collected packages: ninja, cmake
Successfully installed cmake-3.30.2 ninja-1.11.1.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip
-- CPM: Adding package [email protected] (jitify2)
-- Populating jitify
-- Configuring done (0.1s)
-- Generating done (0.1s)
-- Build files have been written to: /usr/local/src/build_static/_deps/jitify-subbuild
[0/7] Performing update step for 'jitify-populate'
-- Fetching latest from the remote origin
[1/7] No patch step for 'jitify-populate'
[3/7] No configure step for 'jitify-populate'
[4/7] No build step for 'jitify-populate'
[5/7] No install step for 'jitify-populate'
[6/7] No test step for 'jitify-populate'
[7/7] Completed 'jitify-populate'
-- CPM: Adding package [email protected] (v3.1.0)
-- CPM: Using local package [email protected]
-- CPM: Adding package [email protected] (e21d607157218540cd7c45461213fb96adf720b7)
-- rapids-cmake [CCCL]: applied diff cccl_symbol_visibility.diff to fix issue: 'Correct symbol visibility issues in libcudacxx [https://github.com/NVIDIA/cccl/pull/1832/]'
-- rapids-cmake [CCCL]: applied diff thrust_disable_64bit_dispatching.diff to fix issue: 'Remove 64bit dispatching as not needed by libcudf and results in compiling twice as many kernels [https://github.com/rapidsai/cudf/pull/11437]'
-- rapids-cmake [CCCL]: applied diff thrust_faster_sort_compile_times.diff to fix issue: 'Improve Thrust sort compile times by not unrolling loops for inlined comparators [https://github.com/rapidsai/cudf/pull/10577]'
-- rapids-cmake [CCCL]: applied diff thrust_faster_scan_compile_times.diff to fix issue: 'Improve Thrust scan compile times by reducing the number of kernels generated [https://github.com/rapidsai/cudf/pull/8183]'
-- CPM: Adding package [email protected] (branch-24.10)
-- RMM: RMM_LOGGING_LEVEL = 'INFO'
-- CPM: Adding package [email protected] (10.1.1)
-- Version: 10.1.1
-- Build type: Release
-- rapids-cmake [fmt]: applied diff fix_10_1_1_version.diff to fix issue: 'fmt 10.1.1 produces a CMake package with version 10.1.0'
-- CPM: Adding package [email protected] (v1.12.0)
-- Build spdlog: 1.12.0
-- Build type: Release
-- Generating install
-- rapids-cmake [spdlog]: applied diff nvcc_constexpr_fix.diff to fix issue: 'Fix constexpr mismatch between spdlog and fmt [https://github.com/gabime/spdlog/issues/2856]'
-- CPM: Adding package [email protected] (apache-arrow-16.1.0)
-- Building using CMake version: 3.30.2
-- Arrow version: 16.1.0 (full: '16.1.0')
-- Arrow SO version: 1601 (full: 1601.0.0)
-- clang-tidy 14 not found
-- clang-format 14 not found
-- Could NOT find ClangTools (missing: CLANG_FORMAT_BIN CLANG_TIDY_BIN) 
-- infer not found
-- Found cpplint executable at CPPLINT_BIN-NOTFOUND
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
-- Using ld linker
-- Build Type: RELEASE
-- Using  approach to find dependencies
CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:449 (file):
  file STRINGS file
  "/usr/local/src/build_static/_deps/arrow-src/thirdparty/versions.txt"
  cannot be read.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:604 (string):
  string sub-command REPLACE requires at least four arguments.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


CMake Error at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:764 (string):
  string sub-command SUBSTRING requires four arguments.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


-- Building without OpenSSL support. Minimum OpenSSL version 1.0.2 required.
CMake Warning at /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:291 (find_package):
  By not providing "Findxsimd.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "xsimd", but
  CMake did not find one.

  Could not find a package configuration file provided by "xsimd" with any of
  the following names:

    xsimdConfig.cmake
    xsimd-config.cmake

  Add the installation prefix of "xsimd" to CMAKE_PREFIX_PATH or set
  "xsimd_DIR" to a directory containing one of the above files.  If "xsimd"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2491 (resolve_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


-- Building xsimd from source
CMake Error at /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject/shared_internal_commands.cmake:1099 (message):
  URL_HASH is set to

    SHA256=

  but must be ALGO=value where ALGO is

    MD5|SHA1|SHA224|SHA256|SHA384|SHA512|SHA3_224|SHA3_256|SHA3_384|SHA3_512

  and value is a hex string.
Call Stack (most recent call first):
  /pyenv/versions/3.11.9/lib/python3.11/site-packages/cmake/data/share/cmake-3.30/Modules/ExternalProject.cmake:3035 (_ep_add_download_command)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2465 (externalproject_add)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:226 (build_xsimd)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:304 (build_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/cmake_modules/ThirdpartyToolchain.cmake:2491 (resolve_dependency)
  /usr/local/src/build_static/_deps/arrow-src/cpp/CMakeLists.txt:543 (include)


-- Configuring incomplete, errors occurred!

Pinning rapids-cmake back to 0e65a84 resolves those failures.

diff --git a/rapids_config.cmake b/rapids_config.cmake
index 3a88769f6e..60aef012ff 100644
--- a/rapids_config.cmake
+++ b/rapids_config.cmake
@@ -26,11 +26,14 @@ else()
   )
 endif()
 
-if(NOT EXISTS "${CMAKE_CURRENT_BINARY_DIR}/CUDF_RAPIDS-${RAPIDS_VERSION_MAJOR_MINOR}.cmake")
-  file(
-    DOWNLOAD
-    "[https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-${RAPIDS_VERSION_MAJOR_MINOR}/RAPIDS.cmake](https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-$%7BRAPIDS_VERSION_MAJOR_MINOR%7D/RAPIDS.cmake)"
-    "${CMAKE_CURRENT_BINARY_DIR}/CUDF_RAPIDS-${RAPIDS_VERSION_MAJOR_MINOR}.cmake"
-  )
-endif()
+include(FetchContent)
+# GIT_TAG = last commit prior to https://github.com/rapidsai/rapids-cmake/pull/678
+FetchContent_Declare(
+  rapids-cmake
+  GIT_REPOSITORY https://github.com/rapidsai/rapids-cmake.git
+  GIT_TAG        0e65a8481b9895b5075d2b601690540a21b45deb
+)
+file(DOWNLOAD https://raw.githubusercontent.com/rapidsai/rapids-cmake/branch-24.10/RAPIDS.cmake
+  ${CMAKE_CURRENT_BINARY_DIR}/CUDF_RAPIDS-${RAPIDS_VERSION_MAJOR_MINOR}.cmake)
+

Expected behavior

N/A

Environment details (please complete the following information):

CMake

Additional context

N/A
@jameslamb jameslamb added bug Something isn't working ? - Needs Triage Need team to review and classify labels Aug 20, 2024
@jameslamb jameslamb mentioned this issue Aug 20, 2024
Merged
7 tasks
rapids-bot bot pushed a commit that referenced this issue Aug 20, 2024
@jameslamb
Revert "Update to CPM 0.40.2 to fix CMake 3.30 deprecation warnings (#…
82bc63b 
…678)" (#679)

Proposes reverting #678.

See #680 for context.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Robert Maynard (https://github.com/robertmaynard)
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #679
@robertmaynard robertmaynard self-assigned this Aug 22, 2024
@robertmaynard
Copy link
Contributor

Given that the computed path for the STRINGS read is "/usr/local/src/build_static/_deps/arrow-src/thirdparty/versions.txt" we can infer that the arrow_SOURCE_DIR variable isn't being computed properly. Which indicates that the SOURCE_SUBDIR argument to CPMFindPackage is being ignored when computing the value.

Will continue to dig into this.

@robertmaynard
Copy link
Contributor

Minimal reproducer: https://github.com/robertmaynard/cpm_40.2_repro

Going to look at necessary CPM changes next.

@robertmaynard
Copy link
Contributor

robertmaynard commented Aug 22, 2024 •
edited
Loading

Hole goes deeper. Looks to not be an issue with CPM but a CMake FetchContent bug

@robertmaynard
Copy link
Contributor

Reproduced to a FetchContent + Policy126 bug in CMake.

Upstream PR: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9770

@robertmaynard
Copy link
Contributor

Upstream PR was revised to: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9779

Which has been merged into the release branch, and will be part of CMake 3.30.3

@robertmaynard robertmaynard reopened this Aug 29, 2024
@robertmaynard robertmaynard mentioned this issue Nov 4, 2024
Merged
2 tasks
rapids-bot bot pushed a commit that referenced this issue Nov 6, 2024
@robertmaynard
rapids_export_write_language no longer generates cmake dev warnings (#…
e867d76 
…712)

The `rapids_export_write_language` function would generate CMake code that would try and do `set(var PARENT_SCOPE)` in the root directory. This would cause CMake developer warnings to appear.

This resolves the issues by adding root level guards around the `set` commands, and setting up each rapids-cmake test to have developer warnings as errors.  Due to #680 we mark some tests with `NO_DEV_ERRORS` so that they continue to pass while we wait to 2025 to bump the minimum required CMake version.

Authors:
  - Robert Maynard (https://github.com/robertmaynard)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #712
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@robertmaynard robertmaynard

Labels
? - Needs Triage Need team to review and classify bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@robertmaynard @jameslamb