Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev Container support for building and testing RAPIDS Spark #268

Open
harrism opened this issue Apr 17, 2024 · 0 comments
Open

Dev Container support for building and testing RAPIDS Spark #268

harrism opened this issue Apr 17, 2024 · 0 comments

Comments

@harrism
Copy link
Member

harrism commented Apr 17, 2024

Is your feature request related to a problem? Please describe.

More developers are using dev containers for RAPIDS development. We can build most of RAPIDS from the provided dev containers. But when changes need to be tested for impacts on the Java bindings, we have to use a completely separate process.

This would also enable experimenting with cuDF / RAPIDS changes while running Spark workloads.

Describe the solution you'd like

Add the proper dependencies and scripts for building and testing JNI.

Describe alternatives you've considered

  • Use the documented process for building libcudf for Spark and then install JDK and maven and build them.
  • Use Spark-RAPIDS containers for testing.

Both of these require me to commit my changes from the dev container, and then checkout the branch inside the Spark environment in order to build and test. An integrated environment will be more productive.

Additional context

I have successfully installed JDK and Maven in a dev container with cuDF, but was unable to build cuDF because of a CMake error.

[exec] CMake Error at /home/coder/cudf/java/target/cmake-build/_deps/rapids-cmake-src/rapids-cmake/find/package.cmake:125 (find_package):
     [exec]   By not providing "Findcudf.cmake" in CMAKE_MODULE_PATH this project has
     [exec]   asked CMake to find a package configuration file provided by "cudf", but
     [exec]   CMake did not find one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant