Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-42149: [C++] Use FetchContent for bundled ORC #43011

Merged
merged 24 commits into from
Jul 8, 2024

Conversation

kou
Copy link
Member

@kou kou commented Jun 22, 2024

Rationale for this change

This also has a workaround for https://issues.apache.org/jira/browse/ORC-1732 .

What changes are included in this PR?

ORC 2.0.1 has a dependency detection problem. We can't override the detection with ExternalProject but can override the detection with FetchContent.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

Copy link

⚠️ GitHub issue #42149 has been automatically assigned in GitHub to PR creator.

@raulcd
Copy link
Member

raulcd commented Jun 26, 2024

@github-actions crossbow submit verify-rc-source--macos-

This comment was marked as outdated.

@kou
Copy link
Member Author

kou commented Jun 28, 2024

(I'll complete this in the next week...)

@raulcd
Copy link
Member

raulcd commented Jun 28, 2024

Thanks @kou !

@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 3, 2024
@kou kou marked this pull request as ready for review July 4, 2024 00:19
@kou
Copy link
Member Author

kou commented Jul 4, 2024

@github-actions crossbow submit -g nightly-tests -g nightly-packaging -g nightly-release

This comment was marked as outdated.

set(SNAPPY_HOME
${Snappy_ROOT}
CACHE BOOL "" FORCE)
set(SNAPPY_LIBRARY
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we set LZ4_STATIC_LIB but do not set SNAPPY_STATIC_LIB or PROTOBUF_STATIC_LIB?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. We don't need to specify *_STATIC_LIB by defining ORC_PREFER_STATIC_*=OFF.


get_target_property(ORC_ZSTD_ROOT ${ARROW_ZSTD_LIBZSTD} INTERFACE_INCLUDE_DIRECTORIES)
get_filename_component(ORC_ZSTD_ROOT "${ORC_ZSTD_ROOT}" DIRECTORY)
# TODO: This should be fixed in upstream.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this after upgrading to v2.0.2 which should the fix? If yes, perhaps add a comment to we won't forget.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I've updated the comment.

if(CMAKE_CXX_COMPILER_VERSION VERSION_LESS "9")
target_link_libraries(orc::orc INTERFACE stdc++fs)
endif()
elseif(CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean we no longer support these legacy versions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. But it seems that this is not needed with ORC 2.0.1...?
https://github.com/ursacomputing/crossbow/actions/runs/9786486604/job/27021493582 passed

Did you fix this in ORC 2.0.1? #41023 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou
Copy link
Member Author

kou commented Jul 4, 2024

@github-actions crossbow submit -g nightly-tests -g nightly-packaging -g nightly-release

@github-actions github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Jul 4, 2024

This comment was marked as outdated.

Copy link
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@kou
Copy link
Member Author

kou commented Jul 4, 2024

(I'm still fixing CI failures...)

@kou
Copy link
Member Author

kou commented Jul 6, 2024

@github-actions crossbow submit -g nightly-tests -g nightly-packaging -g nightly-release

Copy link

github-actions bot commented Jul 6, 2024

Revision: cf33494

Submitted crossbow builds: ursacomputing/crossbow @ actions-f3d88eefb6

Task Status
almalinux-8-amd64 GitHub Actions
almalinux-8-arm64 GitHub Actions
almalinux-9-amd64 GitHub Actions
almalinux-9-arm64 GitHub Actions
amazon-linux-2023-amd64 GitHub Actions
amazon-linux-2023-arm64 GitHub Actions
centos-7-amd64 GitHub Actions
centos-8-stream-amd64 GitHub Actions
centos-8-stream-arm64 GitHub Actions
centos-9-stream-amd64 GitHub Actions
centos-9-stream-arm64 GitHub Actions
conan-maximum GitHub Actions
conan-minimum GitHub Actions
conda-clean Azure
conda-linux-aarch64-cpu-py3 Azure
conda-linux-aarch64-cuda-py3 Azure
conda-linux-ppc64le-cpu-py3 Azure
conda-linux-ppc64le-cuda-py3 Azure
conda-linux-x64-cpu-py3 Azure
conda-linux-x64-cuda-py3 Azure
conda-osx-arm64-cpu-py3 Azure
conda-osx-x64-cpu-py3 Azure
conda-win-x64-cpu-py3 Azure
conda-win-x64-cuda-py3 Azure
debian-bookworm-amd64 GitHub Actions
debian-bookworm-arm64 GitHub Actions
debian-trixie-amd64 GitHub Actions
debian-trixie-arm64 GitHub Actions
example-cpp-minimal-build-static GitHub Actions
example-cpp-minimal-build-static-system-dependency GitHub Actions
example-cpp-tutorial GitHub Actions
example-python-minimal-build-fedora-conda GitHub Actions
example-python-minimal-build-ubuntu-venv GitHub Actions
homebrew-cpp GitHub Actions
java-jars GitHub Actions
nuget GitHub Actions
python-sdist GitHub Actions
r-binary-packages GitHub Actions
test-alpine-linux-cpp GitHub Actions
test-build-cpp-fuzz GitHub Actions
test-build-vcpkg-win GitHub Actions
test-conda-cpp GitHub Actions
test-conda-cpp-valgrind GitHub Actions
test-conda-python-3.10 GitHub Actions
test-conda-python-3.10-cython2 GitHub Actions
test-conda-python-3.10-hdfs-2.9.2 GitHub Actions
test-conda-python-3.10-hdfs-3.2.1 GitHub Actions
test-conda-python-3.10-pandas-latest-numpy-1.26 GitHub Actions
test-conda-python-3.10-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.10-pandas-nightly-numpy-nightly GitHub Actions
test-conda-python-3.10-spark-v3.5.0 GitHub Actions
test-conda-python-3.10-substrait GitHub Actions
test-conda-python-3.11 GitHub Actions
test-conda-python-3.11-dask-latest GitHub Actions
test-conda-python-3.11-dask-upstream_devel GitHub Actions
test-conda-python-3.11-hypothesis GitHub Actions
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly GitHub Actions
test-conda-python-3.11-spark-master GitHub Actions
test-conda-python-3.12 GitHub Actions
test-conda-python-3.8 GitHub Actions
test-conda-python-3.8-pandas-1.0-numpy-1.19 GitHub Actions
test-conda-python-3.8-spark-v3.5.0 GitHub Actions
test-conda-python-3.9 GitHub Actions
test-conda-python-3.9-pandas-latest-numpy-latest GitHub Actions
test-conda-python-emscripten GitHub Actions
test-cuda-cpp GitHub Actions
test-cuda-python GitHub Actions
test-debian-12-cpp-amd64 GitHub Actions
test-debian-12-cpp-i386 GitHub Actions
test-debian-12-docs GitHub Actions
test-debian-12-go-1.21 GitHub Actions
test-debian-12-go-1.22 GitHub Actions
test-debian-12-python-3-amd64 GitHub Actions
test-debian-12-python-3-i386 GitHub Actions
test-debian-c-glib GitHub Actions
test-debian-ruby GitHub Actions
test-fedora-39-cpp GitHub Actions
test-fedora-39-python-3 GitHub Actions
test-r-arrow-backwards-compatibility GitHub Actions
test-r-clang-sanitizer GitHub Actions
test-r-depsource-bundled Azure
test-r-depsource-system GitHub Actions
test-r-dev-duckdb GitHub Actions
test-r-devdocs GitHub Actions
test-r-gcc-11 GitHub Actions
test-r-gcc-12 GitHub Actions
test-r-install-local GitHub Actions
test-r-install-local-minsizerel GitHub Actions
test-r-linux-as-cran GitHub Actions
test-r-linux-rchk GitHub Actions
test-r-linux-valgrind GitHub Actions
test-r-minimal-build Azure
test-r-offline-maximal GitHub Actions
test-r-offline-minimal Azure
test-r-rhub-debian-gcc-devel-lto-latest Azure
test-r-rhub-debian-gcc-release-custom-ccache Azure
test-r-rhub-ubuntu-release-latest Azure
test-r-rocker-r-ver-latest Azure
test-r-rstudio-r-base-4.1-opensuse155 Azure
test-r-rstudio-r-base-4.2-focal Azure
test-r-ubuntu-22.04 GitHub Actions
test-r-versions GitHub Actions
test-skyhook-integration GitHub Actions
test-ubuntu-20.04-cpp GitHub Actions
test-ubuntu-20.04-cpp-bundled GitHub Actions
test-ubuntu-20.04-cpp-minimal-with-formats GitHub Actions
test-ubuntu-20.04-cpp-thread-sanitizer GitHub Actions
test-ubuntu-20.04-python-3 GitHub Actions
test-ubuntu-22.04-cpp GitHub Actions
test-ubuntu-22.04-cpp-20 GitHub Actions
test-ubuntu-22.04-cpp-emscripten GitHub Actions
test-ubuntu-22.04-cpp-no-threading GitHub Actions
test-ubuntu-22.04-python-3 GitHub Actions
test-ubuntu-24.04-cpp GitHub Actions
test-ubuntu-24.04-cpp-gcc-14 GitHub Actions
test-ubuntu-c-glib GitHub Actions
test-ubuntu-r-sanitizer GitHub Actions
test-ubuntu-ruby GitHub Actions
ubuntu-focal-amd64 GitHub Actions
ubuntu-focal-arm64 GitHub Actions
ubuntu-jammy-amd64 GitHub Actions
ubuntu-jammy-arm64 GitHub Actions
ubuntu-noble-amd64 GitHub Actions
ubuntu-noble-arm64 GitHub Actions
verify-rc-source-cpp-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-cpp-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-cpp-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-cpp-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-cpp-macos-amd64 GitHub Actions
verify-rc-source-cpp-macos-arm64 GitHub Actions
verify-rc-source-cpp-macos-conda-amd64 GitHub Actions
verify-rc-source-csharp-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-csharp-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-csharp-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-csharp-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-csharp-macos-amd64 GitHub Actions
verify-rc-source-csharp-macos-arm64 GitHub Actions
verify-rc-source-go-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-go-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-go-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-go-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-go-macos-amd64 GitHub Actions
verify-rc-source-go-macos-arm64 GitHub Actions
verify-rc-source-integration-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-integration-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-integration-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-integration-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-integration-macos-amd64 GitHub Actions
verify-rc-source-integration-macos-arm64 GitHub Actions
verify-rc-source-integration-macos-conda-amd64 GitHub Actions
verify-rc-source-java-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-java-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-java-macos-amd64 GitHub Actions
verify-rc-source-js-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-js-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-js-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-js-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-js-macos-amd64 GitHub Actions
verify-rc-source-js-macos-arm64 GitHub Actions
verify-rc-source-python-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-python-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-python-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-python-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-python-macos-amd64 GitHub Actions
verify-rc-source-python-macos-arm64 GitHub Actions
verify-rc-source-python-macos-conda-amd64 GitHub Actions
verify-rc-source-ruby-linux-almalinux-8-amd64 GitHub Actions
verify-rc-source-ruby-linux-conda-latest-amd64 GitHub Actions
verify-rc-source-ruby-linux-ubuntu-20.04-amd64 GitHub Actions
verify-rc-source-ruby-linux-ubuntu-22.04-amd64 GitHub Actions
verify-rc-source-ruby-macos-amd64 GitHub Actions
verify-rc-source-ruby-macos-arm64 GitHub Actions
verify-rc-source-windows GitHub Actions
wheel-macos-big-sur-cp310-arm64 GitHub Actions
wheel-macos-big-sur-cp311-arm64 GitHub Actions
wheel-macos-big-sur-cp312-arm64 GitHub Actions
wheel-macos-big-sur-cp38-arm64 GitHub Actions
wheel-macos-big-sur-cp39-arm64 GitHub Actions
wheel-macos-catalina-cp310-amd64 GitHub Actions
wheel-macos-catalina-cp311-amd64 GitHub Actions
wheel-macos-catalina-cp312-amd64 GitHub Actions
wheel-macos-catalina-cp38-amd64 GitHub Actions
wheel-macos-catalina-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-arm64 GitHub Actions
wheel-manylinux-2-28-cp311-amd64 GitHub Actions
wheel-manylinux-2-28-cp311-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp38-amd64 GitHub Actions
wheel-manylinux-2-28-cp38-arm64 GitHub Actions
wheel-manylinux-2-28-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp39-arm64 GitHub Actions
wheel-manylinux-2014-cp310-amd64 GitHub Actions
wheel-manylinux-2014-cp310-arm64 GitHub Actions
wheel-manylinux-2014-cp311-amd64 GitHub Actions
wheel-manylinux-2014-cp311-arm64 GitHub Actions
wheel-manylinux-2014-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp38-amd64 GitHub Actions
wheel-manylinux-2014-cp38-arm64 GitHub Actions
wheel-manylinux-2014-cp39-amd64 GitHub Actions
wheel-manylinux-2014-cp39-arm64 GitHub Actions
wheel-windows-cp310-amd64 GitHub Actions
wheel-windows-cp311-amd64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions
wheel-windows-cp38-amd64 GitHub Actions
wheel-windows-cp39-amd64 GitHub Actions

@kou
Copy link
Member Author

kou commented Jul 6, 2024

@github-actions crossbow submit debian-* ubuntu-*

Copy link

github-actions bot commented Jul 6, 2024

Revision: 2e4112d

Submitted crossbow builds: ursacomputing/crossbow @ actions-510dfad054

Task Status
debian-bookworm-amd64 GitHub Actions
debian-bookworm-arm64 GitHub Actions
debian-trixie-amd64 GitHub Actions
debian-trixie-arm64 GitHub Actions
ubuntu-focal-amd64 GitHub Actions
ubuntu-focal-arm64 GitHub Actions
ubuntu-jammy-amd64 GitHub Actions
ubuntu-jammy-arm64 GitHub Actions
ubuntu-noble-amd64 GitHub Actions
ubuntu-noble-arm64 GitHub Actions

@kou
Copy link
Member Author

kou commented Jul 7, 2024

@raulcd This is ready to merge.

@raulcd raulcd merged commit e8a795b into apache:main Jul 8, 2024
36 checks passed
@raulcd raulcd removed the awaiting change review Awaiting change review label Jul 8, 2024
@github-actions github-actions bot added the awaiting merge Awaiting merge label Jul 8, 2024
raulcd pushed a commit that referenced this pull request Jul 8, 2024
### Rationale for this change

This also has a workaround for https://issues.apache.org/jira/browse/ORC-1732 .

### What changes are included in this PR?

ORC 2.0.1 has a dependency detection problem. We can't override the detection with ExternalProject but can override the detection with FetchContent.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #42149

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
Copy link

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit e8a795b.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 102 possible false positives for unstable benchmarks that are known to sometimes produce them.

zanmato1984 pushed a commit to zanmato1984/arrow that referenced this pull request Jul 9, 2024
### Rationale for this change

This also has a workaround for https://issues.apache.org/jira/browse/ORC-1732 .

### What changes are included in this PR?

ORC 2.0.1 has a dependency detection problem. We can't override the detection with ExternalProject but can override the detection with FetchContent.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#42149

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
@kou kou deleted the cpp-orc-fetch-content branch July 11, 2024 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants