Pin pyarrow to <19 due to apparent incompatibility with vendored 16.0 Arrow C++ code on x86 macos #429
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… Arrow C++ code Signed-off-by: Adam Glustein <[email protected]>
AdamGlustein
requested review from
robambalu,
svatasoiu,
alexddobkin,
ptomecek and
timkpaine
as code owners
AdamGlustein
changed the title
svatasoiu
approved these changes
Jan 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It seems there is some ABI incompatibility on x86 Mac between our vendored C++ Pyarrow code (version 16.0) and the recently-released Pyarrow 19.0 Python code, which is resulting in files being produced from the ParquetWriter that are unreadable in Python. See: https://github.com/Point72/csp/actions/runs/12863970430/job/35865078010
Discussed with @timkpaine and there are two options: 1) pin back pyarrow to <19 to avoid the Python side upgrade and 2) upgrade our vendored C++ code to a higher version which is compatible. Due to other bugs in pyarrow 19 (see apache/arrow#45283) we decided to hold back on the upgrade until pyarrow 20.0, which (based on previous schedules) should come out in 2-3 months.