GH-36346: [C++] Safe S3 finalization #36442
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is an alternative approach to #36437. @westonpace @kou |
pitrou
commented
Jul 3, 2023
cpp/src/arrow/filesystem/s3fs.cc
Outdated
| AwsInstance() : is_initialized_(false), is_finalized_(false) {} | ||
| ~AwsInstance() { Finalize(/*from_destructor=*/true); } | ||
|
|
||
| // Returns true iff the instance was newly initialized with `options` | ||
| Result<bool> EnsureInitialized(const S3GlobalOptions& options) { | ||
| bool expected = false; | ||
| // XXX The individual accesses are atomic but the entire sequence below is not. |
There was a problem hiding this comment.
Note that I don't think this should be a problem in practice, as calls to EnsureInitialized and Finalize should be serialized at the application level.
There was a problem hiding this comment.
I agree with your opinion.
How about adding this note to this comment or documents for InitializeS3()/FinalizeS3()?
(I think that we can remove the "XXX" mark here.)
github-actions
bot
added
awaiting committer review
|
@github-actions crossbow submit -g python -g wheel |
This comment was marked as outdated.
This comment was marked as outdated.
kou
reviewed
Jul 4, 2023
cpp/src/arrow/filesystem/s3fs.cc
Outdated
| AwsInstance() : is_initialized_(false), is_finalized_(false) {} | ||
| ~AwsInstance() { Finalize(/*from_destructor=*/true); } | ||
|
|
||
| // Returns true iff the instance was newly initialized with `options` | ||
| Result<bool> EnsureInitialized(const S3GlobalOptions& options) { | ||
| bool expected = false; | ||
| // XXX The individual accesses are atomic but the entire sequence below is not. |
There was a problem hiding this comment.
I agree with your opinion.
How about adding this note to this comment or documents for InitializeS3()/FinalizeS3()?
(I think that we can remove the "XXX" mark here.)
github-actions
bot
added
awaiting review
|
@github-actions crossbow submit -g python -g wheel |
github-actions
bot
added
awaiting review
|
Thanks a lot for the review @kou . I agree with all your comments and made the necessary changes. |
|
Revision: c171b31 Submitted crossbow builds: ursacomputing/crossbow @ actions-1c2290f238
|
|
@github-actions crossbow submit -g cpp |
|
Revision: c171b31 Submitted crossbow builds: ursacomputing/crossbow @ actions-0d14e99a00
|
This was referenced Jul 4, 2023
|
The remaining CI failures seem unrelated. I think I'm going to merge so as to improve CI, as we're nearing the release. |
westonpace
pushed a commit
to westonpace/arrow
that referenced
this pull request
Jul 7, 2023
### Rationale for this change It is required to call `FinalizeS3` at process end to cleanly shut down resources used by the AWS SDK before it's too late. However, once `FinalizeS3` has been called, another problem appears: no AWS SDK API can be called safely anymore. Even object destructors can be dangerous. We therefore need a way to prevent unsafe use of the AWS SDK APIs, regardless of how the application issues filesystems calls. ### What changes are included in this PR? Shield all uses of the internal `S3Client` class behind a safe RAII facility `S3ClientLock`. Obtaining a `S3ClientLock` ensures that S3 finalization has not been called yet _and_ that it will not be called until the `S3ClientLock` goes out of scope. ### Are these changes tested? Yes, some Python tests exercise calling S3 APIs after explicit finalization. ### Are there any user-facing changes? Not for correctly written application code, ideally. * Closes: apache#36346 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>
|
Conbench analyzed the 6 benchmark runs on commit There were 7 benchmark results indicating a performance regression:
The full Conbench report has more details. |
kou
pushed a commit
that referenced
this pull request
Aug 3, 2023
### Rationale for this change In #12227 we decided to use a bundled version of the AWS SDK when compiling Python wheels, in order to downgrade the AWS SDK version. Now that we have fixed S3 finalization issues (#36442), it should be ok to rely on the vcpkg-installed version of the AWS SDK again. ### What changes are included in this PR? Remove use of bundled AWS SDK and use S3 vcpkg feature for requirements. ### Are these changes tested? On CI and via crossbow ### Are there any user-facing changes? No * Closes: #36752 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
|
Did this make it into /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp/src/arrow/filesystem/s3fs.cc:2829: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit
zsh: segmentation fault ./test_routing_pybind
(venv) test % pip list | grep pyarrow
pyarrow 13.0.0 |
|
Yes, it should be in 13.0.0. Can you please open an issue with a reproducer script? |
loicalleyne
pushed a commit
to loicalleyne/arrow
that referenced
this pull request
Nov 13, 2023
…apache#36925) ### Rationale for this change In apache#12227 we decided to use a bundled version of the AWS SDK when compiling Python wheels, in order to downgrade the AWS SDK version. Now that we have fixed S3 finalization issues (apache#36442), it should be ok to rely on the vcpkg-installed version of the AWS SDK again. ### What changes are included in this PR? Remove use of bundled AWS SDK and use S3 vcpkg feature for requirements. ### Are these changes tested? On CI and via crossbow ### Are there any user-facing changes? No * Closes: apache#36752 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
It is required to call
FinalizeS3at process end to cleanly shut down resources used by the AWS SDK before it's too late.However, once
FinalizeS3has been called, another problem appears: no AWS SDK API can be called safely anymore. Even object destructors can be dangerous.We therefore need a way to prevent unsafe use of the AWS SDK APIs, regardless of how the application issues filesystems calls.
What changes are included in this PR?
Shield all uses of the internal
S3Clientclass behind a safe RAII facilityS3ClientLock. Obtaining aS3ClientLockensures that S3 finalization has not been called yet and that it will not be called until theS3ClientLockgoes out of scope.Are these changes tested?
Yes, some Python tests exercise calling S3 APIs after explicit finalization.
Are there any user-facing changes?
Not for correctly written application code, ideally.