fix(rust): Fix sink_ipc_cloud panicking with a Tokio runtime error
#14262
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
make CloudWriter call `get_runtime().block_on` from a newly spawned thread.
Qqwy
force-pushed
the
13614-ipc_cloud_tokio_error2
branch
from
ef4ae59
to
87af454
Compare
Qqwy
force-pushed
the
13614-ipc_cloud_tokio_error2
branch
from
e8f3e67
to
eca2306
Compare
Qqwy
requested review from
ritchie46,
stinodego,
orlp and
c-peters
as code owners
stinodego
changed the title
sink_ipc_cloud panicking with a Tokio runtime error
|
Hi @stinodego! I am sorry for the ping, but is there something blocking this PR or something we could do to get it merged? Thank you! ❤️ |
|
I'm still available to (help) fix(ing) the merge conflicts once more if needed, but it would be nice if it can be merged soon afterwards before new conflicts would arise 😅 |
|
I think this one can be closed as there has changed a lot since then. |
|
Is a fix still accepted? The functionality is currently broken and we would be happy users of it. Thank you :) |
|
Yes, it is accepted. But I believe it was fixed by #18027 |
|
I'm afraid this was not fixed yet. I could reproduce the error with the latest version from the main branch. Here is the backtrace: https://gist.github.com/philss/a81fab6f9cf85d23a0f1c364633e992b |
philss
added a commit
to philss/polars
that referenced
this pull request
Aug 8, 2024
Fixes pola-rs#13614 When writing to ObjectStore-compatible storage using the IPC format, it seems like the `block_on` calls inside the constructed `CloudWriter` might sometimes get called inside another `block_on` call. Tokio does not like this, resulting in a panic. This PR resolves this issue by using `block_on_potential_spawn` in the necessary places instead. This is a fix that was originally written by @Qqwy in another PR: pola-rs#14262 Co-Authored-By: Qqwy / Marten <[email protected]>
philss
added a commit
to philss/polars
that referenced
this pull request
Aug 8, 2024
Fixes pola-rs#13614 When writing to ObjectStore-compatible storage using the IPC format, it seems like the `block_on` calls inside the constructed `CloudWriter` might sometimes get called inside another `block_on` call. Tokio does not like this, resulting in a panic. This PR resolves this issue by using `block_on_potential_spawn` in the necessary places instead. This is a fix that was originally written by @Qqwy in another PR: pola-rs#14262 Co-Authored-By: Qqwy / Marten <[email protected]>
philss
added a commit
to philss/polars
that referenced
this pull request
Aug 8, 2024
Fixes pola-rs#13614 When writing to ObjectStore-compatible storage using the IPC format, it seems like the `block_on` calls inside the constructed `CloudWriter` might sometimes get called inside another `block_on` call. Tokio does not like this, resulting in a panic. This PR resolves this issue by using `block_on_potential_spawn` in the necessary places instead. This is a fix that was originally written by @Qqwy in another PR: pola-rs#14262 Co-Authored-By: Qqwy / Marten <[email protected]>
philss
added a commit
to philss/polars
that referenced
this pull request
Aug 8, 2024
Fixes pola-rs#13614 When writing to ObjectStore-compatible storage using the IPC format, it seems like the `block_on` calls inside the constructed `CloudWriter` might sometimes get called inside another `block_on` call. Tokio does not like this, resulting in a panic. This PR resolves this issue by using `block_on_potential_spawn` in the necessary places instead. This is a fix that was originally written by @Qqwy in another PR: pola-rs#14262 Co-Authored-By: Qqwy / Marten <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #13614
When writing ot ObjectStore-compatible storage in the IPC format, it seems like the
.block_oncalls inside the constructedCloudWritermight sometimes get called inside anotherblock_oncall. Tokio does not like this, resulting in a panic.This PR resolves this issue by using
block_on_potential_spawnin the necessary places instead.