Use sink cloud versions of ipc

[josevalim]

We need a new Explorer version:

• feat(rust): ParquetCloudSink to allow streaming pipelines into remote ObjectStores pola-rs/polars#10060
• feat(rust): Adds sink_ipc_cloud pola-rs/polars#11008

[Qqwy]

@josevalim FYI I briefly spoke with Ritchie on the Polars Discord today and he merged the sink_ipc_cloud PR just now, so work on the Explorer side can now continue!

[philss]

First of all: awesome work there on Polars, @Qqwy !! 🎉

Just to let you know that I'm working on this to bring the functionality to Explorer :)

[philss]

Hey @Qqwy, I had some issues trying to make the Sink IPC work, so I opened an issue upstream: pola-rs/polars#13614

I tried to find a fix for it, but I couldn't understand where the "block_on" is happening (twice). Would you mind to take a look?

[Qqwy]

Hi there! Thanks for tagging me, I'll dig into this problem. block_on is probably called from within the #[tokio::main] decorators. The crash happens because for some reason an attempt is made to start an async Tokio runtime from within a pre-existing Tokio runtime. Why that happens is of course the thing to figure out as it shouldn't.

Just to double-check: Is this only a problem with sink_ipc_cloud and not with sink_parquet_cloud?

[josevalim]

As far as I know, only ipc. @philss will be able to confirm though!

[philss]

@Qqwy yeah, it's only with the sink_ipc_cloud. The sink_parquet_cloud is working good, and it's already in main.

PS: sorry, I didn't see the ping before 😬

[Qqwy]

@philss I have created a PR with a fix on the Polars repo: pola-rs/polars#14262