feat(python): IO plugins

[ritchie46]

This sets up all the architecture for IO plugins. Different from expression plugins these won't go over FFI directly, but will use python as intermediary. This is fine as this IO can release the GIL and do plenty of work before it needs it again.

The IO sources will be consumed by the batch engine and the streaming engine. The sources can act as a python generator yielding new dataframes. They will accept columns: str | None to apply projections, predicate: Expr | None to apply predicates and slice: tuple[int, int] | None to apply slices/early stopping.

After this is in, I will follow up with a hdf5 native reader.
The goal is to support many more file formats as separate wheels. This way Polars can support many IO formats in an efficient manner without worrying about binary size.

[codecov]

Codecov Report

Attention: Patch coverage is 69.75806% with 75 lines in your changes missing coverage. Please review.

Project coverage is 80.32%. Comparing base (82b6388) to head (239a4cc).
Report is 3 commits behind head on main.

Files	Patch %	Lines
crates/polars-plan/src/plans/python/predicate.rs	37.20%	27 Missing ⚠️
py-polars/src/lazyframe/visitor/nodes.rs	0.00%	27 Missing ⚠️
...olars-mem-engine/src/executors/scan/python_scan.rs	92.94%	6 Missing ⚠️
crates/polars-plan/src/plans/ir/dot.rs	0.00%	5 Missing ⚠️
py-polars/polars/io/plugins.py	80.00%	4 Missing ⚠️
crates/polars-plan/src/plans/anonymous_scan.rs	0.00%	3 Missing ⚠️
crates/polars-plan/src/plans/ir/schema.rs	0.00%	1 Missing ⚠️
crates/polars-plan/src/plans/options.rs	0.00%	1 Missing ⚠️
crates/polars-plan/src/plans/python/pyarrow.rs	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17939      +/-   ##
==========================================
- Coverage   80.35%   80.32%   -0.04%     
==========================================
  Files        1492     1494       +2     
  Lines      196330   196453     +123     
  Branches     2813     2817       +4     
==========================================
+ Hits       157764   157802      +38     
- Misses      38045    38131      +86     
+ Partials      521      520       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

Alright, major bump it is. 👍

[stevebuildboats]

will this eventually replace anonymous_scan? with a streaming support

[ldacey]

Would this be the approach for a native delta table reader as well? An IO plugin?

[hugopendlebury]

This sounds very interesting.

I have a project called gribtoarrow which uses c++ and pybind11 to read a grib file (World Meteorology Organisation) format (basically same a CDF file - any anything more of a standard than CDF).... Internally it uses an iterator and I currently use it with polars in a loop, do you have an example of using the IO Plugin ?
I'm stuck with an older version of polars at work, 0.19.17 or similar which has a bug writing Decimals to Parquet and I'm having to use from_arrow() and pyarrow.ParquetWriter to convert CDS Gribs to parquet (the are around 7 billion rows per file).
It would be nice to be able to read / write directly with polars use a lazy / batch based frame and not have mix and match so many tools.
Using my hybrid approach I've been able to extract 2.4 trillion rows of weather data from gribs which takes 1.5 days with 12 concurrent processes (1 per month). However I'm still a bit disappointed with the performance and a lot of this seems to be the fact I'm using Decimals and pyarrow is a bit slow with these. Would be interesting to see what would be possible with a pure polars approach.

[ritchie46]

Would this be the approach for a native delta table reader as well

Yes, this would work for that. I am looking for a good example source. It might be native delta.

will this eventually replace anonymous_scan? with a streaming support

For rust AnonymousScan would already work, we just need to adapt it to support streaming. Which we will.

I'm stuck with an older version of polars at work, 0.19.17

Why? :/

[hugopendlebury]

Would this be the approach for a native delta table reader as well

Yes, this would work for that. I am looking for a good example source. It might be native delta.

will this eventually replace anonymous_scan? with a streaming support

For rust AnonymousScan would already work, we just need to adapt it to support streaming. Which we will.

I'm stuck with an older version of polars at work, 0.19.17

Why? :/

On the version, long story and not in my control.

For a source have you considered GDAL https://gdal.org/index.html ? This enables a huge amount of files to be read, including most of the scientific formats such as HDF5, Grib and zarr. In fact GADL is used under the hood in the mosaic library to allow spark to read such formats.
There is a rust wrapper available, but you can also pip install a python wrapper. It would provide a lot of bang for your buck and provide access to a lot of file formats.