Add narwhals engine as dataframe compatibility layer to support multiple engine at once

[dkapitan]

Current status: feature set differs per engine

Pandera supports different engines, including pandas, polars, modin, Dask and pyspark. Feature completeness differs between these engines, most notably support for pydantic v2 (see #1874 and beda-software/FHIRPathMappingLanguage#18).

Narwhals aims to address this issue by providing a lightweight and extensible compatibility layer between dataframe libraries. It has gained traction and Python projects, as for example Altair, have recently removed the dependency from pandas and integrated Narwhals instead.

Desired situation: more complete feature set across engines

While this issue was triggers by #1874, I see a larger benefit by implementing Narhwals to support multiple dataframe libraries at once. Provided that the Narwhals API provides the functionality required by pandera (I have done a quick check and haven't spotted a blocking issue yet; no guarantees though, since I am quite new to pandera).

Alternatie situation: implement PydanticModel for polars

pandas_engine.PydanticModel into polars_engine in case Narwhals doesn't support the functionality needed by pandera.

[cosmicBboy]

Thanks for creating this @dkapitan !

I see several steps for this to become mainstream:

• Implement narwhals backend, equivalent to the polars backend
• Register this backend against the pandas and polars pandera API objects
• Incrementally get feature-parity with the pandera's pandas backend implementation
• (eventually) swap out the default backends for pandas, polars (and beyond) with the narwhals backend

Narwhals still needs the corresponding libraries to be installed to work right? i.e. If I want to use pandas with narwhals, I still need pandas installed.

[dkapitan]

@cosmicBboy

Makes sense to implement feature parity with polars through narwals first and then take it from there.

Indeed you need to install the engine if you want to work with it, but the pandera library itself can remove the dependency on polars.