Optional dependencies

[EtienneT]

We are polars instead of pandas for managing dataframes. Our production server has an older processor that the polars package doesn't support. We have to use polars-lts-cpu. Both package are imported the same way in code... import polars.

So when we deploy and install dependencies in production, I setuped my .toml like so, made two optional groups and polars and polars-lts-cpu are not listed in the main dependencies:

[project.optional-dependencies]
prodserver = ["polars-lts-cpu==0.20.6"]
devlibs = ["polars==0.20.6"]

When deploying in production I simply install the optional group prodserver. This way only polars-lts-cpu is installed and not polars.

Here is my problem though, I have one package in my dependencies called dagster-duckdb-polars and this one has a dependency on polars if I look at the lock file.

So now even though I don't specify polars in my [dependencies], it gets installed anyway because dagster-duckdb-polars has a dependency on it.

Any way for me to prevent that?

Thanks!

[franzhaas]

I think this is a bug in dagster-duckdb-polars.

Thus said, i really don't think that your solution is how it should be. @frostming any chance to get a

if "AVX" in cpu_features

conditional dependencies? Or is there already a trick i am not aware of?

[frostming]

Or is there already a trick i am not aware of?

Nope, PDM fully complies with PEP 508 and doesn't allow dynamic dependencies as such. You can however create separate lock files resolved from different dependency set.

https://pdm-project.org/en/latest/usage/lockfile/#specify-another-lock-file-to-use

[EtienneT]

Just to clarify, is what I am doing the best solution?

Thanks,

[franzhaas]

@frostming thanks! good for me

[noirbizarre]

Hello 👋🏼

I think this not something that can be fixed by pdm but on polars itself (cf. pola-rs/polars#12880).

This case is a complex one as Python PEP508 doesn't handle CPU features at all.

Anyway, I think it's worth the try: given polars-lts-cpu will install as polars, maybe try this:

[project]
dependencies = [
    "polars==0.20.6",
    "dagster-duckdb-polars",
]

[project.optional-dependencies]
prodserver = ["polars-lts-cpu==0.20.6"]

So when installing project[prodserver], polars-lts-cpu should just overwrite the polars package.

IMHO, the proper solution for polars would be:

• distribute a single polars metapackage with lts-cpu and u64-idx extras depending on polars-lts-cpu and polars-u64-idx
• have the build publish polars-lts-cpu and polars-u64-idx but ensure they do not overwrite the polar package
• make the polars package detect available supports and dynamically import the proper support.

It can be something basic like:

try:
   import polars_lts_cpu.feature as feature
except ImportError:
   import polars.feature as feature

Note

I believe it's even possible to use importlib.metadata and entrypoints to avoid try/except and be able to be more dynamic on support loading at runtime

This way, package depending on polars just need to add polars as dependency.
End user installations can specify polars, polars[lts-cpu] or polars[u64-idx], it will always be resolved as polars but will install the expected extra CPU support and properly resolve it at runtime.

Edit: I cross-posted my proposal in their issue. Who knows, maybe they'll use it