feat(rust,python): Add config to specify GPU polars as the default engine

[Matt711]

Apart of #19797. We should also follow up the PR to allow other engines to be the default.

[ion-elgreco]

Wouldn't it be more practical if you can set any engine then as default?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.63%. Comparing base (79eedcf) to head (e297dd0).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20717      +/-   ##
==========================================
- Coverage   79.64%   79.63%   -0.01%     
==========================================
  Files        1571     1567       -4     
  Lines      223186   223060     -126     
  Branches     2554     2554              
==========================================
- Hits       177751   177638     -113     
+ Misses      44850    44837      -13     
  Partials      585      585              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[orlp]

I think this isn't as simple as it seems, because we do also call Polars methods in the Python code internally which are only supported on, for example, the eager engine.

[wence-]

I think this isn't as simple as it seems, because we do also call Polars methods in the Python code internally which are only supported on, for example, the eager engine.

Can you expand on what you mean here? Effectively what this is trying to do is add a global config option for the default value in LazyFrame.collect(..., engine="cpu") (to make that literal "cpu" string optionally "gpu" instead). The Python-side logic that then perhaps nonetheless turns off the gpu engine still exists.

Why would hooking this up via a config option not work?

[orlp]

@wence- I have no idea what the full scope of such examples are throughout the codebase, but one example is pl.align_frames dispatching to other calls. Such a function might suddenly stop working if you change the default engine, because not all engines have the exact same supported feature set.

[wence-]

OK, thanks, I think I see. I think there's some amount of tension here between making this "easy" to configure globally and I guess some things breaking at a distance from where the user might expect.

[Matt711]

Ok tests are passing now. Gentle ping for reviews from @ritchie46 @orlp et al