expose mechanism for enabling/disabling certian optimization rules

[universalmind303]

Is your feature request related to a problem?

In some scenarios, some of our heuristic optimization may actually be more costly than naively running the plan as is. As seen here #3759

I'd like a way to enable/disable certain optimizations when executing daft.

Describe the solution you'd like

example usage

# not exactly sure what the most ergonomic api for this would be, but something like this maybe? 
daft.set_planning_config(disable_optimizer_rules=["MaterializeScans"])

Describe alternatives you've considered

No response

Additional Context

No response

Would you like to implement a fix?

No

[aditanase]

@universalmind303 thanks for reporting. Quick input on the API and format, we can probably take inspiration from the existing engines like pyspark and datafusion:

• https://spark.apache.org/docs/latest/configuration.html#spark-properties
• https://github.com/apache/datafusion-python?tab=readme-ov-file#configuration
  • https://datafusion.apache.org/user-guide/configs.html

Turning specific rules on and off might be too granular, you may want to consider higher level knobs.

[universalmind303]

@aditanase thanks for the additional context & resources! We'll take a look and see what's the best way we could expose similar settings.