chore: WIP prepare for upgrading to DataFusion 46

[andygrove]

Which issue does this PR close?

Closes #1414

Rationale for this change

Upgrade to latest DataFusion in to make sure that there are no regressions. We need to do this before DataFusion 46 is released.

What changes are included in this PR?

How are these changes tested?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.55%. Comparing base (f09f8af) to head (2042512).
Report is 55 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1423      +/-   ##
============================================
+ Coverage     56.12%   58.55%   +2.43%     
- Complexity      976     1015      +39     
============================================
  Files           119      122       +3     
  Lines         11743    12249     +506     
  Branches       2251     2304      +53     
============================================
+ Hits           6591     7173     +582     
+ Misses         4012     3919      -93     
- Partials       1140     1157      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andygrove]

There are some failures with hash functions.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 6204.0 failed 1 times, most recent failure: Lost task 4.0 in stage 6204.0 (TID 14921) (c8336c0abbf7 executor driver): org.apache.comet.CometNativeException: Function sha256 does not implement invoke but called