[KYUUBI #5738][FEATURE] Add a spark commit protocol to support compact small file before commit job

[AngersZhuuuu]

🔍 Description

Issue References 🔗

This pull request fixes #5738

Describe Your Solution 🔧

Add a spark commit protocol to support compact small file before commit job.
Support committer algorithm version:

• v1
• v2 (TODO later)

Support format:

• ORC
• PARQUET
• other

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

---

Checklists

📝 Author Self Checklist

• My code follows the style guidelines of this project
• I have performed a self-review
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• This patch was not authored or co-authored using Generative Tooling

📝 Committer Pre-Merge Checklist

• Pull request title is okay.
• No license issues.
• Milestone correctly set?
• Test coverage is ok
• Assignees are selected.
• Minimum number of approvals
• No changes are requested

Be nice. Be informative.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.48%. Comparing base (8480605) to head (71748d8).
Report is 208 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #5758      +/-   ##
============================================
+ Coverage     61.41%   63.48%   +2.07%     
  Complexity       23       23              
============================================
  Files           607      624      +17     
  Lines         35882    41960    +6078     
  Branches       4920     5977    +1057     
============================================
+ Hits          22036    26639    +4603     
- Misses        11468    12792    +1324     
- Partials       2378     2529     +151     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Thanks for the PR! This PR is being closed due to inactivity. This isn't a judgement on the merit of the PR in any way. If this is still an issue with the latest version of Kyuubi, please reopen it and ask a committer to remove the Stale tag!

Thank you for using Kyuubi!

[gabrywu]

if you are looking for a solution to merge small files, please refer to this issue #6691