Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release v0.12.0 #297

Merged
merged 6 commits into from
May 1, 2024
Merged

Release v0.12.0 #297

merged 6 commits into from
May 1, 2024

Conversation

fdosani
Copy link
Member

@fdosani fdosani commented Apr 30, 2024

Release v0.12.0

This release has some breaking changes users should be aware of.

  • For the new Pandas on Spark implementation only Pandas 1.5.3 and below will be supported due to Pandas 2.0 compatibility issues with Spark 3.
    • Users wishing to use the older SparkCompare can continue to do so using LegacySparkCompare
  • Spark 3.1.x support will be dropped
  • Python 3.8 support is dropped

Breaking changes

(#275)

  • Pandas on Spark refactor and deprecation of the legacy SparkCompare
  • Drop Python 3.8 and Spark 3.1.x support

Updates

fdosani and others added 5 commits March 25, 2024 14:56
* refactor SparkCompare

* tweaking SparkCompare and adding back Legacy

* conditional import

* cleaning up tests and using pytest-spark for legacy

* adding docs

* caching and some typo fixes

* adding in doc and pandas 2 changes

* adding pandas to testing matrix

* drop 3.8

* drop 3.8

* refactoring ^

* rebase fix for #277

* fixing legacy uncode column names

* unicode fix for legacy

* unicode test for new spark logic

* typo fix

* changes from PR review
* Fix all_mismatch report attribute to use defined df*_name by users

* Add contributor credit

* adding in additional changes for df names

* adding in additional changes for df names

---------

Co-authored-by: enzorooo <[email protected]>
* adding in count_matching_rows

* linting / cleanup
@fdosani fdosani merged commit 794d55b into main May 1, 2024
55 checks passed
rhaffar pushed a commit to rhaffar/datacompy that referenced this pull request Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants