Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addition of Hydra's Helper Functions #371

Merged

Conversation

allyfranken
Copy link
Contributor

The Hydra Team regularly uses datacompy for dataframe comparison. We have added helper functions to the main codebase to allow for additional formatting, control over the compared dataframes, and to capture edge case comparisons.

The major updates in this PR include:

  • The addition of detailed_compare. When using detailed_compare, if column_to_join is empty, a new column is added to each dataframe called "row_number". Dataframes are then compared based on the row_number instead of a user-provided column.
  • Two additional dataframe format helper functions are added: sort_columns and format_numeric_fields
  • Added test cases for helper functions

@CLAassistant
Copy link

CLAassistant commented Jan 15, 2025

CLA assistant check
All committers have signed the CLA.

Copy link
Member

@fdosani fdosani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some initial thoughts. I'm going to push some changes to your branch to discuss a bit further.

datacompy/spark/sql.py Outdated Show resolved Hide resolved
datacompy/spark/sql.py Outdated Show resolved Hide resolved
datacompy/spark/sql.py Outdated Show resolved Hide resolved
datacompy/spark/helper.py Outdated Show resolved Hide resolved
datacompy/spark/helper.py Outdated Show resolved Hide resolved
datacompy/spark/helper.py Outdated Show resolved Hide resolved
@allyfranken allyfranken force-pushed the RCAODF-12933-add-helper-functions branch 2 times, most recently from c51636f to 3eaf760 Compare January 22, 2025 15:30
@allyfranken allyfranken force-pushed the RCAODF-12933-add-helper-functions branch from 3eaf760 to 264fffd Compare January 22, 2025 15:42
@fdosani fdosani self-requested a review January 22, 2025 19:24
Copy link
Member

@fdosani fdosani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fdosani fdosani merged commit afa683f into capitalone:develop Jan 22, 2025
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants