Data Validation Framework: Source + Product data

[sf-dcp]

Partially resolves #650

What

This PR implements a raw architecture of data checks with Pandera in dcpy.

In order to run data checks on a dataframe with Pandera, you need to construct a pandera object, pandera.DataFrameSchema. The Pandera object takes inpandera.Columns which consist of pandera.Checks. Most of the PR is about creating these pandera objects in validate/pandera_utils.py module. The rest of the PR is creating a format for data checks in our templates.

Example of running data checks from Pandera docs:

import pandas as pd
import pandera as pa

# data to validate
df = pd.DataFrame({
    "column1": [1, 4, 0, 10, 9],
    "column2": [-1.3, -1.4, -2.9, -10.1, -20.4],
    "column3": ["value_1", "value_2", "value_3", "value_2", "value_1"],
})

# define schema
schema = pa.DataFrameSchema({
    "column1": pa.Column(int, checks=pa.Check.le(10)),
    "column2": pa.Column(float, checks=pa.Check.lt(-1.2)),
    ]),
})

validated_df = schema(df)
print(validated_df)

Example of data check format in ingest templates/metadata files:

columns:
  - id: bbl
    data_type: text
    is_required: True
    checks:
      - str_contains:
          args:
            pattern: "75"
          warn_only: False

  - id: custom_value
    data_type: integer
    is_required: False
    checks:
      - greater_than:
          description: This column must be greater than zero
          args:
            min_value: 0
      - unique_values_eq:
          description: This column must contain at least these values.
          warn_only: True
          args:
            values: [5, 10]

Check names in a template correspond to pandera check names or our custom checks (same approach as in preprocessing steps). In the last commit, I created a custom check as an example how we would do it; the check itself can be scraped later.

I recommend to review this PR commit by commit or starting with the validate.pandera_utils.run_data_checks() fn and work your way backwards.

🚨 Feedback needed

• Naming of models/functions
• validate.pandera_utils.create_check() fn: I have some validation logic in it where it checks correct input values (i.e. such check name or check parameters exist). I would like to move it under dataset.CheckAttributes model -- does it make sense to do that?
• Better test structure?
• How to appease mypy in the meantime, before pandera checks are fully implemented.

Next steps/PRs

• Implement data types in pandera (map our custom defined data types in templates to pandera data types and custom defined data types like bbl)
  • Ex: text -> pandera.dtypes.DataType.String
• Implement custom data checks.
• Implement validate.run() function which reads in a local file into df, pulls in a template, and runs data checks.
• Refactor ingest/distribution code to use new checks data type. Refactor distribution metadata files.

References

• Discussion: Syntax for data validation framework #902
• Alex's draft PR: Take Pandera for a Spin #859
• Finn's draft PR: Implement Pandera for ingest validation #862

[codecov]

Codecov Report

Attention: Patch coverage is 84.41558% with 12 lines in your changes missing coverage. Please review.

Project coverage is 70.74%. Comparing base (76fbe85) to head (4227bdf).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
dcpy/lifecycle/validate/data.py	0.00%	3 Missing ⚠️
dcpy/lifecycle/validate/pandera_utils.py	92.68%	1 Missing and 2 partials ⚠️
dcpy/lifecycle/package/validate.py	0.00%	1 Missing and 1 partial ⚠️
dcpy/lifecycle/validate/pandera_custom_checks.py	71.42%	2 Missing ⚠️
dcpy/models/dataset.py	90.47%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1241      +/-   ##
==========================================
+ Coverage   70.58%   70.74%   +0.16%     
==========================================
  Files         115      119       +4     
  Lines        5966     6041      +75     
  Branches      695      706      +11     
==========================================
+ Hits         4211     4274      +63     
- Misses       1609     1617       +8     
- Partials      146      150       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fvankrieken]

A couple small notes/thoughts but

1. this all seems great to me, big picture-wise
2. you and Alex seem to be in a good spot (and agreement) about some of the details, so I'll continue to peek and give any wanted feedback, but carry on!

[damonmcc]

don't wanna miss out on the approval party! great stuff!

[sf-dcp]

@fvankrieken @damonmcc @alexrichey Should I merge with unhappy mypy?

[fvankrieken]

@fvankrieken @damonmcc @alexrichey Should I merge with unhappy mypy?

No, those should be resolved in some way, particularly the couple failures outside of validate.

You also have a pytest failure - looks like import issues due to renaming