Feature: Customizable column names and extra config placeholder #127
Conversation
|
@hrfmartins thank you for the contribution. Really appreciated. I have a few requests. Can you please sign your commits? We require all commits to be signed with GPG key. Please also run We currently have an issue with running integration tests triggered from forks. Your PR may be blocked at the moment. |
hrfmartins
force-pushed
the
feature/customizable-column-names
branch
7 times, most recently
from
dbc3dd2 to
a335d2a
Compare
|
@mwojtyczka Signing with GPG done and lint + fmt ran and issues fixed :) Sorry for the inconvenience. Is there anything I can/need to do about the fork issue? Thank you |
Thank you! We are working on fixing the fork issue. Will keep you posted. |
…r other future configurations
hrfmartins
force-pushed
the
feature/customizable-column-names
branch
from
ed300d9 to
a09b0fe
Compare
There was a problem hiding this comment.
@hrfmartins
Can you please extend the existing guide to show how to customize the reporting columns. Perhaps a new section, sth like "Additional configuration" before the custom checks:
https://github.com/databrickslabs/dqx/blob/main/docs/dqx/docs/guide.mdx#quality-rules-and-creation-of-custom-checks
Can you please also extend demos, probably a new cell here:
https://github.com/databrickslabs/dqx/blob/main/demos/dqx_demo_library.py#L286
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
…custom column mappings against apply_checks_by_metadata_and_split
…s' into feature/customizable-column-names
|
@mwojtyczka Just added documentation and a new entry on the notebook of the demos. |
| dq_engine = DQEngine(ws, extra_params=ExtraParams(column_names={'errors': 'ERROR', 'warnings': 'WARN'})) | ||
| test_df = spark.createDataFrame([[1, 3, 3], [2, None, 4], [None, 4, None], [None, None, None]], SCHEMA) | ||
|
|
||
| checks = [{"criticality": "warn", "check": {"function": "col_test_check_func", "arguments": {"col_name": "a"}}}] |
There was a problem hiding this comment.
Let's make this test more focused. Please use regular check not a custom one. Currently col_test_check_func is used which is a custome check.
|
|
||
|
|
||
| checks = [ | ||
| {"criticality": "warn", "check": {"function": "col_test_check_func", "arguments": {"col_name": "a"}}}, |
There was a problem hiding this comment.
Please use one of standard checks, not a custom check.
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Marcin Wojtyczka <[email protected]>
Thank you! |
In this PR I implemented a placeholder for extra configurations for DQEngine. I also included customizable column names to replace the custom names.
Changes
Linked issues
Resolves #46
Tests