Don't handle threshold and threshold_ratio in core.classifier.choose_classifier() #172
Labels
Milestone
Comments
riley-harper
changed the title
riley-harper
added a commit
that referenced
this issue
Dec 5, 2024
The output type of choose_classifier() is really hard to write down precisely because of the way PySpark types are set up. It's something like tuple["Classifier", "Transformer"], but for some reason SQLTransformer is not a subtype of Transformer.
riley-harper
added a commit
that referenced
this issue
Dec 5, 2024
The caller is responsible for passing a dictionary of hyper-parameters to choose_classifier(), and this dictionary should not include hlink's threshold or threshold_ratio. Both of the places where we call choose_classifier() (training and model exploration) already handle this.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Some of the code in
choose_classifier()explicitly excludes thethresholdandthreshold_ratiokeys from theparamsdict. This is because these attributes are stored alongside the parameters in the config file. Butchoose_classifier()should not be responsible for handling that. It should just accept a dictionary of hyper-parameters and pass that along to the correct ML model class.Each time we call
choose_classifier(), we have already extractedthresholdandthreshold_ratiofrom the hyper-parameters dict. So this change should just require removing some code fromchoose_classifier()and maybe updating some documentation. This is technically a breaking change, so we should add it in with v4.The text was updated successfully, but these errors were encountered: