Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

carlg/alternative-metrics #950

Open
wants to merge 29 commits into
base: main
Choose a base branch
from

Conversation

carl-offerfit
Copy link

  • Adds an optional scoring argument to _OrthoLearner.score so that the score can be viewed according to several other sklearn metrics. Currently supported are: mean_absolute_error, mean_squared_error, r2_score.
  • The original default behavior of using MSE for the model fitting is preserved, now calculated in sklearn mean_square_error function instead of written out in code.
  • Adds a new public method score_nuisances to _OrthoLearner.score so that the first stage models can be evaluated by other sklearn metrics. If sample weights are not used, any metric supported by sklearn get_scoring can be used. If sample weights are used, f1_score, log_loss, roc_auc_score are supported for binary outcomes/treatments; and for real valued outcome treatments it will be the same as _OrthoLearner.score for the final model mean_absolute_error, mean_squared_error, r2_score.

carl-offerfit and others added 29 commits January 27, 2025 12:14
Fix names of scoring methods in wrappers (don't end in "score")
Add Log Loss as a weighted option
Fixes calling the ModelFinal.wrap_scoring function
Add pearsonr as a score and test
…nds on the _rlearner _ModelNuisance implementation of score
…scoring is used without an _rlearner._ModelFinal

Test that logic in the _otho_learner test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant