Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

warnings about probabilities from weighted classifiers and calibration for simulation #56

Open
jason-bentley opened this issue Sep 10, 2020 · 0 comments
Assignees
Labels
API New feature or request

Comments

@jason-bentley
Copy link
Contributor

Is your feature request related to a problem? Please describe.
When sample weights are applied to a learner classifier the up weighting on one class will be reflected in a higher predicted probability than what was observed in the unweighted data.

Describe the solution you'd like
There are two aspects to an ideal solution:

  1. when ever using weights with a classifier and performing simulation a warning should be displayed noting as such and that uncalibrated probabilities may not align with observed rates for the positive class.
  2. post training calibration could be added as a toggled option for the ClassifierPipelineDF class, where the default is set to true but can be turned off if desired. This could help ensure that naively even with weights applied to learning the probabilities shown in the simulation outputs align reasonably well with those observed in the data.

Describe alternatives you've considered
None.

Additional context
None.

@jason-bentley jason-bentley added the API New feature or request label Sep 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants