-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
All MAD predictions should be positive. #50
Comments
Hi @JulianHidalgo ! Thanks for opening this issue. I could indeed reproduce the error when using xgboost models with Actually, I've noticed that this behavior happens when the number of estimators mu_model = xgb.XGBRegressor()
sigma_model = xgb.XGBRegressor(n_estimators=5)
# Wrap models in a mean/variance predictor
mean_var_predictor = MeanVarPredictor(
models=[mu_model, sigma_model]
)
cp = LocallyAdaptiveCP(mean_var_predictor)
cp.fit(X_fit=X_train, y_fit=y_train, X_calib=X_test, y_calib=y_test) Can you see if that works for you ? PS: we will look into a suitable solution to "correct" models that predict negative values. We could simply force the absolute value of |
Thanks for checking this out! Reducing the number of estimator helps, but it also decreases the accuracy of the model and it's not reliable: the same number of estimators works fine with a dataset and fails with another. I noticed LightGBM also generates negative values some times, but less often than XGBoost. At least now I know it's not something in the way I'm using the library or my datasets in particular. I will be watching the issue, thank you again! |
Hello @JulianHidalgo, thanks again for using PUNCC and for rising this issue! After having discussed it with the team, we have decided to take the following steps to fix this issue:
We hope this fixes your issue for you. We expect that negative values of sigma are rare and that our procedure does not have a big impact in the size of the prediction sets. Of course, the probabilistic guarantees given by conformal prediction will remain true after this modification. # |
Hey @jdalch! |
Hey @JulianHidalgo, @jdalch has implemented his solution to address the problem. Could you please test it and let us know if it works? |
Hi!
Thank you for creating Puncc. I'm trying to use LocallyAdaptiveCP as described here https://deel-ai.github.io/puncc/regression.html#deel.puncc.regression.LocallyAdaptiveCP
But I get an error: All MAD predictions should be positive. Any idea of what am I missing?
I think the error comes from
puncc/deel/puncc/api/nonconformity_scores.py
Line 248 in 6e0a8f8
But I don't know how to avoid it. Any pointers would be greatly appreciated!
The text was updated successfully, but these errors were encountered: