feat(calibration): add CalibratedClassifier wrapper with online Platt scaling - #1988
Open
Shizoqua wants to merge 1 commit into
Open
feat(calibration): add CalibratedClassifier wrapper with online Platt scaling#1988Shizoqua wants to merge 1 commit into
Shizoqua wants to merge 1 commit into
Conversation
… scaling Adds calibration.CalibratedClassifier, a wrapper that recalibrates a binary classifier's probability estimates via Platt scaling. A logistic function sigma(a*s+b) is fit on top of the wrapped model's score, where the score is read out-of-sample (before the wrapped model learns on the current sample). The two parameters a and b are updated online, one stochastic gradient step per sample, minimizing the log-loss; they start from the identity (a=1, b=0) so the wrapper is a no-op until data flows. The output of predict_proba_one is always keyed on False and True, and the sigmoid is monotonic, so the predicted label never changes. Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
Merging this PR will not alter performance
Performance Changes
Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1421.
Adds
calibration.CalibratedClassifier, a wrapper that recalibrates a binary classifier's probability estimates using online Platt scaling.What it does
σ(a·s + b)on top of the wrapped model's scores.predict_proba_oneis called on the wrapped model before it learns on the current sample, which prevents the calibration from overfitting to the same points it scores.aandbare updated online with one stochastic gradient step per sample, minimizing log-loss. They start ata=1, b=0(the identity), so the wrapper is a no-op until data flows in.predict_proba_onealways returns a distribution keyed onFalseandTrue.Implementation notes
_SigmoidCalibration(Platt, 1999) but in the online setting, usingriver.optim-style gradient steps.utils.math.sigmoidfor numerical stability and a clampedlogitfor extreme probabilities.base.Wrapper,base.Classifier,_unit_test_paramsprovided.Tests
tests/calibration/test_calibrated_classifier.py:a=1, b=0),predict_proba_onereturns a valid distribution,Phishing,Notes
This PR was produced with the assistance of an AI coding agent (opencode), in line with the project's coding-agent rules. The commit carries the author's
Signed-off-by.