Skip to content

feat(calibration): add CalibratedClassifier wrapper with online Platt scaling - #1988

Open
Shizoqua wants to merge 1 commit into
online-ml:mainfrom
Shizoqua:feat/1421-platt-calibration
Open

feat(calibration): add CalibratedClassifier wrapper with online Platt scaling#1988
Shizoqua wants to merge 1 commit into
online-ml:mainfrom
Shizoqua:feat/1421-platt-calibration

Conversation

@Shizoqua

Copy link
Copy Markdown

Closes #1421.

Adds calibration.CalibratedClassifier, a wrapper that recalibrates a binary classifier's probability estimates using online Platt scaling.

What it does

  • Fits a logistic function σ(a·s + b) on top of the wrapped model's score s.
  • The score is read out-of-sample: predict_proba_one is called on the wrapped model before it learns on the current sample, which prevents the calibration from overfitting to the same points it scores.
  • a and b are updated online with one stochastic gradient step per sample, minimizing log-loss. They start at a=1, b=0 (the identity), so the wrapper is a no-op until data flows in.
  • Because the sigmoid is monotonic, the predicted label never changes — only the probabilities are spread/pulled.
  • predict_proba_one always returns a distribution keyed on False and True.

Implementation notes

  • Mirrors the approach of scikit-learn's _SigmoidCalibration (Platt, 1999) but in the online setting, using river.optim-style gradient steps.
  • Uses utils.math.sigmoid for numerical stability and a clamped logit for extreme probabilities.
  • Passes the wrapper/classifier conventions: base.Wrapper, base.Classifier, _unit_test_params provided.

Tests

  • tests/calibration/test_calibrated_classifier.py:
    • identity mapping initially (a=1, b=0),
    • predict_proba_one returns a valid distribution,
    • calibration never changes the predicted label,
    • log-loss improves over the raw model on Phishing,
    • parameters move away from the identity after learning.

Notes

This PR was produced with the assistance of an AI coding agent (opencode), in line with the project's coding-agent rules. The commit carries the author's Signed-off-by.

… scaling

Adds calibration.CalibratedClassifier, a wrapper that recalibrates a
binary classifier's probability estimates via Platt scaling. A logistic
function sigma(a*s+b) is fit on top of the wrapped model's score, where
the score is read out-of-sample (before the wrapped model learns on the
current sample). The two parameters a and b are updated online, one
stochastic gradient step per sample, minimizing the log-loss; they start
from the identity (a=1, b=0) so the wrapper is a no-op until data flows.

The output of predict_proba_one is always keyed on False and True, and
the sigmoid is monotonic, so the predicted label never changes.

Signed-off-by: Lanre Shittu <136805224+Shizoqua@users.noreply.github.com>
@codspeed-hq

codspeed-hq Bot commented Aug 13, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 98 untouched benchmarks
🆕 1 new benchmark
⏩ 16 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Simulation test_calibrated_classifier_learn N/A 27.4 ms N/A

Comparing Shizoqua:feat/1421-platt-calibration (2b12a47) with main (ab09239)

Open in CodSpeed

Footnotes

  1. 16 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Platt’s scaling for probability calibration

1 participant