Probability Calibration

A small Python toolkit for calibrating and evaluating binary classifier probabilities.

The package focuses on a deliberately narrow problem: a classifier can discriminate well while still producing probabilities that are too confident, too conservative, or otherwise poorly calibrated. This project provides three calibration approaches, reliability diagnostics, and a scikit-learn-compatible holdout-calibration wrapper.

Features

• Isotonic regression calibration
• Platt-style sigmoid calibration over probability log-odds
• Temperature scaling with bounded scalar optimization
• ECE, MCE, Brier score, and reliability-bin data
• Uniform and quantile binning
• Held-out calibration for scikit-learn binary classifiers
• Held-out comparison of calibration methods against an uncalibrated baseline
• Scikit-learn-compatible estimator behavior for pipelines, cross-validation, and grid search
• Optional sample weights during estimator and calibrator fitting

Example

from sklearn.linear_model import LogisticRegression
from probability_calibration import ProbabilityCalibratedClassifier

model = ProbabilityCalibratedClassifier(
    LogisticRegression(max_iter=1000),
    method="temperature",
    calibration_fraction=0.2,
)

model.fit(X_train, y_train)
probabilities = model.predict_proba(X_test)

Standalone probability arrays can also be calibrated directly:

from probability_calibration import TemperatureCalibrator

calibrator = TemperatureCalibrator()
calibrator.fit(calibration_probabilities, calibration_labels)

calibrated = calibrator.transform(test_probabilities)

Diagnostics are returned as structured data rather than printed or plotted:

from probability_calibration import compute_calibration_diagnostics

diagnostics = compute_calibration_diagnostics(
    calibrated,
    y_test,
    n_bins=10,
    strategy="quantile",
)

print(diagnostics.ece)
print(diagnostics.brier_score)
print(diagnostics.bin_edges)

Calibration methods can also be compared on a held-out evaluation split:

from probability_calibration import compare_calibration_methods

results = compare_calibration_methods(
    probabilities,
    labels,
    methods=("isotonic", "platt", "temperature"),
)

print(results["uncalibrated"].brier_score)
print(results["temperature"].brier_score)

Binary scope

Version 0.1 intentionally supports binary classification only. Standalone calibrators accept one-dimensional positive-class probabilities and labels encoded as 0/1. ProbabilityCalibratedClassifier accepts arbitrary binary class labels and preserves the wrapped estimator's classes_ ordering.

Multiclass calibration requires different semantics for one-vs-rest calibration, probability renormalization, temperature scaling, and diagnostics, so it is left outside the initial API rather than partially supported.

Calibration methods

Isotonic

Fits a non-parametric monotonic mapping from uncalibrated to calibrated probabilities. It is flexible, but can overfit small calibration sets.

Platt-style sigmoid

Fits a sigmoid mapping after transforming input probabilities to log-odds. The standalone API works from probabilities, so it cannot recover an estimator's original decision margins; the log-odds transform avoids fitting logistic regression directly to bounded probability values.

Temperature

Transforms probabilities to logits, optimizes one positive temperature against negative log-likelihood, and maps the scaled logits back through the sigmoid. The implementation uses bounded scalar optimization rather than a fixed temperature grid.

Numerical stability

Platt and temperature calibration convert probabilities to log-odds internally. Exact 0 and 1 inputs are therefore clipped to [eps, 1 - eps] before the logarithm for numerical stability. The clip_output option controls only whether returned calibrated probabilities are clipped to that same range.

Scikit-learn wrapper

ProbabilityCalibratedClassifier is a binary meta-estimator. With the default prefit=False, it clones the supplied estimator, fits it on one subset, and fits the calibrator on a disjoint holdout subset.

prefit=True leaves an already-fitted estimator untouched and uses the supplied data only for calibration.

The wrapper deliberately uses one holdout split rather than reproducing the cross-validated behavior of CalibratedClassifierCV.

Development

python -m pip install -e ".[dev]"
pytest

License

MIT