Test that the calibrators calibrate

[soodoku]

The gap

StreamingIsotonicCalibrator, NearlyIsotonicCalibrator, IsotonicCalibrator, PlattScaling and TemperatureScaling exist to return calibrated probabilities. Nothing tested that they do.

tests/test_calibrators.py:38-96 asserts monotonicity, range in [0,1], shape and rank preservation. The identity function satisfies all four on already-monotone input, so a calibrator that returned p_raw unchanged passed the entire suite. The only statistical assertions in the repo were assert ece < 0.05 and assert ece > 0.5 in test_metrics.py — hard-coded thresholds on a binned estimator, the first computed from a single np.random.seed(42) draw.

The design

The truth is set by construction. True probabilities p ~ U(0.05, 0.95) are drawn, the reported probabilities are a deterministic strictly monotone distortion q = g(p), and outcomes are Bernoulli(p). Because g is a bijection, E[y | q] = g^-1(q) exactly, so the calibration error of any map is available in closed form:

CE(f) = sqrt( E_q [ (f(q) - g^-1(q))^2 ] )

No binning, no estimator bias, no threshold to choose. Two distortions: a pure temperature shift (logit q = 0.5 logit p, every method correctly specified) and a power law (q = p^0.6, temperature scaling misspecified — a study where every method is well specified cannot tell a method that works from one the fixture is flattering).

Both distortions are strictly monotone, which is precisely why the existing monotonicity and rank tests cannot see them.

expected_calibration_error is deliberately not the yardstick: the plugin estimator charges a perfectly calibrated forecaster for its own binning noise, and test_binned_ece_charges_a_perfect_forecaster_for_noise measures it — 0.0104 at 5 bins, 0.0144 at 20, 0.0481 at 200, on a forecaster whose true calibration error is exactly zero. The one binned statistic used here is a max-|Z| test whose null distribution is known; its size is measured at 0.0485 against a nominal 0.05 over 2000 replicates before anything is concluded from it.

What the studies measured

All claims HOLD.

distortion	raw CE	Streaming	Nearly	Isotonic	Platt	Temperature
slope	0.1067	0.0293	0.0286	0.0284	0.0081	0.0049
power	0.1430	0.0294	0.0268	0.0284	0.0121	0.1381
already calibrated	0.0000	0.0300	0.0267	0.0284	0.0080	0.0079

• Error reduction: every calibrator beats the input it was handed in every replicate, both distortions.
• Already-calibrated input is not destroyed: the damage is 0.0079–0.0300 (worst replicate 0.0460), against sd(p) = 0.2597 for a forecaster that has collapsed and the 0.1067 the distortion costs. Every calibrator clears both yardsticks in every replicate. Both yardsticks come from the DGP, not from a number picked by hand.
• The detector is honest: max-|Z| size on the oracle map 0.0485 vs nominal 0.05; on undistorted input, identical; on either distortion, rejection rate 1.000.

One measured limitation, reported rather than smoothed over

TemperatureScaling removes 3.4% of a power miscalibration (0.1430 → 0.1381) where Platt removes 91.5% and the isotonic methods 79–80%. It technically reduces the error in every replicate, so the reduction gate passes — but "it reduces calibration error" would badly misdescribe it. It has one parameter and p^gamma is not in its family. test_temperature_scaling_barely_helps_outside_its_family pins the gap relative to the weakest other calibrator so it cannot drift into a claim.

Negative controls

Every gate has a case that makes it fail. Two broken calibrators, both built to survive the assertions the suite already had:

• _ConstantCalibrator returns the base rate. It is monotone (weakly), in range, right-shaped, rank preserving, and calibrated in the large — its mean output matches the mean outcome to within 0.01, so any check on E[p] - E[y] passes it. CE 0.2598.
• _ShufflingCalibrator fits correctly then permutes the answers. Its output has the exact marginal distribution of a correct calibrator — same values, same histogram, same distinct levels — and none attached to the right input. CE 0.3684.

Both are required to be rejected by assert_proportion on the reduction gate and on the resolution gate, via pytest.raises. test_the_constant_calibrator_passes_a_calibration_in_the_large_check shows in one measurement why the conditional test is the one being run.

Tolerances

Every gate is a simcheck gate, so its tolerance comes from the replicate count. There is no assert ece < x anywhere in the new file. simcheck enters as a git reference in a test dependency-group, not a published extra — PyPI rejects direct URL references in project metadata.

CI was installing nothing

uv sync --dev resolves the dev group; this project only had a dev extra. Nothing was installed, so ruff, mypy and pytest were never present and all three jobs have been failing on main. They now sync --extra dev --group test. Turning them back on surfaced one pre-existing formatting drift in calibrators.py and two pre-existing no-any-return errors on np.interp; both fixed here. Actions bumped to checkout@v5 / setup-uv@v7.

Counts

uv run pytest tests/ -v: 58 passed (30 pre-existing, 28 new), 0 failed, 0 skipped.
uv run ruff check src tests: clean. uv run ruff format --check src tests: clean. uv run mypy src: clean.

🤖 Generated with Claude Code