Skip to content

fix(stattests): reference quantile binning for numeric drift (#1962) - #1987

Open
Qoder12 wants to merge 1 commit into
evidentlyai:mainfrom
Qoder12:fix/1962-reference-quantile-bins
Open

Qoder12 wants to merge 1 commit into
evidentlyai:mainfrom
Qoder12:fix/1962-reference-quantile-bins

Conversation

@Qoder12

@Qoder12 Qoder12 commented Nov 11, 2026

Copy link
Copy Markdown

Summary

  • Bin numeric PSI/KL/Jensen-Shannon inputs using reference quantiles (with open-ended outer bins) instead of equal-width bins over reference and current combined.
  • Align Hellinger distance on numeric features with the same reference-based binning so a single current outlier cannot collapse drift scores.

Fixes #1962.

Root cause

get_binned_data built histogram edges with np.histogram_bin_edges on the concatenation of reference and current samples. One extreme current value widened the range so almost all mass landed in the first bin, making shifted and outlier-augmented distributions look alike (PSI dropped from ~1.0 to ~0.02). Hellinger used a similar union-based equal-width scheme on unique values.

Technical solution

  • Add reference_quantile_bin_edges() and use it in get_binned_data for numerical features with more than 20 unique reference values.
  • Use the same helper for numeric Hellinger histograms, computing Hellinger distance on normalized bin proportions.

Test plan

…yai#1962)

Equal-width bins over reference and current let one outlier hide drift in
PSI, KL, Jensen-Shannon, and Hellinger. Bin on reference quantiles instead.

Signed-off-by: Qoder12 <175736904+Qoder12@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

psi, kl_div, jensenshannon and hellinger miss a clear drift when current has one outlier (equal width bins over both samples)

1 participant