Repository navigation
Conversation
…yai#1962) Equal-width bins over reference and current let one outlier hide drift in PSI, KL, Jensen-Shannon, and Hellinger. Bin on reference quantiles instead. Signed-off-by: Qoder12 <175736904+Qoder12@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1962.
Root cause
get_binned_databuilt histogram edges withnp.histogram_bin_edgeson the concatenation of reference and current samples. One extreme current value widened the range so almost all mass landed in the first bin, making shifted and outlier-augmented distributions look alike (PSI dropped from ~1.0 to ~0.02). Hellinger used a similar union-based equal-width scheme on unique values.Technical solution
reference_quantile_bin_edges()and use it inget_binned_datafor numerical features with more than 20 unique reference values.Test plan
test_numeric_divergence_stattests_ignore_current_outliersintests/stattests/test_stattests.py(repro from psi, kl_div, jensenshannon and hellinger miss a clear drift when current has one outlier (equal width bins over both samples) #1962: N(0,1) vs N(1,1) with and without a 1e3 outlier).pytest tests/stattests/test_stattests.py -k outlier(CI)pytest tests/stattests/test_stattests.py::test_num_features(CI)