Repository navigation
Conversation
ptimizeroracle
left a comment
There was a problem hiding this comment.
checked against math.erfc over |z| <= 37: main's 2 * (1 - norm.cdf(|z|)) returns exactly 0.0 from |z| about 8.3 (z=9.5 gives 0 where the truth is 2.1e-21), your norm.sf form stays within 2.4e-13 relative everywhere, and inf and nan behave the same. decisions only differ for thresholds under about 1e-16 so the visible change is the reported drift_score. cramer_von_mises_stattest.py:190 has the same 1.0 - cdf shape if you want it here
(ai-assisted, checked by me)
|
Thanks for checking the tail values independently. The main observable benefit here is preserving the reported nonzero p-value; the drift decision changes only when the configured threshold falls between the rounded and corrected values. The Cramér–von Mises asymptotic path uses a custom limiting-distribution CDF whose implementation already documents reduced accuracy for large statistics. Replacing that subtraction needs a stable tail formulation and separate accuracy tests, so I am keeping this PR focused on the proportion z-test. The Cramér–von Mises tail calculation can be handled separately. The current head |
Problem
The proportion z-test computes right-tail probabilities by subtracting norm.cdf from 1. At large positive z statistics, the CDF rounds to 1 even though the tail probability is representable, so the result becomes zero. This loses the reported p-value and can change drift decisions at small custom thresholds. For two binary populations of 500 rows with proportions 0.4 and 0.7, the two-sided p-value is approximately 1.50427e-21; the current result is zero.
Change
Use norm.sf for the two-sided and greater alternatives. The mathematical test and threshold comparison remain the same, and the less alternative is unchanged.
Validation
This improves tail-probability accuracy. It does not imply a different decision at conventional thresholds such as 0.05, or prevent underflow for probabilities below the floating-point range. The full repository suite was not run.