Skip to content

Preserve representable tail probabilities in the proportion z-test - #1993

Open
Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/z-test-tail-probabilities-20261112
Open

Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/z-test-tail-probabilities-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

The proportion z-test computes right-tail probabilities by subtracting norm.cdf from 1. At large positive z statistics, the CDF rounds to 1 even though the tail probability is representable, so the result becomes zero. This loses the reported p-value and can change drift decisions at small custom thresholds. For two binary populations of 500 rows with proportions 0.4 and 0.7, the two-sided p-value is approximately 1.50427e-21; the current result is zero.

Change

Use norm.sf for the two-sided and greater alternatives. The mathematical test and threshold comparison remain the same, and the less alternative is unchanged.

Validation

  • 18 new tests on the upstream base: 10 failed, 8 passed.
  • Fixed tests plus the two existing z-test cases: 20 passed on CPU.
  • Independent math.erfc references cover positive, negative, zero, and tail statistics for every alternative, with zero absolute tolerance so a zero p-value cannot pass as a tiny nonzero value.
  • Actual categorical Series cover both population orders and thresholds on either side of the expected p-value. A native ValueDrift Report verifies the reported result.
  • Ruff 0.3.7 lint/format and git diff --check pass.

This improves tail-probability accuracy. It does not imply a different decision at conventional thresholds such as 0.05, or prevent underflow for probabilities below the floating-point range. The full repository suite was not run.

@ptimizeroracle ptimizeroracle left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked against math.erfc over |z| <= 37: main's 2 * (1 - norm.cdf(|z|)) returns exactly 0.0 from |z| about 8.3 (z=9.5 gives 0 where the truth is 2.1e-21), your norm.sf form stays within 2.4e-13 relative everywhere, and inf and nan behave the same. decisions only differ for thresholds under about 1e-16 so the visible change is the reported drift_score. cramer_von_mises_stattest.py:190 has the same 1.0 - cdf shape if you want it here

(ai-assisted, checked by me)

@Afloat16

Afloat16 commented Nov 15, 2026 •

Copy link
Copy Markdown
Author

Thanks for checking the tail values independently. The main observable benefit here is preserving the reported nonzero p-value; the drift decision changes only when the configured threshold falls between the rounded and corrected values.

The Cramér–von Mises asymptotic path uses a custom limiting-distribution CDF whose implementation already documents reduced accuracy for large statistics. Replacing that subtraction needs a stable tail formulation and separate accuracy tests, so I am keeping this PR focused on the proportion z-test. The Cramér–von Mises tail calculation can be handled separately.

The current head ab2a23a41ae5d8de1c5436153f8bc67bd42bb1ea shows one workflow awaiting maintainer approval. Could a maintainer approve the required CI so the regression can also be checked upstream?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants