Keep the interval endpoints, gate width and power, and derive the last hand-picked tolerance - #3
Conversation
Coverage cannot see a vacuous interval. `assert_coverage` is satisfied by one so wide it always covers, because a rate of 1.0 sits inside the binomial band at any study smaller than about sixty replicates, and `coverage > 0.9` written by hand is satisfied by it always. Three consuming repositories hit this independently and each left a comment where a test should have been; one of them after an inflation heuristic drove a reported standard error to 3e7 times the estimation error while coverage stayed high. All three are reproduced as tests here. The runner computed the endpoints, used them once and dropped them, so the evidence was being thrown away. It now keeps them, and `MonteCarloResult` gained widths. `assert_intervals_informative` fails only when the width exceeds what the study can still see fail *and* the study never saw it fail. The conjunction is the point: a t interval at n=5 is 1.33 times the oracle width and correct, so width alone cannot separate conservatism from vacuity. The reference width is measured by the study rather than chosen; what remains a convention is written down in `vacuous_width_ratio`'s docstring rather than presented as a derivation. `assert_power` and `assert_more_powerful` fill the hole under the package's own claim to answer whether a test has power. Power is a floor, so the first is one-sided; the second replaces `a.rejection_rate > b.rejection_rate`, which is satisfied by a gap of one replicate in four hundred. `assert_se_calibrated`'s tolerance was 0.15, the one number here chosen by hand. The sampling distribution of the ratio gives it: three sigma is 0.21 at 100 replicates and 0.05 at 2000, so the fixed value was tight enough to fail correct estimators in a fast tier and loose enough to certify a 12% error in a deep one. It is now derived by default. geoinference is the only consumer relying on the default and its suite still passes: 43 passed, 4 subtests passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All three were reproduced before being fixed and watched failing after the fix was reverted. **A width comparison could be won on one replicate.** `_mean_gap_se` returned zero when either study had fewer than two replicates, and a zero standard error reads as certainty, so `assert_narrower` certified whichever method the single draw happened to favour. It now refuses the comparison. **The plug-in Wald standard error is exactly zero at a rejection rate of 0 or 1.** One replicate rejecting against one not rejecting was a gap of 1.0 with no uncertainty, so `assert_more_powerful` certified a three-sigma difference from two observations. Agresti-Caffo gives that case 2.6 sigma and fails it, while total separation over four hundred replicates is still hundreds of sigma. **The derived SE tolerance assumed normality.** The relative noise in a sample standard deviation is `sqrt((kappa-1)/(4n))`, not `1/sqrt(2n)`, so a calibrated estimator with Student t(5) sampling error was flagged in 19 of 200 studies against the 0.3% three sigma advertises -- the "gate that fires on correct code" failure this package exists to prevent. The kurtosis is now estimated from the study and floored at 3, since the sample kurtosis is downward-biased at Monte Carlo replicate counts and must not be allowed to narrow the band. 2 of 200 now. geoinference, the only consumer relying on the default tolerance, still passes: 43 passed, 4 subtests passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7743eb4682
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "reject/accept decisions, so the two cannot be compared" | ||
| ) | ||
| strong, weak = more.rejection_rate, less.rejection_rate | ||
| gap = strong - weak |
There was a problem hiding this comment.
When the studies have unequal replicate counts near a boundary, this combines the raw rate gap with an adjusted variance, which can certify an unresolved difference. For example, 1/1 rejections versus 0/100 gives a raw gap of 1 and passes this three-sigma gate at about 3.67 SE, whereas applying the stated add-one-success/add-one-failure adjustment to both rates gives a gap of about 0.657, only 2.41 SE. Compute the gap from the adjusted rates as well as using their adjusted variance.
Useful? React with 👍 / 👎.
| reported = np.asarray(result.standard_errors, dtype=float) | ||
| mean_reported = float(np.mean(reported)) | ||
| variation = ( | ||
| float(np.std(reported, ddof=1)) / mean_reported if mean_reported else 0.0 | ||
| ) |
There was a problem hiding this comment.
When a study uses Estimate's default missing standard error, mean_reported and variation become NaN and this public helper returns NaN rather than raising. The README exposes this helper for callers implementing their own checks, and a check written in the same form as the gate (if deviation > tolerance: raise) then silently passes because comparisons with NaN are false. Validate that the reported standard errors produce a finite usable mean before deriving the tolerance.
Useful? React with 👍 / 👎.
Both reproduced before the fix and watched failing after it was reverted. **Agresti-Caffo was applied to the variance but not to the gap**, which is not Agresti-Caffo and leaves the hole open whenever the two studies are different sizes: one replicate rejecting against a hundred not rejecting is a raw gap of 1.0, and it cleared three sigma against an adjusted standard error. Both halves now come from the adjusted rates, which puts that case at 2.4 sigma and the one-against-one case at 0.9, while total separation over four hundred replicates is unchanged at hundreds of sigma. **`se_ratio_tolerance` returned NaN for a study that reported no standard error.** The helper is public and the README points callers at it, so the obvious hand-written check -- `if abs(ratio - 1) > tolerance: raise` -- compared against NaN, got False, and passed silently on a study that measured nothing. It raises now. The gate itself was never affected: it rejects a non-finite ratio first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Update: two further P1/P2 defects from the hosted Codex review, both fixedFetched from both endpoints ( [P1] Agresti-Caffo was applied to the variance but not to the gap ( [P2] Both fixes watched failing with the check reverted, passing with it in place. After: 84 passed (was 82), |
The changelog entry was written before midnight rolled over. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
Coverage cannot see a vacuous interval, the package had no gate for the power it
advertises, and its own
tolerance=0.15was the one number here chosen by hand.This closes all three, and prepares 0.1.0.
1. Interval endpoints, widths, and a vacuity gate
monte_carlocomputed the endpoints, used them once to buildcovered, anddropped them. They are now kept:
MonteCarloResultgainedlowers,uppers,widths,mean_widthandmedian_width. Asking for a width on a study thatrecorded no endpoints raises and says why, exactly as
coveragealready did.assert_intervals_informative(result, nominal, label)fails a vacuous interval.How the threshold is derived. The reference width is a measurement, not a
choice: for an estimator with spread
sigma, the shortest interval covering at1 - alphahas width2 * z * sigma, and the study measuressigmaassampling_sd.width_ratio(result, nominal)reports the observed mean width inunits of it, so no absolute width appears anywhere.
vacuous_width_ratio(nominal, reps)is the width multiple at which a study of that size stops being able toobserve a miss: widening a calibrated interval by
rdrops its miss rate to2(1 - Phi(r z)), and the threshold is therat which the whole study expectsfewer than
alphamisses. It comes out at 1.78 at 100 replicates, 1.96 at 400,2.15 at 2000 — rising with
reps, which inverts the usual direction and ismeant to: more replicates resolve rarer failures.
What is not derived, stated plainly in the docstring rather than hidden. How
many expected misses per study counts as "could not have failed" is a
convention. No sampling distribution fixes it, because correct procedures occupy
the whole range of widths above the oracle — a t interval at n=5 is 1.33 times
it, an anytime-valid interval more. So the gate does not fail on width alone. It
fails on width and a study that never once saw the interval miss (fewer than
sigmasmisses; rule of three). A procedure that fails at its nominal rate isnot vacuous however wide it is, and no threshold on width can know that.
assert_narrower(narrow, wide)is the efficiency comparison sortition wants,banded by the Monte Carlo standard error of the difference in mean width.
2. Power
assert_power(result, minimum)— one-sided, floor frombinomial_bandaroundthe claim. Power is a floor, and
assert_proportion's two-sided band fails atest for being better than promised.
assert_more_powerful(more, less)— bands the gap between two rejection rates.It replaces
a.rejection_rate > b.rejection_rate, which passes on a gap of onereplicate in four hundred.
3. The hard-coded 0.15 — derived, and the default changed
The default changed from
0.15to a value derived fromreps. Flaggingthis explicitly, as asked, because it changes behaviour for consumers.
se_ratiodivides a mean ofrepsreported standard errors by a sample standarddeviation of
repsestimates. Both are noisy at a rate the study knows:cv / sqrt(reps),cvthe coefficient of variation of the reportedstandard errors — zero for an estimator that reports the same one every time;
sqrt((kappa - 1) / (4 * reps)),kappathe estimator'skurtosis, measured from the study and floored at 3.
In quadrature at three sigma: 0.21 at 100 replicates, 0.11 at 400, 0.05 at
2000 for a normal estimator. The fixed 0.15 was wrong in both directions at
once — loose enough to certify a 12% error in a 2000-replicate study, tight
enough to fail correct estimators in a shallow one.
tolerance=still overridesit;
se_ratio_tolerance(result)returns the band.Blast radius, measured. Of the six consuming repos only
geoinferenceusesthe default (
test_inference.py:410); the rest passtolerance=or do not callthe gate. Its suite was run against this branch unmodified —
PYTHONPATHpointedat this checkout, no consumer file touched — and reports 43 passed, 4 subtests
passed.
Verification the ticket asked for
The gate reproduces all three real-world cases, in
test_the_three_workarounds_in_the_wild_are_caught:test_inference.py:311— "so wide it covers every single time"test_econometrics.py:18— "covering 100% of the time"test_bayesian_calibration.py:133— "ratio to 3e7 while coverage stayed high"And it stays silent on the case that separates conservatism from vacuity: a real
t interval at n=5, 1.33 times the oracle width, coverage 0.95 — passes, because
the study watches it miss.
Negative tests, watched failing
Every new gate has one, and each was watched fail with its check removed (a
pytest plugin that no-ops the gate) and pass with it restored. Failures observed
per neutered check:
assert_intervals_informative→ 4 failed (incl. all three real cases)assert_narrower→ 3 failedassert_power→ 4 failedassert_more_powerful→ 3 failedMonteCarloResult.__post_init__→ 4 failed(
test_the_same_se_error_passes_a_shallow_study_and_fails_a_deep_one)With the checks in place: 82 passed.
The
-Osubprocess test now covers all nine gates, so none of them can be turnedinto a no-op by
python -O.Codex review (release gate 3) — three P1s, all fixed
codex exec review --base mainfound three real defects. Each was reproducedfirst, then fixed, then watched failing again with the fix reverted:
_mean_gap_sereturnedzero for a study with fewer than two replicates, and zero uncertainty reads as
certainty. Reproduced:
assert_narrowerpassed on one replicate each. Now aValueError.Reproduced: one replicate rejecting vs one not rejecting passed a three-sigma
claim from two observations. Now Agresti-Caffo — that case is 2.6 sigma and
fails, while 400/400 against 0/400 still passes.
Var(s)/sigma^2is(kappa-1)/(4n), not1/(2n). Reproduced: a calibrated estimator witht(5) sampling error was flagged in 19 of 200 studies, against the 0.3%
three sigma advertises. Now 2 of 200.
Release preparation (0.1.0)
CHANGELOG.md[Unreleased]→## [0.1.0] - 2026-08-08.release.ymltriggers ontags: ["v*"], soit must be
v0.1.0.README.md'spip install simcheckis not yet true — a git-install line isadded alongside it until the first PyPI release lands.
v0.1.0tag push the build and GitHub-release jobs succeed and thepublishjob fails, because no PyPI trusted publisher exists for
finite-sample/simcheckyet. That failure is accepted. It is recoverable without a version bump: a
failed publish uploads nothing, and
workflow_dispatchagainst the tag withpublish: truere-builds the same version and completes the upload once thepublisher is registered.
Checks
uv run ruff check .clean ·ruff format --check .13 files formatted ·pyright0 errors ·pydoclint srcno violations ·pytest82 passed ·python -O -m pytest82 passed · doctests 4 passed.No consuming repo was modified.