Keep the interval endpoints, gate width and power, and derive the last hand-picked tolerance

CHANGELOG.md

@@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.0] - 2026-08-09
+
 ### Added
 
 - Initial extraction from `incline/tests/_statistics.py`, generalised so it
@@ -30,16 +32,89 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Tier helpers `reps_for`, `deep_tier`, `FAST_REPS` and `DEEP_REPS`, driven by
   `SIMCHECK_DEEP` and `SIMCHECK_REPS`.
 
+- **Interval endpoints are kept, not just whether each interval covered.**
+  `MonteCarloResult` gained `lowers`, `uppers`, `widths`, `mean_width` and
+  `median_width`; `monte_carlo` passes the endpoints through instead of computing
+  them, using them once and dropping them. Supplying one endpoint without the
+  other, an upper endpoint below its lower one, or a `covered` array that
+  contradicts the endpoints are all rejected; supplying endpoints without
+  `covered` fills `covered` in, since the endpoints and the truth determine it.
+
+- `assert_intervals_informative`, which fails an interval so wide that the study
+  never saw it miss. `assert_coverage` cannot: a coverage rate of 1.0 sits inside
+  the binomial band at any study smaller than about 60 replicates, and a
+  hand-written `coverage > 0.9` is satisfied by an interval that always covers.
+  Three consuming repositories hit this independently and left a comment where a
+  test should have been — one of them after an inflation heuristic drove a
+  reported standard error to 3e7 times the estimation error while coverage stayed
+  high. The gate fails only when the width exceeds `vacuous_width_ratio(nominal,
+  reps)` *and* the study observed fewer than three misses, so a correct but
+  conservative procedure — a t interval at n=5 is 1.33 times the oracle width —
+  is not flagged.
+
+- `vacuous_width_ratio` and `width_ratio`, the threshold and the measured
+  quantity behind that gate. The reference width, `2 * z * sampling_sd`, is
+  measured by the study, so no absolute width appears anywhere. How many expected
+  misses per study counts as "could not have failed" is a convention rather than
+  a derivation, and `vacuous_width_ratio`'s docstring says so and says why: correct
+  procedures occupy the whole range of widths above the oracle, so no sampling
+  distribution separates conservatism from vacuity on width alone.
+
+- `assert_narrower`, for the efficiency half of an interval comparison, banded by
+  the Monte Carlo standard error of the difference in mean width. A study of one
+  replicate is refused rather than treated as certain.
+
+- `assert_power` and `assert_more_powerful`. The package documented power as one
+  of the four questions it answers and had no gate for it; consumers were
+  reaching for `assert_proportion`, which is two-sided and needs a nominal you
+  already know analytically, or hand-rolling a two-sample standard error.
+  `assert_power` is one-sided, because power is a floor and a two-sided band
+  fails a test for being better than claimed. `assert_more_powerful` compares
+  Agresti-Caffo adjusted rates rather than raw ones: the plug-in Wald standard
+  error is exactly zero at a rejection rate of 0 or 1, so one replicate rejecting
+  against one not rejecting would otherwise be a three-sigma finding from two
+  observations.
+
 - **Negative tests for every gate.** Each is exercised on input that satisfies
   its property, where it must stay silent, and on input that violates it, where
   it must raise. Plus gates run against estimators whose behaviour is known
   analytically: the sample mean trips nothing, the `ddof=0` variance is caught
   with its textbook `-sigma^2/n` bias, a 1.96 interval at n=5 is caught
-  under-covering at about 0.875, and a false-positive check confirms correct
-  estimators are essentially never flagged.
+  under-covering at about 0.875, a two-sided z test shows the power its formula
+  gives (0.323 at n=25 and 0.851 at n=100 for delta=0.3), an interval built at a
+  known scale comes out at a width ratio of exactly one, and a false-positive
+  check confirms correct estimators are essentially never flagged.
+
+### Changed
+
+- **`assert_se_calibrated`'s tolerance now comes from the replicate count.** It
+  was `0.15`, the one number in the package chosen by hand rather than derived,
+  and it was wrong in both directions: `se_ratio` divides a mean of `reps`
+  reported standard errors by a sample standard deviation of `reps` estimates, so
+  its Monte Carlo spread is `sqrt(cv^2/reps + (kappa-1)/(4*reps))`, and three of
+  those is 0.21 at 100 replicates and 0.05 at 2000 for a normal estimator. The fixed value was therefore tight
+  enough to fail correct estimators in a fast tier and loose enough to certify a
+  12% error in a deep one. Passing `tolerance=` explicitly still overrides it,
+  and `se_ratio_tolerance(result)` returns the derived band.
+
+  This changes behaviour for callers that relied on the default. Of the six
+  consuming repositories only `geoinference` does, and its suite was re-run
+  against this branch: 43 passed, 4 subtests passed. Every other consumer passes
+  `tolerance=` explicitly or does not call the gate.
+
+  The band uses the estimator's own fourth moment, not a normal assumption:
+  `Var(s)/sigma^2 = (kappa-1)/(4*reps)`, with `kappa` estimated from the study and
+  floored at 3 so a downward-biased sample kurtosis cannot narrow it. Assuming
+  normality flagged a *calibrated* estimator with Student t(5) sampling error in
+  19 of 200 studies; with the fourth-moment term it is 2 of 200.
 
 ### Fixed
 
+- **`assert_se_calibrated` diagnosed a missing standard error as a constant
+  estimator.** `Estimate` documents that leaving `standard_error` as NaN means
+  the gate "will have nothing to check and will say so"; it said the estimator
+  did not vary across replicates, which is a different and false diagnosis.
+
 - **The extracted `assert_rate` could report the worst possible result as the
   best possible one.** It took either a count or a rate and guessed which:
   `observed = successes / reps if successes > 1 else float(successes)`. For a

README.md

@@ -9,6 +9,8 @@ under the null, and does it have power under an alternative.
 
 ```bash
 pip install simcheck
+# until the first PyPI release lands:
+pip install "simcheck @ git+https://github.com/finite-sample/simcheck@v0.1.0"
 ```
 
 Python 3.11+. Depends on numpy, and nothing else.
@@ -86,12 +88,74 @@ rate.
 |---|---|
 | `assert_unbiased` | the mean estimate is more than 3 Monte Carlo standard errors from the truth |
 | `assert_coverage` | interval coverage falls outside the binomial band around the nominal level |
+| `assert_intervals_informative` | the intervals are so wide the study never saw one miss, so their coverage measures the width |
+| `assert_narrower` | one method's intervals are not measurably narrower than another's |
 | `assert_se_calibrated` | the reported standard error misstates the spread actually observed |
-| `assert_proportion` | an observed **rate** — size, power — is inconsistent with the claimed one |
+| `assert_power` | a test rejects less often than claimed under an alternative (one-sided) |
+| `assert_more_powerful` | one test does not reject measurably more often than another at the same alternative |
+| `assert_proportion` | an observed **rate** — size, coverage — is inconsistent with the claimed one |
 | `assert_count_rate` | the same, given a **count** of successes |
 
-`binomial_band(nominal, reps)` gives the band directly if you want to assert
-something else against it.
+`binomial_band(nominal, reps)`, `vacuous_width_ratio(nominal, reps)` and
+`se_ratio_tolerance(result)` give the three thresholds directly, if you want to
+assert something else against them. `width_ratio(result, nominal)` is the
+measured quantity behind the first of those.
+
+## Coverage cannot see a vacuous interval
+
+`assert_coverage` is satisfied by an interval so wide it always covers, whenever
+the study is small enough that a rate of 1.0 still sits inside the binomial band
+— and `assert coverage > 0.9`, written by hand, is satisfied by it always.
+Three repositories hit this independently and each left a comment where a test
+should have been; one had shipped an inflation heuristic that drove the reported
+standard error to 3×10⁷ times the estimation error while coverage stayed high,
+because a vacuous interval covers everything.
+
+The fix is to keep the endpoints, not just the hit:
+
+```python
+result = monte_carlo(replicate, truth=2.0, reps=2000, seed=11)
+assert_coverage(result, 0.95, "t interval")
+assert_intervals_informative(result, 0.95, "t interval")
+```
+
+Two things must both be true before that fails, and the conjunction is the
+point. The interval must be far wider than the width its own level requires
+against the spread the estimator actually has — `2 * z * sampling_sd`, measured
+by the study, so no absolute width is written down anywhere. *And* the study must
+never have seen it miss. A Student t interval at n=5 is 1.33 times the normal
+oracle width and an anytime-valid interval is wider still; both are correct, both
+miss at their nominal rate, and the study watches them do it. Width alone cannot
+tell conservatism from vacuity. Width plus a study that never saw a failure can.
+
+`vacuous_width_ratio(0.95, reps)` is the width multiple at which a study of that
+size stops being able to observe a miss: 1.78 at 100 replicates, 1.96 at 400,
+2.15 at 2000. It rises with the replicate count, which inverts the usual
+direction and is meant to — more replicates resolve rarer failures.
+
+The one thing here that is a judgement rather than a derivation is how many
+expected misses per study counts as "could not have failed". Its docstring says
+so, and says why no sampling distribution fixes it: correct procedures occupy the
+whole range of widths above the oracle.
+
+## Power
+
+The package claims to answer whether a test has power under an alternative, so
+there is a gate for it:
+
+```python
+assert_power(result, 0.80, "score test at delta=0.3")
+assert_more_powerful(robust, naive, "robust against naive at the same alternative")
+```
+
+`assert_power` is one-sided, unlike `assert_proportion`: power is a floor, and a
+two-sided band would fail a test for being *better* than promised. Size, which is
+a target rather than a floor, still belongs in `assert_proportion`.
+
+`assert_more_powerful` bands the gap between two rejection rates by the standard
+error of the difference. The assertion it replaces — `a.rejection_rate >
+b.rejection_rate` — passes on a gap of one replicate in four hundred and reports
+whichever method the seed favoured as the winner.
 
 ## Two failure modes this is built to prevent
 
@@ -118,6 +182,24 @@ reported as the best possible one. That is why counts and rates are separate
 functions here, and why `assert_proportion` raises rather than guesses when it
 is handed something outside `[0, 1]`.
 
+## The rule applies to simcheck too
+
+`assert_se_calibrated` used to take `tolerance=0.15`, which was the one number in
+the package chosen by hand rather than derived — and it was wrong in both
+directions at once. `se_ratio` is `mean(reported se) / sd(estimates)`, and both
+halves are estimated from the same replicates, so it is noisy even when the
+estimator is perfect: the numerator's relative standard error is `cv/sqrt(reps)`
+and the denominator's is `sqrt((κ-1)/(4·reps))`, where κ is the estimator's
+kurtosis — measured, not assumed, because a normal assumption flags a correct
+heavy-tailed estimator. Added in quadrature and taken at three sigma, that is
+0.21 at 100 replicates and 0.05 at 2000 for a normal estimator. A fixed 0.15 was
+therefore tight enough to fail correct estimators in a fast tier and loose enough
+to certify a 12% error in a deep one.
+
+The tolerance is now derived from `reps` by default; passing a number still
+overrides it, which is worth doing when the claim really is about a fixed
+accuracy at a fixed sample size. `se_ratio_tolerance(result)` returns the band.
+
 ## Negative tests
 
 Every gate has one: an input that violates the property, and a check that the
@@ -129,9 +211,11 @@ itself as tested.
 The gates are also run against estimators whose behaviour is known analytically
 (`tests/test_against_known_statistics.py`): the sample mean must trip nothing,
 the uncorrected `ddof=0` variance must be caught with its textbook bias of
-`-σ²/n`, and a 1.96 interval at n=5 must be caught under-covering at ≈0.875.
-There is also a false-positive check, because a gate that fires on 5% of correct
-code gets disabled within a week.
+`-σ²/n`, a 1.96 interval at n=5 must be caught under-covering at ≈0.875, a
+two-sided z test must show the power its formula gives (0.323 at n=25, 0.851 at
+n=100 for δ=0.3), and an interval built at a *known* scale must come out at a
+width ratio of exactly one. There is also a false-positive check, because a gate
+that fires on 5% of correct code gets disabled within a week.
 
 ## Tiers
 

src/simcheck/__init__.py

@@ -26,6 +26,12 @@
 helper that silently passes everything is worse than no helper, because it
 converts an untested codebase into one that reports itself as tested.
 
+**A gate that a vacuous answer satisfies.** Coverage is satisfied by an interval
+so wide it always covers, which is why the endpoints are kept and
+:func:`~simcheck.assert_intervals_informative` exists; and one-sided power was
+being asserted by hand as ``a > b``, which is satisfied by a gap of one
+replicate, which is why :func:`~simcheck.assert_more_powerful` exists.
+
 Examples:
 >>> import numpy as np
 >>> from simcheck import MonteCarloResult, assert_coverage, assert_unbiased
@@ -51,10 +57,17 @@
     GATE_SIGMAS,
     assert_count_rate,
     assert_coverage,
+    assert_intervals_informative,
+    assert_more_powerful,
+    assert_narrower,
+    assert_power,
     assert_proportion,
     assert_se_calibrated,
     assert_unbiased,
     binomial_band,
+    se_ratio_tolerance,
+    vacuous_width_ratio,
+    width_ratio,
 )
 from .results import MonteCarloResult
 from .runner import Estimate, monte_carlo
@@ -74,11 +87,18 @@
     "__version__",
     "assert_count_rate",
     "assert_coverage",
+    "assert_intervals_informative",
+    "assert_more_powerful",
+    "assert_narrower",
+    "assert_power",
     "assert_proportion",
     "assert_se_calibrated",
     "assert_unbiased",
     "binomial_band",
     "deep_tier",
     "monte_carlo",
     "reps_for",
+    "se_ratio_tolerance",
+    "vacuous_width_ratio",
+    "width_ratio",
 ]