Repository navigation
repo compare: an interval that agrees with the sign-flip p #292
Description
Problem
repo compare takes a row's 95% interval from one method and its p from another. Gate and score rows use a percentile bootstrap, the other rows a t interval, and p always comes from a sign-flip test (repos/compare.py docstring and paired_summary). Near 0.05 the two disagree, and a reader can't tell which to believe. Two rows from a replay of IntelPython/dpnp#3099 (26 review_replay tasks):
| row | delta | interval | p (sign-flip) |
|---|---|---|---|
metric:n_findings, 26 tasks |
+0.89 | t: [−0.091, +1.860] (includes 0) | 0.0497 (t-test 0.074) |
| score, 6 tasks | +0.139 | bootstrap: [+0.028, +0.250] (excludes 0) | 0.25 |
In the second row, three non-zero deltas, all positive, can't give an exact sign-flip p below 0.25, while a bootstrap over six tasks is too narrow. A separate change adds a note line wherever the two disagree. This issue is about removing the cause.
Proposal
Take every row's interval from the sign-flip test itself, by inverting it.
- The interval: every shift θ for which the test on
d − θdoes not reject at 0.05. Find each end by bisection between the mean and a far bound. - Exactness: exact up to
EXACT_MAX(16) non-zero deltas, as the p is. Above that, Monte Carlo with the fixed seed and common random numbers across θ, so the ends are stable. - The result: "interval excludes 0" holds exactly when p < 0.05 (exactly in the exact range, up to Monte Carlo error above it), on every row.
- Few non-zero deltas: the interval is wide, or open on one side, which is the honest answer.
This is a change of method, so it goes with a research-register decision record and with §9 of the contract.
Open questions
- Discreteness: the exact test's p jumps as θ crosses a delta, so the interval ends sit at data points. Is that acceptable as is, or should a mid-p version be used?
- Speed: 2^16 sign patterns × ~30 bisection steps × 2 ends × rows. Fine in NumPy? Measure it.
- Comparability: keep the bootstrap behind a flag, so a comparison can be redone the old way?
- Other tracks: should the arms track (
eval/stats.py) follow too?
Done when
repo compare computes every row's interval by inversion. Tests pin the interval on known delta sets, including the two rows above, and check that it agrees with p. §9 and the module docstring describe it, and a research record holds the decision.
Activity
antonwolfy commentedon Nov 1, 2026
Proposed answers to the open questions, plus two points the issue missed.
p_holm, and on those rowsinterval_p_disagreereadsp_holm(repos/compare.py). Inverting the test at 0.05 gives an interval that agrees with the raw p. In a family of 2+ rows, the disagreement can therefore still occur wherever Holm lifts p across 0.05. Holm's step-down has no simple compatible interval (the 0.05/m level is compatible with Bonferroni, not with Holm). Proposal: invert at the uncorrected 0.05, label the interval as uncorrected, and keep the note only for that case, reworded to "the interval is uncorrected;p_holmdecides this row".mdeandp_holm_floordescribe p, so they stay as they are.null) for 5 or fewer tasks. The text prints an open end as such. The existing floor note keeps describing p at θ = 0, where only the changed tasks count.eval/stats.py, which feat(stats): keep the unrounded delta, CI and corrected p next to the rounded ones #345 is editing) should follow in its own issue once this one settles the method. The decision record covers both.Order. Do this after #332 and #336 merge, and after the compare flags of #290: all three edit
repos/compare.py, and this change rewrites the core ofpaired_summary. The research-register decision record can be written now.