Skip to content

Speed up the enabled matching-capacity census 4.92x with exact traversal - #5340

Merged
Mister-Meeseeks merged 1 commit into
CrocSwap:mainfrom
aliromman:codex/five-pr-04
Oct 23, 2026
Merged

Mister-Meeseeks merged 1 commit into
CrocSwap:mainfrom
aliromman:codex/five-pr-04

Conversation

@aliromman

Copy link
Copy Markdown
Contributor

This algorithmic improvement replaces repeated cube/prefix scans in PR #5033's matching-capacity predicate with exact incremental state. In seven serialized alternating runs on the complete n=6, L=14, split (7,7) root shard, enabled median runtime falls from 200.42 ms to 40.73 ms (4.92x) with identical search reports. Capacity disabled still takes 20.68 ms, so the option remains off by default. This improves an existing optional verifier; it does not change a pruning theorem, exclusion, supplier or the complete kappa calculation. Selected kappa remains 60403/20000000.

Count invariant and scope, complete source delta, deterministic receipt, timing samples, and source pins are at reviewed head 08bba0dcd71e79f0d517740792c9b5a7ca9cecb8. Base and all consumed input pins: ff3cc58e370d0f678e529246f8c73e8094e18fcf.

For each omitted coordinate and each functional class, the engine stores visited vertices a, used stay edges e, and matching pairs with both endpoints visited b. The unused matching-pair count is exactly 2^(n-2)-a+b; unused vertices are 2^(n-1)-a. The two endpoint mates give one-vertex capacity. A child copies its parent's depth row and updates these counts on a fresh insertion. Siblings overwrite only the child row; each shard resets the initial state. The original completion inequality, simple-path gate, closing edge, search order, output contract and cap behavior remain intact. The new fixed state uses at most 3,696 bytes on the tested four-byte-int platform.

Both enabled engines visit exactly 495,919 nodes, retain 89,782 restored leaves and make 2,922 capacity prunes. Disabled capacity visits 506,239 nodes with the same leaves. There are no universal hits in this timing domain. Samples use the same compiler, uninstrumented native programs and complete no-budget root shard; compile time is excluded and process startup is included. These are local timings, not a production seven-label prediction or a CI performance threshold.

From the repository root, Python 3.12, standard library, and C11 compiler with unsigned __int128 and undefined-behavior/bounds sanitizer support:

python3 -I -W error -B research/incremental-prefix-capacity/verify.py
python3 -I -O -W error -B research/incremental-prefix-capacity/verify.py
python3 -B scripts/research_package.py --package research/incremental-prefix-capacity
python3 -I -B research/incremental-prefix-capacity/verify.py --timings

Expected direct marker: PASS_INCREMENTAL_PREFIX_CAPACITY. Normal and optimized CPython 3.12.13, plus optimized 3.14.7, pass. Twenty complete configurations compare all non-timing report fields and full restored/hit Counters at (2,4), (3,6), (4,8), (4,10), (5,12), with budgets and capacity independently on/off. Separate controls cover repeat-capable bypass, multiple shards and a 17-node cap. An independent split checker compares 1,988 prefix/functional cases; the sanitizer build agrees. Four source mutants fail: lost occupied pairs, wrong mate, lost parent state and lost final endpoint. Committed-byte checks and the official 13-file export pass; the Git-free optimized extraction passes and rejects source and receipt corruption. Full-history preflight is not claimed from this shallow checkout.

The preserved predicate is the theorem from PR #5001, with its pinned proof. That theorem and the predecessor traversal remain explicit premises. This PR proves the new count invariant and checks implementation equivalence. No full seven-label census, selected cold replay, Lean/Docker run, native supplier admission, independent human or proof-kernel acceptance is claimed. Hosted CI and external review remain separate.

The advance claim takes the predecessor's first open question. Final overlap covered 381 open PRs, 4,566 Discussion records and all 7,091 tracked README/TASK notes; the source handoff and this claim were the relevant matches. Canonical main still matched the base. Unpublished work and arbitrary diffs remain outside coverage.

Credit @JoyboyBrian for the maintained integration, @aliromman for the earlier completion theorem, @petalas for matching balance and the union census, Nicholas Stone for the parity correction, Mihir Srivastava for the handoff, Ross Richardson for the prefix model, and Aaron Wang/opencode and all retained source notices. Prepared by @aliromman with substantial OpenAI Codex assistance in proof, code, verification and review; Apache-2.0.

@jwr404

jwr404 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Independent review (James Reed, with Claude Code). Head 08bba0dcd71e79f0d517740792c9b5a7ca9cecb8.

Verdict: the incremental state is an exact replacement for the old scan on every path where it is used, and a differential run beyond your domain agrees byte for byte.

Code check against DELTA.diff

  • Count identity. Per side there are 2^(N−1) vertices and 2^(N−2) matching pairs. The mate v ^ (1<<omitted) has the same side, because mask drops the omitted bit. With a visited and b fully visited pairs, touched pairs are (a − 2b) + b, so untouched pairs are 2^(N−2) − a + b.
  • Redundant exclusion. The old mate ∉ {0, q} exclusion was redundant for pairs, since 0 and q are always seen. The old cheap scan counts unseen mates of 0 and q, which is exactly the new endpoint loop.
  • crossing = depth − stays[0] − stays[1] and the per-side stays attribution (side of the new point) match the old loop.
  • The two edge cases are both unreachable while capacity is active:
    • q = 0 double count. The endpoint loop {0, q} would count the mate of 0 twice when q = 0, where the old scan counted it once. But capacity_splits only runs at i > 0, and q can be 0 there only after a non-fresh step.
    • Stale state. extend_capacity runs only on fresh insertions, so a non-fresh step would leave capacity[i+1] stale from a sibling.
    • Why both are unreachable. With capacity_active (requires S = T = L/2), every position 0..L−1 must be a fresh vertex. rem_even[i]/rem_odd[i] count positions i+2..L−1 by parity, so any non-fresh non-final step leaves one parity class one short and hits the continue on line 160 before rec.
    • Suggestion. A one-line comment (or an assert under the sanitizer build) stating this invariant next to extend_capacity would protect it if the activation condition is ever relaxed.

Independent differential run

I compiled the pinned predecessor research/f7-prefix-capacity-engine/census.c and this PR's census.c with identical flags. I ran complete root shards with --prefix-capacity, with and without budgets, and compared the full JSON reports (timing fields removed) and the hits files.

N, L, split budgets old / new time reports hits
5, 12, (6,6) on / off 5 / 3 ms equal equal (828)
5, 14, (7,7) on / off 45 / 28 ms equal equal (24,787)
6, 14, (7,7) on 36 / 11 ms equal equal (0)
6, 14, (7,7) off 106 / 28 ms equal equal (0)
6, 16, (8,8) on 1,862 / 439 ms equal equal (6,744)
6, 16, (8,8) off 2,375 / 622 ms equal equal (6,744)

The (5,14) and (6,16) cases have many universal hits and 11.3M nodes, which your timing domain lacked, so they also exercise the hit path. The speedup (3–4×) is consistent with your 4.92×.

No objections. This review covers implementation equivalence only, not the #5001 capacity theorem, which remains a pinned premise as you state.

@Mister-Meeseeks
Mister-Meeseeks merged commit 3b785d5 into CrocSwap:main Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants