diff --git a/research/butterfly-word-frame-floor/README.md b/research/butterfly-word-frame-floor/README.md new file mode 100644 index 0000000000..1153915830 --- /dev/null +++ b/research/butterfly-word-frame-floor/README.md @@ -0,0 +1,150 @@ +# The frame floor of PR217's butterfly word + +**No new κ, no certificate change.** PR212 bounded the bit coarse saving of one *word* -- +#200's physical bit word, shared by #205, #206 and #207 -- over every admissible choice of +operation frames. PR217 builds a **different word**: 440 isolated four-input XOR +reassociations (880 graph nodes and scalar controls) on top of PR211's cascade frames and +PR200's construction. This note runs PR212's method unchanged on that word. + +## Result + +At 1,200 Lagrangian rounds, the round count of PR212's published run, on PR217's head +`b739fc226a54a4a507989aa1bef092d491287451`: + +| Ledger | PR200's word (PR212) | PR217's butterfly word (here) | +|---|---|---| +| own: floor on `C` | 2,804,210 | 2,800,700 | +| own: coarse saving < | 6.904e-4 | **6.913e-4** | +| own: κ < | 6.900e-4 | 6.908e-4 | +| completed banks: floor on `C` | 2,780,144 | 2,776,634 | +| completed banks: coarse saving < | 6.964e-4 | **6.973e-4** | +| completed banks: κ < | 6.959e-4 | 6.968e-4 | + +The like-for-like pair at 400 rounds moves together as well (own 6.909e-4 vs 6.918e-4, +banked 6.969e-4 vs 6.978e-4), so the comparison does not depend on the round count. + +The word's own numbers, recomputed here from its certificate: + +| quantity | PR200's word | PR217's butterfly word | +|---|---|---| +| certified coarse saving, own ledger | 6.7777395e-4 | 6.7938424e-4 | +| `D/C` of the certified layout, own | 6.7837287e-4 | 6.7998615e-4 | +| certified coarse saving, completed banks | 6.8366e-4 (PR207) | 6.8516596e-4 | +| `D/C` of the certified layout, banks | 6.8427e-4 (PR207) | 6.8578301e-4 | +| κ of the composition | 6.831904550365e-4 (PR207) | 6.84696826673891e-4 | + +**So the butterflies raised the ceiling and the achieved value by about the same amount, +and the room left is slightly smaller than before.** On the completed-bank ledger the +distance from the achieved `D/C` to the ceiling is +1.68% here against +1.77% on #200's +word; on the κ basis, +1.77% against +1.86%. Changing the DAG is not a way to enlarge the +frame slack -- it is a different word with its own, slightly higher, ceiling. PR212's own +scope statement already said that non-frame changes are unbounded by it; this note only +confirms that the bounded part still binds after the change. + +## Can PR216's admitted cuts be carried onto this word? + +PR216 (`research/coordinated-crossover-pr200-v2`, head `34abdc5`) is a third layout on +this same lineage: it keeps PR211's cascade frames, adds 220 further operation-frame +changes, and admits them natively (κ = 6.838643131376e-4). Because the ceiling above +bounds *every* frame layout of PR217's word, the natural next question -- combine the +two -- has a bounded answer, and `transfer_check.py` checks its structural side against +the two packages themselves: + +* **The two words are one operation-index space.** Both have 41,288 operations, and the + parts that fix the DAG are identical: 11,856 arcs, 25,900 order entries, 1,760 reuse + pairs, 3,960 gauges, 6,624 root roles, 17,688 phase-1 and 7,728 plain operations, and + the same 1,760 sources and reads. Exactly **880 operation contents** differ -- PR217's + reassociation set -- and 7,068 operation frames differ. +* **PR216's admitted frames are keyed by that same index.** Its records are + `[op_index, [basis vectors …]]` pairs: 6,191 current bases, 6,258 override records and + 2,268 canonical subspaces, all with indices inside the shared list. Only **3 of the + 6,258** override records land on a rewritten operation, and **none of the 2,268** + genuinely different subspaces does. +* **So a merge is well defined at the index level, and that is the whole of what is + free.** PR216 admits its cuts with "scalar operations, source/target contracts, …" + fixed, and PR217's change is exactly in those operations: a merged layout is a new + submission that needs fresh global admission and a fresh column replay, not a + transplant of an existing certificate. +* **And the prize is capped.** Any frame layout of this word gives κ < 6.968e-4, i.e. at + most **+1.77%** over the 6.84696826673891e-4 record, while PR216's own layout on the + plain word sits **0.122% below** that record -- the merge would have to recover the + butterflies' whole contribution before it recovers anything. For scale, the bank + absorption of #219 claims +2.30% over the same record with an accounting ledger, which + is the route this comparison actually favours. + +## How the word was obtained and checked + +1. PR217's own constructor, `research/butterfly-coordinated-bit-211/candidate.py`, was + called with `--bit-root` at PR202's head `8d8d67bcf69c5ea67d3a29dbc64ba588156d6e8d` + (fetched from `pull/202/head`). It rebuilds the changed word from PR200's frozen word, + PR211's pinned frame payload (`opframe-bases.json.gz.b64`) and the frozen butterfly + overlay, checking each of those hashes itself. +2. The three emitted payloads reproduce PR217's own declared hashes exactly: + `graph_p12.json` `2bb271df49780365…`, `word_p12.json` `ef77a7d30390a21c…`, + `frames_p12.json` `4e0fce06f07a044e…`, and the word rebuilt here has 41,288 + operations, 18,908 roles, 1,760 reuse pairs, 3,960 gauges and **11,671 changed + operation frames**, the number its `expected-bit.json` declares. That match is what + ties this note's word to PR217's. +3. The tree handed to `ceiling.py --tree` carries PR217's emitted `graph_p12.json`, + `word_p12.json` and `frames_p12.json`; PR202's unchanged `kchron_p12.json`, + `sinks.json`, `profile_p12.json` and vendored PR168-v4 checker; and a certificate + shim whose `bit` record is PR217's own post-terminal profile plus the pre-sink profile + restored by the terminal record (`{3, 21} x -102`, i.e. +34 children of width 3 and + +34 of width 21 and +34 roles, giving `W = 20,668` and rank mass 1,486,160). +4. `ceiling.py` accepted the tree and passed every self-test on this word: the certified + layout lies inside every derived joint frame bound, **its slot chains reproduce + PR217's certified ledger child for child** (`W 20,668`, rank 1,486,160) and the + terminal substitution is exactly `[3, 21]` per sink and stage, with every per-slot + floor below the actual chain cost. Fact 1 reproduces the certificate to eight digits + (`6.7938424e-4 < D/C = 6.7998615e-4`, gap 0.089%). + +## Reproduce + +`rebuild_word.py` rebuilds the emitted payloads and checks them against PR217's +`expected-bit.json`; `build_tree.py` assembles the `--tree` input; then run PR212's +script unmodified: + +```sh +python3 rebuild_word.py --package /research/butterfly-coordinated-bit-211 \ + --bit-root --out +python3 build_tree.py --emit --bit-root \ + --package /research/butterfly-coordinated-bit-211 \ + --tree +python3 /research/bit-frame-ceiling-200/ceiling.py --tree --rounds 1200 +``` + +`transfer_check.py` compares PR217's emitted word with PR216's, from the same emit +directory: + +```sh +python3 transfer_check.py --emit \ + --pr216-package /research/coordinated-crossover-pr200-v2 +``` + +## Scope and limits + +* This bounds **frame layouts of PR217's word only**, and only in the sense PR212's floor + has: joint per-operation frame bounds, an exact per-slot dynamic programme and + Lagrangian coupling between the two registers of an operation, with the certified + layout as its own control. +* The butterflies themselves are part of the word. Nothing here bounds further + reassociation, other circuits, other reuse, gauges or sinks -- the escapes PR212 lists. +* The pre-sink profile in the shim is reconstructed, not read from PR217's own receipt: + the delta is exact arithmetic and `ceiling.py`'s child-for-child self-test is the + check, but a run of PR217's own `run_bit.py` would settle it directly. That run needs + Linux (`resource` limits and `signal.alarm`) and the pinned numpy 2.3.5 / scipy 1.17.0; + this machine has numpy 2.4.4 / scipy 1.18.1 and Windows, so the emission was driven + through the package's own `candidate.py` instead, which is the stage that produces the + word. +* All κ remain conditional on the interfaces each submission states. + +## Credits + +PR212 (Chafik Boukhalfa) supplies the method and the script, unmodified; PR217 (hcg890) +supplies the word, its overlay and `candidate.py`; PR211 (Rohan Arun) supplies the pinned +cascade frames; PR200/PR168 v4 (Chafik Boukhalfa, eumemic, icekylinx) supply the base +word and the vendored checker; PR216 (Dugongue) supplies the cut package whose admitted +frames are compared above -- that comparison reads PR216's own witness files, its native +admission was not run here. This note adds only the run, the shim, the comparison and the +reading. +Prepared with Anthropic Claude assistance; Apache-2.0. diff --git a/research/butterfly-word-frame-floor/build_tree.py b/research/butterfly-word-frame-floor/build_tree.py new file mode 100644 index 0000000000..f54543ce13 --- /dev/null +++ b/research/butterfly-word-frame-floor/build_tree.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +"""Assemble a PR212-shaped `--tree` input for PR217's changed bit word. + +The tree is a faithful shim: the word, frames and graph are PR217's emitted payloads +(their hashes were checked against the package's `expected-bit.json` by +`rebuild_word.py`), the partner chronology, sinks, base profile and the vendored +PR168-v4 checker come from the pinned PR202 checkout, and `certificate.json['bit']` +carries PR217's own post-terminal profile plus the pre-sink profile restored by the +terminal record (`{3, 21} x -102`, exactly as PR200's certificate records it, i.e. +34 +children of width 3 and +34 of width 21 and +34 roles). + +Every field of that shim is re-derived by `ceiling.py`'s own self-tests: it rebuilds the +children multiset from the word and requires it to equal the shim's pre-sink histogram, +requires the sink record to be one width-3 and one width-21 child per sink and stage, and +requires `2v + roles - pairs` and the rank mass to match. A wrong shim fails loudly +instead of quietly lowering the floor. + + python3 build_tree.py --emit --bit-root \ + --package /research/butterfly-coordinated-bit-211 \ + --tree +""" +import argparse +import gzip +import json +import shutil +import sys +from pathlib import Path + +PKG_REL = Path('research/paired-cube-diagonal-bit-168') + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--emit', type=Path, required=True, help='directory written by rebuild_word.py') + ap.add_argument('--bit-root', type=Path, required=True, help='the #202 checkout used for the rebuild') + ap.add_argument('--package', type=Path, required=True, help='the butterfly package directory') + ap.add_argument('--tree', type=Path, required=True, help='fresh tree directory to create') + args = ap.parse_args() + sys.set_int_max_str_digits(0) + pr202 = (args.bit_root.resolve() / PKG_REL) + package = args.package.resolve() + tree = args.tree.resolve() / PKG_REL + bit = tree / 'selected' / 'bit' + if tree.exists(): + shutil.rmtree(tree) + bit.mkdir(parents=True) + shutil.copytree(pr202 / 'references', tree / 'references') + for name in ('kchron_p12.json', 'sinks.json', 'profile_p12.json'): + shutil.copy2(pr202 / 'selected' / 'bit' / name, bit / name) + for name in ('graph_p12.json', 'word_p12.json', 'frames_p12.json'): + payload = (args.emit.resolve() / name).read_bytes() + (bit / name).write_bytes(payload) + (bit / (name + '.gz')).write_bytes(gzip.compress(payload, compresslevel=9, mtime=0)) + post = json.loads((package / 'expected-bit.json').read_text())['profile'] + pre = dict(post) + pre['child_histogram'] = dict(post['child_histogram']) + pre['child_histogram']['3'] = pre['child_histogram'].get('3', 0) + 102 + pre['child_histogram']['21'] = pre['child_histogram'].get('21', 0) + 102 + pre['rank_per_vertex'] = sum(int(r) * n for r, n in pre['child_histogram'].items()) + pre['W_per_vertex'] = post['W_per_vertex'] + 34 + pre['physical_target_histogram'] = dict(post['physical_target_histogram']) + pre['physical_target_histogram']['21'] = pre['physical_target_histogram'].get('21', 0) + 34 + cert = json.loads((package / 'certificate.json').read_text()) + shim = {'bit': { + 'overridden_profile': pre, + 'profile': post, + 'terminal': {'child_delta': {'3': -102, '21': -102}}, + 'coarse': {'coarse_saving': cert['bit_coarse']}, + 'kappa': cert['kappa'], + 'packed_profile': cert['packed_profile'], + 'shim_note': 'pre-sink profile restored from the post-terminal profile by the {3,21} x -102 sink record', + }} + (tree / 'certificate.json').write_text(json.dumps(shim, indent=1, sort_keys=True) + '\n') + print('tree:', tree) + print('pre-sink W %s rank %s deficit %s' + % (pre['W_per_vertex'], pre['rank_per_vertex'], pre['deficit_per_vertex'])) + print('post-terminal W %s rank %s deficit %s' + % (post['W_per_vertex'], post['rank_per_vertex'], post['deficit_per_vertex'])) + + +if __name__ == '__main__': + main() diff --git a/research/butterfly-word-frame-floor/rebuild_word.py b/research/butterfly-word-frame-floor/rebuild_word.py new file mode 100644 index 0000000000..6057c9a69d --- /dev/null +++ b/research/butterfly-word-frame-floor/rebuild_word.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Rebuild PR217's changed bit word through PR217's own constructor. + +Calls `candidate(bit_root)` from the butterfly package (the stage that produces the +changed graph, word and frames), writes the three emitted payloads into --out, and +checks them against the package's own `expected-bit.json` hashes. Nothing in the +butterfly package or the PR202 checkout is modified, and no network is used. + + python3 rebuild_word.py --package /research/butterfly-coordinated-bit-211 \ + --bit-root --out +""" +import argparse +import hashlib +import importlib.util +import json +import sys +import time +from pathlib import Path + + +def load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + mod = importlib.util.module_from_spec(spec) + sys.modules[name] = mod + spec.loader.exec_module(mod) + return mod + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--package', type=Path, required=True, help='the butterfly package directory') + ap.add_argument('--bit-root', type=Path, required=True, help="a checkout of #202's head 8d8d67b") + ap.add_argument('--out', type=Path, required=True, help='fresh directory for the emitted payloads') + args = ap.parse_args() + sys.set_int_max_str_digits(0) + start = time.monotonic() + pkg = args.package.resolve() + if str(pkg) not in sys.path: + sys.path.insert(0, str(pkg)) + cand = load('bfly_candidate', pkg / 'candidate.py') + word, emitted, meta = cand.candidate(args.bit_root.resolve()) + print('[%.1fs] candidate built' % (time.monotonic() - start), flush=True) + expected = json.loads((pkg / 'expected-bit.json').read_text())['emitted_sha256'] + got = {k: hashlib.sha256(v).hexdigest() for k, v in emitted.items()} + ok = True + for k in sorted(got): + same = got[k] == expected.get(k) + ok &= same + print(' %-18s %s %s' % (k, got[k][:16], 'matches expected-bit.json' if same else 'DIFFERS')) + if not ok: + raise SystemExit('emitted payloads differ from the package\'s own expected-bit.json') + print('word: ops %d roles %d v %d h %d pairs %d gauges %d changed frames %d' + % (len(word.ops), word.R, word.v, word.h, len(word.w.get('pairs', [])), + len(word.w.get('gauges', [])), len(word.changed_frames))) + args.out.mkdir(parents=True, exist_ok=True) + for name, payload in emitted.items(): + (args.out / name).write_bytes(payload) + (args.out / 'word-meta.json').write_text(json.dumps( + {'emitted_sha256': got, 'changed_operation_frames': len(word.changed_frames), + 'source_head': meta.get('source_head') if isinstance(meta, dict) else None}, + indent=1, sort_keys=True) + '\n') + print('wrote', args.out) + + +if __name__ == '__main__': + main() diff --git a/research/butterfly-word-frame-floor/transfer_check.py b/research/butterfly-word-frame-floor/transfer_check.py new file mode 100644 index 0000000000..1e82644f14 --- /dev/null +++ b/research/butterfly-word-frame-floor/transfer_check.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +"""Can PR216's admitted cut list be carried onto PR217's butterfly word? + +PR216 (`research/coordinated-crossover-pr200-v2`, head 34abdc5) changes 220 further +operation frames of the word it inherits from PR211 and admits them with a native +checker over a *fixed* scalar word. PR217 (head b739fc2) rewrites part of the same +lineage with paid four-input XOR reassociations. This script compares the two words +structurally, so that the question "can the two be combined?" is answered by counted +facts rather than by the two submissions' prose: + + * the skeleton -- arcs, partner order, reuse pairs, gauges, root roles, phase-1 and + plain operations, sources and reads -- is identical, and both words have the same + operation count, so the two live in one operation-index space; + * the operations that differ are exactly PR217's reassociation set, and the frames + that differ are counted; + * PR216's admitted frames are keyed by that same operation index (each entry is + `[op_index, [basis vectors...]]`), so the indices line up; + * the frame floor of PR217's word bounds every frame layout of it, hence any + combination that keeps that word. + +It reads PR217's emitted payloads (from `rebuild_word.py`) and one file of PR216's +package; it writes nothing and touches neither package. + + python3 transfer_check.py --emit /tmp/bfly/emit \ + --pr216-package /research/coordinated-crossover-pr200-v2 +""" +import argparse +import json +import sys +import zipfile +from pathlib import Path + +# #217's own published numbers, for the like-for-like comparison. +PR217_KAPPA = '684696826673891/10^15' # 6.84696826673891e-4 +PR216_KAPPA = '427415195711/625000000000000' # 0.000683864313137600 +# The completed-bank floor of #217's word, from this package's README (ceilings.py run +# at 1200 rounds): coarse saving < 6.973e-4, so kappa < 6.968e-4. +FLOOR_KAPPA = '6968/10^7' # 6.968e-4, upper bound on any layout +SKELETON = ('arcs', 'order', 'pairs', 'gauges', 'rootroles', 'phase1', 'plain', + 'sources', 'reads') + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--emit', type=Path, required=True, + help='the directory written by rebuild_word.py') + ap.add_argument('--pr216-package', type=Path, required=True, + help="#216's package directory (native-inputs.zip + witnesses/)") + args = ap.parse_args() + sys.set_int_max_str_digits(0) + pkg = args.pr216_package.resolve() + + a = json.loads((args.emit.resolve() / 'word_p12.json').read_text()) + with zipfile.ZipFile(pkg / 'native-inputs.zip') as z: + b = json.loads(z.read('word_p12.json').decode('utf-8')) + current = json.loads(z.read('current-bases.json').decode('utf-8')) + + print('skeleton (must be identical for one operation-index space to exist)') + bad = [] + for k in SKELETON: + same = a[k] == b[k] + print(' %-10s A=%d B=%d %s' % (k, len(a[k]), len(b[k]), + 'identical' if same else 'DIFFERS')) + if not same: + bad.append(k) + ok = not bad and len(a['ops']) == len(b['ops']) + print(' ops A=%d B=%d %s' % (len(a['ops']), len(b['ops']), + 'same length' if len(a['ops']) == len(b['ops']) + else 'DIFFERENT LENGTH')) + if not ok: + raise SystemExit('the two words are not on one operation-index space') + + changed_ops = sum(1 for x, y in zip(a['ops'], b['ops']) if x != y) + changed_frames = sum(1 for x, y in zip(a['op_frame'], b['op_frame']) if x != y) + print('\nwhat PR217 rewrote in that shared index space') + print(' operations whose content differs: %d of %d' % (changed_ops, len(a['ops']))) + print(' operation frames that differ: %d of %d' % (changed_frames, len(a['ops']))) + + rewritten = {i for i in range(len(a['ops'])) if a['ops'][i] != b['ops'][i]} + print('\nPR216 admission records, keyed by operation index') + for label, path, records in ( + ('current bases (input)', pkg / 'native-inputs.zip', current), + ('complete override list', pkg / 'witnesses' / 'additional-bases.json', None), + ('canonical subspaces', pkg / 'witnesses' / 'merged-bases.json', None)): + if records is None: + records = json.loads(Path(path).read_text()) + idx = [int(e[0]) for e in records] + if max(idx) >= len(a['ops']): + raise SystemExit('%s: index beyond the shared operation list' % label) + print(' %-22s %5d records, %5d distinct indices, max %5d, on a rewritten op: %d' + % (label, len(idx), len(set(idx)), max(idx), + sum(1 for i in idx if i in rewritten))) + + print('\nthe ceiling this package measured for PR217\'s own word') + print(' #217 achieved kappa = %s' % PR217_KAPPA) + print(' #216 achieved kappa (plain word) = %s' % PR216_KAPPA) + print(' floor: any frame layout of the word < %s' % FLOOR_KAPPA) + head = (0.0006968 / 0.000684696826673891 - 1) * 100 + print(' so a combined layout on this word can add at most +%.2f%% to the record' % head) + print(' while #216\'s own layout sits %.3f%% below it, on the plain word' + % ((0.000683864313137600 / 0.000684696826673891 - 1) * 100)) + print('\nPASS structural comparison (one index space, %d rewritten operations,' + ' %d rewritten frames)' % (changed_ops, changed_frames)) + print('OPEN combination: the merge is well defined at the index level -- almost every') + print(' admitted record addresses an operation PR217 left untouched -- but PR216') + print(' admits its cuts with the scalar word fixed and PR217\'s rewrite is in those') + print(' operations, so a merged layout needs fresh global admission and a fresh') + print(' column replay, and the ceiling above bounds what it can reach.') + + +if __name__ == '__main__': + main()