Stream complete R437 high-prefix directories into the actual leaf/XOR consumer

research/r437-streaming-directory/NOTICE.md

@@ -0,0 +1,10 @@
+JoyboyBrian; substantial OpenAI Codex assistance. Apache-2.0.
+
+The directory.Context/Odometer/old_prefix bodies and original compiler, numerical,
+record, leaf and XOR sources retain their existing authors and notices. The
+source-independent fixture and proof build on #4981 (JoyboyBrian), and its actual
+physical bridge retains Chris R / Clear1nsight attribution. Source source files
+are authenticated live inputs; none are edited here. actual64.py compiles a
+restricted AST of the pinned predecessor fixture without modifying that file.
+The implementation's original contributions are persistent emission-role binding,
+complete producer-to-consumer composition, proof, resource ledger and controls.

research/r437-streaming-directory/PROOF.md

@@ -0,0 +1,158 @@
+# Complete streamed high-prefix interface
+
+JoyboyBrian; substantial OpenAI Codex assistance; Apache-2.0. Source numerical,
+record, odometer, leaf and XOR primitives retain their predecessor authors.
+
+## Contract and scope
+
+Let C>0 be the encoded physical old-prefix count, h>=0 the encoded old-field
+count, d the fixed source dimension, q the compiled radix and q^w the Ring
+modulus. The authenticated directory.Context supplies cap=q^(w-1) and
+H=cap^(d²). The old directory is canonical and contains exactly C prefixes of
+h fields. Equal byte strings at different physical positions are different
+occurrences. Every old high digit is in [0,cap). Field count h+1 must fit the
+index arena. Enough source numerical inventory is provided. Source/target/old
+are distinct, borrowed, rewound tapes; both payloads have exactly CH complete
+fibres. The trace is a closed prefix-local leaf, its callback reads only that
+complete prefix and borrowed numerical/program context, restores prefix heads,
+and returns owned coefficients according to the actual source contract.
+
+The callback passed to visit is synchronous, read-only on its prefix, restores
+its head and does not retain the physical prefix role after return. The fixture
+observer copies bytes only outside measured semantics. These are interface
+premises: arbitrary callbacks that remember the prefix object are excluded.
+
+visit returns the owned encoded product CH. run returns the same owned full
+XOR result as directory.produce, the eager actual packet_engine leaf and actual
+episode.xor_packet. No full production GL64 or q127 payload is claimed.
+
+## Source order and finite control
+
+The original directory.old_prefix body is cloned with only Tape and Counter
+constructors rebound to persistent roles; no source bytecode is edited. The
+Counter constructor body is itself cloned with its Tape constructor rebound.
+All source finite-code, word-extent, high-range and record-delimiter checks,
+reader movement and counter arithmetic execute unchanged. Each close clears
+its persistent backing before the next initialization. The reader still holds
+at most one decoded old numerical Word, freed by its original finally clause.
+The arena erases and reuses freed Word slots rather than keeping a new register
+for every old occurrence.
+
+There is one source Odometer, initially all zero. Its actual advance body
+increments the last row-major digit first; reaching cap resets that digit to
+zero and carries left. At its unique terminal carry every digit has been reset
+to zero. Resetting only its finite finished flag therefore gives precisely the
+same next old-prefix odometer initialization as constructing fresh zero Words.
+No numerical value is decoded into Python control.
+
+Induct on the outer physical occurrence i and the inner odometer index j.
+Before emission, saved is exactly original old_prefix(i), the current digits
+are exactly the original j-th high matrix and prefix is empty. Source append
+primitives emit saved, the new code, all d² high Words and RECORD_END. Thus the
+callback receives exactly the (iH+j)-th original produced prefix, in the same
+byte order. After callback return prefix is erased, the encoded emitted Count
+is incremented, and the unchanged odometer advances. At the terminal carry,
+saved is erased and outer is decremented. This proves all C*H occurrences,
+including duplicate byte prefixes and every high matrix, occur exactly once.
+The final original-input BLANK check rejects a wrong old count/trailing bytes.
+The encoded Count.product/emitted equality independently checks multiplicity.
+
+Context.highs is the predecessor Count class, while the extended cardinal
+product requires cardinal_counts.Count. The adapter makes and closes an
+encoded copy, rather than silently treating the distinct classes as identical.
+Count.product is paid setup. It never becomes a host integer.
+
+## Actual producer-to-consumer composition
+
+For each emitted prefix, run consumes one complete source fibre and the
+corresponding dirty target fibre into owned local tapes. It executes the
+unchanged actual packet_engine.run on that one prefix and then the unchanged
+episode.xor_packet on its computed return. Only this consumed fibre is appended
+to the full output. All local returned/output/left/right stores are erased at
+the end of the callback. The source and target heads advance to the next fibre;
+the prefix head is restored before visit clears it.
+
+The #4981 prefix-factorization theorem applies: a closed prefix-local leaf acts
+independently on each fibre. Exchanging the eager cross-prefix chronology for
+complete leaf/XOR execution at each prefix preserves every within-prefix
+callback and primitive order. Induction on iH+j therefore proves the complete
+returned packet equality for arbitrary dirty payload bits. The final two BLANK
+checks prevent omitted payload fibres. Borrowed source, target and old directory
+heads are rewound in finally; their bytes are never written.
+
+The separate actual64 calibration uses exactly this run code object with only
+its visitor rebound to a two-prefix reader. It exercises the actual SOURCE87
+compiled program/callback and dirty XOR consumer, but does not test the full
+64-dimensional producer. Its AST harness changes only the package adapter path,
+the obsolete adapter-specific rejection block and status label; an exact edit
+count pins this restriction. Full81-high production and actual64 sampling are
+separate evidence, never multiplied into a claim of 3**4096 execution.
+
+## Directory backing and complete remaining ledger
+
+Let L be the fixed residue Word bit extent and b the fixed compiled-code width.
+A field has B=(b+1)+d²(L+1)+1 record cells. The old saved role retains at most
+hB cells and the current role at most (h+1)B, each plus LEFT. Tape.clear erases
+these cells without shrinking its bytearray; reuse is essential. Their retained
+backing is at most (2h+1)B+2, independent of C*H. The persistent reader counter
+has at most index_bits+2 cells. There is one outer countdown (O(log(C+1)) cells),
+one emitted Count and one expected Count (O(log(CH+1)) each), and d² odometer
+Words of L+2 cells each. The original old-word reader needs one additional
+reused numerical slot; compare/increment are actual in-place source operations.
+The encoded field increment and complete source leaf have their own paid slots.
+
+Count.product setup uses copied factor/scan/result and the source multiply/add
+work tapes. Every setup allocation, transient backing and erasure remains paid;
+there is no claim that the whole setup uses only the three prefix roles. Its
+bit widths are bounded by input/product widths, and its constructor count is
+independent of the number of emissions performed after setup. The fixed-role
+claim concerns the producer emission loop only. The verifier observes no record
+Tape construction between first and last emissions, even across old prefixes.
+
+The original complete new directory occupies CH(h+1)B cells. That whole object
+is removed. This is a source-order representation theorem, not a complete
+physical reserve saving: the borrowed old directory still occupies ChB, complete
+source and dirty target still occupy CHv each, the owned full result retains
+CHv, and local left/right/leaf/XOR work retains its full fibre backing. The
+unchanged leaf/consumer may allocate new record tapes on each call; its allocator
+and all cleared backing remain a retention path requiring a persistent pool or
+an explicit physical retirement tariff. Gather/scatter, all spectators, Context
+and codebook construction, precision, stock, compiler, native and outer admission
+remain unpriced. No global kappa or total fixed-tape tariff follows.
+
+Every emitted prefix is still written and parsed by every actual callback.
+The setup Counts, old validation, copying saved to current, clearing current,
+full source/target copying, encoded counters and terminal rewinds are replacements
+that must be charged. This package does not infer a speedup from deleting the
+large directory or compare elapsed time across different machines.
+
+## Cleanup and rejecting paths
+
+On valid success all producer numerical temporaries and record roles are erased
+and retired, the returned Count/output is caller-owned, and all borrowed bytes
+and heads are preserved. Finally clauses also preserve callback exception
+identity and clear producer roles for an exception after a complete prefix.
+The new-code guard is before allocations; old high-range failure has the source
+Word finally, and a trailing-input failure occurs after complete reads. The
+verifier covers these paths. It does not promise transaction rollback of
+arbitrary packet callbacks, arena allocation failures or malformed read_word
+extents: inherited primitives can allocate before raising, and rollback of
+those failure classes is a separate contract.
+
+## Evidence and family decision
+
+The full generic q3,w2,d2 cube tests81 high matrices for each of three physical
+old prefixes, the first and last byte-identical. All243 emitted prefixes match
+both original produce and an independent row-major serializer. The complete
+19683-bit dirty source/target result matches original eager execution and an
+independent permutation/XOR oracle, with 1458 real callbacks per arm and Word.inspect
+forbidden. A second actual64 calibration uses nonzero high charts and the real
+source-owned incoming coefficient/program and following SOURCE87 XOR statement.
+Those are bounded differential certificates supporting the uniform scoped proof.
+
+This is one justified expansion after #4981: it closes the complete producer
+obligation and removes an entire serialized interface object. It does not repeat
+a callback census, extend the admitted Count boundary of #5025, or solve the
+active #4927 full-directory serialization/reserve audit. The next decision is
+whether a complete physical caller can pool/reuse the retained payload/leaf
+roles and pay the above full ledger; directory deletion alone is insufficient.

research/r437-streaming-directory/README.md

@@ -0,0 +1,61 @@
+# Stream the complete high-prefix directory into its real consumer
+
+**Scoped source-order theorem and bounded executable certificates; no new bound.**
+
+The original producer writes every complete old/new prefix to a whole directory
+before its consumer starts. [streaming.py](streaming.py) instead reuses one saved
+old-prefix role, one current-prefix role, one reader counter and one odometer,
+then invokes the actual leaf and XOR consumer synchronously. It preserves all
+physical occurrences, including repeated bytes, and every high matrix in original
+source order. [PROOF.md](PROOF.md) gives the uniform contract, induction, backing
+bound, cleanup scope and complete remaining ledger.
+
+The complete generic q3,w2,d2 test produces all 81 high matrices for each of three
+old physical prefixes. All243 prefixes and the full 19683-bit dirty-payload result
+match the original eager pipeline and independent byte/permutation oracles;
+1458 actual coefficient callbacks remain in each execution arm. The separate
+[actual64.py](actual64.py) calibration substitutes the same consumer code into
+#4981's authenticated actual R437 SOURCE87 nonzero-high fixture. It samples two
+complete prefixes of the 3**4096 domain and makes no full-domain claim.
+
+The retained current-directory backing is `(2h+1)B+2` for the two prefix roles,
+with `B=(code_bits+1)+d²(word_bits+1)+1`; the original new directory needs
+`C H (h+1) B` cells. Encoded counters, odometer, setup and erasure remain paid.
+The full borrowed payloads and owned result remain; the unchanged leaf allocator
+still creates per-call tapes, so this is not an overall physical reserve saving.
+
+## Reproduce
+
+```sh
+python3 scripts/research_package.py --package research/r437-streaming-directory --check-only
+python3.12 scripts/research_package.py --package research/r437-streaming-directory
+```
+
+The registered normal and optimized hooks authenticate all declared live inputs
+before execution and compare typed deterministic receipts. [SOURCES.json](SOURCES.json)
+pins exact Git/SHA256 provenance and the historical selected-loader projection.
+The actual64 harness reads the pinned predecessor verifier and restricts its AST
+without changing frozen source files. [ACTUAL64.json](ACTUAL64.json) and
+[RESULT.json](RESULT.json) are separate deterministic receipts.
+
+## Family gate
+
+#4981 proved prefix-local functional fusion but retained the full producer. This
+package closes that named obligation once, with complete generic production,
+rather than adding another selected statement or tiny prefix census. The active
+#4927 existing-directory audit remains separate. The actual complete GL64 gather,
+Context/codebook setup and q127 huge payload are not executed or priced.
+
+## Open next questions
+
+The next useful task is a source-bound persistent pool for the complete leaf/XOR
+workspace plus its movement/erasure tariff. Start by tracing every per-prefix
+Tape constructor and lifetime in the actual leaf; retain all full input/output
+and spectator backing. Without that whole-caller result, do not infer a supplier
+admission, kappa change or a complete multiplication theorem.
+
+## Credit
+
+JoyboyBrian; substantial OpenAI Codex assistance; Apache-2.0. Retains Chris R /
+Clear1nsight's physical bridge and all original directory, numerical, leaf and
+XOR source attribution. The #4981 fixture and theorem are pinned predecessors.

research/r437-streaming-directory/RESULT.json

@@ -0,0 +1,36 @@
+{
+  "complete_consumer": {
+    "actual_leaf_and_xor": true,
+    "callbacks": 1458,
+    "fibre_bits": 81,
+    "output_sha256": "c74d323c6de93c1b73f8cc5235cc4407e564248735c3725d3ff40b87c00b7665",
+    "packet_bits": 19683,
+    "prefixes": 243,
+    "word_inspect_disabled": true
+  },
+  "controls": [
+    "duplicate physical prefixes retained",
+    "nonzero high-cube order",
+    "independent complete payload oracle",
+    "callback failure clears fixed producer stores",
+    "invalid new code rejected before callbacks",
+    "complete out-of-range old high rejected with cleanup",
+    "omitted physical prefix rejected after162 callbacks"
+  ],
+  "live_inputs": 406,
+  "producer": {
+    "dimension": 2,
+    "directory_bytes": 14580,
+    "emitted": 243,
+    "highs": 81,
+    "max_current_prefix_bytes": 60,
+    "no_record_constructor_in_emission_loop": true,
+    "old_physical_prefixes": 3,
+    "one_current_prefix_role": true,
+    "q": 3,
+    "sha256": "2eb4b8a1637eaba05348053ad195b29bda1c228932f3b31e9af2478abba01337",
+    "w": 2
+  },
+  "scope": "Complete generic high cube and actual source leaf/XOR consumer; production GL64 and total physical reserve remain unproved.",
+  "status": "PASS_R437_STREAMING_DIRECTORY"
+}