Tooling: 2.63× faster production hosted digest via fragment reuse with identical receipts

research/bit-hosted-digest-fastpath/README.md

@@ -112,13 +112,19 @@ is inferred from this bounded search. The active joint-dual containment-mask wor
 uses a different verifier and fixture; PR1736 checks PR589 hosting windows but
 does not optimize instruction hashing. [Claim and final handoff](https://github.com/CrocSwap/integer-mult-bounds/discussions/1871).
 
+## Production follow-up
+
+`research/bit-hosted-digest-production/` wires the equivalent fragment-reuse
+digest into the p111-tuned production `audit_hosted_k.py`. This opt-in package
+now loads the pairwise-era bytes from
+`research/bit-hosted-digest-fastpath/baseline.py` for differential comparison.
+
 ## Open next questions
 
-A maintainer could integrate the adapter into one actively maintained replay
-driver, preserving its pins and comparing its complete receipt once. That would
-make the runtime benefit available without opting into this package. Driver
-integration and its applicable cold-replay obligations are outside this change;
-see the structured question in [TASK.json](TASK.json).
+Driver-level documentation may still mention this opt-in entrypoint; the shipped
+p111-tuned auditor already includes the fragment digest after the production
+package tip. See that package's open questions for remaining vendor copies.
+
 
 ## Credit
 

research/bit-hosted-digest-fastpath/SOURCES.json

@@ -64,9 +64,10 @@
     },
     {
       "path": "research/e8-bit-p111-tuned/vendor/pr510/e8-r480-nine-stage/vendor/e8-r715-eleven-stage-whole/code/audit_hosted_k.py",
-      "usage": "input",
+      "usage": "provenance",
       "git_blob": "8864ca9b942a3507171edca0cf6cbaaf10fcf434",
-      "sha256": "57e558c80a2fea6e291587b04afd76dfaee3550729997798b61f33daf90b8ec9"
+      "sha256": "57e558c80a2fea6e291587b04afd76dfaee3550729997798b61f33daf90b8ec9",
+      "note": "Pairwise-era digest at package base_commit. After research/bit-hosted-digest-production/, working production audit_hosted_k.py ships fragment reuse; this package compares against research/bit-hosted-digest-fastpath/baseline.py."
     }
   ]
 }

research/bit-hosted-digest-fastpath/VERIFY.json

@@ -12,7 +12,7 @@
     "research/community-hyperplane599-audit/pr589/bit589/v9/nine/GLOBAL-TEMPLATE.json.gz",
     "research/community-hyperplane599-audit/pr589/bit589/v9/nine/GLOBAL.json",
     "research/e8-bit-p111-tuned/code/adapt9p.py",
-    "research/e8-bit-p111-tuned/vendor/pr510/e8-r480-nine-stage/vendor/e8-r715-eleven-stage-whole/code/audit_hosted_k.py"
+    "research/bit-hosted-digest-fastpath/baseline.py"
   ],
   "commands": [
     {

research/bit-hosted-digest-fastpath/baseline.py

@@ -0,0 +1,157 @@
+# Independent ownership/window audit, extending DaysSky PR442 and chafreaky PR444 (OpenAI Codex); no generator imports.
+# Generalised to an arbitrary odd stage count, any replica count and host-every-residual templates by DreamingOfClouds with
+# Claude Opus 5.5 (Anthropic), via Claude Code.  Apache-2.0.
+"""Independent ownership/window audit of an idle-axis-hosted compressed S-stage bit template.
+Usage: audit_hosted_k.py PARENT_DIR HOSTED_DIR OUT.json
+PARENT_DIR holds the parent's records/ and nine/-style GLOBAL files via --parent-nine (default PARENT_DIR/nine).
+At the nine-stage defaults this performs PR #444's checks and fourteen mutations; at S = 11 the mutations that need a
+bank are replaced by stage-count, window and exterior-coordinate controls."""
+import copy, gzip, hashlib, json, sys
+from collections import Counter, defaultdict
+from pathlib import Path
+V=120
+read=lambda p:json.loads(Path(p).read_text())
+template=lambda p:json.loads(gzip.decompress(Path(p).read_bytes()))
+
+def windows(schedule):
+    last={}; seen=[]; win={}; rank={}; pair={}
+    for pos,e in enumerate(schedule):
+        if e['kind']=='local_cover_sweep':
+            seen.append((pos,e['stage'],e['pair']));last[e['pair']]=pos
+        elif e['kind']=='bridge':
+            last[e['dst']]=pos;last[e['src']]=pos
+        elif e['kind']=='exchange':
+            assert pos==len(schedule)-1
+        else:
+            assert e['kind']=='idle'
+            i=e['idx'];assert i not in win
+            win[i]={s for q,s,p in seen if last.get(e['pair'],-1)<q<pos and p!=e['pair']}
+            rank[i]=e['rank'];pair[i]=e['pair']
+    return win,rank,pair
+
+def audit(t,parent,widths,depth):
+    L,T,M=t['stages'],t['replicas'],t['m']; K=(L-1)//2; R=len(widths); DATA=2*K*V*T; DEFICIT=T*(2*K*V-9*8*L)
+    assert L>=5 and L%2==1 and M==1+7*L
+    mutable={'version','live','banks_per_stage','hosting','helper_route','calls','schedule'}
+    assert set(t)==set(parent)|{'hosting'}
+    assert {k:v for k,v in t.items() if k not in mutable}=={k:v for k,v in parent.items() if k not in mutable}
+    assert t['replicas']==T and t['m']==M and t['private_centre_reserve']==9
+    sched=copy.deepcopy(parent['schedule']);idx=0
+    for e in sched:
+        if e['kind']=='idle':e['idx']=idx;idx+=1
+    assert t['schedule']==sched
+    W,rank,pair=windows(t['schedule']);hs=t['hosting']
+    assert hs['windows']=={str(i):[pair[i],rank[i],sorted(W[i])] for i in W if rank[i]>0 and W[i]}
+    used=defaultdict(set);hosted={}
+    for i,cp,off,w,s,h in hs['slots']:
+        assert i in W and s in W[i] and 0<=s<L and 0<=h<R and 0<=cp<2*V
+        assert w==widths[h]>0 and 0<=off and off+w<=rank[i]
+        cells=set(range(off,off+w));assert not cells & used[i,cp];used[i,cp]|=cells
+        assert (s,h) not in hosted;hosted[s,h]=(i,cp,off,w)
+    rem={};summaries={}
+    for i in rank:
+        if not rank[i]:continue
+        for cp in range(2*V):rem[i,cp]=rank[i]-len(used[i,cp])
+        if str(i) in hs['windows']:
+            summaries[str(i)]={str(r):n for r,n in sorted(Counter(rem[i,cp] for cp in range(2*V)).items())}
+    assert hs['remainders']==summaries
+    banks={int(s):n for s,n in hs['banks_per_stage'].items()}
+    assert set(banks)==set(range(L)) and all(isinstance(n,int) and n>=0 for n in banks.values())
+    assert {int(s):n for s,n in t['banks_per_stage'].items()}==banks
+    base={};end=DATA
+    for s in range(L):base[s]=end;end+=banks[s]
+    assert t['live']==end
+    assert len(t['calls'])==L*T and len(parent['calls'])==L*T
+    occ=defaultdict(set);bank_count=0;host_count=0
+    assert {(c['stage'],c['replica']) for c in t['calls']}=={(s,r) for s in range(L) for r in range(T)}
+    for c,pc in zip(t['calls'],parent['calls']):
+        assert {k:v for k,v in c.items() if k!='owners'}=={k:v for k,v in pc.items() if k!='owners'}
+        s,rep,p=c['stage'],c['replica'],c['pair'];own=c['owners']
+        assert len(own)==2*V+R and own[:2*V]==pc['owners'][:2*V]
+        for h,(f,r) in enumerate(own[2*V:]):
+            if (s,h) in hosted:
+                i,cp,off,w=hosted[s,h];side,port=divmod(cp,V)
+                assert r==['hosted',i,off,w] and f==rep*(2*K*V)+(2*pair[i]+side)*V+port
+                assert pair[i]!=p and 0<=f<DATA;host_count+=1
+            else:
+                assert len(r)==3 and r[0]=='bank' and r[2]==widths[h]
+                assert base[s]<=f<base[s]+banks[s]
+                assert 0<=r[1] and r[1]+r[2]<=M
+                cells=set(range(r[1],r[1]+r[2]));assert not cells & occ[f];occ[f]|=cells
+                bank_count+=1
+    assert set(occ)==set(range(DATA,end)) and all(x==set(range(M)) for x in occ.values())
+    assert host_count==len(hosted)*T and host_count+bank_count==L*T*R
+    paid=Counter()
+    for c in t['calls']:
+        for z in t['local_programs'][c['orientation']]:
+            if z[0]==0 and z[4]:paid[z[4]]+=1
+            if z[0]==2:paid[z[5]]+=1
+    for r in rem.values():
+        if r:paid[r]+=T
+    assert M*end-sum(r*n for r,n in paid.items())==DEFICIT
+    assert M**depth>2*max(paid)**depth
+    return paid,host_count,bank_count
+
+def digest(t):
+    # Independent streamed hashing uses the same specified JSON instruction format.
+    h=hashlib.sha256();n=0;R=len(t['calls'][0]['owners'])-2*V
+    for call in t['calls']:
+        for z in t['local_programs'][call['orientation']]:
+            x=[z[0],call['owners'][z[1]]]
+            if z[0]==0:x.extend(z[2:])
+            else:
+                x.append(call['owners'][z[2]] if z[2]<2*V+R else [t['live'],['private',0]])
+                x.extend(z[3:])
+            h.update(json.dumps(x,separators=(',',':')).encode());h.update(b'\n');n+=1
+    return h.hexdigest(),n
+
+def main():
+    par,bd,out=map(Path,sys.argv[1:4]);pt=template(par/'nine/GLOBAL-TEMPLATE.json.gz')
+    t=template(bd/'GLOBAL-TEMPLATE.json.gz');g=read(bd/'GLOBAL.json');pg=read(par/'nine/GLOBAL.json')
+    st=read(par/'records/249-states.json');frames=read(par/'records/frames.json')['frames']
+    L=t['stages'];T=t['replicas'];M=t['m'];K=(L-1)//2;R=pg['R'];depth=g['halving_depth'];DATA=2*K*V*T
+    widths=[8-frames[str(st['initial'][str(h)])]['dim'] for h in range(2*V,2*V+R)]
+    P,host,bank=audit(t,pt,widths,depth)
+    th=hashlib.sha256(gzip.decompress((bd/'GLOBAL-TEMPLATE.json.gz').read_bytes())).hexdigest()
+    assert th==g['template_sha256'] and g['parent_template_sha256']==pg['template_sha256']
+    assert P==Counter({int(r):n for r,n in g['profile'].items()})
+    DEFICIT=T*(2*K*V-9*8*L)
+    assert (g['live'],g['physical'],g['rank_mass'],g['deficit'],g['paid_calls'],g['maxchild'])==(t['live'],t['live']+9,sum(r*n for r,n in P.items()),DEFICIT,sum(P.values()),max(P))
+    assert g['hosted_residuals']==host and g['bank_assignments']==bank
+    dg,n=digest(t);assert dg==g['emitted_program_sha256'] and n==g['emitted_events']==pg['emitted_events']
+    assert dg!=pg['emitted_program_sha256']
+    controls=[]
+    def reject(name,fn):
+        q=copy.deepcopy(t);fn(q)
+        try:audit(q,pt,widths,depth)
+        except (AssertionError,KeyError,IndexError,TypeError):controls.append(name)
+        else:raise AssertionError('accepted '+name)
+    def owner(q,kind):
+        return next(o for c in q['calls'] for o in c['owners'][2*V:] if o[1][0]==kind)
+    Wd,rk,pr=windows(t['schedule'])
+    def outside_stage(q):
+        i,cp,off,w,s,h=q['hosting']['slots'][0]; q['hosting']['slots'][0][4]=next(x for x in range(L) if x not in Wd[i])
+    reject('slot outside window',outside_stage)
+    reject('copy index outside family',lambda q:q['hosting']['slots'][0].__setitem__(1,2*V))
+    reject('negative slot offset',lambda q:q['hosting']['slots'][0].__setitem__(2,-1))
+    reject('wrong residual width',lambda q:q['hosting']['slots'][0].__setitem__(3,1+q['hosting']['slots'][0][3]))
+    reject('duplicate helper',lambda q:q['hosting']['slots'].append(q['hosting']['slots'][0]))
+    reject('omitted helper',lambda q:q['hosting']['slots'].pop())
+    reject('hosted owner in another replica or outside the data stock',lambda q:owner(q,'hosted').__setitem__(0,owner(q,'hosted')[0]+2*K*V))
+    reject('hosted owner on a wrong data family',lambda q:owner(q,'hosted').__setitem__(0,q['calls'][0]['owners'][0][0]))
+    if bank:
+        reject('wrong bank slot',lambda q:owner(q,'bank')[1].__setitem__(1,M))
+        reject('wrong bank stage',lambda q:owner(q,'bank').__setitem__(0,t['live']-1))
+    reject('wrong call orientation',lambda q:q['calls'][0].__setitem__('orientation','reverse'))
+    reject('duplicate call',lambda q:q['calls'].__setitem__(1,q['calls'][0]))
+    reject('changed idle rank',lambda q:next(e for e in q['schedule'] if e['kind']=='idle').__setitem__('rank',1))
+    reject('understated live stock',lambda q:q.__setitem__('live',q['live']-1))
+    reject('missing remainder summary',lambda q:q['hosting']['remainders'].pop(next(iter(q['hosting']['remainders']))))
+    reject('stale stage count',lambda q:q.__setitem__('stages',L-2))
+    reject('hosted coordinate beyond the pending set',lambda q:q['hosting']['slots'][0].__setitem__(2,rk[q['hosting']['slots'][0][0]]))
+    reject('overlapping hosted coordinates',lambda q:q['hosting']['slots'].append(q['hosting']['slots'][0][:5]+[(q['hosting']['slots'][0][5]+1)%R]))
+    res=dict(status='PASS_INDEPENDENT_REPLICATED_HOSTING_AUDIT',template_sha256=th,emitted_program_sha256=dg,emitted_events=n,
+             profile={str(r):z for r,z in sorted(P.items())},live=t['live'],physical=t['live']+9,hosted_residuals=host,
+             bank_assignments=bank,rank_mass=sum(r*n for r,n in P.items()),deficit=DEFICIT,maxchild=max(P),mutations_rejected=controls)
+    out.write_text(json.dumps(res,indent=2)+'\n');print(json.dumps(res))
+if __name__=='__main__':main()

research/bit-hosted-digest-fastpath/fast_digest.py

@@ -12,8 +12,12 @@
 
 PACKAGE = Path(__file__).resolve().parent
 ROOT = PACKAGE.parents[1]
-AUDITOR = ('research/e8-bit-p111-tuned/vendor/pr510/e8-r480-nine-stage/'
-           'vendor/e8-r715-eleven-stage-whole/code/audit_hosted_k.py')
+# Pairwise-era auditor bytes (production now ships fragment reuse).
+AUDITOR = 'research/bit-hosted-digest-fastpath/baseline.py'
+PAIRWISE_AUDITOR = (
+    'research/e8-bit-p111-tuned/vendor/pr510/e8-r480-nine-stage/'
+    'vendor/e8-r715-eleven-stage-whole/code/audit_hosted_k.py'
+)
 PARAMETERIZER = 'research/e8-bit-p111-tuned/code/adapt9p.py'
 FIXTURE = 'research/community-hyperplane599-audit/pr589/bit589'
 
@@ -31,12 +35,19 @@ def pins():
             if x['usage'] == 'input'}
 
 
+def provenance_pins():
+    return {x['path']: x['sha256'] for x in
+            json.loads((PACKAGE / 'SOURCES.json').read_text(encoding='utf-8'))['files']
+            if x['usage'] == 'provenance'}
+
+
 def load_auditor():
     """Apply only the five hosted-auditor edits listed in frozen adapt9p.py."""
     if not __debug__:
         raise RuntimeError('the inherited auditor requires assertions; remove -O/PYTHONOPTIMIZE')
     sources = pins()
-    original = checked_bytes(ROOT / AUDITOR, sources[AUDITOR]).decode('utf-8')
+    # baseline.py vendors the pairwise-era bytes pinned as provenance on PAIRWISE_AUDITOR.
+    original = checked_bytes(ROOT / AUDITOR, provenance_pins()[PAIRWISE_AUDITOR]).decode('utf-8')
     adapter = checked_bytes(ROOT / PARAMETERIZER, sources[PARAMETERIZER]).decode('utf-8')
     assignments = [n for n in ast.parse(adapter).body if isinstance(n, ast.Assign)
                    and any(isinstance(t, ast.Name) and t.id == 'EDITS' for t in n.targets)]

research/bit-hosted-digest-production/NOTICE.md

@@ -0,0 +1,19 @@
+# Notice
+
+This package wires Matthieu Pascal's proved opt-in hosted-digest adapter
+(`research/bit-hosted-digest-fastpath/`, CrocSwap/integer-mult-bounds#1874) into
+the production p111-tuned `audit_hosted_k.py`.
+
+Predecessor credit:
+
+- DaysSky (PR442) and Chafik Boukhalfa / chafreaky (PR444) for the ownership /
+  window audit lineage.
+- DreamingOfClouds for the PR463/510/589 nine-stage hosted chain and the PR589
+  word retained under `research/community-hyperplane599-audit/`.
+- eumemic for PR555 and the P111 `adapt9p.py` parameterizer.
+- Matthieu Pascal (@mtpsl) for the exact fragment-reuse digest and differential
+  controls in #1874 / discussion #1871.
+
+Prepared by Wentao Li (GitHub VictorLiwentao, X @Is__Victor) with substantial AI
+assistance, under Apache-2.0. Finite certificate checks are not a formal proof
+of the full multiplication theorem.

research/bit-hosted-digest-production/README.md

@@ -0,0 +1,103 @@
+# Production fragment-reuse digest for p111-tuned hosted audits
+
+**Tooling improvement; no new kappa.** At base
+`0fdd7d82de03e43c9e5a1000dfbcd380d1b687d2`, production
+`research/e8-bit-p111-tuned/vendor/pr510/e8-r480-nine-stage/vendor/e8-r715-eleven-stage-whole/code/audit_hosted_k.py`
+still hashed every mapped instruction with a fresh `json.dumps`. Matthieu
+Pascal's opt-in adapter `research/bit-hosted-digest-fastpath/` already proved
+exact JSON-fragment reuse with identical PR589 hosted-audit receipts. This
+package wires that equivalent digest into the **production** p111-tuned source
+consumed by `adapt9p.py`.
+
+Other byte-identical `audit_hosted_k.py` inventories under `research/*/vendor/`
+are left as provenance. `research/bit-hosted-digest-production/baseline.py`
+vendors the pairwise-era bytes. Dependent VERIFY packages refresh or demote
+those pins to provenance in this tip. Selected frontier untouched.
+
+## Reproduce
+
+Python **3.12**, standard library only, from the repository root:
+
+```sh
+python3.12 -I -B research/bit-hosted-digest-production/verify.py
+python3.12 -I -O -B research/bit-hosted-digest-production/verify.py
+python3.12 -B scripts/research_package.py --package research/bit-hosted-digest-production
+python3.12 -I -B research/bit-hosted-digest-production/benchmark.py --rounds 5 --write-result
+```
+
+Expected:
+
+```text
+PASS_BIT_HOSTED_DIGEST_PRODUCTION events=7843392 … corruption=… certs=1 no_new_kappa
+PASS_REGISTERED_RESEARCH_PACKAGE research/bit-hosted-digest-production outcome=completed
+PASS_IDENTICAL_BIT_HOSTED_DIGEST_PRODUCTION_RECEIPTS
+```
+
+Default verify is read-only. Temp-copy digest foil rejects via receipt mismatch
+or audit assertion. Package gates under `-O` re-exec an asserting child
+(`python -I -B` with `PYTHONOPTIMIZE` cleared). Full `make verify`, selected
+cold replay, Lean, and multiplication-theorem obligations were not rerun for
+this isolated historical-auditor change.
+
+## Result
+
+On Linux / CPython 3.12.3 with `taskset -c 0`, five alternating fresh-process
+pairs gave median complete hosted-audit wall time **34.880 → 13.268 s
+(2.63×)**. Timing is host-descriptive and is not a correctness gate; all
+samples and environment pins are in [`RESULT.json`](RESULT.json). Every
+parameterized hosted-audit receipt matches digest
+`96c7ad2e744e9a503a4ef5ecd52d0f6a8899bdb5efac2e3aafa0db18adf99523`
+with 7,843,392 events and all 16 inherited mutation rejections.
+
+## Why the fragment digest is equivalent
+
+Compact JSON arrays concatenate at element boundaries. Encoding each local
+instruction's constant prefix/tail once per orientation, and each call's owner
+descriptors once per call, then concatenating the mapped owner fragments,
+yields the same byte stream as building each full instruction array and calling
+`json.dumps` per event. Private owners still use `[live, ["private", 0]]` past
+the first call's owner count. There is no persistent cache across invocations.
+See `research/bit-hosted-digest-fastpath/README.md`.
+
+All five `adapt9p.py` hosted-auditor anchors remain unique after the rewrite.
+
+## Overlap and credit
+
+Closest priors (different delivery):
+
+- `research/bit-hosted-digest-fastpath/` / [#1874](https://github.com/CrocSwap/integer-mult-bounds/pull/1874)
+  — opt-in fragment digest; this is the production wiring.
+- [Discussion #1871](https://github.com/CrocSwap/integer-mult-bounds/discussions/1871)
+  — author handoff inviting driver integration.
+- [#756](https://github.com/CrocSwap/integer-mult-bounds/pull/756)
+  — identical-receipt tooling quality bar.
+- Open leave-alone tooling on other objects (#1922 joint frames, #2039/#2054
+  shared-point, #2100 ternary, #2117 DisjointTensor, #2150 mixed-point,
+  #2194/#2210/#2231/#2239/#2257/#2263/#2276/#2320/#2329 production rewrites;
+  #1076 quadratic-affine) was not reopened. Soft note: #2263/#2276 touch
+  `e8-bit-r420/MANIFEST.sha256`; this tip does not.
+
+Credit Matthieu Pascal, DreamingOfClouds, eumemic, DaysSky, Chafik Boukhalfa
+and the retained hyperplane599 archive. Prepared by Wentao Li
+([VictorLiwentao](https://github.com/VictorLiwentao),
+[@Is__Victor](https://x.com/Is__Victor)) with substantial AI assistance, under
+Apache-2.0.
+
+## Open next questions
+
+1. **Question:** Should the other byte-identical vendored `audit_hosted_k.py`
+   copies under `research/*/vendor/` adopt the same fragment-reuse digest in a
+   follow-up tip?
+   **Why it matters:** Those trees still pin pairwise-era digests; a coordinated
+   refresh would remove stale inventory digests without changing receipts.
+   **First step:** Diff each copy against the production transform site and list
+   which SOURCE inventories would need provenance demotion or honest refresh.
+   **Blocked by:** Out of this session’s one-object stop rule.
+
+2. **Question:** Should active bit-hosting drivers prefer the production digest
+   path in their documented reproduce commands?
+   **Why it matters:** Some notes still point at the opt-in `fast_digest.py`
+   entrypoint.
+   **First step:** List drivers that still shell out to
+   `research/bit-hosted-digest-fastpath/` paths.
+   **Blocked by:** Maintainer documentation preference.