Research (no κ claim): reorder, re-frame and re-ranking ceilings on the #4610 productive-helper word

research/frontier-word-reorder-ceilings/README.md

@@ -0,0 +1,52 @@
+# Lever ceilings on the #4610 round-three productive-helper word (no κ claim)
+
+Two screening rounds on Cao Yifan's #4610 word (`research/kappa-round3-warp-20261022/ACTUAL-WORD.json.gz`,
+κ 0.0030217758). The word is unchanged in the current #5413 caller, so these bounds still describe it.
+They say where the remaining headroom is *not*, so that effort goes elsewhere. Unless a line says
+"exact", every bound is a floating-point screen and not a certificate.
+
+## Summary
+
+| Lever | Result | Where |
+|---|---|---|
+| Productive re-ranking (finish rank-4/7/8 helpers as rank 9) | loses for every k by strict subadditivity of x^(1−C): dM = f(9) − f(k) − f(9−k) < 0 (4→9: dC ≈ −1.35e-6) | `r12-levers/` §1 |
+| Longer controllers, query thinning | already closed by #4658/#4659 and #4664/#4666/#4674 | `r12-levers/` §2–3 |
+| Centre merging; query/local climb fusion | worth ≈ +4.1e-5 and +2.8e-5 if legal, but not legal (#4698/#4700; the query temporary shares no frame) | `r12-levers/` |
+| One climb per register (fragmentation fully removed, same masses) | ceiling 0.0071457 | `r12-levers/` |
+| Reorder freely + re-frame, keeping the ADD graph | ceiling 0.0071108 | `r13-reorder/` §2 |
+| **Reorder within the exact commutation poset + re-frame** (dimension-only MIP, HiGHS optimal) | 0.0033076 (free warps) / 0.0032280 (pinned warps); **does not lift to real subspaces** (332 clashes) | `r13-reorder/` §2 |
+| **Same, exact subspace LP over full frame domains (3.03M variables)** | local cost ≥ 5251.946 vs current 5253.117, i.e. ≤ 1.17 units per local, **κ ≤ ≈ 0.003107** | `r13-reorder/` §4 |
+| Exact subspace MIP on restricted and medium frame domains (forward and backward) | **optimum equals the current word (gap 0)** | `r13-reorder/` §2 |
+| One-step non-commuting swaps with exact correction ADDs | 0 gains (1005 + 1466 greedy sites; 36 MIP-reframed feasible rewrites all at exactly the current cost; 153 infeasible) | `r13-reorder/` §4 |
+
+**Takeaway.** Each of the 632 registers climbs in 3–9 steps, because a scalar ADD needs both operands
+at the identical frame. That fragmentation is the whole concave loss. Reordering within the true
+dependencies and re-choosing frames can remove at most about 1.17 cost units per local. The helper
+frames forced in *every* valid reordering already cost 3458.82 of the helpers' 3464.74. What remains
+needs a redesigned helper factorization, not reordering or local rewrites. Our #5462 shows the
+owned-relation lever is also at its optimum.
+
+## Contents
+- `r12-levers/`
+  - `README.md`: notes.
+  - `DIGEST.md`: a working digest of the productive-helper accounting, with commands.
+  - `tools/`: `moment.py` reproduces #4610's raw crossing; `marg.py`, `consolidate.py`, `stab.py`,
+    `regdp.py`, `dimmilp.py`, `dimlp.py`.
+- `r13-reorder/`
+  - `README.md`: model, ceilings, why the fragmentation is forced, and probes.
+  - `tools/`: `poset.py`, `ceil.py`, `sdom.py`, `backbone.py`, `construct.py`, `realize.py`, `rewrite.py`,
+    `mipeval.py`, `lift.py`, `price.py`, and others.
+  - `logs/`: solver logs, renamed `.txt`.
+  - `data/`: gzipped MIP/LP solutions.
+
+`realize.py` rebuilds the current word exactly from the poset model and passes both native
+interpreters on all 632 arbitrary-input columns. That is the positive control for the model.
+
+**Running the tools.** They expect a checkout of #4610 (head 9fd59d9b) as the working directory, and a
+few contain absolute paths from the machine that produced them; adjust those paths before running.
+Python 3.12+ with numpy, scipy and highspy.
+
+**Credits.** Cao Yifan (#4610 word and native semantics), zxv (#4359 productive-helper paradigm, native
+and compiler machinery), Cinna Davis and 1337isnot1337 (#4440/#4441/#4447 controllers; #4658/#4659),
+#4664/#4666/#4674/#4698/#4700, Dugongue (#4713), aliromman and Clear1nsight (#5383/#5413). Prepared by
+Joel Pulikkan (GamingPuzzled) with Claude Opus 5.5 (Anthropic) via Claude Code. Apache-2.0.

research/frontier-word-reorder-ceilings/r12-levers/DIGEST.md

@@ -0,0 +1,76 @@
+# Productive-helper paradigm digest (R12, 2026-10-22)
+
+Base for all numbers: PR4610 head 9fd59d9b08ec (worktree ~/imb-r12, branch pr4610).
+Frontier: #4610 kappa 0.0030217758 (round6 compiler tightening of the round-three word);
+raw crossing (0.00302177587077, 0.00302177587078). #4635 (Word RAM) 0.0030217754 is lower.
+Selected main still 0.00302015 (#4470 / joint-helper-lattice).
+
+## 1. Accounting (how kappa is computed)
+- Saving C solves M(C) = sum_k T_k k^(1-C) - sum_r P_r r^(1-C) = 0 (T = productive targets, P = paid
+  recursive jobs, ranks = Gaussian ranks / axis counts). kappa = slightly below the raw root after the
+  strict compiler (paid tensor completion #4416, splitting at c^v, higher-moment tails #4433,
+  v=1024, c=20, b=21, u=1/5, contraction reserve now 1e-11; finite integer threshold j, H, N=2^H).
+- Code: research/kappa-round6-compiler-tightening-20261022/arithmetic.py
+  `assemble(local)` builds (T,P) from the p16 controller: for each physical event
+  LOCAL -> T += helper census H, P += local paid profile N; QUERY_SHEAR -> P[1] += 112;
+  IDLE(stages) -> P[8*len] += 224; diagonal term -> T[1+8*sum] += 112.
+  Exact intervals: research/higher-moment-strictification/verify.py (E.moment),
+  second engine research/kappa-rebuild/architecture/mixed_target_probe.py.
+- Quick float pricer: ~/imb-harness/r12/tools/moment.py (run from ~/imb-r12 with python3.14 -B);
+  marginals: ~/imb-harness/r12/tools/marg.py.
+- Current invoice (PR4610 round3): local paid N={1:2201,2:412,3:510,4:127,5:40,8:8} (3298 jobs,
+  mass 5327); helpers H={4:28,7:7,8:47,9:326} (408, mass 3471). Controller p16, a=7 prefix pairs,
+  18 LOCAL (full native), 15 QUERY_SHEAR, 15 IDLE, 90 ADD, 16 SIGNED_SWAP; ambient rank 145.
+  T={4:504,7:126,8:846,9:5868, 17..65 step 8: 448 each, 73:224, 145:224};
+  P={1:41298,2:7416,3:9180,4:2286,5:720,8:144, 16..64 step 8: 448 each, 72:224}.
+  10,928 productive fields, 64,404 paid calls.
+- Rank-mass ledger (KEY INVARIANT): surplus sum T k - sum P r = 752 =
+  35840 (diagonal 177408 - idle 141568) - 18*1856 (local deficit) - 1680 (queries).
+  Local deficit 1856 = 224*8 (main D_s transverse work) + 64 (eight paid rank-8 owned-centre jobs,
+  no productive credit). Helper/native MILPs (#4470,#4610,#4672) move helper and paid mass
+  TOGETHER (deficit unchanged); gains are purely concave redistribution.
+- Linearization: M ~ 752 - C*S, S = sum T k ln k - sum P r ln r; dM/dC = -247222 at root.
+  Marginal dC per local-type change (x18 copies): remove one paid rank-1: +7.28e-5;
+  free helper rank +1 (8->9): +7.2e-5 (but violates mass invariant); paid 4+4->3+5: +5.5e-8;
+  merge paid 1+1->2: +3.05e-7; split helper 9->4+5: +1.35e-6; drop the 64 centre charge: C~0.00806 (#4659).
+
+## 2. Structure
+- Native local word: research/kappa-round3-warp-20261022/ACTUAL-WORD.json.gz
+  (canonical: v=112 ports, h=9, 632 start/final columns, 4429 frames, 72 warps, 180 warp edges;
+  programs forward/backward 21449 ops). 408 helpers arbitrary independent inputs; 1 owned centre
+  reused sequentially (8 centre calls, rank 8 each).
+- Controller (#4440 query-thinned): paired path, 2p+1 stages, full native at {0,2p} u odd, other
+  stages replaced by rank-one query shears S_y(Q^-1)/S_x(-Q) (1 paid rank-1 per port); idles drop
+  thin-stage multipliers. Code: research/query-thinned-controllers/controller.py (keep_mask,
+  thick_stages, inventories, replay_events); p16 pinned in research/joint-helper-lattice/
+  {CONTROLLER.json, verify.py:controller16}.
+- Boundaries: exact rational L A R = diagonal check over Laurent coefficients (32x32), verify.py.
+- LP column generation / controller discovery (#4359): research/kappa-rebuild/architecture/
+  {controller_menu_search.py, mixed_forest_search.py, mixed_target_probe.py, controller_tariff.py};
+  DP over native/thin stages: research/kappa-hybrid-controller/search/direct_stage_dp.py.
+- Helper/native MILPs (#4610): research/kappa-round3-warp-20261022/{line_free_milp.py,
+  helper_line_milp.py, both_boundary_milp.py} (HiGHS, 1.6-2.0M vars, --operator all --seconds 180).
+- Hyperplane-retiming min-cut search (#4672): pr4672 research/hyperplane-retiming/search/*.py.
+
+## 3. Admission checks (what a new supplier must pass)
+all-632-column arbitrary-input replay both orientations (two interpreters); exact Clifford
+admission of changed native G maps with absolute phase on 512 columns; warp classes; helper
+boundaries; controller Laurent factorization; two rational moment engines; strict compiler +
+54 outer predicates; mutation controls. Registered replay:
+`python3.12 -B scripts/research_package.py --package research/<pkg> [--check-only]`
+(PR4610 packages themselves were run with python3.14; joint-helper-lattice is registered).
+
+## 4. Levers already used / closed
+Used: productive helper jobs + mixed balance (#4359), paid target padding (#4416), native tensor
+strictification (#4412), p6 native controller (#4414), query thinning p17->p16 (#4440), helper
+retiming (#4441), contact span (#4447), joint helper entrance/exit lattice MILP (#4470),
+joint helper entrances + native maps (#4610 round3), compiler reserve 1e-9 -> 1e-11 (#4610 round6),
+helper621 chart (#4626, does not combine), hyperplane retiming (#4672, round-two only; its
+round-three screen found nothing).
+Closed (proved no gain): any pair count / split / full-vs-thin mask with fixed profile (#4658 p<=64
+exact, #4659 all p: root < 0.0030217759); independent idle factors (#4664); idle-free binary
+controllers (#4666); paid symmetric phase changes on 4610 word (#4696); typed A/B quotient (#4654
+-> #4660, #4674 surcharge >= 8*7^-C); ambient quotients (#4596/#4629/#4633 closed, #4655);
+centre rank floor 64 under monotone chart retiming (#4700); scalar trace compression no moment
+change (#4692). Active (1337isnot1337, today): centre interface (#4694 640-field, #4698 helper
+normal form) -- the 64-per-local centre charge is the dominant waste.

research/frontier-word-reorder-ceilings/r12-levers/README.md

@@ -0,0 +1,95 @@
+# R12 notes: lever screens on the PR4610 round-three productive-helper supplier (no new kappa)
+
+Base: PR4610 head 9fd59d9b08ec (Cao Yifan), word `research/kappa-round3-warp-20261022/ACTUAL-WORD.json.gz`,
+p16 query-thinned controller (`research/joint-helper-lattice/CONTROLLER.json`), pricer
+`research/kappa-round6-compiler-tightening-20261022/arithmetic.py::assemble`.
+Frontier: kappa 0.0030217758; raw crossing 0.00302177587078 (reproduced in floating point by
+`tools/moment.py`). Tools are in `~/imb-harness/r12/tools/` and run from `~/imb-r12`.
+All bounds below are floating-point screens. None is an exact certificate unless stated.
+
+## 0. Accounting facts used by every screen
+- M(C) = sum T_k k^(1-C) - sum P_r r^(1-C); dM/dC = -247,222 at the root, so a moment change
+  dM gives dC = dM/247,222.
+- Rank-mass surplus is 752 = 112*(2 + 30 windows - 15 thin) - 18*64. The local deficit is
+  1856 = 224*8 (main transverse) + 64 (eight paid rank-8 owned-centre transports). In every
+  admitted chain step (#4470, #4610, #4672) the helper and paid masses move together. All gains
+  since #4470 are concave redistribution at fixed mass.
+- Native semantics (from `kappa-rebuild/complex/native_extensions.py::literal_replay`):
+  - Every paid job is a `climb` of one register to a strictly larger F2^9 chart. Its rank is
+    the dimension jump.
+  - A scalar ADD needs both registers at the identical frame.
+  - Warps are free G-map transports.
+  - Helper credit is final dim minus start dim.
+  - Every one of the 632 registers climbs in 3-9 steps; this fragmentation is the entire
+    concave loss.
+
+## 1. Productive re-ranking (lever 1): dead
+Finishing a rank-k helper as rank 9 adds 9-k rank mass to the targets. The mass invariant
+means it must add at least 9-k paid mass. With #4416 padding semantics (one paid job on the
+missing axes), the change is dM = f(9) - f(k) - f(9-k) < 0 by strict subadditivity of
+x^(1-C).
+- Per local type (x18): 4->9 costs -0.334 (dC -1.35e-6); 7->9 and 8->9 also lose.
+- Paying with rank-one jobs loses more. 8->9 with one rank-one job gives dM = 17.83 - 18 < 0.
+- The opposite move, splitting rank-9 helpers, wins (+1.35e-6 for 9->4+5). That would need
+  a helper whose output is a product of independent fields, which no native form currently
+  provides.
+- A re-ranking gains only if completion costs less paid mass than the rank it adds. That
+  would be a new circuit, not a completion.
+
+## 2. Controller length scaling (lever 2): dead with the fixed local profile
+- #4658 (Cinna Davis): exact, p <= 64, all splits, all native/thin subsets: C < 0.003022.
+- #4659: all p, all binary full/query masks: best raw crossing is in
+  (0.0030217758, 0.0030217759), attained by the existing p16.
+- In closed form (my derivation), surplus(p) = 112(p+1) - 64(p+2) = 48p - 16. The growth of
+  S = sum T k ln k - sum P r ln r outruns it beyond p16.
+
+## 3. Query-thinning of idles and adapters (lever 3): dead or covered
+- Idles are already maximally merged paid jobs of rank 8*len. Replacing an idle of rank 8s by
+  rank-one queries costs 8s*f(1) - f(8s) > 0.
+- Independent idle factors: #4664. Idle-free binary controllers: #4666. Typed A/B quotient
+  query interfaces: #4660 and #4674 (surcharge >= 8*7^-C).
+- Thin stages are already priced by #4659.
+
+## 4. New screens: chart/order ceilings for the redistribution lever
+All of these keep the helper census, endpoint dims and the eight centre jobs.
+
+| relaxation | what is freed | ceiling (raw root) | tool |
+|---|---|---:|---|
+| single climb per register | everything except register masses | 0.0071457 | consolidate.py |
+| keep ADD graph, free order + frames (dims only) | scalar order, warps | 0.0071108 | stab.py |
+| fixed per-register order, decoupled registers | frames, warps, coupling of ADD levels | 0.0063826 | regdp.py |
+| fixed global order, coupled, dims only (free warps) | frames/warps only | LP: local paid cost >= 5297.54 vs 5316.72 current (saving <= 19.18/local, linearized root <= ~0.0044) | dimlp.py |
+| current admitted word | (none) | 0.0030217759 | moment.py |
+
+How to read the table:
+- The first three rows show that the add graph and per-register order constrain little.
+- The binding constraint is the coupling: both ends of an ADD share one frame, under
+  monotone climbing in a common time order.
+- Row 4 is the right relaxation for "new warp gauges / native maps at fixed scalar order".
+  The gap between rows 4 and 5 is what exact-frame rigidity (available G maps) costs.
+- A full MILP on row 4's model (170k binaries, 516k rows; LP via HiGHS IPM without crossover, 624 s; primal objective, so not a certified dual bound) did not beat the current word's
+  cost in 900 s with HiGHS single-thread. Its incumbent local cost was 5318.99, against
+  5316.72 for the current word.
+
+## 5. Other levers considered
+- Centre amortization: merge the 18 sequential centre transports per centre index into one
+  rank-144 job. The fixed-mass ceiling is dM = 1152(8^-C - 144^-C) ~ 10.1, about +4.1e-5.
+  It is illegal: each centre call reads and writes main data inside its local (#4698 shows
+  z_out = z_in + h + R_X X), so the transports neither commute nor sit idle. #4700 proves the
+  64-per-local floor under monotone chart retiming. The centre is the dominant waste: 1152 of
+  surplus against 752 actual, and removing it gives 0.00806 (#4659). It is under active attack
+  by 1337isnot1337 (#4694, #4698, #4700).
+- Query/local fusion: fuse each rank-one query with an adjacent rank-one local climb on the
+  same register, on disjoint axes. The ceiling is 1680(2 - 2^(1-C)) ~ 7.0, about +2.8e-5.
+  Query copies live in an owned temporary Z that is erased, with no common frame with the
+  local's climbs. Not realizable without a new native interface.
+- Compiler-only: the raw crossing is 7.1e-11 above the claimed 0.0030217758, so a further
+  reserve tightening could state about 0.00302177586. This is not a lever, and #4610 round6
+  already marks it as numerical-only.
+
+## Conclusion
+No lever beats 0.0030217758. The quantified headroom is in climb fragmentation (rows 1-4), and
+only a changed scalar circuit or order can reach it, because frame-equal ADD coupling forces
+the fragmentation. The next concrete step is a coupled-relaxation solve restricted to one
+connected block of the add graph, with exact G-map availability, to locate which registers'
+fragmentation is forced by order rather than by available warps.

research/frontier-word-reorder-ceilings/r12-levers/tools/consolidate.py

@@ -0,0 +1,18 @@
+"""Ceiling for any chart/retiming lever that keeps per-register climb mass:
+replace each register's climb chain by one climb of the same total rank."""
+import sys,gzip,json; sys.path.insert(0,str(__import__('pathlib').Path.home()/'imb-harness/r12/tools'))
+from moment import *
+from collections import Counter,defaultdict
+w=json.loads(gzip.decompress(open(ROOT/'research/kappa-round3-warp-20261022/ACTUAL-WORD.json.gz','rb').read()))
+c=w['canonical'];fr=c['frames'];dim=lambda i:len(fr[i])
+helpers={4:28,7:7,8:47,9:326}
+for side in ['forward','backward']:
+    steps=defaultdict(list)
+    for op in w['programs'][side]:
+        if op[0]=='climb': steps[op[2]].append(op[5])
+    nsteps=Counter(len(v) for v in steps.values())
+    one=Counter(sum(v) for v in steps.values()); one[8]+=8  # centres
+    print(side,'chain lengths',dict(sorted(nsteps.items())),'consolidated paid',dict(one))
+    for name,N in [('current',LOCAL['paid']),('consolidated',dict(one))]:
+        H,_,T,P,_=invoice({'helpers':helpers,'paid':N}); print(f'  {name:13s} root {root(T,P):.10f}')
+    # helper-only consolidation (main chains kept) : classify main as registers with total 8 and start dim>=0 not helper — ambiguous; report split