diff --git a/research/direct-flow-leaf-walk/README.md b/research/direct-flow-leaf-walk/README.md new file mode 100644 index 0000000000..02ddde948f --- /dev/null +++ b/research/direct-flow-leaf-walk/README.md @@ -0,0 +1,104 @@ +# The leaf-exit walk of #4436's direct flow: a √G regime now, saturation later + +**Claim type:** exact finite computation of the leaf log-depth up to lockstep depth 60, plus a standard +large-deviation heuristic for its limit. **No new κ.** It explains the growth of the log-gap `g` in my +[#4465](https://github.com/CrocSwap/integer-mult-bounds/pull/4465). That growth comes from the leaf exits +alone, and `g` sets the refined-induction ceiling `s*·g/(g + 1 + ln m)` on zxv's #4436 direct flow. + +## The walk + +A leaf exit at rank `N` is `rρ`, where `ρ` is a product of `G = ⌊log₅(N/m)⌋` letters of `P`, under +`mρr ≤ N` at each step. Its `t*`-potential weight is `cnt(ρ)·P_r·(rρ)^{t*}`. Each step `ln r` is +nonnegative, so only the final constraint binds: `S_G = ln ρ ≤ B = ln(N/m)`. The leaf log-depth +therefore splits as + + D(G) = E[ln(N/(rρ))] = (ln m − E ln r) + E[B − S_G | S_G ≤ B]. + +At level starts `B = G ln 5`. The `t*`-tilted step law has mean `μ = 1.62592`, which is above +`ln 5 = 1.60944` by `ε = 0.01648`, and standard deviation `σ = 1.48391`. + +## The √G regime (exact data) + +`D(G)` is computed exactly from `F[G]`, which lives on {2,3,5}-smooth states, at `N = 121·5^G`. The +verifier covers `G ≤ 40` in about 12 seconds. `evidence/leafwalk.py` extends this to `G = 60`, with +output in `evidence/LEAF-DEPTH-G60.txt`. + +| G | 2 | 5 | 10 | 20 | 30 | 40 | 60 | +| --- | --- | --- | --- | --- | --- | --- | --- | +| D(G) | 5.005 | 5.683 | 6.777 | 8.297 | 9.447 | 10.354 | 11.940 | + +Fits over `7 ≤ G ≤ 40`: + +| law | max residual | +| --- | --- | +| `a + b√G` | **0.054**, with `a = 3.26`, `b = 1.125` | +| linear | 0.38 | +| logarithmic | 0.32 | + +The intercept matches `ln m − μ = 3.17`. The slope is close to the CLT value `σ√(2/π) = 1.18`. +That is the law of `E[B − S | S ≤ B]` while `B` sits inside the CLT window (`εG ≪ σ√G`). + +## Saturation (heuristic) + +Once `G ≫ G× = σ²/ε² ≈ 8,100`, the barrier lies far below the mean. By the exponential change of +measure with Esscher parameter `λ* = 0.0075`, which tilts the step mean to `ln 5`, the overshoot +`B − S_G` given survival tends to `Exp(λ*)`. Hence + + D(G) → ln m − μ + 1/λ* ≈ 136.5. + +So `g` stays bounded and the refined-induction cap stays strictly below `s*`. + +- **Estimate with stable shares.** Keeping the measured shares (leaf about 0.914) and depths (boundary + about 1.84, sources about 2.45) gives `g∞ ≈ 122` and a cap near `0.955·s* ≈ 0.00256`. +- **Lower estimate.** My #4459 bound `β ≤ 0.321` gives a leaf share of at least 0.68, which yields + `g∞ ≳ 88` and a cap of at least about 0.00252. + +Either way, saving 0.0023 is reached at `g ≈ 34.5`. That happens inside the √G regime, near +`G ≈ 1,340`, or `N ≈ 10^{940}`. `diagnostics/SATURATION.md` refines the earlier `10^{700}` and observes the +saturation directly by importance sampling up to `G = 10⁶`: the overshoot reaches 131.7, against `1/λ* = 133.3`. + +## Reading + +- **#4436's direct flow certifies at most roughly 0.0025–0.0026 under #4405's single-step test.** This + heuristic asymptotic ceiling sits strictly below the root `s* = 0.0026865`, because the tilted + P-step mean exceeds `ln k_min`. +- **The only practical gap is the rank.** The selected 0.0023 is certifiable only at astronomically + large rank. +- **What the shortfall comes from.** It is caused by the leaf exits sitting `Θ(1/λ*)` nats below `N`, + not by the boundary exits. + +## Not covered + +- A proof of the saturation, including the lattice and non-Gaussian corrections, and the stability + of the shares at large `G`. +- Any multi-step test. +- Native premises. + +## Reproduce + +```sh +python3.12 -B scripts/research_package.py --package research/direct-flow-leaf-walk --check-only +python3.12 -B scripts/research_package.py --package research/direct-flow-leaf-walk +python3.12 -B research/direct-flow-leaf-walk/evidence/leafwalk.py 60 # ~3 min, the G = 60 table +``` + +The verifier prints `PASS_DIRECT_FLOW_LEAF_WALK`, using only the standard library. + +## Open next questions + +1. **Can the leaf walk be steered to `μ ≤ ln k_min`?** + - *Why it matters:* with zero or negative relative drift, the overshoot would grow without bound, + and the direct cap would tend to `s*`. + - *First step:* check whether a different job order or leaf depth (for example `G + 1` levels) + changes the effective barrier slope. +2. **A rigorous Bahadur–Rao bound for `E[B − S_G | S_G ≤ B]` with explicit constants.** + - *First step:* the three-dimensional {2,3,5} lattice local CLT. + +## Credit + +- The direct flow and the gate are zxv's (#4436). +- The refined induction is 1337isnot1337's (#4405). +- The lockstep construction is Jacob Alan Sussman's Theorem 1G. + +Prepared by Ryanson Jonathan with substantial Claude Code (Claude Opus 5.5) assistance. +Apache-2.0. diff --git a/research/direct-flow-leaf-walk/SOURCES.json b/research/direct-flow-leaf-walk/SOURCES.json new file mode 100644 index 0000000000..9124bb10fe --- /dev/null +++ b/research/direct-flow-leaf-walk/SOURCES.json @@ -0,0 +1,18 @@ +{ + "schema_version": 1, + "base_commit": "2a13eb1827784206202db90e5dc8327cf8a250db", + "files": [ + { + "path": "certificates/selected-result.json", + "usage": "provenance", + "git_blob": "ed33caef5c5827693e9572d252a66f99cbb73d40", + "sha256": "5732707cc1e460b090b53f2463162749ad860de878a18f7408938671d295a9d9" + }, + { + "path": "docs/research/agent-task-catalog.json", + "usage": "provenance", + "git_blob": "64bfc5af6709ed9811171e0a2108f05e1219cb65", + "sha256": "352a6184713b8843d96fd2c2071eb7aeaa655906b7c51d45ad444912666c8a96" + } + ] +} diff --git a/research/direct-flow-leaf-walk/TASK.json b/research/direct-flow-leaf-walk/TASK.json new file mode 100644 index 0000000000..ac1d757ef4 --- /dev/null +++ b/research/direct-flow-leaf-walk/TASK.json @@ -0,0 +1,100 @@ +{ + "schema_version": 1, + "task_id": "independent-moment", + "priority_revision": 7, + "base_commit": "2a13eb1827784206202db90e5dc8327cf8a250db", + "original_card": { + "id": "independent-moment", + "question": "Does a new paid-profile or mixture-ceiling claim have independent source-bound coverage of its actual changed ledger?", + "starting_points": [ + "docs/research/research-family-status.md", + "certificates/selected-result.json", + "research/staged544-mixed-fourth-entry/README.md", + "research/staged-fourth-mixture-ceiling/README.md", + "research/r437-two-window-carrier/README.md" + ], + "methods": [ + "Independently reconstruct one changed paid ledger or pooled dual with exact source and model binding.", + "Check a genuinely uncovered moment implication with a different enclosure method and semantic controls." + ], + "acceptance_gate": "Demonstrate independence, source binding, positive cases and semantic failure controls. Name the existing checks and the missing coverage. More digits or another equivalent evaluation of an unchanged root is not by itself new verification coverage.", + "first_step": { + "command": [ + "python3", + "-B", + "scripts/research_baseline.py" + ], + "expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY", + "inspect": "Choose one uncovered implication in #4088 literal mixed fill, #4091 pooled dual/model binding or #4067 changed frame/host bill. #4091 pins an earlier #4088 head: validate word, capacity and stock equivalence before applying that ceiling to the later fill.", + "scope": "Source-binding orientation only; no scientific replay, theorem acceptance or supplier admission. Budget any heavy replay separately." + }, + "first_experiment": "Independently reconstruct one changed ledger or pooled certificate with a positive case and a semantic failure control. Compare exact sources and name what existing checks miss. Keep this separate from supplier-boundary-audit (ordinary/outer dispatch) and uniform-transfer-audit (all-size proof). Stop after the chosen implication; more digits of an unchanged root are not additional mechanism coverage.", + "bottleneck_link": "Independent pricing prevents unsupported frontier claims and can reveal genuine accounting slack.", + "allowed_variations": [ + "Reconstruct either a source ledger or a numeric enclosure with a different method." + ], + "scope_change_triggers": [ + "Importing the primary function and calling its output independent verification.", + "Calling an aggregate histogram or source archive an independently accepted physical supplier." + ], + "overlap_terms": [ + "independent moment", + "rank histogram", + "exact enclosure", + "2979", + "rank7 joint endframes", + "paired extraction", + "joint lowering", + "4088", + "4091", + "4067", + "staged544-mixed-fourth-entry", + "staged-fourth-mixture-ceiling", + "r437-two-window-carrier" + ], + "existing_checks": [ + { + "scope": "PR #4088: mixed local/owner/phase ledger and moment reported.", + "evidence": "research/staged544-mixed-fourth-entry/README.md", + "remaining": "Independent binding of changed construction to paid histogram, distinct from full admission.", + "status": "author-checked", + "source_commit": "72d7efc181edb21806a878f174847c96dd410ab0", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + }, + { + "scope": "PR #4091: sharp pooled ceiling proposed for fixed THREE/FOUR words and inventory.", + "evidence": "research/staged-fourth-mixture-ceiling/README.md", + "remaining": "Validate dual, exhaustive pattern scope and exact source equivalence before applying to newer fills.", + "status": "author-checked", + "source_commit": "a29bdd8f56cd18b805917775cdec09ff935eddc4", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + }, + { + "scope": "PR #4067: paid (2,2) to (1,3) exchange and fixed OQ4 host reported.", + "evidence": "research/r437-two-window-carrier/README.md", + "remaining": "Independent full cost delta; remaining caller/precision/stock obligations are separate.", + "status": "author-checked", + "source_commit": "a95f11c52e3ca1e4c5171b3c20d8f4b0815aee2e", + "selected_record_sha256": "bb2eaa58647b353f88d0aec2eafb9d30df1be53589231b23e12ceef3e4bad129" + } + ] + }, + "scope_status": "refined", + "actual_scope": "Leaf-exit walk behind #4465's log-gap: exact sqrt(G) regime to G=60 and predicted saturation (Esscher) giving a refined-induction cap strictly below s*.", + "scope_change_reason": "Follow-up to #4465.", + "bottleneck_link": "Bounds what a direct-flow (no tensor/padding) exponent can certify via the refined induction.", + "budget_minutes": 40, + "stop_rule": "Stop at exact data to G=60 and the asymptotic heuristic.", + "outcome": "completed", + "evidence": "D(G) = 3.26 + 1.125 sqrt(G) (max residual 0.054, G=7..40; linear 0.38, log 0.32); mu - ln5 = 0.0165 > 0, lambda* = 0.0075, G_x ~ 8100; predicted saturation D ~ 136.5 and cap ~0.0025-0.00256 < s* = 0.0026865.", + "author": "Ryanson Jonathan, with Claude Code (Claude Opus 5.5) assistance", + "ai_assistance": "Substantial Claude Code (Claude Opus 5.5) assistance; no independent human review.", + "open_questions": [ + { + "question": "Can the leaf walk be steered to tilted mean <= ln k_min?", + "why_it_matters": "Then the overshoot grows without bound and the direct cap tends to s*.", + "first_step": "Test alternative leaf depths/job orders for the effective barrier slope.", + "blocked_by": "None." + } + ] +} diff --git a/research/direct-flow-leaf-walk/VERIFY.json b/research/direct-flow-leaf-walk/VERIFY.json new file mode 100644 index 0000000000..ef13be973c --- /dev/null +++ b/research/direct-flow-leaf-walk/VERIFY.json @@ -0,0 +1,18 @@ +{ + "schema_version": 1, + "status": "ready", + "python": "3.12", + "requirements": [], + "inputs": [], + "commands": [ + { + "argv": [ + "{python}", + "-B", + "{package}/verify.py" + ], + "timeout_seconds": 180, + "checks": "Exact leaf log-depth D(G) of #4436's direct p7 flow at N=121*5^G, G=2..40: sqrt(G) fit residual <0.08 vs linear/log >0.25, slope near sigma*sqrt(2/pi), intercept ln m - mu, concave 10-level increments; tilted step mean > ln 5; Esscher lambda* and crossover G_x reported." + } + ] +} diff --git a/research/direct-flow-leaf-walk/diagnostics/DEPTH-VARIANTS.md b/research/direct-flow-leaf-walk/diagnostics/DEPTH-VARIANTS.md new file mode 100644 index 0000000000..a874789ab7 --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/DEPTH-VARIANTS.md @@ -0,0 +1,43 @@ +# Diagnostic: forcing a shallower lockstep depth + +This answers open question 1 of the README negatively, for the obvious lever. It is not registered +and not part of the verifier. Run it with `python3.12 -B depth_variants.py`. +`overlap_flow_depth.py` is the package's flow with one optional `G` override added. + +| N | G0 | G | exact identity | g | κ(0.001), η = 0.99422 | source mass ≤ N/m | +| --- | --- | --- | --- | --- | --- | --- | +| 10⁵ | 4 | 4 | yes | 2.823 | 0.995257 | 0 | +| 10⁵ | 4 | 3 | yes | 3.220 | 0.994592 | 2.0·10⁻⁶ | +| 10⁵ | 4 | 2 | yes | 3.783 | 0.993648 | 0.296 | +| 2·10⁶ | 6 | 6 | yes | 3.219 | 0.994593 | 0 | +| 2·10⁶ | 6 | 5 | yes | 3.618 | 0.993927 | 1.3·10⁻¹⁰ | +| 2·10⁶ | 6 | 3 | yes | 5.022 | 0.991578 | 0.327 | + +## What the table shows + +- A shallower depth keeps the rank-one potential identity, and the `t*` balance, exactly. It also + raises `g` substantially. +- But it puts sources at or below `N/m`. +- `G0 = ⌊log₅(N/m)⌋` is the only depth with every source in `(N/m, N]`. + +## Open question + +`G0 − 1` violates the source range only by a mass of about `10⁻⁶` to `10⁻¹⁰`, yet gains about 0.4 in +`g`. Can #4405's induction absorb a tiny low-source mass, for example through the induction hypothesis +at smaller rank? If so, a mixed-depth flow could lift the single-step ceiling. That is a question for +#4436 and #4405, and it is not claimed here. + +## Conservative mixed depth (`depth_mixed_scan.py`, about 10 min) + +Low sources (`≤ N/m`) get **no credit**: they are dropped from the denominator. The table gives +`sup κ` over the 5% grid with `3·10⁵ ≤ N < 2.5·10⁶`. + +| saving | η | depth `G0` | depth `G0−1` | per-rank best of the two | +| --- | --- | --- | --- | --- | +| 0.001 | 0.994220 | 0.995037 | 0.995786 | **0.994521** | +| 0.0012 | 0.993068 | 0.995623 | 0.996568 | 0.995199 | +| 0.0015 | 0.991343 | 0.996504 | 0.997743 | 0.996165 | + +Choosing the depth rank by rank helps, but only modestly: about 0.0005 in κ. Under this conservative +reading it does not yet certify 0.001 by `2.5·10⁶`. Depth `G0−1` alone is worse in the supremum, +because it loses near the level changes. diff --git a/research/direct-flow-leaf-walk/diagnostics/SATURATION.md b/research/direct-flow-leaf-walk/diagnostics/SATURATION.md new file mode 100644 index 0000000000..418c78f909 --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/SATURATION.md @@ -0,0 +1,25 @@ +# Diagnostic: the predicted saturation, observed by importance sampling + +The walk is monotone, because every step `ln r ≥ 0`. So the leaf law is exactly the `G`-fold tilted +P-product restricted to `S_G ≤ B`, where `B = G ln 5` at level starts. `saturate.py` (numpy, about +1 minute) samples `S_G` as a multinomial sum at the Esscher tilt whose mean is `B/G`. It reweights by +`e^{λ(S−B)}`, and estimates the conditioned overshoot `E[B − S_G | S_G ≤ B]` from 400,000 samples per +row. + +| G | 50 | 200 | 1,000 | 4,000 | 16,000 | 64,000 | 256,000 | 10⁶ | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| overshoot | 8.01 | 15.54 | 32.01 | 55.16 | 84.73 | 111.53 | 125.74 | 131.66 | + +- **The limit.** The overshoot tends to `1/λ* = 133.35`, as predicted. +- **Consistency with the exact data.** At `G ≈ 50` the exact computation gives + `D − (ln m − μ) ≈ 11.10 − 3.17 = 7.93`, consistent with the sampled 8.01. +- **The crossover.** It sits near `G× ≈ 8,100`, as predicted. + +## Refined estimates + +These keep the measured shares (leaf about 0.914, boundary depth about 1.84, source depth about 2.45). + +- **The limit:** `g∞ ≈ 0.914·(3.17 + 133.35) − 2.29 ≈ 122.5`, so the cap is + `s*·g∞/(g∞ + 5.796) ≈ 0.00257`. +- **Saving 0.0023:** this needs `g ≈ 34.5`, an overshoot of about 37, so `G ≈ 1,340` and + `N ≈ 121·5^{1340} ≈ 10^{940}`. That replaces the README's rougher `10^{700}`. diff --git a/research/direct-flow-leaf-walk/diagnostics/SHARES.md b/research/direct-flow-leaf-walk/diagnostics/SHARES.md new file mode 100644 index 0000000000..4f8de91d49 --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/SHARES.md @@ -0,0 +1,20 @@ +# Diagnostic: share and depth stability through N ≈ 1.2·10⁹ + +This addresses the "stability of the shares" caveat in the README and in `SATURATION.md`. Run it with +`python3.12 -B leaf_share.py 2 9` (about 2 minutes). It uses the package flow, copied as +`overlap_flow.py`, at the start and near the end of each level. + +| G | N | leaf share of exit potential | boundary depth | source depth | +| --- | --- | --- | --- | --- | +| 2 | 3,026 / 14,822 | 0.885 / 0.917 | 1.845 / 1.900 | 2.438 / 2.345 | +| 3 | 15,126 / 74,112 | 0.923 / 0.904 | 1.914 / 1.773 | 2.390 / 2.440 | +| 4 | 75,626 / 370,562 | 0.918 / 0.902 | 1.798 / 1.904 | 2.438 / 2.521 | +| 5 | 378,126 / 1,852,812 | 0.921 / 0.910 | 1.818 / 1.873 | 2.425 / 2.502 | +| 6 | 1,890,626 / 9,264,062 | 0.914 / 0.911 | 1.851 / 1.845 | 2.458 / 2.480 | +| 7 | 9,453,126 / 46,320,312 | 0.915 / 0.906 | 1.830 / 1.864 | 2.453 / 2.490 | +| 8 | 47,265,626 / 231,601,562 | 0.908 / 0.908 | 1.867 / 1.853 | 2.482 / 2.487 | +| 9 | 236,328,126 / 1,158,007,812 | 0.911 / 0.910 | 1.861 / 1.839 | 2.483 / 2.474 | + +The leaf share, the boundary-exit depth and the source depth all settle, at about 0.91, 1.85 and 2.48, +while the leaf depth keeps growing (README). This supports the stable-share assumption behind the +asymptotic cap of about 0.00257. It is finite evidence, not a proof. diff --git a/research/direct-flow-leaf-walk/diagnostics/depth_mixed_scan.py b/research/direct-flow-leaf-walk/diagnostics/depth_mixed_scan.py new file mode 100644 index 0000000000..962f48bcad --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/depth_mixed_scan.py @@ -0,0 +1,23 @@ +exec(open('overlap_flow_depth.py').read()) # run from this directory +W={5:420,6:270,7:840,8:1395,9:3195,25:448,41:448,57:448,121:224} +P={1:37305,2:10695,3:3360,4:1530,5:300,8:1464,16:448,32:448,48:448} +m=121 +sav=(0.001,0.0012,0.0015) +worst={(d,s):0 for d in (0,1,2) for s in sav} +for N in [int(1000*1.05**i) for i in range(0,200) if 1000*1.05**i<2.5e6]: + G0=0 + while 5**(G0+1)<=N//m: G0+=1 + ks={} + for d in (0,1): + if G0-d<1: continue + R=flow(N,W,P,G0-d); S,X=R['S'],R['X'] + for s in sav: + z=1-s + ks[(d,s)]=sum(v*e**z for e,v in X.items())/sum(v*x**z for x,v in S.items() if m*x>N) + if N>=3e5: + for s in sav: + worst[(0,s)]=max(worst[(0,s)],ks[(0,s)]); worst[(1,s)]=max(worst[(1,s)],ks[(1,s)]) + worst[(2,s)]=max(worst[(2,s)],min(ks[(0,s)],ks[(1,s)])) +for s in sav: + eta=(1-s)*121**(-s) + print(f"saving {s}: eta {eta:.6f} | depth G0: sup kappa(N>=3e5) {worst[(0,s)]:.6f} | depth G0-1, low sources dropped: {worst[(1,s)]:.6f} | per-N best of the two: {worst[(2,s)]:.6f}") diff --git a/research/direct-flow-leaf-walk/diagnostics/depth_variants.py b/research/direct-flow-leaf-walk/diagnostics/depth_variants.py new file mode 100644 index 0000000000..669f551c22 --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/depth_variants.py @@ -0,0 +1,20 @@ +# Diagnostic (not registered): #4436's direct flow with a forced lockstep depth G <= G0 = floor(log5(N/m)). +# Run from this directory: python3.12 -B depth_variants.py +import math +exec(open('overlap_flow_depth.py').read()) +W = {5: 420, 6: 270, 7: 840, 8: 1395, 9: 3195, 25: 448, 41: 448, 57: 448, 121: 224} +P = {1: 37305, 2: 10695, 3: 3360, 4: 1530, 5: 300, 8: 1464, 16: 448, 32: 448, 48: 448} +m, ts = 121, 0.9973135109511491 +for N in (20000, 100000, 400000, 2000000): + G0 = 0 + while 5 ** (G0 + 1) <= N // m: + G0 += 1 + for G in range(max(0, G0 - 3), G0 + 1): + R = flow(N, W, P, G) + S, X = R['S'], R['X'] + ident = sum(s * v for s, v in S.items()) - sum(e * v for e, v in X.items()) == 608 * R['col'] + st = sum(v * s ** ts for s, v in S.items()); xt = sum(v * e ** ts for e, v in X.items()) + g = sum(v * s ** ts * math.log(s) for s, v in S.items()) / st - sum(v * e ** ts * math.log(e) for e, v in X.items()) / xt + k = sum(v * e ** 0.999 for e, v in X.items()) / sum(v * s ** 0.999 for s, v in S.items()) + low = sum(v for s, v in S.items() if m * s <= N) / sum(S.values()) + print(N, G0, G, ident, f"g={g:.4f} kappa(0.001)={k:.6f} (eta 0.994220) source mass <= N/m: {low:.3g}") diff --git a/research/direct-flow-leaf-walk/diagnostics/leaf_share.py b/research/direct-flow-leaf-walk/diagnostics/leaf_share.py new file mode 100644 index 0000000000..e8d9c88f6d --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/leaf_share.py @@ -0,0 +1,15 @@ +# Diagnostic (not registered): leaf share and depths at level starts/ends. Run from this directory: +# python3.12 -B leaf_share.py 2 9 (about 2 minutes) +import math, time +exec(open('overlap_flow.py').read()) +W={5:420,6:270,7:840,8:1395,9:3195,25:448,41:448,57:448,121:224} +P={1:37305,2:10695,3:3360,4:1530,5:300,8:1464,16:448,32:448,48:448} +ts=0.9973135109511491; t0=time.time() +import sys +for G in range(int(sys.argv[1]), int(sys.argv[2]) + 1): + for N in (121*5**G+1, 121*5**G*49//10): + R=flow(N,W,P); X=R['X']; BX=R['BX']; S=R['S'] + xt=sum(v*e**ts for e,v in X.items()); bt=sum(v*e**ts for e,v in BX.items()) + srcd=sum(v*s**ts*math.log(N/s) for s,v in S.items())/sum(v*s**ts for s,v in S.items()) + bdd=sum(v*e**ts*math.log(N/e) for e,v in BX.items() if v)/bt + print(G,N,f"leaf share {1-bt/xt:.4f} bd depth {bdd:.4f} src depth {srcd:.4f}",f"{time.time()-t0:.0f}s",flush=True) diff --git a/research/direct-flow-leaf-walk/diagnostics/overlap_flow.py b/research/direct-flow-leaf-walk/diagnostics/overlap_flow.py new file mode 100644 index 0000000000..60c1997786 --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/overlap_flow.py @@ -0,0 +1,59 @@ +from fractions import Fraction as Q +from math import factorial +import itertools, json, sys +def multis(n_types, size): + if n_types == 0: + if size == 0: yield () + return + for x in range(size + 1): + for rest in multis(n_types - 1, size - x): yield (x,) + rest +def flow(N, W, P): + ks = sorted(W); m = ks[-1]; kmin = ks[0] + def c(a): + v = 1 + for k, x in zip(ks, a): v *= k ** x + return v + def u(a): + v = factorial(sum(a)) + for k, x in zip(ks, a): v = v // factorial(x) * W[k] ** x + return v + G = 0 + while kmin ** (G + 1) <= N // m: G += 1 + F = [{1: 1}] + for d in range(1, G + 1): + nxt = {} + for rho, cnt in F[d - 1].items(): + for r, pr in P.items(): + if m * rho * r <= N: nxt[rho * r] = nxt.get(rho * r, 0) + cnt * pr + F.append(nxt) + S, X, BX = {}, {}, {} + col = units = matched = 0 + add = lambda D, k, v: D.__setitem__(k, D.get(k, 0) + v) + present = lambda rho, a: m * rho * c(a) <= N + for d in range(G + 1): + for rho, cnt in F[d].items(): + for b in multis(len(ks), G - d): + if not present(rho, b): continue + units += cnt * u(b); col += cnt * u(b) * rho * c(b) + for r, pr in P.items(): + issued = cnt * pr * u(b); rank = r * rho * c(b) + if d == G: + add(X, rank, issued); continue + D = sum(W[k] * u(tuple(x - (i == j) for j, x in enumerate(b))) + for i, k in enumerate(ks) if b[i] > 0 and present(rho * r, tuple(x - (i == j) for j, x in enumerate(b)))) + matched += cnt * pr * D + if issued - cnt * pr * D: + add(X, rank, issued - cnt * pr * D); add(BX, rank, issued - cnt * pr * D) + for a in multis(len(ks), G): + if present(1, a): + for k in ks: add(S, k * c(a), W[k] * u(a)) + for d in range(1, G + 1): + for rho, cnt in F[d - 1].items(): + for r, pr in P.items(): + for a in multis(len(ks), G - d): + if not present(rho * r, a): continue + for i, k in enumerate(ks): + b = list(a); b[i] += 1 + if not present(rho, tuple(b)): + add(S, rho * r * c(tuple(b)), cnt * pr * u(a) * W[k]) + return dict(G=G, S=S, X=X, BX=BX, col=col, units=units, matched=matched) diff --git a/research/direct-flow-leaf-walk/diagnostics/overlap_flow_depth.py b/research/direct-flow-leaf-walk/diagnostics/overlap_flow_depth.py new file mode 100644 index 0000000000..3bed7b0bea --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/overlap_flow_depth.py @@ -0,0 +1,60 @@ +from fractions import Fraction as Q +from math import factorial +import itertools, json, sys +def multis(n_types, size): + if n_types == 0: + if size == 0: yield () + return + for x in range(size + 1): + for rest in multis(n_types - 1, size - x): yield (x,) + rest +def flow(N, W, P, Gover=None): + ks = sorted(W); m = ks[-1]; kmin = ks[0] + def c(a): + v = 1 + for k, x in zip(ks, a): v *= k ** x + return v + def u(a): + v = factorial(sum(a)) + for k, x in zip(ks, a): v = v // factorial(x) * W[k] ** x + return v + G = 0 + while kmin ** (G + 1) <= N // m: G += 1 + if Gover is not None: G = Gover + F = [{1: 1}] + for d in range(1, G + 1): + nxt = {} + for rho, cnt in F[d - 1].items(): + for r, pr in P.items(): + if m * rho * r <= N: nxt[rho * r] = nxt.get(rho * r, 0) + cnt * pr + F.append(nxt) + S, X, BX = {}, {}, {} + col = units = matched = 0 + add = lambda D, k, v: D.__setitem__(k, D.get(k, 0) + v) + present = lambda rho, a: m * rho * c(a) <= N + for d in range(G + 1): + for rho, cnt in F[d].items(): + for b in multis(len(ks), G - d): + if not present(rho, b): continue + units += cnt * u(b); col += cnt * u(b) * rho * c(b) + for r, pr in P.items(): + issued = cnt * pr * u(b); rank = r * rho * c(b) + if d == G: + add(X, rank, issued); continue + D = sum(W[k] * u(tuple(x - (i == j) for j, x in enumerate(b))) + for i, k in enumerate(ks) if b[i] > 0 and present(rho * r, tuple(x - (i == j) for j, x in enumerate(b)))) + matched += cnt * pr * D + if issued - cnt * pr * D: + add(X, rank, issued - cnt * pr * D); add(BX, rank, issued - cnt * pr * D) + for a in multis(len(ks), G): + if present(1, a): + for k in ks: add(S, k * c(a), W[k] * u(a)) + for d in range(1, G + 1): + for rho, cnt in F[d - 1].items(): + for r, pr in P.items(): + for a in multis(len(ks), G - d): + if not present(rho * r, a): continue + for i, k in enumerate(ks): + b = list(a); b[i] += 1 + if not present(rho, tuple(b)): + add(S, rho * r * c(tuple(b)), cnt * pr * u(a) * W[k]) + return dict(G=G, S=S, X=X, BX=BX, col=col, units=units, matched=matched) diff --git a/research/direct-flow-leaf-walk/diagnostics/saturate.py b/research/direct-flow-leaf-walk/diagnostics/saturate.py new file mode 100644 index 0000000000..1dc6bbed6c --- /dev/null +++ b/research/direct-flow-leaf-walk/diagnostics/saturate.py @@ -0,0 +1,26 @@ +import numpy as np, math +P={1:37305,2:10695,3:3360,4:1530,5:300,8:1464,16:448,32:448,48:448} +ts=0.9973135109511491 +rs=np.array(sorted(P)); lr=np.log(rs); base=np.array([P[r]*r**ts for r in rs],float) +def law(lam): + w=base*np.exp(-lam*lr); return w/w.sum() +def mean(lam): return float((law(lam)*lr).sum()) +ln5=math.log(5) +rng=np.random.default_rng(7) +print("G overshoot_estimate (B = G ln5)") +for G in (50,200,1000,4000,16000,64000,256000,1000000): + B=G*ln5 + # choose tilt so the tilted mean equals B/G (Esscher); exact reweighting factor exp(lam*S) up to a constant + lo,hi=0.0,1.0 + for _ in range(80): + mid=(lo+hi)/2 + if mean(mid)>B/G: lo=mid + else: hi=mid + lam=(lo+hi)/2; q=law(lam) + n=rng.multinomial(G,q,size=400000) + S=n@lr + keep=S<=B + w=np.exp(lam*(S[keep]-B)) # original/tilted density ratio up to a constant + est=float(((B-S[keep])*w).sum()/w.sum()) + print(G, round(est,3), "lambda",round(lam,5), "kept", keep.mean().round(3), flush=True) +print("1/lambda* =", 1/0.007499128505171482) diff --git a/research/direct-flow-leaf-walk/evidence/LEAF-DEPTH-G60.txt b/research/direct-flow-leaf-walk/evidence/LEAF-DEPTH-G60.txt new file mode 100644 index 0000000000..342b57de04 --- /dev/null +++ b/research/direct-flow-leaf-walk/evidence/LEAF-DEPTH-G60.txt @@ -0,0 +1,59 @@ +2 start 5.0054 end 5.5632 states 15 0s +3 start 5.1886 end 5.9708 states 33 0s +4 start 5.4929 end 6.3133 states 64 0s +5 start 5.6833 end 6.4766 states 110 0s +6 start 6.0081 end 6.6652 states 175 0s +7 start 6.1907 end 6.9586 states 261 0s +8 start 6.4957 end 7.1078 states 371 0s +9 start 6.6320 end 7.2605 states 508 0s +10 start 6.7770 end 7.4914 states 676 0s +11 start 7.0118 end 7.6275 states 878 0s +12 start 7.1432 end 7.8554 states 1117 0s +13 start 7.2903 end 7.9756 states 1396 0s +14 start 7.4771 end 8.0980 states 1716 0s +15 start 7.6069 end 8.2775 states 2082 0s +16 start 7.7888 end 8.4006 states 2497 0s +17 start 7.9102 end 8.5215 states 2965 0s +18 start 8.0278 end 8.6815 states 3488 0s +19 start 8.1635 end 8.7966 states 4069 1s +20 start 8.2965 end 8.9434 states 4711 1s +21 start 8.4073 end 9.0598 states 5417 1s +22 start 8.5548 end 9.1617 states 6190 1s +23 start 8.6638 end 9.2760 states 7034 2s +24 start 8.7787 end 9.4001 states 7951 2s +25 start 8.9024 end 9.4972 states 8945 2s +26 start 8.9983 end 9.6324 states 10018 3s +27 start 9.0887 end 9.7310 states 11173 4s +28 start 9.2175 end 9.8313 states 12414 4s +29 start 9.3297 end 9.9462 states 13743 5s +30 start 9.4470 end 10.0339 states 15164 6s +31 start 9.5338 end 10.1164 states 16680 7s +32 start 9.6151 end 10.2342 states 18294 8s +33 start 9.7299 end 10.3366 states 20009 10s +34 start 9.8152 end 10.4458 states 21828 11s +35 start 9.8869 end 10.5275 states 23754 13s +36 start 10.0038 end 10.6031 states 25790 15s +37 start 10.0964 end 10.7072 states 27938 17s +38 start 10.2047 end 10.7884 states 30204 19s +39 start 10.2860 end 10.9013 states 32589 22s +40 start 10.3536 end 10.9848 states 35096 24s +41 start 10.4516 end 11.0565 states 37728 28s +42 start 10.5335 end 11.1518 states 40488 31s +43 start 10.6421 end 11.2293 states 43379 35s +44 start 10.7252 end 11.2943 states 46406 39s +45 start 10.7906 end 11.3831 states 49570 44s +46 start 10.8795 end 11.4620 states 52875 49s +47 start 10.9579 end 11.5625 states 56323 54s +48 start 11.0199 end 11.6430 states 59917 60s +49 start 11.1007 end 11.7065 states 63663 67s +50 start 11.1809 end 11.7879 states 67562 74s +51 start 11.2766 end 11.8632 states 71616 81s +52 start 11.3578 end 11.9239 states 75829 90s +53 start 11.4193 end 12.0154 states 80204 99s +54 start 11.4935 end 12.0913 states 84744 108s +55 start 11.5690 end 12.1714 states 89453 119s +56 start 11.6293 end 12.2449 states 94333 130s +57 start 11.7191 end 12.3039 states 99387 142s +58 start 11.7928 end 12.3725 states 104618 155s +59 start 11.8667 end 12.4446 states 110029 170s +60 start 11.9400 end 12.5038 states 115624 185s diff --git a/research/direct-flow-leaf-walk/evidence/leafwalk.py b/research/direct-flow-leaf-walk/evidence/leafwalk.py new file mode 100644 index 0000000000..7485ee36e6 --- /dev/null +++ b/research/direct-flow-leaf-walk/evidence/leafwalk.py @@ -0,0 +1,26 @@ +import math, time, sys +P={1:37305,2:10695,3:3360,4:1530,5:300,8:1464,16:448,32:448,48:448} +m=121; ts=0.9973135109511491 +def leaf_depth(N): + # F[d]: rho -> cnt, with m*rho*r <= N at each step; G = floor(log5(N/m)) steps + G=0 + while 5**(G+1) <= N//m: G+=1 + F={1:1.0}; scale=0.0 + for d in range(G): + nxt={} + for rho,c in F.items(): + for r,pr in P.items(): + if m*rho*r<=N: nxt[rho*r]=nxt.get(rho*r,0.0)+c*pr + s=sum(nxt.values()); scale+=math.log(s) + F={k:v/s for k,v in nxt.items()} + num=den=0.0 + for rho,c in F.items(): + for r,pr in P.items(): + w=c*pr*(r*rho)**ts + num+=w*math.log(N/(r*rho)); den+=w + return G, num/den, len(F) +t0=time.time() +for G in range(2,int(sys.argv[1])+1): + lo=121*5**G; hi=121*5**G*49//10 + a=leaf_depth(lo); b=leaf_depth(hi) + print(G, f"start {a[1]:.4f} end {b[1]:.4f} states {a[2]}", f"{time.time()-t0:.0f}s", flush=True) diff --git a/research/direct-flow-leaf-walk/verify.py b/research/direct-flow-leaf-walk/verify.py new file mode 100644 index 0000000000..42440e630a --- /dev/null +++ b/research/direct-flow-leaf-walk/verify.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""The leaf-exit walk of #4436's direct p7 flow: sqrt(G) regime now, predicted saturation later. + +A leaf exit of the direct flow at rank N is r*rho, rho a product of G = floor(log5(N/m)) letters of P with +m*rho*r <= N at each step; its t*-potential weight is cnt(rho) P_r (r rho)^t*. Since every step ln r >= 0, only the +final constraint binds: S_G = ln rho <= B = ln(N/m). This verifier computes the leaf log-depth + D(G) = E[ln(N / (r rho))] (t*-potential weights) at N = 121 * 5^G, exactly from F[G], for G = 2..40, +and checks: D fits a + b sqrt(G) (max residual < 0.08, b close to sigma sqrt(2/pi)) far better than a linear or a +logarithmic law; increments are decreasing. It also computes the tilted step law: mean mu > ln 5, sigma, the Esscher +parameter lambda* with tilted mean ln 5, and the crossover G_x = sigma^2 / (mu - ln 5)^2. For G >> G_x the conditioned +overshoot B - S_G tends to Exp(lambda*), so D saturates near ln m - E[ln r] + 1/lambda*: the log-gap g of #4465 then +stays bounded and the refined-induction cap stays strictly below s*. Floating point; the saturation is an asymptotic +heuristic, not a proof. Ryanson Jonathan with Claude Code (Claude Opus 5.5) assistance. Apache-2.0. +""" +import json +import math +import sys + +P = {1: 37305, 2: 10695, 3: 3360, 4: 1530, 5: 300, 8: 1464, 16: 448, 32: 448, 48: 448} +W = {5: 420, 6: 270, 7: 840, 8: 1395, 9: 3195, 25: 448, 41: 448, 57: 448, 121: 224} +M = 121 + + +def need(condition, message): + if not condition: + raise AssertionError(message) + + +def root(): + f = lambda t: sum(W[k] * k ** t for k in W) - sum(P[r] * r ** t for r in P) + lo, hi = 0.99, 1.0 + for _ in range(80): + mid = (lo + hi) / 2 + lo, hi = (lo, mid) if f(mid) > 0 else (mid, hi) + return (lo + hi) / 2 + + +def leaf_depth(N, ts): + G = 0 + while 5 ** (G + 1) <= N // M: + G += 1 + F = {1: 1.0} + for _ in range(G): + nxt = {} + for rho, c in F.items(): + for r, pr in P.items(): + if M * rho * r <= N: + nxt[rho * r] = nxt.get(rho * r, 0.0) + c * pr + s = sum(nxt.values()) + F = {k: v / s for k, v in nxt.items()} + num = den = 0.0 + for rho, c in F.items(): + for r, pr in P.items(): + w = c * pr * (r * rho) ** ts + num += w * math.log(N / (r * rho)) + den += w + return G, num / den + + +def fit(xs, ys): + n = len(xs) + mx, my = sum(xs) / n, sum(ys) / n + b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs) + a = my - b * mx + return a, b, max(abs(y - a - b * x) for x, y in zip(xs, ys)) + + +def tilted(s): + w = {r: P[r] * r ** s for r in P} + z = sum(w.values()) + m1 = sum(w[r] * math.log(r) for r in P) / z + return m1, sum(w[r] * math.log(r) ** 2 for r in P) / z - m1 * m1 + + +def main(): + ts = root() + D = {} + for G in range(2, 41): + g, d = leaf_depth(M * 5 ** G, ts) + need(g == G, "level") + D[G] = d + sel = [G for G in D if G >= 7] + a, b, res = fit([math.sqrt(G) for G in sel], [D[G] for G in sel]) + _, _, res_lin = fit(sel, [D[G] for G in sel]) + _, _, res_log = fit([math.log(G) for G in sel], [D[G] for G in sel]) + mu, var = tilted(ts) + sigma = math.sqrt(var) + need(res < 0.08 and res_lin > 0.25 and res_log > 0.25, "sqrt(G) law fits far better than linear or logarithmic") + need(abs(b - sigma * math.sqrt(2 / math.pi)) < 0.15, "slope close to the CLT value sigma sqrt(2/pi)") + inc = [D[G + 10] - D[G] for G in range(10, 31, 10)] + need(all(x > y for x, y in zip(inc, inc[1:])), "increments decrease (concave growth)") + need(mu > math.log(5), "tilted mean exceeds the barrier slope ln 5") + lo, hi = 0.0, 1.0 + for _ in range(100): + mid = (lo + hi) / 2 + lo, hi = (mid, hi) if tilted(ts - mid)[0] > math.log(5) else (lo, mid) + lam = (lo + hi) / 2 + offset = math.log(M) - mu + need(abs(offset - a) < 0.3, "fitted intercept matches ln m - E[ln r]") + print(json.dumps({"status": "PASS_DIRECT_FLOW_LEAF_WALK", "t_star": f"{ts:.12f}", + "leaf_depth": {G: round(D[G], 4) for G in (2, 5, 10, 20, 30, 40)}, + "sqrt_fit": {"a": round(a, 4), "b": round(b, 4), "max_residual": round(res, 4)}, + "linear_fit_max_residual": round(res_lin, 4), "log_fit_max_residual": round(res_log, 4), + "tilted_step": {"mu": round(mu, 6), "sigma": round(sigma, 6), "mu_minus_ln5": round(mu - math.log(5), 6)}, + "esscher_lambda": round(lam, 6), "saturation_depth_estimate": round(offset + 1 / lam, 2), + "crossover_G": round(var / (mu - math.log(5)) ** 2), "new_kappa_claimed": False}, indent=1)) + + +if __name__ == "__main__": + sys.exit(main())