diff --git a/research/x54-mixed-t37-saturation/README.md b/research/x54-mixed-t37-saturation/README.md new file mode 100644 index 0000000000..a0a20fa967 --- /dev/null +++ b/research/x54-mixed-t37-saturation/README.md @@ -0,0 +1,155 @@ +# Saturation of the X54 mixed-T37 construction (#1193/#1218) + +**No new κ.** This is a scoped negative result plus a screening tool for the highest open claim, +κ = 0.001675809683500501 (#1193, #1218, #1239). It asks whether that construction can still be improved by +(a) more width-neutral local moves on its two final words, or (b) a better packing of the same helpers. +Within the families and the model below, the answer is no. It also gives exact prices for helper-width changes, +so later inventory proposals can be screened before anyone builds them. + +> **Superseded in part during this session.** Gains appeared after the base snapshot (21:03Z), and both escape +> the scope stated here: +> - #1274 found a local gain on the same words outside the #1186 move families. It replaces the pivot-531 cold +> kernel with a helper-donor kernel, giving H1 −3, H3 +1. +> - #1252 (Gonzalo-Mier), applied in #1268 and #1280, repartitions completely idle rank-48 children +> (48+48 → 45+51). +> +> So statement 2 below is about the **bin-remainder model**, where an idle child is a bin's unfilled remainder. +> Idle-axis exchanges after packing are not covered by it. The sweep tallies, the Φ invariant on occupied bins +> and the width/Φ prices remain valid for what they state. The highest open claim is now +> 0.001676377648122875 (#1280). + +## Inputs + +These are vendored byte-for-byte from open PRs and pinned in `verify.py`: + +| File | Source | +|---|---| +| `inputs/D3-WORD.json.gz` | #1193 `vendor/pr1207/BASE-X54-WORD.json.gz`: #1148 joint entrances + #1195 fusions + X54 (125 calls) | +| `inputs/D4-WORD.json.gz` | #1193 `data/FOUR-WORD.json.gz`: D3 + entrance group 627 (208 calls) | +| `inputs/FLOW.json.gz`, `inputs/MIX.json` | #1193's frozen integer pattern flow and call dispatch | + +The #1193 head is `4e392e90`; #1218 at `0aa14797` reuses the same words, flow and dispatch. The T37 bin +capacities are recomputed from files already on main: the retained #1042 replay archive and PR858/PR870's +unchanged `actual_windows`. + +## Results + +### 1. No local gain is left in the toolkit's move families + +I used the #1186 toolkit at head `9254a3cc`, unchanged, with exact frame pricing. Both final words were swept. +#1195 had swept only the pre-slot and seven-slot words, before X54 and the joint entrances. + +| Family | Candidates per word | Feasible | Gains (D3 / D4) | +|---|---:|---:|---:| +| Read fusion + single/pair swap repairs + full exchange (`repair.py`) | 2,238 | 969 | 0 / 0 | +| Read-set exchange, every subset of swappable reads, 14 copy pairs | 50 | 0 | 0 / 0 | +| Joint triangle-hybrid, ≤3 fused reads, partial swaps, both restore modes | 161 | 24 | 0 / 0 | + +The baseline frames are also exactly optimal: the frame-solver optimum equals the published profile. The +per-candidate results are in `evidence/sweeps/`. + +### 2. The incumbent idle profile is optimal for its mix in the bin-remainder model (exact) + +`packing.py` uses standard-library integer arithmetic. It recounts every helper width from the two words and +checks that the flow hosts all 159,840 helpers exactly once. The bins hold `{8:24864, 16:16576, 32:16576, 48:16576}` +and the idle remainders are `{1:2, 48:11374}`, with mass 545,954. + +**Width-class invariant.** Let φ(4) = φ(8) = 0, φ(5) = 3, φ(7) = 1 and φ(9) = −1. Enumerating every multiset +of widths gives these bounds: +- a bin filled exactly to capacity 8, 16 or 32 has Φ ≥ 0; +- a bin filled exactly to capacity 48 has Φ ≥ −4. + +Φ is additive, so its total is fixed by the helper inventory: −20,814 for 125 D3 + 208 D4. + +**Consequence.** The only cheaper idle histogram of the same mass is `{2:1, 48:11374}`. A histogram with fewer +rank-48 parts is dearer by Karamata; this is checked for every j up to 11,374 at #1193's root endpoint. +Realizing `{2:1, 48:11374}` would need exactly 5,202 occupied capacity-48 bins, one remainder-two bin and every +other bin full. Then Φ ≥ −20,810 > −20,814, a contradiction. + +The incumbent is tight. Every occupied capacity-48 bin attains −4, and the two `4+9+9+9` bins give −3 each. +**So the two leftover rank-one idle units cannot be merged into one rank-two child.** That would have been the +only remaining helper-packing gain, about +3.3·10⁻¹⁰ in κ, for this mix. This holds even with time windows, +ports and family rules ignored. It does not hold once the idle children themselves are reshaped (#1252). + +### 3. The mix is optimal among demotion counts (numerical screening) + +`aggregate.py` (SciPy/HiGHS, floating point) lets every call use k ∈ [−3, 3] extra four-helper entrance groups. +Each group is taken to change the profile by `−8H1+8H2+4H3−4H4` and four widths by 4→5. Only k = 0, 1 are +admitted words; the others are a linear extrapolation. The model ranges over all 1,163 bin patterns and relaxes +time windows. Its integer optimum equals the incumbent objective exactly: 208 groups in total. The fractional LP +gains only half a group, which is not integral, worth ≈2.5·10⁻¹⁰ in κ. + +The LP duals have an exact two-parameter structure, π_w = −λ·w − μ·φ(w), with λ = 11.731 per unit of helper +width and μ = 25.041 per unit of Φ. The units are 10⁶ × moment × W, at the #1193 root. + +### Screening rule for a proposed local change + +Take a per-call change with paid-profile delta Δn, total helper-width delta ΔWid and Φ delta ΔΦ. Its marginal +score is + +``` +score = Σ_r Δn_r · 1e6·((r/73)^(1-a) − r/73) − 11.731·ΔWid − 25.041·ΔΦ +``` + +summed over the calls that adopt it, with a = 0.0016786227. A negative score is a candidate gain. +Δκ ≈ −1.028·10⁻¹¹ × score. The prices are marginal LP duals. An integral packing must still exist, and there is +almost no slack (2 units), so rerun the ILP with the new variant and then admit it natively. + +Examples: +- An extra entrance group scores ≈ −1, which is break-even. That is why #1193 stops where it does. +- Shrinking one 9-wide helper to 8 earns 13.3. Turning a 9 into a 7 earns 26.6. +- Turning a 4 into a 5 earns 86.9, but growing 7→8 costs 13.3. + +## Reproduce + +```sh +python3 -B research/x54-mixed-t37-saturation/verify.py # ~2 s, stdlib +python3 -B research/x54-mixed-t37-saturation/aggregate.py --out /tmp/agg.json # ~25 s, SciPy +python3 -B research/x54-mixed-t37-saturation/sweep.py --toolkit PR1186/research/exact-frame-toolkit \ + --word D3 --out /tmp/sweeps --workers 6 # ~10 min per word +``` + +`verify.py` does the following: +- checks the input pins; +- replays the exact packing statement against `evidence/PACKING.json`; +- rejects five controls: a changed call mix, a changed helper width, a non-conserving pattern, a dropped bin + and non-invariant φ weights; +- recounts the stored sweep tallies. + +It does not rerun the sweeps or the ILP. + +## Scope and limitations + +- The sweeps cover the #1186 move families as implemented. #1274's helper-donor kernel shows that gains do exist + outside them, with their default limits: `repair.py` tries at most + 40 repairs per fusion. They do not cover other delivery rewrites, entrance or end changes, or different + helper interfaces. A failed finite search is not an impossibility theorem. +- The packing statements relax time windows, ports and per-call family rules, so they are upper bounds on what + any helper packing could achieve when an idle child is a bin's remainder. Repartitioning idle axes, as in + #1252, is outside that model. They assume the retained T37 bin geometry (capacities ≤ 48) and the {4,5,7,8,9} + width alphabet. +- All inherited conditions of the incumbent remain conditions here: all-size compiler, common-label and + mutable-source realization, whole-call hosting, precision and tape, exact recovery, analytic transfer, and + bit H1–H5. + +## Open next questions + +1. **Can a helper-width change pass the screening rule?** For example, a 9→8 end at local cost below 13.3 units, + or a 4→5 entrance change below 86.9 per helper. *Why:* the packing is saturated, so only a changed inventory + or local word can move κ. *First step:* price candidate single-helper entrance and end changes with the + #1186 `harness.Pricer`, then score them with the rule above. *Blocked by:* there is no generator for + single-helper interface changes; #1176 found that entrance demotions need whole four-helper groups. +2. **How do the width/Φ prices move once idle children are reshaped?** #1252/#1280 turn idle 48s into 45/51. + *Why:* those exchanges change the marginal value of idle mass, and so λ, and with it the break-even price of + an entrance group. The mix may then want more or fewer than 208 groups. *First step:* rerun `aggregate.py` + with an idle-cost function in which empty capacity-48 bins can pair into 45+51 at #1280's achieved rate. + *Blocked by:* that rate depends on #1252's per-(replica, port) commuting-axis condition, which is not + modelled here. + +## Credit + +The #1193 global allocation is Jeremy Diviney's; #1218 is Rohan Arun's composition. The words come from +Thomas DiFiore's #1148, Marius-Paul Dumitrean's #1195 fusions, Jeremy Kothe's #1130 X54 and #1196/#1207's +composition. The search toolkit is #1186, by Marius-Paul Dumitrean. The windows are from Hao Yan's PR858/PR870 +geometry. The demotion-group lead is #1111/#1176. #1240's packing-ceiling note motivated the idle-rank +question. Prepared by Wronskode with substantial AI assistance (Anthropic Claude, via Claude Code). Apache-2.0. diff --git a/research/x54-mixed-t37-saturation/RESULT.json b/research/x54-mixed-t37-saturation/RESULT.json new file mode 100644 index 0000000000..ccf65d48f9 --- /dev/null +++ b/research/x54-mixed-t37-saturation/RESULT.json @@ -0,0 +1,54 @@ +{ + "incumbent": { + "heads": { + "1193": "4e392e90e3fca8f395e35d6a23644920160a9f0f", + "1218": "0aa147977ad226f9d777d6a21a610ab037985b1e" + }, + "kappa": "1675809683500501/1000000000000000000", + "prs": [ + 1193, + 1218 + ] + }, + "status": "NO_NEW_KAPPA_SCOPED_SATURATION", + "sweeps": { + "D3": { + "exchange": { + "candidates": 50, + "feasible": 0, + "gains": 0 + }, + "joint": { + "candidates": 161, + "feasible": 24, + "gains": 0 + }, + "repair": { + "candidates": 2238, + "feasible": 969, + "gains": 0 + } + }, + "D4": { + "exchange": { + "candidates": 50, + "feasible": 0, + "gains": 0 + }, + "joint": { + "candidates": 161, + "feasible": 24, + "gains": 0 + }, + "repair": { + "candidates": 2238, + "feasible": 969, + "gains": 0 + } + } + }, + "toolkit": { + "head": "9254a3cc610a72e9c4600da089d5cdbcab996b9c", + "pr": 1186 + } +} diff --git a/research/x54-mixed-t37-saturation/SOURCES.json b/research/x54-mixed-t37-saturation/SOURCES.json new file mode 100644 index 0000000000..d0131dd934 --- /dev/null +++ b/research/x54-mixed-t37-saturation/SOURCES.json @@ -0,0 +1,18 @@ +{ + "schema_version": 1, + "base_commit": "dd6403f774a0891093a006ae9716c02e371b1fdb", + "files": [ + { + "path": "certificates/selected-result.json", + "usage": "provenance", + "git_blob": "ae6a99f28aa468649eb66a361e764d769363bf11", + "sha256": "10b9e340fd8e354a37a0c4813c90e5de1dd42044b5f6aca4a3fc5a42b685fa32" + }, + { + "path": "docs/research/agent-task-catalog.json", + "usage": "provenance", + "git_blob": "c05fafaa25ba95b2d5af86b1800427198e3d20eb", + "sha256": "89b5928bc92c81e2647b6d702a7ad85a6795b8f44e2eb267b2ff3145d8f2b2ce" + } + ] +} diff --git a/research/x54-mixed-t37-saturation/TASK.json b/research/x54-mixed-t37-saturation/TASK.json new file mode 100644 index 0000000000..56447d0cfd --- /dev/null +++ b/research/x54-mixed-t37-saturation/TASK.json @@ -0,0 +1,66 @@ +{ + "schema_version": 1, + "task_id": "selected-packing", + "priority_revision": 2, + "base_commit": "dd6403f774a0891093a006ae9716c02e371b1fdb", + "original_card": { + "id": "selected-packing", + "question": "Is there a genuinely unsearched, explicitly bounded improvement to the selected packing?", + "starting_points": [ + "certificates/selected-result.json", + "research/heterogeneous-owner-assembly/README.md" + ], + "methods": [ + "Search a named unclaimed canonical candidate domain with a reproducible partition.", + "Prove a stronger scoped relaxation or exact optimum for one named residual subproblem." + ], + "acceptance_gate": "Check live PR overlap first, price both suppliers and distinguish a search best from an optimum.", + "first_step": { + "command": [ + "python3", + "-B", + "scripts/research_baseline.py" + ], + "expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY", + "inspect": "Locate the selected packing source in the review and check live PRs for the same residual problem.", + "scope": "Positive control / source-bound orientation only; this is not a new construction or full cold replay." + }, + "first_experiment": "Define an unclaimed canonical domain and a favorable bound before starting a new search.", + "bottleneck_link": "An actual paid-profile gain can improve the frontier up to the other supplier cap.", + "allowed_variations": [ + "One bounded residual packing instance or exact relaxation." + ], + "scope_change_triggers": [ + "Rerunning an existing search with only a different seed as a separate mathematical contribution." + ], + "overlap_terms": [ + "T16", + "owner exchange", + "packing" + ] + }, + "scope_status": "refined", + "actual_scope": "Is there an unsearched width-neutral local move or packing improvement for the highest open claim (#1193/#1218 X54 mixed-T37, kappa 0.001675809683500501), rather than the selected #1094 packing?", + "scope_change_reason": "The selected #1094 packing is superseded by open claims on the same lineage; the bounded residual question that matters is whether the strongest open construction still has local or packing headroom.", + "bottleneck_link": "The complex supplier binds; on this line every recent gain came from width-neutral local moves or packing. This bounds both for the incumbent words/mix and prices the inventory changes that could still help.", + "budget_minutes": 120, + "stop_rule": "Stop after exhaustive toolkit sweeps on both final words and an exact or ILP packing bound; build an admitted construction only if a gain appears.", + "outcome": "negative-result", + "evidence": "evidence/sweeps (0 gains in 2x(2238+50+161) candidates), evidence/PACKING.json (exact Phi invariant and idle optimum), evidence/AGGREGATE.json (ILP = incumbent; dual prices). Superseded in part by #1274 (local gain outside the #1186 families) and #1252/#1268/#1280 (idle-axis repartition outside the bin-remainder model).", + "author": "Wronskode", + "ai_assistance": "Substantial assistance from Anthropic Claude (Claude Code): search runs, analysis, code and text.", + "open_questions": [ + { + "question": "Is there a single-helper entrance/end change whose local frame cost passes the width/Phi screening rule (e.g. 9->8 below 13.3 units, 4->5 below 86.9 per helper)?", + "why_it_matters": "Packing and width-neutral moves are saturated for the incumbent; only a changed helper inventory or new local word can move kappa on this line.", + "first_step": "Price candidate single-helper interface changes with the #1186 Pricer and apply the README score; rerun aggregate.py with the new variant.", + "blocked_by": "No generator for single-helper interface changes; #1176 reports demotions need whole four-helper groups." + }, + { + "question": "How do the width/Phi shadow prices and the optimal demotion-group count change once idle rank-48 children can be repartitioned (48+48 -> 45+51, #1252/#1280)?", + "why_it_matters": "Idle reshaping changes the marginal value of idle mass, hence lambda and the break-even of an entrance group; the 208-group mix may no longer be optimal.", + "first_step": "Rerun aggregate.py with an idle-cost function allowing empty capacity-48 bins to pair into 45+51 at the rate #1280 achieves.", + "blocked_by": "The achievable pairing rate depends on #1252's per-(replica, port) commuting-axis condition, which is not modelled here." + } + ] +} diff --git a/research/x54-mixed-t37-saturation/VERIFY.json b/research/x54-mixed-t37-saturation/VERIFY.json new file mode 100644 index 0000000000..9662e6f2fb --- /dev/null +++ b/research/x54-mixed-t37-saturation/VERIFY.json @@ -0,0 +1,21 @@ +{ + "schema_version": 1, + "status": "ready", + "python": "3.12", + "requirements": [], + "inputs": [ + "research/community-frontier1042-audit/selected-replay.tar.gz", + "research/complex-t21-owners/vendor/pr870/t16/build_parent.py" + ], + "commands": [ + { + "argv": [ + "{python}", + "-B", + "{package}/verify.py" + ], + "timeout_seconds": 120, + "checks": "input pins; exact width recount, flow conservation, Phi lemma and idle optimum vs evidence/PACKING.json; five rejected mutation controls; sweep tally recount; aggregate ILP receipt." + } + ] +} diff --git a/research/x54-mixed-t37-saturation/aggregate.py b/research/x54-mixed-t37-saturation/aggregate.py new file mode 100644 index 0000000000..e9c4af710e --- /dev/null +++ b/research/x54-mixed-t37-saturation/aggregate.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Numerical screening (HiGHS via SciPy): is any demotion-count mix plus bin packing better than #1193's? + +Relaxation. Each call uses a variant with k extra four-helper entrance groups relative to D3 (k in -3..3). +Following #1193/#1176, one group changes the local profile by -8H1+8H2+4H3-4H4 and turns four width-4 helpers +into width-5 helpers. k = 0, 1 are the admitted D3/D4 words; other k are hypothetical (linear extrapolation, +not admitted). Bins: the retained T37 capacities with their exact counts, every multiset of widths up to the +capacity, time windows and family rules ignored. Objective: the complex moment at #1193's root endpoint, with +the mass-linear part r/m removed (total mass is fixed by the coverage identity) and scaled by 1e6. + +Outputs the integer optimum, the LP optimum and the LP dual prices. The dual width prices have the exact +structure pi_w = -lambda*w - mu*phi(w) with packing.py's phi, i.e. one price per unit of helper width and one +per unit of Phi. This is floating-point screening, not a certificate; packing.py carries the exact statement +for the incumbent mix. +""" +import argparse, itertools, json +from pathlib import Path +import numpy as np +from scipy.optimize import milp, linprog, LinearConstraint, Bounds + +WIDTHS = (4, 5, 7, 8, 9) +PHI = {4: 0, 5: 3, 7: 1, 8: 0, 9: -1} +M, LIVE, CALLS = 73, 33152, 333 +A_ROOT = 0.001678622735735992 # #1193 complex root +D3 = {1: 2358, 2: 504, 3: 410, 4: 130, 5: 82, 8: 8} +INV3 = {4: 44, 5: 44, 7: 38, 8: 114, 9: 240} +GROUP = {1: -8, 2: 8, 3: 4, 4: -4} +CAPS = {8: 24864, 16: 16576, 32: 16576, 48: 16576} +INCUMBENT = {1: 783552, 2: 169496, 3: 137362, 4: 42458, 5: 27306, 8: 2664, 48: 11374} +KS = range(-3, 4) + + +def cost(r, a=A_ROOT): + return 0.0 if r == 0 else 1e6 * ((r / M) ** (1 - a) - r / M) + + +def local(k): + return {r: D3.get(r, 0) + k * GROUP.get(r, 0) for r in set(D3) | set(GROUP)} + + +def inventory(k): + return {4: 44 - 4 * k, 5: 44 + 4 * k, 7: 38, 8: 114, 9: 240} + + +def model(): + pats = [(c, p) for c in CAPS for n in range(13) for p in itertools.combinations_with_replacement(WIDTHS, n) if sum(p) <= c] + obj = [sum(n * cost(r) for r, n in local(k).items()) for k in KS] + [cost(c - sum(p)) for c, p in pats] + rows, rhs = [[1] * len(KS) + [0] * len(pats)], [CALLS] + for c, n in CAPS.items(): + rows.append([0] * len(KS) + [int(cc == c) for cc, _ in pats]); rhs.append(n) + for w in WIDTHS: + rows.append([-inventory(k)[w] for k in KS] + [p.count(w) for _, p in pats]); rhs.append(0) + return pats, np.array(obj), np.array(rows, float), np.array(rhs, float) + + +def run(): + pats, obj, A, b = model() + incumbent = sum(n * cost(r) for r, n in INCUMBENT.items()) + ilp = milp(obj, constraints=LinearConstraint(A, b, b), integrality=np.ones(len(obj)), bounds=Bounds(0, np.inf), + options=dict(time_limit=600, mip_rel_gap=1e-12)) + assert ilp.status == 0, ilp.message + lp = linprog(obj, A_eq=A, b_eq=b, bounds=(0, None), method='highs') + assert lp.status == 0, lp.message + y = lp.eqlin.marginals + pi = dict(zip(WIDTHS, y[1 + len(CAPS):])) + lam = -pi[4] / 4 + mu = (-pi[5] - 5 * lam) / 3 + residual = max(abs(pi[w] + lam * w + mu * PHI[w]) for w in WIDTHS) + used = lambda x: {str(k): round(v) for k, v in zip(KS, x[:len(KS)]) if v > 0.5} + # dM/da at the root, to convert objective units into kappa: dkappa = -dObj / (1e6 * LIVE * dM/da) / (1 + a)^2 + dMda = sum(n * (r / M) ** (1 - A_ROOT) * np.log(M / r) for r, n in INCUMBENT.items()) / LIVE + return dict( + status='SCREENING_ONLY_FLOAT_HIGHS', + incumbent_objective=incumbent, ilp_objective=ilp.fun, ilp_minus_incumbent=ilp.fun - incumbent, + ilp_total_extra_groups=int(round(sum(k * v for k, v in zip(KS, ilp.x[:len(KS)])))), ilp_mix=used(ilp.x), + lp_objective=lp.fun, lp_minus_incumbent=lp.fun - incumbent, + lp_total_extra_groups=float(sum(k * v for k, v in zip(KS, lp.x[:len(KS)]))), + dual_width_price={str(w): pi[w] for w in WIDTHS}, dual_bin_price={str(c): v for c, v in zip(CAPS, y[1:1 + len(CAPS)])}, + lambda_per_width_unit=lam, mu_per_phi_unit=mu, price_structure_residual=residual, + dM_da=dMda, kappa_per_objective_unit=1 / (1e6 * LIVE * dMda) / (1 + A_ROOT) ** 2, + patterns=len(pats), variants=list(KS)) + + +if __name__ == '__main__': + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--out', type=Path) + a = ap.parse_args() + r = run() + if a.out: + a.out.write_text(json.dumps(r, indent=1, sort_keys=True) + '\n') + print(json.dumps({k: r[k] for k in ('ilp_minus_incumbent', 'ilp_total_extra_groups', 'lp_minus_incumbent', + 'lambda_per_width_unit', 'mu_per_phi_unit', 'price_structure_residual')})) diff --git a/research/x54-mixed-t37-saturation/evidence/AGGREGATE.json b/research/x54-mixed-t37-saturation/evidence/AGGREGATE.json new file mode 100644 index 0000000000..78785f3f60 --- /dev/null +++ b/research/x54-mixed-t37-saturation/evidence/AGGREGATE.json @@ -0,0 +1,45 @@ +{ + "dM_da": 2.923437411057431, + "dual_bin_price": { + "16": 187.6947259783875, + "32": 375.38945195677536, + "48": 462.92018110882816, + "8": 93.84736298919373 + }, + "dual_width_price": { + "4": -46.92368149459692, + "5": -133.7775994879974, + "7": -107.15744182212849, + "8": -93.84736298919378, + "9": -80.53728415625932 + }, + "ilp_minus_incumbent": 0.0, + "ilp_mix": { + "-1": 150, + "-2": 1, + "-3": 1, + "2": 180, + "3": 1 + }, + "ilp_objective": 162191780.82192433, + "ilp_total_extra_groups": 208, + "incumbent_objective": 162191780.82192433, + "kappa_per_objective_unit": 1.02834692831779e-11, + "lambda_per_width_unit": 11.73092037364923, + "lp_minus_incumbent": -24.321204900741577, + "lp_objective": 162191756.50071943, + "lp_total_extra_groups": 208.5000000000019, + "mu_per_phi_unit": 25.040999206583752, + "patterns": 1163, + "price_structure_residual": 1.2434497875801753e-13, + "status": "SCREENING_ONLY_FLOAT_HIGHS", + "variants": [ + -3, + -2, + -1, + 0, + 1, + 2, + 3 + ] +} diff --git a/research/x54-mixed-t37-saturation/evidence/PACKING.json b/research/x54-mixed-t37-saturation/evidence/PACKING.json new file mode 100644 index 0000000000..ce74e02cd3 --- /dev/null +++ b/research/x54-mixed-t37-saturation/evidence/PACKING.json @@ -0,0 +1,60 @@ +{ + "bins_by_capacity": { + "8": 24864, + "16": 16576, + "32": 16576, + "48": 16576 + }, + "concentration_test": { + "endpoint": "0.001678622735735992236999", + "j_range": [ + 1, + 11374 + ], + "min_gap_over_j": "0.000035875727023542260129333503" + }, + "helper_widths": { + "4": 13820, + "5": 15484, + "7": 12654, + "8": 37962, + "9": 79920 + }, + "idle_mass": 545954, + "idle_profile": { + "1": 2, + "48": 11374 + }, + "mix": { + "demote3": 125, + "demote4": 208 + }, + "phi": { + "4": 0, + "5": 3, + "7": 1, + "8": 0, + "9": -1 + }, + "phi_lower_bound_with_remainder_two_bin": { + "16": -20806, + "32": -20810, + "48": -20806, + "8": null + }, + "phi_min_full_bin": { + "8": 0, + "16": 0, + "32": 0, + "48": -4 + }, + "phi_min_remainder_two_bin": { + "8": null, + "16": 2, + "32": -2, + "48": -2 + }, + "phi_total": -20814, + "scope": "Fixed 125 D3 / 208 D4 mix, the retained T37 bin capacities, whole helpers, time windows and family rules relaxed. No statement about other local words, mixes, templates or outer recipes.", + "status": "PASS_EXACT_IDLE_REMAINDER_OPTIMUM_FOR_FIXED_MIX_RELAXATION" +} diff --git a/research/x54-mixed-t37-saturation/evidence/sweeps/EXCHANGE-D3.json.gz b/research/x54-mixed-t37-saturation/evidence/sweeps/EXCHANGE-D3.json.gz new file mode 100644 index 0000000000..cb5d85c419 Binary files /dev/null and b/research/x54-mixed-t37-saturation/evidence/sweeps/EXCHANGE-D3.json.gz differ diff --git a/research/x54-mixed-t37-saturation/evidence/sweeps/EXCHANGE-D4.json.gz b/research/x54-mixed-t37-saturation/evidence/sweeps/EXCHANGE-D4.json.gz new file mode 100644 index 0000000000..7db01e6881 Binary files /dev/null and b/research/x54-mixed-t37-saturation/evidence/sweeps/EXCHANGE-D4.json.gz differ diff --git a/research/x54-mixed-t37-saturation/evidence/sweeps/JOINT-D3.json.gz b/research/x54-mixed-t37-saturation/evidence/sweeps/JOINT-D3.json.gz new file mode 100644 index 0000000000..4718ca84ed Binary files /dev/null and b/research/x54-mixed-t37-saturation/evidence/sweeps/JOINT-D3.json.gz differ diff --git a/research/x54-mixed-t37-saturation/evidence/sweeps/JOINT-D4.json.gz b/research/x54-mixed-t37-saturation/evidence/sweeps/JOINT-D4.json.gz new file mode 100644 index 0000000000..44a649133a Binary files /dev/null and b/research/x54-mixed-t37-saturation/evidence/sweeps/JOINT-D4.json.gz differ diff --git a/research/x54-mixed-t37-saturation/evidence/sweeps/REPAIR-D3.json.gz b/research/x54-mixed-t37-saturation/evidence/sweeps/REPAIR-D3.json.gz new file mode 100644 index 0000000000..7aaf8398c8 Binary files /dev/null and b/research/x54-mixed-t37-saturation/evidence/sweeps/REPAIR-D3.json.gz differ diff --git a/research/x54-mixed-t37-saturation/evidence/sweeps/REPAIR-D4.json.gz b/research/x54-mixed-t37-saturation/evidence/sweeps/REPAIR-D4.json.gz new file mode 100644 index 0000000000..a4e3c69bbb Binary files /dev/null and b/research/x54-mixed-t37-saturation/evidence/sweeps/REPAIR-D4.json.gz differ diff --git a/research/x54-mixed-t37-saturation/inputs/D3-WORD.json.gz b/research/x54-mixed-t37-saturation/inputs/D3-WORD.json.gz new file mode 100644 index 0000000000..ba44a29122 Binary files /dev/null and b/research/x54-mixed-t37-saturation/inputs/D3-WORD.json.gz differ diff --git a/research/x54-mixed-t37-saturation/inputs/D4-WORD.json.gz b/research/x54-mixed-t37-saturation/inputs/D4-WORD.json.gz new file mode 100644 index 0000000000..9ed9bb09c6 Binary files /dev/null and b/research/x54-mixed-t37-saturation/inputs/D4-WORD.json.gz differ diff --git a/research/x54-mixed-t37-saturation/inputs/FLOW.json.gz b/research/x54-mixed-t37-saturation/inputs/FLOW.json.gz new file mode 100644 index 0000000000..71246bb8e4 Binary files /dev/null and b/research/x54-mixed-t37-saturation/inputs/FLOW.json.gz differ diff --git a/research/x54-mixed-t37-saturation/inputs/MIX.json b/research/x54-mixed-t37-saturation/inputs/MIX.json new file mode 100644 index 0000000000..52e8bdd775 --- /dev/null +++ b/research/x54-mixed-t37-saturation/inputs/MIX.json @@ -0,0 +1,1667 @@ +[ + [ + "demote4", + 0, + 0 + ], + [ + "demote4", + 0, + 1 + ], + [ + "demote4", + 0, + 2 + ], + [ + "demote4", + 0, + 3 + ], + [ + "demote4", + 0, + 4 + ], + [ + "demote4", + 1, + 0 + ], + [ + "demote4", + 0, + 5 + ], + [ + "demote4", + 1, + 1 + ], + [ + "demote4", + 0, + 6 + ], + [ + "demote4", + 1, + 2 + ], + [ + "demote4", + 0, + 7 + ], + [ + "demote4", + 1, + 3 + ], + [ + "demote4", + 0, + 8 + ], + [ + "demote4", + 1, + 4 + ], + [ + "demote4", + 0, + 9 + ], + [ + "demote4", + 2, + 0 + ], + [ + "demote4", + 1, + 5 + ], + [ + "demote4", + 0, + 10 + ], + [ + "demote4", + 2, + 1 + ], + [ + "demote4", + 1, + 6 + ], + [ + "demote4", + 0, + 11 + ], + [ + "demote4", + 2, + 2 + ], + [ + "demote4", + 1, + 7 + ], + [ + "demote4", + 0, + 12 + ], + [ + "demote4", + 2, + 3 + ], + [ + "demote4", + 1, + 8 + ], + [ + "demote4", + 0, + 13 + ], + [ + "demote4", + 2, + 4 + ], + [ + "demote4", + 1, + 9 + ], + [ + "demote4", + 3, + 0 + ], + [ + "demote4", + 0, + 14 + ], + [ + "demote4", + 2, + 5 + ], + [ + "demote4", + 1, + 10 + ], + [ + "demote4", + 3, + 1 + ], + [ + "demote4", + 0, + 15 + ], + [ + "demote4", + 2, + 6 + ], + [ + "demote4", + 1, + 11 + ], + [ + "demote4", + 3, + 2 + ], + [ + "demote4", + 0, + 16 + ], + [ + "demote4", + 2, + 7 + ], + [ + "demote4", + 1, + 12 + ], + [ + "demote4", + 3, + 3 + ], + [ + "demote4", + 0, + 17 + ], + [ + "demote4", + 2, + 8 + ], + [ + "demote4", + 1, + 13 + ], + [ + "demote4", + 3, + 4 + ], + [ + "demote4", + 0, + 18 + ], + [ + "demote4", + 2, + 9 + ], + [ + "demote4", + 4, + 0 + ], + [ + "demote4", + 1, + 14 + ], + [ + "demote4", + 3, + 5 + ], + [ + "demote4", + 0, + 19 + ], + [ + "demote4", + 2, + 10 + ], + [ + "demote4", + 4, + 1 + ], + [ + "demote4", + 1, + 15 + ], + [ + "demote4", + 3, + 6 + ], + [ + "demote4", + 0, + 20 + ], + [ + "demote4", + 2, + 11 + ], + [ + "demote4", + 4, + 2 + ], + [ + "demote4", + 1, + 16 + ], + [ + "demote4", + 3, + 7 + ], + [ + "demote4", + 0, + 21 + ], + [ + "demote4", + 2, + 12 + ], + [ + "demote4", + 4, + 3 + ], + [ + "demote4", + 1, + 17 + ], + [ + "demote4", + 3, + 8 + ], + [ + "demote4", + 0, + 22 + ], + [ + "demote4", + 2, + 13 + ], + [ + "demote4", + 4, + 4 + ], + [ + "demote4", + 1, + 18 + ], + [ + "demote4", + 3, + 9 + ], + [ + "demote4", + 0, + 23 + ], + [ + "demote4", + 5, + 0 + ], + [ + "demote4", + 2, + 14 + ], + [ + "demote4", + 4, + 5 + ], + [ + "demote4", + 1, + 19 + ], + [ + "demote4", + 3, + 10 + ], + [ + "demote4", + 0, + 24 + ], + [ + "demote4", + 5, + 1 + ], + [ + "demote4", + 2, + 15 + ], + [ + "demote4", + 4, + 6 + ], + [ + "demote4", + 1, + 20 + ], + [ + "demote4", + 3, + 11 + ], + [ + "demote4", + 0, + 25 + ], + [ + "demote4", + 5, + 2 + ], + [ + "demote4", + 2, + 16 + ], + [ + "demote4", + 4, + 7 + ], + [ + "demote4", + 1, + 21 + ], + [ + "demote4", + 3, + 12 + ], + [ + "demote4", + 0, + 26 + ], + [ + "demote4", + 5, + 3 + ], + [ + "demote4", + 2, + 17 + ], + [ + "demote4", + 4, + 8 + ], + [ + "demote4", + 1, + 22 + ], + [ + "demote4", + 3, + 13 + ], + [ + "demote4", + 0, + 27 + ], + [ + "demote4", + 5, + 4 + ], + [ + "demote4", + 2, + 18 + ], + [ + "demote4", + 4, + 9 + ], + [ + "demote4", + 1, + 23 + ], + [ + "demote4", + 6, + 0 + ], + [ + "demote4", + 3, + 14 + ], + [ + "demote4", + 0, + 28 + ], + [ + "demote4", + 5, + 5 + ], + [ + "demote4", + 2, + 19 + ], + [ + "demote4", + 4, + 10 + ], + [ + "demote4", + 1, + 24 + ], + [ + "demote4", + 6, + 1 + ], + [ + "demote4", + 3, + 15 + ], + [ + "demote4", + 0, + 29 + ], + [ + "demote4", + 5, + 6 + ], + [ + "demote4", + 2, + 20 + ], + [ + "demote4", + 4, + 11 + ], + [ + "demote4", + 1, + 25 + ], + [ + "demote4", + 6, + 2 + ], + [ + "demote4", + 3, + 16 + ], + [ + "demote4", + 0, + 30 + ], + [ + "demote4", + 5, + 7 + ], + [ + "demote4", + 2, + 21 + ], + [ + "demote4", + 4, + 12 + ], + [ + "demote4", + 1, + 26 + ], + [ + "demote4", + 6, + 3 + ], + [ + "demote4", + 3, + 17 + ], + [ + "demote4", + 0, + 31 + ], + [ + "demote4", + 5, + 8 + ], + [ + "demote4", + 2, + 22 + ], + [ + "demote4", + 4, + 13 + ], + [ + "demote4", + 1, + 27 + ], + [ + "demote4", + 6, + 4 + ], + [ + "demote4", + 3, + 18 + ], + [ + "demote4", + 0, + 32 + ], + [ + "demote4", + 5, + 9 + ], + [ + "demote4", + 2, + 23 + ], + [ + "demote4", + 7, + 0 + ], + [ + "demote4", + 4, + 14 + ], + [ + "demote4", + 1, + 28 + ], + [ + "demote4", + 6, + 5 + ], + [ + "demote4", + 3, + 19 + ], + [ + "demote4", + 0, + 33 + ], + [ + "demote4", + 5, + 10 + ], + [ + "demote4", + 2, + 24 + ], + [ + "demote4", + 7, + 1 + ], + [ + "demote4", + 4, + 15 + ], + [ + "demote4", + 1, + 29 + ], + [ + "demote4", + 6, + 6 + ], + [ + "demote4", + 3, + 20 + ], + [ + "demote4", + 0, + 34 + ], + [ + "demote4", + 5, + 11 + ], + [ + "demote4", + 2, + 25 + ], + [ + "demote4", + 7, + 2 + ], + [ + "demote4", + 4, + 16 + ], + [ + "demote4", + 1, + 30 + ], + [ + "demote4", + 6, + 7 + ], + [ + "demote4", + 3, + 21 + ], + [ + "demote4", + 0, + 35 + ], + [ + "demote4", + 5, + 12 + ], + [ + "demote4", + 2, + 26 + ], + [ + "demote4", + 7, + 3 + ], + [ + "demote4", + 4, + 17 + ], + [ + "demote4", + 1, + 31 + ], + [ + "demote4", + 6, + 8 + ], + [ + "demote4", + 3, + 22 + ], + [ + "demote4", + 0, + 36 + ], + [ + "demote4", + 5, + 13 + ], + [ + "demote4", + 2, + 27 + ], + [ + "demote4", + 7, + 4 + ], + [ + "demote4", + 4, + 18 + ], + [ + "demote4", + 1, + 32 + ], + [ + "demote4", + 6, + 9 + ], + [ + "demote4", + 3, + 23 + ], + [ + "demote4", + 8, + 0 + ], + [ + "demote4", + 5, + 14 + ], + [ + "demote4", + 2, + 28 + ], + [ + "demote4", + 7, + 5 + ], + [ + "demote4", + 4, + 19 + ], + [ + "demote4", + 1, + 33 + ], + [ + "demote4", + 6, + 10 + ], + [ + "demote4", + 3, + 24 + ], + [ + "demote4", + 8, + 1 + ], + [ + "demote4", + 5, + 15 + ], + [ + "demote4", + 2, + 29 + ], + [ + "demote4", + 7, + 6 + ], + [ + "demote4", + 4, + 20 + ], + [ + "demote4", + 1, + 34 + ], + [ + "demote4", + 6, + 11 + ], + [ + "demote4", + 3, + 25 + ], + [ + "demote4", + 8, + 2 + ], + [ + "demote4", + 5, + 16 + ], + [ + "demote4", + 2, + 30 + ], + [ + "demote4", + 7, + 7 + ], + [ + "demote4", + 4, + 21 + ], + [ + "demote4", + 1, + 35 + ], + [ + "demote4", + 6, + 12 + ], + [ + "demote4", + 3, + 26 + ], + [ + "demote4", + 8, + 3 + ], + [ + "demote4", + 5, + 17 + ], + [ + "demote4", + 2, + 31 + ], + [ + "demote4", + 7, + 8 + ], + [ + "demote4", + 4, + 22 + ], + [ + "demote4", + 1, + 36 + ], + [ + "demote4", + 6, + 13 + ], + [ + "demote4", + 3, + 27 + ], + [ + "demote4", + 8, + 4 + ], + [ + "demote4", + 5, + 18 + ], + [ + "demote4", + 2, + 32 + ], + [ + "demote4", + 7, + 9 + ], + [ + "demote4", + 4, + 23 + ], + [ + "demote4", + 6, + 14 + ], + [ + "demote3", + 3, + 28 + ], + [ + "demote3", + 8, + 5 + ], + [ + "demote3", + 5, + 19 + ], + [ + "demote3", + 2, + 33 + ], + [ + "demote3", + 7, + 10 + ], + [ + "demote3", + 4, + 24 + ], + [ + "demote3", + 6, + 15 + ], + [ + "demote3", + 3, + 29 + ], + [ + "demote3", + 8, + 6 + ], + [ + "demote3", + 5, + 20 + ], + [ + "demote3", + 2, + 34 + ], + [ + "demote3", + 7, + 11 + ], + [ + "demote3", + 4, + 25 + ], + [ + "demote3", + 6, + 16 + ], + [ + "demote3", + 3, + 30 + ], + [ + "demote3", + 8, + 7 + ], + [ + "demote3", + 5, + 21 + ], + [ + "demote3", + 2, + 35 + ], + [ + "demote3", + 7, + 12 + ], + [ + "demote3", + 4, + 26 + ], + [ + "demote3", + 6, + 17 + ], + [ + "demote3", + 3, + 31 + ], + [ + "demote3", + 8, + 8 + ], + [ + "demote3", + 5, + 22 + ], + [ + "demote3", + 2, + 36 + ], + [ + "demote3", + 7, + 13 + ], + [ + "demote3", + 4, + 27 + ], + [ + "demote3", + 6, + 18 + ], + [ + "demote3", + 3, + 32 + ], + [ + "demote3", + 8, + 9 + ], + [ + "demote3", + 5, + 23 + ], + [ + "demote3", + 7, + 14 + ], + [ + "demote3", + 4, + 28 + ], + [ + "demote3", + 6, + 19 + ], + [ + "demote3", + 3, + 33 + ], + [ + "demote3", + 8, + 10 + ], + [ + "demote3", + 5, + 24 + ], + [ + "demote3", + 7, + 15 + ], + [ + "demote3", + 4, + 29 + ], + [ + "demote3", + 6, + 20 + ], + [ + "demote3", + 3, + 34 + ], + [ + "demote3", + 8, + 11 + ], + [ + "demote3", + 5, + 25 + ], + [ + "demote3", + 7, + 16 + ], + [ + "demote3", + 4, + 30 + ], + [ + "demote3", + 6, + 21 + ], + [ + "demote3", + 3, + 35 + ], + [ + "demote3", + 8, + 12 + ], + [ + "demote3", + 5, + 26 + ], + [ + "demote3", + 7, + 17 + ], + [ + "demote3", + 4, + 31 + ], + [ + "demote3", + 6, + 22 + ], + [ + "demote3", + 3, + 36 + ], + [ + "demote3", + 8, + 13 + ], + [ + "demote3", + 5, + 27 + ], + [ + "demote3", + 7, + 18 + ], + [ + "demote3", + 4, + 32 + ], + [ + "demote3", + 6, + 23 + ], + [ + "demote3", + 8, + 14 + ], + [ + "demote3", + 5, + 28 + ], + [ + "demote3", + 7, + 19 + ], + [ + "demote3", + 4, + 33 + ], + [ + "demote3", + 6, + 24 + ], + [ + "demote3", + 8, + 15 + ], + [ + "demote3", + 5, + 29 + ], + [ + "demote3", + 7, + 20 + ], + [ + "demote3", + 4, + 34 + ], + [ + "demote3", + 6, + 25 + ], + [ + "demote3", + 8, + 16 + ], + [ + "demote3", + 5, + 30 + ], + [ + "demote3", + 7, + 21 + ], + [ + "demote3", + 4, + 35 + ], + [ + "demote3", + 6, + 26 + ], + [ + "demote3", + 8, + 17 + ], + [ + "demote3", + 5, + 31 + ], + [ + "demote3", + 7, + 22 + ], + [ + "demote3", + 4, + 36 + ], + [ + "demote3", + 6, + 27 + ], + [ + "demote3", + 8, + 18 + ], + [ + "demote3", + 5, + 32 + ], + [ + "demote3", + 7, + 23 + ], + [ + "demote3", + 6, + 28 + ], + [ + "demote3", + 8, + 19 + ], + [ + "demote3", + 5, + 33 + ], + [ + "demote3", + 7, + 24 + ], + [ + "demote3", + 6, + 29 + ], + [ + "demote3", + 8, + 20 + ], + [ + "demote3", + 5, + 34 + ], + [ + "demote3", + 7, + 25 + ], + [ + "demote3", + 6, + 30 + ], + [ + "demote3", + 8, + 21 + ], + [ + "demote3", + 5, + 35 + ], + [ + "demote3", + 7, + 26 + ], + [ + "demote3", + 6, + 31 + ], + [ + "demote3", + 8, + 22 + ], + [ + "demote3", + 5, + 36 + ], + [ + "demote3", + 7, + 27 + ], + [ + "demote3", + 6, + 32 + ], + [ + "demote3", + 8, + 23 + ], + [ + "demote3", + 7, + 28 + ], + [ + "demote3", + 6, + 33 + ], + [ + "demote3", + 8, + 24 + ], + [ + "demote3", + 7, + 29 + ], + [ + "demote3", + 6, + 34 + ], + [ + "demote3", + 8, + 25 + ], + [ + "demote3", + 7, + 30 + ], + [ + "demote3", + 6, + 35 + ], + [ + "demote3", + 8, + 26 + ], + [ + "demote3", + 7, + 31 + ], + [ + "demote3", + 6, + 36 + ], + [ + "demote3", + 8, + 27 + ], + [ + "demote3", + 7, + 32 + ], + [ + "demote3", + 8, + 28 + ], + [ + "demote3", + 7, + 33 + ], + [ + "demote3", + 8, + 29 + ], + [ + "demote3", + 7, + 34 + ], + [ + "demote3", + 8, + 30 + ], + [ + "demote3", + 7, + 35 + ], + [ + "demote3", + 8, + 31 + ], + [ + "demote3", + 7, + 36 + ], + [ + "demote3", + 8, + 32 + ], + [ + "demote3", + 8, + 33 + ], + [ + "demote3", + 8, + 34 + ], + [ + "demote3", + 8, + 35 + ], + [ + "demote3", + 8, + 36 + ] +] diff --git a/research/x54-mixed-t37-saturation/packing.py b/research/x54-mixed-t37-saturation/packing.py new file mode 100644 index 0000000000..c4fb872b9b --- /dev/null +++ b/research/x54-mixed-t37-saturation/packing.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exact idle-remainder optimality of the #1193/#1218 mixed T37 packing (standard library only). + +Inputs: the two final local words (D3, D4), the frozen integer pattern flow and call mix of #1193, and the +T37 parent geometry retained on main (the #1042 replay archive and PR858/PR870's unchanged `actual_windows`). + +Statement checked here (all integers / exact rationals, 50-digit decimals only for one sign test): + + Width-class invariant. Put phi(4) = phi(8) = 0, phi(5) = 3, phi(7) = 1, phi(9) = -1 and, for a multiset p of + helper widths, Phi(p) = sum phi. Enumerating every multiset over {4,5,7,8,9}: a bin filled exactly to + capacity 8, 16 or 32 has Phi >= 0, and one filled exactly to capacity 48 has Phi >= -4. + + Consequence. With the frozen 125 D3 / 208 D4 mix, the T37 bins (8,16,32,48 capacities) and any assignment + of the 159,840 whole helpers to bins (time windows, ports and family rules ignored, so this is a relaxation), + the idle (remainder) profile {1:2, 48:11374} of #1193 is the cheapest one achievable: the only cheaper idle + histogram of the same mass with ranks <= 48 is {2:1, 48:11374}, and every packing realizing it has + Phi_total > -20814 = Phi(helpers), a contradiction. + +Scope: one mix, one bin geometry, one width alphabet. It does not bound other local words, mixes, templates or +outer recipes; `aggregate.py` screens the demotion-count mixes numerically. +""" +import argparse, gzip, hashlib, importlib.util, itertools, json, sys, tarfile, tempfile +from collections import Counter +from decimal import Decimal, getcontext +from pathlib import Path + +PKG = Path(__file__).resolve().parent +ROOT = PKG.parents[1] +WIDTHS = (4, 5, 7, 8, 9) +PHI = {4: 0, 5: 3, 7: 1, 8: 0, 9: -1} +ARCHIVE = ROOT / 'research/community-frontier1042-audit/selected-replay.tar.gz' +ARCHIVE_SHA = '22c3f4dce4b897a1d423d1388801933e288b6a6b1906cb15e1b16a5eb68db754' +TEMPLATE = 'slots/global/global/base/split-centres/GLOBAL-TEMPLATE.json.gz' +WINDOWS_MODULE = ROOT / 'research/complex-t21-owners/vendor/pr870/t16/build_parent.py' +M, LIVE = 73, 33152 + + +def require(ok, why): + if not ok: + raise AssertionError(why) + + +def sha(path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def read(path): + raw = Path(path).read_bytes() + return json.loads(gzip.decompress(raw) if str(path).endswith('.gz') else raw) + + +def dim(vectors): + """GF(2) dimension of the span of 9-bit integer vectors.""" + basis = [] + for v in vectors: + for b in basis: + v = min(v, v ^ b) + if v: + basis.append(v) + return len(basis) + + +def helper_widths(word): + """Width of every helper: dim(final frame) - dim(start frame); full final frame has dimension 9. + Independent of the toolkit and of the native canonical word (same convention as the lineage recount).""" + v, R = word['report']['v'], word['report']['R'] + out = [] + for q in range(2 * v, 2 * v + R): + start = dim(word['start_extra'].get(str(q), [])) + end = dim(word['helper_final'][str(q)]) if str(q) in word['helper_final'] else 9 + out.append(end - start) + return out + + +def min_phi(total, max_len=12): + """Exact minimum of Phi over multisets of widths with the given sum (None when the sum is unattainable).""" + best = None + for k in range(1, max_len + 1): + for p in itertools.combinations_with_replacement(WIDTHS, k): + if sum(p) == total: + f = sum(PHI[w] for w in p) + best = f if best is None or f < best else best + return best + + +def geometry(flow, order): + """Capacity of every flow group from the retained T37 parent's idle-child ranks (PR858/870 windows).""" + require(sha(ARCHIVE) == ARCHIVE_SHA, 'retained #1042 archive changed') + with tempfile.TemporaryDirectory() as tmp, tarfile.open(ARCHIVE) as t: + member = t.getmember(TEMPLATE) + require(member.isfile(), 'template member') + t.extract(member, tmp, filter='data') + parent = read(Path(tmp) / TEMPLATE) + spec = importlib.util.spec_from_file_location('pr870_build_parent', WINDOWS_MODULE) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + windows, _ = mod.actual_windows(parent, order) + caps = [] + for g in flow: + cs = {windows[k][0] for k in g['physical_windows']} + require(len(cs) == 1, 'one capacity per group') + caps.append(cs.pop()) + return caps, windows + + +def f(r, a): + """(r/m)^(1-a) to 50 digits.""" + getcontext().prec = 60 + return (Decimal(r) / M) ** (1 - a) + + +def load_inputs(inputs=PKG / 'inputs'): + return dict(d3=read(inputs / 'D3-WORD.json.gz'), d4=read(inputs / 'D4-WORD.json.gz'), + flow=read(inputs / 'FLOW.json.gz'), mix=read(inputs / 'MIX.json')) + + +def check(data=None, geometry_cache=None): + data = data or load_inputs() + d3, d4, flow, mix = data['d3'], data['d4'], data['flow'], data['mix'] + order = sorted([[s, r] for s in range(9) for r in range(37)], key=lambda k: (37 * k[0] + 8 * k[1], k[0], k[1])) + require([[s, r] for _, s, r in mix] == order, 'frozen #1131/#1152 wave order') + calls = Counter(pid for pid, _, _ in mix) + require(calls == {'demote3': 125, 'demote4': 208}, 'call mix') + inv = {'demote3': Counter(helper_widths(d3)), 'demote4': Counter(helper_widths(d4))} + require(inv['demote3'] == {4: 44, 5: 44, 7: 38, 8: 114, 9: 240}, 'D3 width inventory') + require(inv['demote4'] == {4: 40, 5: 48, 7: 38, 8: 114, 9: 240}, 'D4 width inventory') + helpers = Counter() + for pid, n in calls.items(): + for w, c in inv[pid].items(): + helpers[w] += n * c + + if geometry_cache is not None and 'caps' in geometry_cache: + caps = geometry_cache['caps'] + else: + caps, _ = geometry(flow, order) + if geometry_cache is not None: + geometry_cache['caps'] = caps + bins, used, idle = Counter(), Counter(), Counter() + phi_bins = 0 + for cap, g in zip(caps, flow): + require(sum(p['count'] for p in g['patterns']) == 112 * len(g['physical_windows']), 'every port is one bin') + for p in g['patterns']: + s = sum(p['widths']) + require(s <= cap, 'pattern fits') + bins[cap] += p['count'] + for w in p['widths']: + used[w] += p['count'] + if cap - s: + idle[cap - s] += p['count'] + phi_bins += p['count'] * sum(PHI[w] for w in p['widths']) + require(used == helpers, 'the flow hosts every whole helper exactly once') + require(dict(idle) == {1: 2, 48: 11374}, '#1193 idle profile') + capacity = sum(c * n for c, n in bins.items()) + idle_mass = capacity - sum(w * n for w, n in helpers.items()) + require(idle_mass == sum(r * n for r, n in idle.items()) == 545954, 'idle mass') + phi_total = sum(PHI[w] * n for w, n in helpers.items()) + require(phi_total == phi_bins == -20814, 'Phi is a sum over helpers') + + # Lemma: exact minima of Phi over full bins and over remainder-two bins. + full = {c: min_phi(c) for c in bins} + require(full == {8: 0, 16: 0, 32: 0, 48: -4}, 'full-bin Phi minima') + rem2 = {c: min_phi(c - 2) for c in bins} + q = idle_mass // 48 + require(idle_mass == 48 * q + 2 and max(bins) == 48, 'mass decomposition') + # Cheapest idle histograms. Remainder ranks never exceed 48 (the largest capacity) and the idle mass is + # 48*q + 2. A histogram with exactly q rank-48 parts has remainder (1, 1) (the incumbent) or (2). One with + # k = q - j rank-48 parts (j >= 1) puts mass 48*j + 2 into parts <= 47; f is concave with f(0) = 0, so by + # Karamata the greedy partition (47, ..., 47, rest) majorizes every such partition and is the cheapest. + # The loop checks that this lower bound already exceeds the incumbent cost for every j. + a = Decimal('0.001678622735735992236999') # #1193 native upper root endpoint (cost increases with a) + f1, f2, f47, f48 = f(1, a), f(2, a), f(47, a), f(48, a) + worst = None + for j in range(1, q + 1): + t, rest = divmod(48 * j + 2, 47) + gap = t * f47 + (f(rest, a) if rest else 0) - j * f48 - 2 * f1 + worst = gap if worst is None or gap < worst else worst + require(worst > 0, 'every less concentrated histogram is dearer') + require(2 * f1 > f2, '{2:1} is cheaper than {1:2}') + # {2:1, 48:q} forces exactly q empty capacity-48 bins, one remainder-two bin and every other bin full. + occupied48 = bins[48] - q + bounds = {} + for c, mphi in rem2.items(): + if mphi is None: + bounds[c] = None + continue + full48 = occupied48 - (1 if c == 48 else 0) + bounds[c] = full[48] * full48 + mphi + require(all(b is None or b > phi_total for b in bounds.values()), 'Phi rules out a remainder-two bin') + # The incumbent is tight: every occupied capacity-48 bin attains -4 and the two remainder-one bins -3 each. + require(full[48] * occupied48 + 2 * sum(PHI[w] for w in (4, 9, 9, 9)) == phi_total, 'tightness') + return dict( + status='PASS_EXACT_IDLE_REMAINDER_OPTIMUM_FOR_FIXED_MIX_RELAXATION', + mix=dict(calls), helper_widths=dict(sorted(helpers.items())), bins_by_capacity=dict(sorted(bins.items())), + idle_profile=dict(sorted(idle.items())), idle_mass=idle_mass, phi={str(w): PHI[w] for w in WIDTHS}, + phi_total=phi_total, phi_min_full_bin=full, phi_min_remainder_two_bin=rem2, + phi_lower_bound_with_remainder_two_bin={str(c): b for c, b in bounds.items()}, + concentration_test=dict(endpoint=str(a), min_gap_over_j=str(worst)[:32], j_range=[1, q]), + scope='Fixed 125 D3 / 208 D4 mix, the retained T37 bin capacities, whole helpers, time windows and family ' + 'rules relaxed. No statement about other local words, mixes, templates or outer recipes.') + + +if __name__ == '__main__': + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--out', type=Path) + a = ap.parse_args() + r = check() + if a.out: + a.out.write_text(json.dumps(r, indent=1, sort_keys=True) + '\n') + print(r['status'], 'Phi', r['phi_total'], 'idle', r['idle_profile']) diff --git a/research/x54-mixed-t37-saturation/sweep.py b/research/x54-mixed-t37-saturation/sweep.py new file mode 100644 index 0000000000..c8f3272ceb --- /dev/null +++ b/research/x54-mixed-t37-saturation/sweep.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exhaustive width-neutral local sweeps on the final X54 mixed-T37 words, using #1186's toolkit unchanged. + +Families (each candidate priced by the toolkit's exact frame solver; negative delta = gain): + repair every read fusion (`fusion.candidates`) with single/pair swap repairs and full read-set exchanges + (`repair.py`, unchanged, default options); + exchange every subset of swappable post-copy reads of every copy pair (BRoute's SW4 read-set exchange); + joint joint triangle-hybrid candidates with up to three fused reads, partial swaps and both restore modes. + +The toolkit is not on main: pass `--toolkit` pointing at research/exact-frame-toolkit from PR #1186 head +9254a3cc610a72e9c4600da089d5cdbcab996b9c (its paths.py must see a checkout containing the retained engines). +Output is nomination data only; a gain would still need native and strict owner admission. + +usage: python -B sweep.py --toolkit DIR --word D3|D4 --out OUT_DIR [--workers 6] [--families repair,exchange,joint] +""" +import argparse, gzip, itertools, json, os, subprocess, sys, time +from pathlib import Path + +PKG = Path(__file__).resolve().parent +WORDS = {'D3': PKG / 'inputs/D3-WORD.json.gz', 'D4': PKG / 'inputs/D4-WORD.json.gz'} + + +def summarize_repair(rows): + out = [] + for r in rows: + b = r.get('best') + out.append(dict(t=r['t'], A=r['A'], H=r['H'], p=r['p'], rest=r['rest'], tried=r['tried'], nrepairs=r['nrepairs'], + best=None if not b else dict(delta=b['delta'], steps=b['steps'], profile=b['profile']))) + return out + + +def exchange_sweep(C, X, J, word): + P = C.Pricer(str(word)); w = P.base; out = [] + for h, (r, ic) in X.copy_pairs(w).items(): + M = X.movable_reads(w, h, r, ic) + if not M: + continue + sw = [(m, J.swap_edit(w, h, r, ic, m)) for m in M] + sw = [(m, s) for m, s in sw if s] + for k in range(1, len(sw) + 1): + for sub in itertools.combinations(sw, k): + ed = J.merge([s for _, s in sub]) + res = dict(status='overlap') if ed is None else P.price(ed, timeout=120) + out.append(dict(h=h, r=r, swapped=[m[0] for m, _ in sub], status=res.get('status'), delta=res.get('delta'), + profile=res.get('profile'))) + return out + + +def joint_sweep(C, J, word): + P = C.Pricer(str(word)); out = [] + for c in J.candidates(P.base, max_fused=3, allow_partial=True, restores=('canon', 'after')): + res = P.price(c['ed'], timeout=120) + out.append(dict(h=c['h'], r=c['r'], fused=c['fused'], swapped=c['swapped'], status=res.get('status'), + delta=res.get('delta'), profile=res.get('profile'))) + return out + + +def tally(rows, key=lambda r: r.get('delta')): + deltas = [key(r) for r in rows] + ok = [d for d in deltas if d is not None] + return dict(candidates=len(rows), feasible=len(ok), gains=sum(d < 0 for d in ok), best_delta=min(ok) if ok else None) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--toolkit', type=Path, required=True) + ap.add_argument('--word', choices=sorted(WORDS), required=True) + ap.add_argument('--out', type=Path, required=True) + ap.add_argument('--workers', type=int, default=6) + ap.add_argument('--families', default='repair,exchange,joint') + a = ap.parse_args() + tk = a.toolkit.resolve(); word = WORDS[a.word]; a.out.mkdir(parents=True, exist_ok=True) + sys.path.insert(0, str(tk)) + import harness as C, exchange as X, joint as J + summary = dict(word=a.word, toolkit=str(tk)) + fams = a.families.split(',') + if 'repair' in fams: + raw = a.out / f'repair-{a.word}.raw.json'; t0 = time.time() + subprocess.run([sys.executable, '-B', str(tk / 'repair.py'), str(word), str(raw), '--workers', str(a.workers)], cwd=tk, check=True) + rows = summarize_repair(json.load(open(raw))) + json.dump(rows, gzip.open(a.out / f'REPAIR-{a.word}.json.gz', 'wt'), sort_keys=True) + summary['repair'] = dict(tally(rows, key=lambda r: r['best']['delta'] if r['best'] else None), seconds=round(time.time() - t0)) + if 'exchange' in fams: + rows = exchange_sweep(C, X, J, word) + json.dump(rows, gzip.open(a.out / f'EXCHANGE-{a.word}.json.gz', 'wt'), sort_keys=True) + summary['exchange'] = tally(rows) + if 'joint' in fams: + rows = joint_sweep(C, J, word) + json.dump(rows, gzip.open(a.out / f'JOINT-{a.word}.json.gz', 'wt'), sort_keys=True) + summary['joint'] = tally(rows) + print(json.dumps(summary, sort_keys=True)) + + +if __name__ == '__main__': + main() diff --git a/research/x54-mixed-t37-saturation/verify.py b/research/x54-mixed-t37-saturation/verify.py new file mode 100644 index 0000000000..ec6a3c2581 --- /dev/null +++ b/research/x54-mixed-t37-saturation/verify.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Replay the exact packing statement, its failure controls and the recorded sweep tallies (standard library). + +Checks: + 1. byte pins of the vendored #1193/#1207 inputs; + 2. packing.check(): exact helper-width recount from both words, flow conservation, the Phi lemma and the + idle-remainder optimum for the fixed mix, compared with evidence/PACKING.json; + 3. failure controls: each mutation must be rejected at the intended check; + 4. the recorded sweep files recount to the tallies in RESULT.json (no gain in any family); + 5. evidence/AGGREGATE.json is present and reports ILP = incumbent (rerun it with aggregate.py; needs SciPy). +""" +import copy, gzip, hashlib, json, sys +from pathlib import Path + +PKG = Path(__file__).resolve().parent +sys.path.insert(0, str(PKG)) +import packing # noqa: E402 + +PINS = { + 'inputs/D3-WORD.json.gz': '4e837da23d932fc1336ef84dd1a802b46dbcb43bcf783b1338eeb7fa260eca1e', + 'inputs/D4-WORD.json.gz': 'eccbead1c65cd56111c3516bd4f5ab0a002df34b1ffda1faa5e1c5eaf5baf0d7', + 'inputs/FLOW.json.gz': 'e8e6dfa66e083ff8f3617a8441b5b2d766c3c9d9ac62ddc37d99edc24051f79e', + 'inputs/MIX.json': 'dd86bbb792a5c9f0d7507d55bd2f8392c66716023934b03b24f48ca737204eea', +} + + +def rejected(mutate, expect, data, cache): + d = copy.deepcopy(data) + mutate(d) + try: + packing.check(d, cache) + except AssertionError as e: + if expect in str(e): + return str(e) + raise AssertionError(f'control rejected for the wrong reason: {e!r} (expected {expect!r})') + raise AssertionError('control accepted: ' + expect) + + +def controls(data, cache): + out = {} + + def swap_mix(d): # one D4 call becomes D3 + i = next(i for i, e in enumerate(d['mix']) if e[0] == 'demote4'); d['mix'][i][0] = 'demote3' + out['mix'] = rejected(swap_mix, 'call mix', data, cache) + + def widen(d): # a width-4 helper of D3 starts one dimension lower (width 5) + q = next(q for q, U in d['d3']['start_extra'].items() if len(U) == 4); d['d3']['start_extra'][q] = d['d3']['start_extra'][q][:3] + out['inventory'] = rejected(widen, 'D3 width inventory', data, cache) + + def fill(d): # make one remainder-one bin full without supplying a width-5 helper + g = next(g for g in d['flow'] if any(p['widths'] == [4, 9, 9, 9] for p in g['patterns'])) + p = next(p for p in g['patterns'] if p['widths'] == [4, 9, 9, 9]); p['widths'] = [5, 9, 9, 9] + f = next(x for x in p['flow'] if x[1] == 4); f[1] = 5 + out['conservation'] = rejected(fill, 'hosts every whole helper', data, cache) + + def drop_bin(d): # remove one empty capacity-48 bin + g = next(g for g in d['flow'] if any(p['widths'] == [] for p in g['patterns'])) + next(p for p in g['patterns'] if p['widths'] == [])['count'] -= 1 + out['bin_count'] = rejected(drop_bin, 'every port is one bin', data, cache) + + saved = dict(packing.PHI) # weights without the invariant property break the full-bin lemma + try: + packing.PHI[9] = 0 + out['phi_weights'] = rejected(lambda d: None, 'Phi', data, cache) + finally: + packing.PHI.clear(); packing.PHI.update(saved) + return out + + +def tallies(): + out = {} + for w in ('D3', 'D4'): + rows = json.load(gzip.open(PKG / f'evidence/sweeps/REPAIR-{w}.json.gz', 'rt')) + ds = [r['best']['delta'] for r in rows if r['best']] + fam = {'repair': dict(candidates=len(rows), feasible=len(ds), gains=sum(x < 0 for x in ds))} + for name in ('exchange', 'joint'): + rows = json.load(gzip.open(PKG / f'evidence/sweeps/{name.upper()}-{w}.json.gz', 'rt')) + ds = [r['delta'] for r in rows if r['delta'] is not None] + fam[name] = dict(candidates=len(rows), feasible=len(ds), gains=sum(x < 0 for x in ds)) + out[w] = fam + return out + + +def main(): + for rel, h in PINS.items(): + assert hashlib.sha256((PKG / rel).read_bytes()).hexdigest() == h, 'input pin ' + rel + data, cache = packing.load_inputs(), {} + result = packing.check(data, cache) + assert json.loads(json.dumps(result)) == json.loads((PKG / 'evidence/PACKING.json').read_text()), 'PACKING.json differs' + ctl = controls(data, cache) + claim = json.loads((PKG / 'RESULT.json').read_text()) + t = tallies() + assert t == claim['sweeps'], 'sweep tallies' + assert all(f['gains'] == 0 for w in t.values() for f in w.values()) + agg = json.loads((PKG / 'evidence/AGGREGATE.json').read_text()) + assert agg['ilp_minus_incumbent'] == 0.0 and agg['ilp_total_extra_groups'] == 208 + assert agg['price_structure_residual'] < 1e-9 + print(json.dumps(dict(packing=result['status'], controls_rejected=sorted(ctl), sweeps=t), sort_keys=True)) + print('PASS_X54_MIXED_T37_SATURATION_NO_NEW_KAPPA') + + +if __name__ == '__main__': + main()