Reduce complete dirty-phase scientific replay time by 17.87% (no κ)

research/dirty-phase-row-products/README.md

@@ -0,0 +1,173 @@
+# Fewer integer products in the complete dirty-phase replay
+
+This implementation was produced by Nick Petalas. It removes **255,310
+scalar products** from the complete scientific replay of Philipp / mightyOil's
+[PR1294](https://github.com/CrocSwap/integer-mult-bounds/pull/1294).
+Five alternating fresh-process pairs show a **17.87% local median wall-time
+reduction**, with identical complete scientific stdout and zero failures.
+Claim type: scoped implementation-preservation proof, exact finite conformance
+and local performance evidence. No new mathematical identity or kappa is claimed.
+
+The opt-in adapter changes only `check_dense.multiply` in a private module.
+The original symbolic checker, substitution loops, counterexamples, source
+files, scientific receipts and six package tests remain intact. The
+predecessor mathematics is the preserved workload, not new verification credit.
+
+## Implementation and proof
+
+For nonempty rectangular integer matrices A of size m by n and B of size n
+by p, the original kernel evaluates every term in
+
+    C[i,j] = sum_k A[i,k] B[k,j].
+
+The new kernel allocates a zero m-by-p result, then visits each row i and
+inner index k. If A[i,k] is nonzero, it adds A[i,k] times row k of B to
+result row i. After processing inner indices less than k, the accumulator
+is the corresponding partial sum. Terms with A[i,k]=0 contribute integer
+zero. Induction on k proves equality of the complete matrices. Exact integer
+addition permits the changed accumulation order. Both inputs remain unchanged
+and each result row is fresh. The empty-output-row boundary is also preserved.
+
+The scalar-product count changes from m*n*p to p times the number of nonzero
+entries of A. This is classical row expansion. Custom scalar objects,
+floating-point arithmetic and malformed matrices are outside the contract.
+
+`adapter.py` authenticates all fourteen declared predecessor files by SHA256
+and Git-blob hash before compiling any predecessor module. It executes those
+authenticated source bytes directly, without predecessor bytecode-cache reads
+or scientific-result caching. Changed dependencies reject before scientific
+execution. Each invocation loads fresh private modules and compares both
+complete generated receipts with their authenticated originals.
+
+## Evidence
+
+The complete dense consumer reaches **16,294** multiplication calls across
+26 shapes with dimensions at most eleven. The checking adapter compares every
+new result against the original kernel and checks both inputs after both
+calls. The old kernel enumerates 562,574 scalar products; the new kernel
+enumerates 307,264. Their difference is exactly 255,310. These counts include
+all direct matrices, normal-form substitutions and four invariant-removal
+counterexamples of that consumer.
+
+Separate conformance covers all 6,561 pairs of two-by-two matrices with entries
+in {-1,0,1}, 64 rectangular fixtures and the empty-output boundary, for **6,626**
+cases. Sixteen source controls reject corruption of each of fourteen actual
+inputs, a missing source and an unknown entry. Four arithmetic faults fail
+explicit kernel fixtures; four additional well-shaped faulty kernels fail the
+actual complete dense consumer. All six retained package tests run unchanged,
+including relocation, source/receipt/order faults and exclusive output creation.
+No assertion is needed for acceptance.
+
+Complete scientific JSON before its trailing newline has SHA256
+`7ab04e8ef093875f43f3d1240eb6bcf224c7d11f069eb7e40bfdc469e6cc9df1`.
+The checking adapter also generates a digest of every reached input/output
+triple and a shape histogram. Standalone verification prints a generated
+receipt bound to every regular package file. No receipt is handwritten.
+
+## Complete scientific caller timing
+
+The timed boundary starts before launching a fresh CPython process and ends
+after successful exit and captured output. Each process includes source
+authentication, both module loads, the complete universal polynomial body,
+the complete dense body, both expected-receipt comparisons and complete JSON
+serialization. The six retained tests, new conformance tests, metadata checks,
+shell startup and CI planning run separately. This does not claim a 17.87%
+reduction of the whole generic registered-package job or a changed default
+predecessor command.
+
+| Pair | Original kernel, seconds | Row expansion, seconds |
+| --- | ---: | ---: |
+| 1 | 0.075181084 | 0.060592416 |
+| 2 | 0.074843250 | 0.061917208 |
+| 3 | 0.074595417 | 0.062196792 |
+| 4 | 0.075584708 | 0.061222917 |
+| 5 | 0.074929500 | 0.061536584 |
+| Median | 0.074929500 | 0.061536584 |
+
+The ranges are 0.074595417 to 0.075584708 seconds before and 0.060592416 to
+0.062196792 seconds after. Every pair improves; the ranges are separated.
+All ten outputs are byte-identical, with stdout SHA256
+`292f0080f85366a0fc96108216970c365fb6c0fd52913133c962eff14902b2ea`,
+including the final newline.
+
+The machine runs macOS 27.0.1 arm64 and CPython 3.12.15, with eighteen logical
+cores and entry load averages 2.10, 2.22 and 2.37. Other research processes were
+present. Both sides use identical options and warm filesystem caches; module
+state is fresh. An earlier five-pair pilot with the same kernel is excluded
+from this final table. An untimed profile places 0.073 of 0.133 profiled seconds
+in the original dense multiply. Python loops and integer arithmetic limit this
+single-process caller. The measured reduction is smaller than that profiled
+share; the profile itself is not a wall-time sample.
+
+`benchmark.py` reproduces the five-pair procedure and emits every duration,
+failure count, complete-output digest, source hashes, program hashes and
+environment. Timing is opt-in, with no CI threshold.
+
+## Reproduce
+
+From the contribution checkout root, using CPython 3.12 and its standard library:
+
+```sh
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/dirty-phase-row-products/verify.py
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -O -W error -B research/dirty-phase-row-products/verify.py
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/research_package.py --package research/dirty-phase-row-products
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/check_research_commit.py --package research/dirty-phase-row-products --check-source-history
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/dirty-phase-row-products/benchmark.py
+```
+
+Another CPython 3.12 path may replace the local path. Expected markers are
+`PASS_DIRTY_PHASE_ROW_PRODUCTS`, `PASS_REGISTERED_RESEARCH_PACKAGE` and
+`PASS_RESEARCH_COMMIT_BYTES`. Direct checks write only stdout. Retained file
+fault tests use system temporary storage; set `TMPDIR` to a private external
+directory when collecting evidence. Exported replay needs this package and
+all fourteen declared inputs at their relative paths, but no Git or network.
+
+## Scope, sources and credit
+
+The base is canonical main `7c49fe32c55de02c2c873655c804b16e4c3fef0a`, priority
+revision 8, selected conditional PR6030. The replay-runtime card is refined
+to this measured integer kernel. Current main, open and recent-account PRs,
+merged object/general-kernel searches, active Discussions and local reservations
+were checked before the
+[advance claim](https://github.com/CrocSwap/integer-mult-bounds/discussions/1166#discussioncomment-19363383).
+Sparse historical notes and unpublished work remain coverage limits.
+
+Credit Philipp / mightyOil PR1294 for both complete scientific consumers,
+identities, receipts, tests and original implementation; Charles Bethin PR1278
+for the interval proof and phase formulas; Hanson Sun PR1244/1085, Chafik
+Boukhalfa PR628/905, Gareth Davidson PR877 and their retained lineage. Their
+source and credit lines are unchanged. The new contribution is this concrete
+implementation, preservation proof, conformance and complete-caller measurement.
+Distributivity and sparse multiplication are classical. New work is Apache-2.0.
+
+Physical caller, rounded leaves, precision/tape, stock, compiler, supplier
+admission and the complete multiplication theorem remain outside this result.
+No selected cold replay, native suite, Lean or Docker run is needed or claimed.
+The selected frontier and its five assumptions are unchanged. Independent
+human and formal proof-kernel review are unclaimed.
+
+## Open next questions
+
+Should a maintained scientific replay adopt this integer kernel? Begin with
+one named caller's profile, exact matrix contract and complete conformance.
+This isolated package supplies no default adoption or selected-caller gain.
+The structured first step and blocker are in `TASK.json`.
+
+## Verifier constraints
+
+New functions are below sixty lines. Fixed fixture domains bound every loop.
+Caller dimensions are at most eleven; independent rectangular fixtures have
+dimensions at most four. Fourteen source files are limited to one MiB each.
+Five benchmark pairs have a thirty-second subprocess timeout. Registered
+commands each have a 120-second outer bound. The unchanged retained six-test
+suite invokes finite children without individual timeouts; the registered
+runner's process-group deadline bounds that inherited path.
+
+Power of Ten rule 3 deviation: CPython allocates managed integers and containers.
+Fixed domains and source sizes bound live collections; no fixed-allocation
+runtime is claimed. Rule 8/9 deviation: authenticated compilation, `runpy` for
+the local adapter, private replacement and the fixed unittest loader use dynamic
+execution/dispatch. Targets are three named predecessor modules, this adapter
+and the explicitly enumerated fault kernels. No source text is rewritten.
+Warning-strict ordinary/optimized checks and source compilation supply
+diagnostics. No standalone Python linter is installed in this environment.

research/dirty-phase-row-products/SOURCES.json

@@ -0,0 +1,102 @@
+{
+  "schema_version": 1,
+  "base_commit": "7c49fe32c55de02c2c873655c804b16e4c3fef0a",
+  "files": [
+    {
+      "path": "certificates/selected-result.json",
+      "usage": "provenance",
+      "git_blob": "d59b5035e7b4cbaa9d688a1d32dd0d6eb79c466a",
+      "sha256": "9b8d0eea95b3bf0dfcb7938a732a6439b87a2685dec2a5ab9e525b3b34debbfa"
+    },
+    {
+      "path": "docs/research/agent-task-catalog.json",
+      "usage": "provenance",
+      "git_blob": "3404fbdd5e7529a0674eef26114b5071e4586cec",
+      "sha256": "ead6cdc9d0af6d58e8a49744d8ec8c7c15e13d2ba93c276c434bcf9b51b93a81"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/CERTIFICATE.json",
+      "usage": "input",
+      "git_blob": "40983343f6839cd98f60878ba5ebdbaaa74a23c2",
+      "sha256": "d7af9d33856ec45d4163bae1493abff2503664b58876210047491c5ba2c383af"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/DENSE-CHECK.json",
+      "usage": "input",
+      "git_blob": "5c6d7295d199e72eaafbe8592d041218b5e67597",
+      "sha256": "43f23f4ba8cce04488662d0c0bc6595b688073420fbb89ad67c40a33670fff9d"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/README.md",
+      "usage": "input",
+      "git_blob": "fdebc9c89099f0f59b2b1cc4353bc33786529011",
+      "sha256": "3620ec8ae93b66af22d9dff045a339ff75160b4e6c334518a01768dd69e6a0ea"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/SOURCES.json",
+      "usage": "input",
+      "git_blob": "0d90ca139d061b286af584fbec5c5b8c3c629a9e",
+      "sha256": "167117bffb8c7587ee0bc676e37b45b9b345a00ae24502ad3a66ee664867d548"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/TASK.json",
+      "usage": "input",
+      "git_blob": "a678d0ad03bc8dbe98c0abb89dc7d80117f89203",
+      "sha256": "94a9e1bb53d005c413564a91eeb42ce3bd611d4011ee2a6669f4945ab0c9677d"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/VALIDATION.json",
+      "usage": "input",
+      "git_blob": "5abd00051e8d8d6b3dd513cd3238d36936f1502f",
+      "sha256": "493dafa182db67f1955d86a414a3090b4f9593da03a098c35f80898bf22ba04c"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/VERIFY.json",
+      "usage": "input",
+      "git_blob": "58792a2654becf30f6fe2ab0bb75e2673db9f5ca",
+      "sha256": "d74193b96e897c9f231a0133b9d6ecec4f4e32d8bf138e18b13a9cba7a853011"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/check_dense.py",
+      "usage": "input",
+      "git_blob": "cf04137c1bcf9a5a0c8df55174c2937d4fc2f177",
+      "sha256": "69629276352e02f335ea71af0c83608821d2c7ced315e4f4b522b48ebbb93740"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/test_package.py",
+      "usage": "input",
+      "git_blob": "cc708f022848e27e37ed0cc59e449f93c693607c",
+      "sha256": "5116b9d96a9eccdc08c3e5db27207b2fedd0dfec8a4a77f4ea1e7d2b5d0c29a2"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/vendor/LICENSE",
+      "usage": "input",
+      "git_blob": "261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64",
+      "sha256": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/vendor/PR1278-NOTICE.md",
+      "usage": "input",
+      "git_blob": "b543c9e72463cb495ed380d5ded4c4235571483a",
+      "sha256": "e5d70a6492de305c914b1bd96b84506f6124c5fbfc3d09434ab241d1f4e2b438"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/vendor/PR1278-PROOF.md",
+      "usage": "input",
+      "git_blob": "de9c8fca358c225e2ef30180c553717208015252",
+      "sha256": "17d346b53eb71780410e1ec5e87d56bc444215a540e98c84283f12553c0c81e9"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/vendor/PR1278-README.md",
+      "usage": "input",
+      "git_blob": "01538b770abfbbca1ac753d9123aa8d0b5003e2a",
+      "sha256": "a05c0daa6ca62a8b619cfd8fbe38c1663b58ba7c258f9a40bdbb79b488fcf1fc"
+    },
+    {
+      "path": "research/universal-dirty-phase-identities/verify.py",
+      "usage": "input",
+      "git_blob": "ac671f6a2ea28745b7dac50aa68cbf564b290b25",
+      "sha256": "3b440adcaa258585b064aeb05ce02ba171acc88c0021ed1ab0a245ccf6eeb15b"
+    }
+  ]
+}

research/dirty-phase-row-products/TASK.json

@@ -0,0 +1,62 @@
+{
+  "schema_version": 1,
+  "task_id": "replay-runtime",
+  "priority_revision": 8,
+  "base_commit": "7c49fe32c55de02c2c873655c804b16e4c3fef0a",
+  "original_card": {
+    "id": "replay-runtime",
+    "question": "Can one measured replay bottleneck be sped up without dropping coverage?",
+    "starting_points": [
+      "scripts/run_isolated_tests.py",
+      "scripts/verify_replay_coverage.py",
+      "certificates/selected-result.json"
+    ],
+    "methods": [
+      "Profile one isolated replay and optimize the measured bottleneck.",
+      "Design source-bound reuse with explicit dependency invalidation and negative controls."
+    ],
+    "acceptance_gate": "Report before/after wall time on the same fixture, retained checks and invalidation behavior.",
+    "first_step": {
+      "command": [
+        "python3",
+        "-B",
+        "scripts/research_selected_baseline.py"
+      ],
+      "expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY",
+      "inspect": "Resolve the current selected review, separate complex/ordinary BIT/admitted BIT/outer budgets, and compare the card against live handoffs before starting. Historical ceilings require exact source/model equivalence.",
+      "scope": "Source-binding orientation only; not a scientific replay or all-size proof. Materialize selected inputs if this checkout is sparse."
+    },
+    "first_experiment": "Measure one unchanged fixture, preserve check coverage, and test one optimization with invalidation controls.",
+    "bottleneck_link": "Faster trustworthy replays increase review throughput and let mechanism candidates be checked sooner.",
+    "allowed_variations": [
+      "Profile one stage or design one source-bound cache boundary."
+    ],
+    "scope_change_triggers": [
+      "Skipping coverage or comparing different fixtures to claim speedup."
+    ],
+    "overlap_terms": [
+      "replay runtime",
+      "cache invalidation",
+      "isolated tests"
+    ]
+  },
+  "scope_status": "refined",
+  "actual_scope": "Remove zero-left-factor scalar products from the complete PR1294 scientific caller while preserving exact source-bound outputs and every retained package test.",
+  "scope_change_reason": "The generic replay-runtime card is refined to one measured dense integer multiplication consumer. This is implementation work, not a scientific audit.",
+  "bottleneck_link": "Shorter complete scientific replays reduce repeated checking costs without changing their evidence or the selected frontier.",
+  "budget_minutes": 60,
+  "stop_rule": "Ship the first verified implementation improvement; do not optimize additional kernels.",
+  "outcome": "completed",
+  "evidence": "README.md; adapter.py; verify.py; benchmark.py. The standalone verifier generates a hash-bound receipt on stdout.",
+  "author": "Nick Petalas",
+  "ai_assistance": "Independent human and formal proof-kernel review are unclaimed.",
+  "open_questions": [
+    {
+      "question": "Should an actual maintained replay adopt this integer multiplication kernel?",
+      "why_it_matters": "The opt-in implementation does not change the existing registered predecessor commands.",
+      "first_step": "Profile a named maintained consumer, verify its exact integer matrix contract and compare its complete command with this implementation.",
+      "blocked_by": "A separate adoption decision and caller-specific conformance are required."
+    }
+  ],
+  "ci_status": "pending"
+}

research/dirty-phase-row-products/VERIFY.json

@@ -0,0 +1,49 @@
+{
+  "schema_version": 1,
+  "status": "ready",
+  "python": "3.12",
+  "requirements": [],
+  "inputs": [
+    "research/universal-dirty-phase-identities/CERTIFICATE.json",
+    "research/universal-dirty-phase-identities/DENSE-CHECK.json",
+    "research/universal-dirty-phase-identities/README.md",
+    "research/universal-dirty-phase-identities/SOURCES.json",
+    "research/universal-dirty-phase-identities/TASK.json",
+    "research/universal-dirty-phase-identities/VALIDATION.json",
+    "research/universal-dirty-phase-identities/VERIFY.json",
+    "research/universal-dirty-phase-identities/check_dense.py",
+    "research/universal-dirty-phase-identities/test_package.py",
+    "research/universal-dirty-phase-identities/vendor/LICENSE",
+    "research/universal-dirty-phase-identities/vendor/PR1278-NOTICE.md",
+    "research/universal-dirty-phase-identities/vendor/PR1278-PROOF.md",
+    "research/universal-dirty-phase-identities/vendor/PR1278-README.md",
+    "research/universal-dirty-phase-identities/verify.py"
+  ],
+  "commands": [
+    {
+      "argv": [
+        "{python}",
+        "-I",
+        "-W",
+        "error",
+        "-B",
+        "{package}/verify.py"
+      ],
+      "timeout_seconds": 120,
+      "checks": "All 16294 reached products; complete primary/dense receipts; finite rectangular integer conformance; actual source and arithmetic corruptions; all six retained package tests."
+    },
+    {
+      "argv": [
+        "{python}",
+        "-I",
+        "-O",
+        "-W",
+        "error",
+        "-B",
+        "{package}/verify.py"
+      ],
+      "timeout_seconds": 120,
+      "checks": "The same complete conformance and rejection checks with assertions disabled."
+    }
+  ]
+}