Speed complete QQ16 replay with exact integer rows

research/qq16-integer-row-replay/BENCHMARK.json

@@ -0,0 +1,45 @@
+{
+  "python": "3.12.7 (tags/v3.12.7:0b05ead, Oct  1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)]",
+  "platform": "Windows-11-10.0.26300-SP0",
+  "pairs": 3,
+  "measurements": [
+    {
+      "pair": 0,
+      "engine": "reference",
+      "seconds": 25.245870400000058
+    },
+    {
+      "pair": 0,
+      "engine": "integer",
+      "seconds": 19.72184849999985
+    },
+    {
+      "pair": 1,
+      "engine": "integer",
+      "seconds": 20.634534199999962
+    },
+    {
+      "pair": 1,
+      "engine": "reference",
+      "seconds": 26.82034079999994
+    },
+    {
+      "pair": 2,
+      "engine": "reference",
+      "seconds": 35.573120899999594
+    },
+    {
+      "pair": 2,
+      "engine": "integer",
+      "seconds": 21.07019629999968
+    }
+  ],
+  "median_seconds": {
+    "reference": 26.82034079999994,
+    "integer": 20.634534199999962
+  },
+  "ratio_of_medians": 1.2997793184980153,
+  "all_original_outputs_and_nine_scalar_calls_equal": true,
+  "adapter_sha256": "2cc8b12e262a1806492e4fe1a1da4455d6b3a003562838eb016e605a126239e6",
+  "scope": "Fresh processes on a shared host; same complete caller, checks and trace instrumentation. OS caches not flushed. Timings are descriptive."
+}

research/qq16-integer-row-replay/README.md

@@ -0,0 +1,166 @@
+# Exact integer rows in the complete QQ16 replay
+
+This opt-in adapter accelerates one function in the complete frozen QQ16
+caller: `research/joint-query/verify_scalar.py:replay`. Every literal event
+still executes on all 712 independent input columns. The original caller,
+source reconstruction, scalar formulas, native frames, Gaussian columns,
+ordered proofs and corruption controls remain in use. The five original
+output files and stdout must be byte-identical; complete scalar maps from
+all nine invocations must also agree, including the corrupted words.
+
+The local words and original verifiers are the contribution of **Ali Romman**
+in [PR1969](https://github.com/CrocSwap/integer-mult-bounds/pull/1969), exact
+head `a6e6ad3936f67b8f63b5b68c5e75d540610a65e7`, integrated with the reviewed
+scope recorded in `docs/research/community-efficient1000-review.md`. The new
+contribution is an exact execution representation and its caller comparison.
+It changes no construction, supplier, selected result, or frozen source.
+
+## Why this representation is equivalent
+
+Represent each sparse rational row by `(d, a)`, where `d` is a positive integer
+and the represented coefficient at key `j` is `a[j]/d`. The denominator is
+shared by the row rather than reconstructed for each coefficient operation.
+At entry, take the least common multiple of the coefficient denominators
+(one for an empty row) and multiply each numerator by the appropriate integer.
+Input dictionaries are copied independently, even when the caller aliases them.
+
+For an ADD with reduced rational coefficient `p/q`, let target and source
+denominators be `d_t` and `d_s`. Set
+
+```
+D = lcm(d_t, d_s*q)
+A = D/d_t
+B = p*D/(d_s*q).
+```
+
+The new target numerator is `A*a_t[j] + B*a_s[j]`; dividing by `D` gives
+exactly the original target plus `(p/q)` times the source. All quantities
+computed in the loop are integers. The original guard requires distinct
+source and target registers, so updating the private target dictionary cannot
+change the source being read. SCALE multiplies numerators by `p` and the
+denominator by `q`; CLEAR replaces only an allowed owned private row by
+`(1,{})`. Non-scalar retained events use the original whitelist and leave
+these common-chart scalar rows unchanged. Induction over the literal event
+sequence proves equality of every represented coefficient, without a dyadic
+assumption or a finite-field heuristic. Conversion to `Fraction` at return
+reduces each exact coefficient.
+
+The sparse dictionary convention is also preserved. SCALE keeps explicit
+zero-valued keys. ADD removes a zero key only when the original source loop
+visits that key; untouched keys remain. This matters for exact dictionary
+equality, even though explicit zeros do not change the linear map. The adapter
+accepts exact `int` or `Fraction` initial values; its production caller supplies
+only Fraction basis rows. Floats, booleans and complex values are outside the
+declared adapter domain and are refused. No general Python duck-typing
+compatibility is claimed.
+
+## Source boundary and coverage
+
+`SOURCES.json` binds 29 runtime inputs by exact Git blob and SHA-256, including
+all five original Python modules, the certificate, source archive and all
+original proof/review evidence consumed by the caller. They are authenticated
+on every invocation. Modules execute from the authenticated source bytes;
+there is no imported bytecode or persistent verdict cache. Historical task
+catalogue and selected-record pins stay historical provenance.
+
+Only the scalar replay function is replaced. `replay.py:main` itself executes
+unchanged with a loader that supplies the authenticated original modules and
+the single function substitution. `source_data.read_git` still reads the
+original archive from commit `bce54fb3f895cbe5cd4c0eb477c3afeed100190f` and
+checks its exact hash. The corresponding current archive is independently
+pinned here; it is not substituted for the historical read.
+
+Both arms retain:
+
+- Nine scalar invocations: B and F, both composed orders, and the five original
+  timing/private-clear corruptions. Each has all 712 initial rows. The receipt
+  binds every returned row and coefficient, including all dirty helper inputs.
+- Both original data inverse orders, 16 conserved selected sums, 480 restored
+  borrowed helpers, eight zeroed owned private outputs and exact scalar bills.
+- All 37 native cases, 16,384 exact Gaussian basis columns, original phase,
+  translation, owner-action and inverse-sign controls, and the ordered actual,
+  noncentral and singular examples.
+
+Original `REPLAY.json` still identifies the original sources. A separate
+`ADAPTER.json` explicitly records the selected implementation, adapter source
+hash and all nine scalar-map digests, so this is not presented as an execution
+of unmodified code. `verify.py` compares every byte of the five original
+outputs and stdout, with no field exclusions. It additionally compares the
+complete nine-call trace and checks 768 exact prefixes of 96 synthetic words,
+including odd denominators, signs, zero coefficients, cancellation and aliased
+initial dictionaries. Six original error guards agree in exception and message.
+Changed scalar-map and original receipt controls are rejected. The separate
+test mutates each of the 29 runtime files in disposable copies and requires
+source-pin refusal; later unrelated provenance files do not invalidate runtime.
+
+## Measurements
+
+`BENCHMARK.json` records paired alternating fresh-process runs on the same
+machine. Timings include the complete original caller, source authentication,
+the same complete-map trace instrumentation and all original checks. They are
+descriptive observations, not acceptance gates or forecasts for other hosts.
+No OS cache flush or process isolation from unrelated machine load is claimed.
+
+On native Windows 11 / CPython 3.12.7, three alternating pairs gave median
+**26.82 s reference versus 20.63 s integer rows: 1.30x, or 23.1% less elapsed
+time**. Individual reference runs were 25.25, 26.82 and 35.57 seconds; integer
+runs were 19.72, 20.63 and 21.07 seconds. The slow third reference run is
+retained, not discarded. Every pair reproduced the same five original output
+files and nine complete scalar-map digests. The comparison is between the
+two instrumented arms described above, not a selected-frontier timing claim.
+
+The initial CPython 3.12.7 profile of the unchanged caller took 90.9 seconds
+under cProfile. Its nine scalar replays accounted for 29.5 seconds; the scalar
+module's sparse `add` routine accounted for 33.5 seconds including formula
+work. These profile numbers diagnose the bottleneck and are not compared to
+unprofiled timings. The separate benchmark is the before/after comparison.
+
+## Reproduce
+
+From a full repository checkout with the source commit available, Python 3.12
+and Git, using only the standard library:
+
+```sh
+python3.12 -I -O -B research/qq16-integer-row-replay/verify.py
+python3.12 -I -O -B research/qq16-integer-row-replay/test_adapter.py
+python3.12 scripts/research_package.py --package research/qq16-integer-row-replay
+python3.12 -I -O -B research/qq16-integer-row-replay/adapter.py --out /tmp/qq16-fast
+python3.12 -I -O -B research/qq16-integer-row-replay/benchmark.py --pairs 3 --out /tmp/qq16-benchmark.json
+```
+
+Expected verification status: `PASS_QQ16_INTEGER_ROW_REPLAY`. The adapter's
+default engine is `integer`; `--engine reference` keeps the original Fraction
+replay. Outputs go only to the explicit output directory. Frozen inputs and
+historical source files are never edited.
+
+## Decision and limits
+
+The assigned verifier-engineering card is refined to this measured whole-caller
+bottleneck. Under revision 5's expansion gate: the input/output contract is
+unchanged and source-bound; exact representation equivalence is proved above;
+local caller equality and source invalidation are executable evidence; native
+hosting, movement, recursive precision and the supplier invoice remain unknown
+and are not charged zero. This experiment narrows verification work and does
+not expand the QQ16 construction family. Stop if complete equality fails or
+paired whole-caller timing gives no material benefit.
+
+The optimization is opt-in. It does not alter the standard selected cold
+replay or claim a full multiplication theorem. Denominators need not be reduced
+after every event and may grow in other words; no universal runtime or memory
+improvement is claimed. Runtime still requires Git and the original source
+commit. No Git-free full-caller or arbitrary source-loader contract is claimed.
+
+## Open next questions
+
+Should a maintained successor caller adopt this representation? The first step
+is an explicit maintainer decision about the frozen source contract, followed
+by the same complete output and failure-control comparison. This package
+supplies an opt-in boundary and measurements, not an automatic repin. No
+further optimization campaign is justified by this result alone; the remaining
+native, ordered and source-reconstruction stages were deliberately unchanged.
+
+Prepared by **ywz65** with substantial **OpenAI Codex** assistance. Apache-2.0.
+Retain `research/joint-query/NOTICE.md` and its source-specific notices for the
+Ali Romman, gauge44/T16 and centre-encoding/cover-algebra/residual-syndrome/
+coherent-port/interface proof lineage. This package claims none of those
+constructions or original verification methods as new work.

research/qq16-integer-row-replay/RESULT.json

@@ -0,0 +1,104 @@
+{
+  "status": "PASS_QQ16_INTEGER_ROW_REPLAY",
+  "original_outputs_sha256": {
+    "CLEAN-J-CHECKS.json": "2946d03237300839c08acb33b3df993ce61334a55589bf2f9eb17c3d5f463791",
+    "NATIVE-CHECKS.json": "fe959be0bf2490f99efcf6b8109b19cc8d79036d9df54b59598f32de92295054",
+    "ORDERED-CHECKS.json": "c0b3474406807199aae80bc48dc188f03035b5869f113572828615de0b41563a",
+    "REPLAY.json": "70fcedb5160b1347f500ce6db0e2e09c182a42839f8a81196f9aa88fb5892561",
+    "SCALAR-CHECKS.json": "7205aa5f24294f6d6422830719aae6d5b01efb08747fdf746fa9f2f5594a0387"
+  },
+  "scalar_calls": [
+    {
+      "events": 26467,
+      "scalar_events": 21401,
+      "input_rows": 712,
+      "event_sha256": "5650db6861cc29fd1e824f2318aaf72f1b4234388084a5b5c01931698976cb85",
+      "result_rows": 712,
+      "result_terms": 2864,
+      "result_sha256": "9804cfa4fdee7162c29946eecb0a77965be5b13db686d1833d1120df4ea386d7"
+    },
+    {
+      "events": 26091,
+      "scalar_events": 21025,
+      "input_rows": 712,
+      "event_sha256": "83e9f0c64c2c22e57f07bb2a1796c5c498807bc009c08b895b3fa3ad42e8615b",
+      "result_rows": 712,
+      "result_terms": 11488,
+      "result_sha256": "1874d9909d9695900fe9974b72486002982c48da25e3acc4214e78bcec5e6c05"
+    },
+    {
+      "events": 52558,
+      "scalar_events": 42426,
+      "input_rows": 712,
+      "event_sha256": "3d2bcc4bac03d2e2eb5cd8f14d1a9aa786dd9dc1f93add8f02462498134e6861",
+      "result_rows": 712,
+      "result_terms": 13928,
+      "result_sha256": "71a60f7718f456fdb5ee361795e180b151152a19338f7ad2f5443c1ccbc9216c"
+    },
+    {
+      "events": 52558,
+      "scalar_events": 42426,
+      "input_rows": 712,
+      "event_sha256": "8facc69fedb6245d49b119b4873559cc38a25ed090bfbb8fc570101297b46cf9",
+      "result_rows": 712,
+      "result_terms": 4152,
+      "result_sha256": "d504c9fb32d1b07431e1a5fd454c1a1c49dd5a19e626ef0131d43b1db5393ca9"
+    },
+    {
+      "events": 26467,
+      "scalar_events": 21401,
+      "input_rows": 712,
+      "event_sha256": "387ebb4578a73d28298f8e3e368083d923e0822e28ae46f6c40406314fe08ca9",
+      "result_rows": 712,
+      "result_terms": 2448,
+      "result_sha256": "6f86ee913b775792cbcbf12087cc1604687bcd5767090e350ebedbc027ea9995"
+    },
+    {
+      "events": 26467,
+      "scalar_events": 21401,
+      "input_rows": 712,
+      "event_sha256": "fa4044b5e7ad5e239ade8686d669279fa0115ffb3b40b8d5037199df909e8310",
+      "result_rows": 712,
+      "result_terms": 7408,
+      "result_sha256": "8c3a3a70e7d6e753087feb347d67ec90de842bd57c93a5572bbb2cb7fa4c3466"
+    },
+    {
+      "events": 26091,
+      "scalar_events": 21025,
+      "input_rows": 712,
+      "event_sha256": "94ac9b9c8906b880ccb74e0c20e9540127a7eec492bac1ca961184c889921ec1",
+      "result_rows": 712,
+      "result_terms": 11976,
+      "result_sha256": "fdf61d66ddf5f3d3e67dcd36119a55cf8a49a9b0331b6d00775cb6945ddebae7"
+    },
+    {
+      "events": 26091,
+      "scalar_events": 21025,
+      "input_rows": 712,
+      "event_sha256": "b38b7afe94d52c3e8844b8a77fe8810d52da309a5c69c7009b28de4fc8dca7ba",
+      "result_rows": 712,
+      "result_terms": 11840,
+      "result_sha256": "3d0d723a082f1ebb9d1465aafd97c8abb3ab6a4f86a5450b8fa5098af9dac402"
+    },
+    {
+      "events": 26466,
+      "scalar_events": 21400,
+      "input_rows": 712,
+      "event_sha256": "d13ff666bb96a6f1e875961fc812c1778eea82b354da2f1287b830cd295d9bf4",
+      "result_rows": 712,
+      "result_terms": 2865,
+      "result_sha256": "d8af7d9427848007dfd04c10f48cbba700b0a5d4d782773937c6dee3806aa077"
+    }
+  ],
+  "controls": {
+    "synthetic_words": 96,
+    "exact_prefix_comparisons": 768,
+    "original_guard_pairs": 6,
+    "unaligned_denominator_control": "REJECTED",
+    "input_alias_and_zero_key_control": "PASS",
+    "changed_map_witness": "REJECTED",
+    "changed_receipt_byte": "REJECTED"
+  },
+  "scope": "Exact pinned full-caller equivalence, all original tests retained; not a supplier or new mathematical construction.",
+  "original_stdout_bytes_equal": true
+}