Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 115 additions & 0 deletions research/strict-payload-paired-walk/BENCHMARK.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
{
"aggregate_median_ratio": 2.446938669690739,
"alternating_order_trials": 3,
"files": [
{
"body_bytes": 14139930,
"file": "global/hosted/GLOBAL-TEMPLATE.json.gz",
"median_ratio": 2.5888696310711405,
"median_seconds": {
"original": 5.755338600000073,
"paired": 2.2231087000000116
},
"seconds": {
"original": [
5.182077499999991,
5.805689799999982,
5.755338600000073
],
"paired": [
2.164645500000006,
2.2231087000000116,
2.4667056999999204
]
}
},
{
"body_bytes": 9769015,
"file": "global/hosted/ACTUAL-SIGMAS.json.gz",
"median_ratio": 1.622032323562568,
"median_seconds": {
"original": 0.9897019999999657,
"paired": 0.6101617000000488
},
"seconds": {
"original": [
0.9385374999999385,
0.9897019999999657,
0.9907008999999789
],
"paired": [
0.5790405000000192,
0.6101617000000488,
0.615403600000036
]
}
},
{
"body_bytes": 5906328,
"file": "parent16/complex-global/base/split-centres/GLOBAL-TEMPLATE.json.gz",
"median_ratio": 2.3186512024457913,
"median_seconds": {
"original": 3.0083660000000236,
"paired": 1.2974638000000596
},
"seconds": {
"original": [
3.1456018000000086,
3.0083660000000236,
2.8887455000000273
],
"paired": [
1.2974638000000596,
1.354051199999958,
1.2128192999999783
]
}
},
{
"body_bytes": 5906328,
"file": "lift/GLOBAL-TEMPLATE.json.gz",
"median_ratio": 2.4257843561088537,
"median_seconds": {
"original": 2.9161374999999907,
"paired": 1.20214210000006
},
"seconds": {
"original": [
2.95317579999994,
2.9161374999999907,
2.8281696999999895
],
"paired": [
1.2499482000000626,
1.178186299999993,
1.20214210000006
]
}
},
{
"body_bytes": 5906328,
"file": "global/base/split-centres/GLOBAL-TEMPLATE.json.gz",
"median_ratio": 2.750362320201454,
"median_seconds": {
"original": 3.4415566999999783,
"paired": 1.2513103
},
"seconds": {
"original": [
2.724268299999949,
3.4896039999999857,
3.4415566999999783
],
"paired": [
1.1993016000000125,
1.2513103,
1.3865067000000408
]
}
}
],
"limits": "Machine/load-dependent observation, no threshold gate. Mismatches are parsed again by the original fallback and may be slower. Equal-byte fast path unchanged.",
"platform": "Windows-11-10.0.26300-SP0",
"python": "3.12.14",
"scope": "five largest frozen compressed JSON files, whitespace-only reformat, complete classify including decompression and parsing; no setup or fixture-generation timing"
}
88 changes: 88 additions & 0 deletions research/strict-payload-paired-walk/OVERLAP.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
{
"coverage": {
"discussions": {
"complete_pagination": true,
"fetched_at": "2026-10-15T15:22:22.047084+00:00",
"records": 1239
},
"open_prs": {
"complete_pagination": true,
"fetched_at": "2026-10-15T17:34:02.879446+00:00",
"records": 786
}
},
"limits": "Remote PR scope is open PRs including drafts. Closed work is searched through local READMEs, TASK metadata and ledger only. No code diffs or arbitrary package files are searched. Recheck relevant heads before submission.",
"local_records": 1690,
"matches": [
{
"excerpt": "## Reproduce\n\nPython 3.12 standard library, from the repository root:\n\n```sh\npython3 -I -B research/strict-replay-payload-comparison/verify.py\npython3 -I -O -B research/strict-replay-payload-comparison/test_compare.py\npython3 scripts/research_package.py --package research/strict-replay-payload-comparison --check-only\npython3 scripts/research_package.py --package research/strict-replay-payload-comparison\n```\n\nExpected audit marker:\n\n```text\nPASS_STRICT_REPLAY_PAYLOAD_COMPARISON files=105 false_eq",
"kind": "local-readme",
"matched_terms": [
"strict-replay-payload-comparison"
],
"path": "research/strict-replay-payload-comparison/README.md",
"score": 3,
"sha256": "65195fa3cf40e61a7287d4fb7c16151080d7ca5e5b99d2691acd080f3a11fbe3",
"title": "research/strict-replay-payload-comparison/README.md"
},
{
"excerpt": "glas Colkitt; retained supplier lineage.\",\n \"artifact_and_reproduction\": \"python3 -I -B research/strict-replay-payload-comparison/verify.py\"\n },\n \"discussion_url\": \"https://github.com/CrocSwap/integer-mult-bounds/discussions/1472#discussioncomment-18930555\",\n \"overlap\": {\n \"mode\": \"offline shared snapshot; complete pagination, expected stale-cache warnings\",\n \"queries\": [\n \"--task portable-replay\",\n \"--path research/selected-final-stage-portability/compare_payloads.py\",\n ",
"kind": "local-task",
"matched_terms": [
"strict-replay-payload-comparison"
],
"path": "research/strict-replay-payload-comparison/TASK.json",
"score": 3,
"sha256": "33cfce0e80cda6db73511d1ada30715d92f7b37108419f0f783f78be62415e38",
"title": "research/strict-replay-payload-comparison/TASK.json"
},
{
"excerpt": "# Paired traversal for strict replay payload comparison\n\nThis opt-in successor accelerates positive comparisons in\n[`strict-replay-payload-comparison`](../strict-replay-payload-comparison/README.md)\nwithout materializing two complete tagged JSON trees. Its parser, path rules,\ntiming validation, directory inventory, CLI and negative diagnostics come from\nthe SHA256-authenticated PR2040 comparator by **commdata2338**. The predecessor\nfiles and selected evidence remain unchanged.\n\n`RESULT.json` rec",
"kind": "local-readme",
"matched_terms": [
"strict-replay-payload-comparison",
"paired traversal"
],
"path": "research/strict-payload-paired-walk/README.md",
"score": 2,
"sha256": "e8682d7aa299e742211b680e50b2807132b06463240c315d746e811e5bcb0bf7",
"title": "research/strict-payload-paired-walk/README.md"
},
{
"excerpt": "if it fits the budget. No cold supplier replay and no new mathematics.\nExpected artifact: `research/strict-replay-payload-comparison/` with a source-bound audit, a strict comparator, an exact receipt and positive and negative controls. Verifier tooling and a scoped audit finding only.\nClaim expires: 2026-10-14T19:36Z\n\nNearest overlap: PR #1824 supplies the cross-interpreter replay and comparator; this audits its untested equality and key-relocation boundaries. The `selected-generated-identity` a",
"kind": "discussion-comment",
"matched_terms": [
"strict-replay-payload-comparison"
],
"number": 1472,
"score": 1,
"title": "DONE: Linux/POSIX portability audit and descendant cleanup regression",
"updated_at": "2026-10-14T17:59:31Z",
"url": "https://github.com/CrocSwap/integer-mult-bounds/discussions/1472#discussioncomment-18930555"
},
{
"excerpt": "declared time fields, and each individual deletion rejects.\n\n**Reproduce:** `python3 -I -B research/strict-replay-payload-comparison/verify.py`, expecting `PASS_STRICT_REPLAY_PAYLOAD_COMPARISON files=105 false_equivalences=39 type_cases=25 path_cases=12 timing_rejections=22 no_new_kappa`. The registered CPython 3.12.3 replay, nine semantic test groups, an optimized minimal-source export, real corrupted-receipt rejection and immutable archive-header rejection pass.\n\n**Also observed:** the unchang",
"kind": "discussion-reply",
"matched_terms": [
"strict-replay-payload-comparison"
],
"number": 1472,
"score": 1,
"title": "DONE: Linux/POSIX portability audit and descendant cleanup regression",
"updated_at": "2026-10-14T17:59:29Z",
"url": "https://github.com/CrocSwap/integer-mult-bounds/discussions/1472#discussioncomment-18930899"
}
],
"matches_total": 5,
"searched_at": "2026-10-15T17:35:17.495407+00:00",
"status": "POSSIBLE_OVERLAP_NOT_A_NOVELTY_CERTIFICATE",
"terms": [
"strict-replay-payload-comparison",
"paired traversal",
"paired walk"
],
"warnings": [
"open_prs: offline; cached data may miss current changes",
"discussions: offline; cached data may miss current changes"
]
}
128 changes: 128 additions & 0 deletions research/strict-payload-paired-walk/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Paired traversal for strict replay payload comparison

This opt-in successor accelerates positive comparisons in
[`strict-replay-payload-comparison`](../strict-replay-payload-comparison/README.md)
without materializing two complete tagged JSON trees. Its parser, path rules,
timing validation, directory inventory, CLI and negative diagnostics come from
the SHA256-authenticated PR2040 comparator by **commdata2338**. The predecessor
files and selected evidence remain unchanged.

`RESULT.json` records complete-result differential agreement on **1,012 calls**:
365 calls from the unchanged predecessor audit (105-file corpus, 39 adverse
witnesses and its other controls), 625 additional semantic cases, and 22
compressed JSON bodies reformatted to exercise the new traversal. The original
complete audit receipt also agrees. This is comparison evidence, not a fresh
execution of the mathematical supplier.

## What is preserved

Both inputs are completely decompressed and parsed. Duplicate object keys and
nonfinite constants are rejected; Decimal values remain exact. Integers,
Decimals and Booleans retain different tags. Every declared timing path is
validated on both inputs, including byte-identical inputs. Root tokens and
canonical POSIX prefixes retain the original injective path tagging, longest
prefix rule and literal-string distinction. Unknown fields and array order
remain significant. No digest or previously cached equality replaces reading
the actual payloads.

`paired` compares matching containers recursively. For objects with root maps,
it makes only a per-object map from normalized key to original key/value; with
empty root maps it compares the original key sets. Arrays compare their lengths
and all corresponding entries. Scalar comparison first checks the exact Python
type produced by the retained parser. Object-key order is irrelevant, as it is
in the original sorted tagged representation. Original keys are retained for
the two timing-path traversals: a normalized key is never mistaken for a path
component.

An induction on these cases shows that a successful traversal gives exactly
the same tagged-tree equality as PR2040. For the timing quotient, the two nodes
are equal as timing tags only when **both** original paths are masked; otherwise
the usual recursion applies. All masks have already passed the original shape,
type, finiteness and nonnegativity checks. Equal payload bytes under equal root
interpretations retain the predecessor's existing shortcut.

If a comparison fails, or the speculative traversal encounters an error, the
adapter invokes the original comparator on the original bytes and policies.
This preserves its entire result, including invalid-input precedence and the
first-difference diagnostic. A new traversal cannot accidentally replace an
invalid sibling by an early mismatch. Binary files always take this original
path. The adapter changes only a private, authenticated module instance; it
does not patch another importer or rewrite a frozen file.

## Measurements and limits

`BENCHMARK.json` contains three alternating-order trials on the five largest
compressed JSON bodies in the pinned corpus. The run changes only leading and
trailing JSON whitespace and measures complete `classify` calls, including
decompression and parsing. Both methods must return the full same result.
Fixture creation, source authentication and corpus extraction are outside the
timed interval. Reproduce the observation on the intended machine; the timing
is not an acceptance threshold and concurrent load can change it.

On CPython 3.12.14 / Windows 11, the sum of the five per-file median times fell
from 16.111 seconds to 6.584 seconds (**2.447x**). Individual ratios were
1.622x–2.750x. These timings include the same full-input checks and do not use a
warm result cache. The separately recorded tree-only profiling was used to
choose the optimization, not as the end-to-end speedup claim.

The saving comes from avoiding complete tagged-tree allocations on positive
cases. Parsing still constructs both JSON values, and wide objects still need
per-object key maps when relocation is enabled. Peak memory was not measured.
Different or invalid payloads can be slower because the original fallback
parses them again. Equal-byte and ordinary gzip-container cases already have a
shortcut in the predecessor. There is no claimed speedup for those cases.

The unchanged contract is JSON plus canonical POSIX root maps. PR2344's Windows
drive-root adaptation is a separate caller and is not replaced here. This does
not add UNC handling, raw digest rebinding, a supplier admission, a new circuit
cost, or a new kappa. Inputs remain trusted local files subject to the existing
parser's recursion and resource limits; arbitrary attacker-sized JSON is not a
new supported service interface.

## Reproduce

Python 3.12 or later, standard library only:

```sh
python3 -I -B research/strict-payload-paired-walk/verify.py
python3 -I -O -B research/strict-payload-paired-walk/test_compare.py
python3 scripts/research_package.py --package research/strict-payload-paired-walk
python3 -I -B research/strict-payload-paired-walk/benchmark.py --rounds 3 --out /tmp/paired-benchmark.json
```

The test file runs all nine retained predecessor test groups against the new
adapter, including its CLI, plus six adapter controls. These check source-change
invalidation before execution, original diagnostic equality, original-key
masking, rejection of invalid identical timing data, detection of a Python
equality mutant, and that positive traversals do not build tagged trees.
`SOURCES.json` authenticates all 12 inherited inputs by SHA256 and Git blob ID;
the replay works without a Git checkout or network. `RESULT.json` is compared
as canonical JSON bytes, so Boolean/integer substitutions cannot satisfy it.

The CLI is the same as the predecessor, with this script path:

```sh
python3 research/strict-payload-paired-walk/compare.py REF_DIR RUN_DIR \
--ref-root W=/old/root --run-root W=/new/root --out comparison.json
```

Use an explicit timing policy only when that caller authorizes those exact
typed paths. This package introduces no default ignored fields.

## Open next questions

Can an actual replay wrapper obtain a useful end-to-end saving after its own
source authentication and unchanged-output shortcuts? First run its complete
output comparison through both implementations with identical policy and
retained negative fixtures; adoption is blocked on caller-specific review.

Can the negative path reuse already parsed values while preserving the
predecessor's exact error ordering? First profile a representative changed
large payload. This is left open because positive-path allocation is the
measured bottleneck addressed here, and moving diagnostic logic would enlarge
the verification boundary.

Prepared by **ywz65**, with substantial OpenAI Codex assistance in implementation,
proof review, testing and writing. Apache-2.0. Credit to commdata2338 for the
strict comparator and controls, Josie Elliston for the historical portability
comparator, Rohan Arun and the retained supplier authors for the frozen corpus.
19 changes: 19 additions & 0 deletions research/strict-payload-paired-walk/RESULT.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"classifications": {
"binary-diff": 1,
"bytes": 252,
"gzip-container": 23,
"invalid": 40,
"json-diff:declared-timing": 6,
"json-diff:proof-relevant-candidate": 582,
"json-format": 106,
"json-relocated": 2
},
"compressed_json_reformats": 22,
"corpus_files": 105,
"extra_differential_cases": 625,
"retained_adverse_witnesses": 39,
"retained_classify_calls": 365,
"scope": "exact comparison equivalence on stated corpus/controls; no supplier replay or new kappa",
"status": "PASS_STRICT_PAYLOAD_PAIRED_WALK"
}
Loading