[petalas]

Summary

Reduce complete research-source metadata validation's peak traced allocation by about 42% on a real large-source package. This is a verifier-engineering contribution with a scoped equivalence proof and finite tests. No new κ is claimed.

 source blob ID and SHA256 checks
-compare pinned bytes with a second whole-file allocation
+compare every working-file byte through a reusable buffer of at most 1 MiB
 retain source-drift rejection and cleanup

The production change adds one helper and replaces one comparison. The existing stable regular-file checkout contract remains required. This contribution was produced by Nick Petalas.

Evidence

Five alternating fresh-process pairs run complete, unchanged validate calls on research/physical-gaussian-core, including its 54,548,480-byte archive input. All ten complete task/manifest results match and no validation fails. The package's scientific verifier is not run or audited.

• Peak traced Python allocation: baseline 127,510,623 bytes in every pair; candidate median 74,011,826 bytes, range 74,011,708–74,011,826.
• Traced validation time has a cost: baseline median 0.232871 s, range 0.225368–0.253928; candidate median 0.330213 s, range 0.329083–0.340037. No runtime speedup is claimed.
• CPython 3.12.15, macOS arm64, warm OS caches. The generated BENCHMARK.json retains raw samples, environment, source hashes and complete-output hashes. These measure Python traced allocation, not RSS or whole scientific replay.
• Seven normal/optimized contract tests cover equal binary/chunk-boundary files, same-size drift, truncation/append, open errors, late read failure after an earlier mismatch, closure, restored inputs and unchanged blob/SHA rejection.
• Both registered commands pass locally; direct normal/optimized checks took 0.60 s and 0.68 s against 30-second limits. Committed bytes/source ancestry and all three export input declarations pass at ac7949b52fb71417bc5d48b8a562042b8e78cf89. Read-only exported normal/optimized replays pass, with all 12 exported files unchanged.
• The complete tooling suite runs 685 tests: 654 pass, 29 skip on this host, and only the two unchanged main-baseline documentation tests below fail. Command: /Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -W error -B -m unittest discover -s tests -p 'test_research_*.py' -v.

From the repository root, set TMPDIR to an external scratch directory and run:

/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/source-comparison-memory/verify.py
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -O -W error -B research/source-comparison-memory/verify.py
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/research_package.py --package research/source-comparison-memory
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/check_research_commit.py --package research/source-comparison-memory --check-source-history

For the optional benchmark, materialize research/physical-gaussian-core and references/semantic-bulk/rad20 first in a sparse checkout, then run:

/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/source-comparison-memory/benchmark.py --work /absolute/new/scratch --pairs 5

Elsewhere substitute a local CPython 3.12 interpreter. Git is needed for source metadata and the temporary test fixtures. The benchmark baseline is main 721c03822c421b7f7fcbb2598993b947c4d88419.

Limitations

The pinned Git blob is still allocated in full. The 1 MiB bound covers the comparison's additional data buffer. Concurrent mutation is outside the existing checkout-stability contract. Native Windows execution, whole-process RSS and scientific/full selected replays remain unrun; mathematical, physical and supplier obligations are unchanged. The complete validator is slower in these traced measurements.

Open next questions

Can the retained pinned Git blob also be streamed while preserving blob identity, SHA256, response framing, final process status and cleanup? First specify a bounded chunk protocol and all existing consumers; the present complete-bytes git_blobs interface blocks that next step.

Related open work

• Reuse publication metadata: 15.7% less complete-check time (no κ) #6162 reuses already authenticated committed metadata in publication byte/history checks; this PR bounds a working-file comparison buffer in registered-package validation. The commands, fixtures and measured quantities differ, and no combined performance is claimed.
• Speed up publication byte/history checks by 12% (no new κ) #6155 removes mismatch-only diagnostic hashing from publication byte/history checks on the same Gaussian fixture; it changes a different command. No combined performance is measured.
• Reuse replay planner input validation within each invocation #6087 reuses planner input-path validation within an invocation; this changes live source-byte comparison during full package validation.
• Use Windows Job Objects in the shared Git blob reader #5721 integrates Windows Job Object ownership in the Git blob reader; the reader and its ownership/cleanup implementation are unchanged here.
• Restore research CI navigation after the PR6030 selection #6050 repairs selected-replay CI navigation; this PR retains that separate documentation issue.

Credit Douglas Colkitt's runner; woklerx/PR1452, Arham-Tahir64/PR1595 and ywz65/PR2539 for the source-reader lineage; later path, deadline, replacement-ref and descendant-cleanup contributors; and Jose Manuel/jmbarrancoml for the unchanged benchmark package. Buffered I/O and memory views are standard methods.

CI notes

Uncertified by CI at head ac7949b52fb71417bc5d48b8a562042b8e78cf89. In run 40725920133, plan fails and replay (research/source-comparison-memory) is skipped. The package replay did not run in CI. A maintainer re-run is needed after main is fixed.

The only two failures are inherited onboarding documentation mismatches on canonical main 721c03822c421b7f7fcbb2598993b947c4d88419, the PR6030 admission committed with [skip ci]. That commit changed the selected record without matching the status and CI navigation pages:

• test_research_current_status.CurrentStatusTests.test_current_section_matches_selected_record expects standalone **0.003125804** in docs/research/current-status.md; the page bolds the larger expression 0.003125804 = 781451/250000000.
• test_research_current_status.CurrentCiNavigationTests.test_current_ci_section_matches_selected_record expects make frontier6030-verify; the CI navigation page, read from docs/ci-verification.md by the test, still names Raise conditional kappa to 0.00310045 with independent helper splits #5599 and make frontier5599-verify.

The hosted run reports Ran 685 tests and FAILED (failures=2, skipped=23). These are the same two failures seen locally and in PR #6120's run 40691776027. Open PR #6050 targets the documentation fix. Neither failing document nor the selected record is changed here.

Local normal/optimized package replay, committed-byte/source-history checks and read-only export replays remain separate evidence. No hosted package certification is claimed. Shared-runner changes select every registered package in the existing planner; no historical scientific suites were launched locally.

Merge Danger

Door: Two-way

Blast Radius: Validation

All registered package source checks use this comparison. The written proof and behavioral controls preserve exact acceptance under the existing checkout contract; the measured memory/time tradeoff is explicit.