Add gated bounded parallel merge preparation

[aaltshuler]

Multi-table merges currently prepare each table sequentially. This adds ordered preparation windows of up to four tables behind scoped diagnostic controls, while retaining the existing graph publication protocol. Production remains at width one, with a separate hard width-one cap for HTTP.

Stacked on #662. This implements the diagnostic stage of #638 under canonical RFC 0057, accepted separately in #638. Production activation remains unfinished, and this PR stays a draft.

Changes

• Extract owned table-preparation results and collect them in canonical order, with serial barriers for adoption and Blob work.
• Add shared accounting for controlled retained allocations, bounded fallback from four to two to one, deterministic error selection, and private scratch ownership. Graph validation, authority fencing, recovery, serial durable effects and single publication retain their existing protocol.
• Bound eligible native scans and add scoped preparation/ObjectStore API measurements. Preserve the serial fast path and boxed-future boundaries required by existing default-stack tests.
• Extend existing correctness owners and small reproducible merge fixtures, including current-format history-depth controls.

Rebase and GQT verification — 2026-09-07

Rebased onto updated #662 at f57268dc; the complete stack includes main 5f94a741.

• Added multi_table_merge_conflict_retry.gqt: four tables, 16 seed rows, conflicting changes in two tables, restart and exact source/target checks after refusal, followed by a corrected retry and another restart. It passed on perf(branches): preserve snapshots and reuse verified views #662 before this refactor and guards unchanged serial-default semantics.
• Freshly rebuilt Add gated bounded parallel merge preparation #668 runner at 2f79bc5e: all 53 GQT cases passed. The new case takes about 0.25 s locally. GQT does not activate private width-four controls; existing Rust scheduler/budget/cancellation owners remain necessary.
• Rebase integration preserves the shared GQ/HTTP merge handler and routes both through the request-owned width-one entry. Extended the existing drop/retry fixture to cover the GQ statement with requested diagnostic widths 1/2/4.
• Preserved main's typed FTS filter-demand and scan-statistics changes alongside bounded merge scans. Restored canonical RFC 0057 references and removed the obsolete duplicate RFC 0054 retained by plain rebase.
• Formatting, typos, documentation, metadata and whitespace checks passed. Current-head CI passed the full workspace/failpoint test graph, AWS-server feature tests, both Clippy graphs, and the genuine V5/V6 format fence. The GQT workflow passed 127 runner tests and 53 cases; the real DST suite passed 78 tests, with 30 ignored instruments. The local runner unit suite also passed all 127 tests. AWS-server logs explicitly confirm the new GQ width-one drop/retry case and existing real HTTP disconnect case. Live RustFS/Azurite integration jobs were skipped by the PR workflow. The second local feature-graph build was stopped before its tests ran, and those checks were completed in CI.

This update does not repeat AWS performance benchmarks; historical measurements below apply to their recorded revisions. The fresh DST run above verifies the serial default, not diagnostic width-four simulation. Production and HTTP stay at width one, and this PR remains a draft.

Native AWS/S3 results — 2026-09-06

Completed: 58 measured merges and two smoke merges passed complete target/source verification; all 58 raw records passed independent admission and provenance checks. Linux release binaries were built in AWS CodeBuild and each passed all 11 benchmark-contract tests. Runs used the existing c6id.2xlarge lab in us-east-1, calling the embedded engine directly against native S3 with no injected delay.

Main matrix: 121 populated node tables, four rows per table, two disjoint updates on each side, five alternating baseline/current pairs per point. Times measure the merge after fresh graph open; setup and verification are excluded. The current build uses diagnostic preparation width four.

Touched tables	Frozen serial	Diagnostic width 4	Ratio of medians
1	3.068 s	2.884 s	1.064×
8	10.407 s	6.375 s	1.632×
29	33.509 s	18.561 s	1.805×

The single-table result does not establish a consistent gain: its paired-ratio median is 1.022×, and one 5.857 s baseline outlier is retained. Eight-table paired ratios range from 1.510× to 1.718×; 29-table ratios range from 1.754× to 1.915×. The 2× multi-table target remains unmet.

Small history matrix: four populated/touched tables, only 16 live rows per branch, two alternating pairs per point. Reachable history and fragmentation grow together in the current storage format.

Extra history commits	Frozen serial	Diagnostic width 4	Ratio of medians	Manifest-read counter, both variants
0	6.185 s	4.104 s	1.507×	72
16	8.063 s	6.244 s	1.291×	264
64	12.898 s	10.645 s	1.212×	840

At H64, initial graph preparation and final manifest publication together consume 72.2% of current merge time; table candidate preparation takes 0.972 s. The unchanged manifest-read counts show the history cost is not removed by preparation concurrency. At 29 touched tables, preparation accounts for 35.5%, physical publication 27.7%, recovery confirmation 17.1%, and final revalidation 11.0%. Publication, confirmation, and revalidation together take 55.8%; the ten disjoint top-level timers cover 99.25% of elapsed time. Phase fractions are calculated within each sample before taking medians; nested timers are not added to their parents, and the serial build's unavailable preparation timer is not inferred.

Controls (two pairs each): production-default width-one S3 merges at eight touched tables are essentially unchanged, 10.792 → 10.752 s. On the same EC2 instance's XFS/NVMe, width-four eight-table merges are 133.748 → 111.715 ms (1.197×). Largest current post-merge S3 fixture: 9.950 MiB; highest operation-process peak RSS: 125.34 MiB. All current-build samples ended with settled preparation/I/O gauges and no fallback or discarded results; observed preparation width never exceeded four.

Runs were sequential, fresh processes, nice 15, with a 256 MiB Lance pool and two Lance CPU, Lance I/O, Tokio, and Rayon threads. OS/device/S3 caches were uncontrolled. These are diagnostic scalar-merge measurements, not HTTP, concurrent-merge, disconnect, crash/recovery, Blob, or width-four DST qualification. Current-format history is not a historical-format or retired-branch test. Wrapped ObjectStore API counts are not wire-request/retry counts, and the Lance pool is not an RSS or pre-decode allocation cap. The earlier branch-context optimization is already in the baseline; this is not a comparison against released 0.10.0.

AWS provenance, reproducible artifacts, and cleanup

The runtime bases remain cbd66386015f890a885d2e22998c7d7834fce141 (serial) and 28488d7c39cb547e551158dbfb614703a042a257 (current). Both received the same harness-only native-S3 patch, producing clean measurement commits 5ed30c55e10bb3d307d4208f161a8ba43dcb5576 and ea0311502689d2ce9ffb731b4ba230d7938842ae. The source bundle and exact binaries are retained; these frozen measurements predate the current rebase and GQT additions.

• CodeBuild: benchmarks-package-al2023:d49d7486-ac4f-4102-87b4-2785f85c1d1c — succeeded.
• Matrix SSM command: 826927e2-6cf1-46ee-b141-cb10922aa84c — succeeded; all 29 pairs independently verified.
• Serial binary SHA-256: 2fd890a8e2549f091c0b86ef16fed28fe59bb9a1442c01520217f53fba904a26.
• Current binary SHA-256: 6573988c91b28e54893179718c2a73fbd88f7828f0bab3ae66b92a2cd00062c0.

Artifacts require AWS access and are retained under:

s3://benchmarks-repo-248194531892-us-east-1/clusters/lab/benchmarks/rfc0057-20260906-ede8fc19/runtime/
  build-artifacts.tar.gz       # frozen Linux binaries, cloud receipts, source.bundle
  matrix-results.tar.gz        # all 58 raw JSONLs/logs, campaign, runner, receipts
  smoke-results.tar.gz        # two successful smoke records
  smoke-first-failure.tar.gz   # preserved setup-only failure
  analysis-and-cleanup-v2.tar.gz  # report, full distributions, independent audit, cleanup receipts

Matrix archive SHA-256: 2e309d1c83253ecde7b2cd24257114304919e52eaa910f5ff5aae418dded5674.
Analysis/cleanup archive SHA-256: 2c7d99cced550467769094ec255446ef0e7f25a537b8b31b1f377451ef1c4caa.

The first smoke attempt failed during setup because an external-runner trailing slash produced a double-slash fixture path; merge and verification never ran. The URI was corrected without changing either binary, and that attempt is preserved separately from measured samples. No failed or replaced matrix measurements were omitted.

All 30,053 temporary graph object versions and 584 delete markers were removed after evidence capture; the final graph-prefix version census is empty. The runtime artifacts, existing lab server, shared fixtures, and production resources were retained.

Local API-delay release results

Five alternating pairs per point, 121 populated node tables with four rows each, H0, and 17 ms added per observed ObjectStore API call. Times measure branch_merge after graph open. Ratios below divide the reported medians.

Touched tables	Frozen serial	Diagnostic width 4	Speedup
1	0.802 s	0.812 s	0.99×
8	3.619 s	1.956 s	1.85×
29	12.604 s	6.598 s	1.91×

The one-table 1.34% regression is within the 10% limit. The required 2× multi-table gate is not met, under either ratios of medians or median paired ratios. Four-worker overlap is observed, with no fallback or residual end gauges. At 29 tables, physical publication still takes 3.096 s, about 47% of the new total; both variants make 741 delayed API calls.

Small history control: four populated/touched tables, 16 live rows, two pairs per point:

History commits	Frozen serial	Diagnostic width 4	Speedup
0	2.022 s	1.174 s	1.72×
16	2.651 s	1.839 s	1.44×
64	4.353 s	3.535 s	1.23×

At H64, outer preparation and manifest publication consume roughly 71% of current operation time. Manifest reads grow from 13 at H0 to 269 at H64 in both variants. Old-history latency remains dominated by this manifest work.

All 58 timed merges passed separate setup/operation/verification checks. Largest fixture: 9.605 MiB; highest observed benchmark process RSS: 71.5 MiB. A two-pair production-default control shows 3.689 → 3.709 s at eight touched tables (+0.55%). These API-delay diagnostics are not wire-level S3 latency, p95 estimates, or worst-case decoder-memory qualification; history controls use current physical formats.

The frozen serial reference is published on codex/merge-serial-reference at cbd66386, based on 86aa5085; accepted-context reuse is already in that base and is not counted again. The measured current build is 28488d7c. Both clean source receipts and hashed release binaries are retained locally; the portable paired receipt is checked in with RFC 0057. The original subsequent commits only reconciled documentation; the current rebase integrates later main changes and is not a new measurement of these binaries. The completed native-S3 follow-up is reported above.

Original implementation validation (before this rebase)

• Native AWS/S3 performance campaign: 58 measured merges + two smoke merges verified; both cloud-built binaries pass 11/11 benchmark-contract tests. Scope and activation limits are recorded above.
• Full workspace/failpoint CI: 2,984 passed, 27 ignored, including the configured genuine v0.9 upgrade regression.
• AWS-server CI: 371 passed, 2 ignored. These are correctness checks at measured 28488d7c, not an AWS performance benchmark.
• Actual local crate-local DST: 78 passed, 30 ignored, including real merge and strict replay cases. This covers the serial default, not width-four simulation behavior.
• Both workspace Clippy graphs, formatting, documentation and repository metadata checks pass.
• Focused width-one/two/four owners and a configured MinIO merge pass; dedicated cost/MinIO owners prove useful four-worker overlap. Existing composite tests pass on the default local stack.

Remaining activation gates

Lance 11 exposes no supported pre-decode allocation cap for highly compressed wide rows. The controlled 128 MiB allowance is not a process-memory limit. A real TCP-disconnect test also shows that the HTTP request owner cannot drain parallel preparation after its future is dropped.

The speed gate and these resource/lifetime gates remain open. Further parallel qualification includes mixed Blob barriers, conflicts followed by fatal errors, authority/recovery after useful overlap, pending scratch-write settlement, more retained data shapes, and width-four DST. No production latency improvement is claimed.