Skip to content

perf: bound capsule catalog capture and proof traffic - #645

Merged
forhappy merged 29 commits into
crab-v2from
codex/crab-v2-small-run-compaction
Nov 4, 2026
Merged

forhappy merged 29 commits into
crab-v2from
codex/crab-v2-small-run-compaction

Conversation

@forhappy

@forhappy forhappy commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Crab/git push repeatedly reconciles repository-wide metadata before learning whether any outgoing Xet keys exist. This change defers visibility reconciliation for publication discovery, coalesces authenticated reads and reuses verified ref-head receipts. Complete capture still grows with repository state; the final team-scale gates remain open.

Implementation

  • PublicationCatalogLookup captures stable epoch-filtered heads, fresh activations, the checkpoint and authenticated run controls. File/chunk/xorb misses are advisory and return no remote values or Git visibility authority. The first hit completes ordinary ref ordering, visibility and closure reconciliation against those same immutable sources before returning values. This includes the final xorb query after residual packing.
  • Probe and authorized query share one admission state; warm pages do not reset budgets. Concurrent queries share successful reconciliation. Cancellation/error leaves immutable inputs available for another initialization attempt, with no partial authority. Ordinary readers, inspection, GC and checkpoint builders retain complete capture. Own-ref expected-old checks and staged recipe/payload/origin verification remain unchanged.
  • Coalesce small visibility controls and point proofs in bounded waves. Preserve shared dictionaries, selected compacted-member paging, source reuse and aggregate admission.
  • Carry ref-head bodies into checkpoints as one authenticated gzip member: separate 8 MiB encoded/decoded limits and at most 16,384 heads. Require the same physical origin/prefix/repository/epoch, fresh matching strong provider identities and stable inventories. Resolve activations afresh; reject malformed, truncated, trailing or conflicting cache bytes.
  • Elect one foreground catalog-checkpoint owner after 128 uncheckpointed capsules, rechecking the count before complete metadata intake. Restore that interval after the 64-capsule experiment worsened p95. Keep complete checkpoint validation, exact root/ref-position CAS and GC admission. Protected staging and replicated publication retain their existing owners; failed postcommit maintenance cannot revoke an acknowledged ref.
  • Run complete run/control/checkpoint/bulk-visibility decoding behind four blocking CPU slots per process. Workers retain immutable inputs and permits through cancellation, with no store, lookup session or publication fence.
  • Preserve bounded contention retries/backoff and identity, epoch and activation checks. Establish one canonical native format-1 baseline and remove experimental compatibility readers, geometry selectors and the development chunk-index converter.

Crab remains v2 (2.0.0-alpha.1). Git protocol-v2, the v2/ namespace and published/external formats remain independently versioned. The prelaunch native reset requires fresh buckets, including bucket-global hints. Earlier experimental bytes are never rewritten or relabeled; inspect retained stores with their original binaries.

Current evidence: 6ba71789ec66c25249bd838829e66af87e4f9626

  • The negative-discovery regression reproduces 257 reads for 64 unrelated refs and passes with a 129-read bound. Hit tests reject inconsistent ref authority after preceding misses, including xorb hits; cancellation/concurrent retry and shared probe/result admission pass. All 41 focused host tests, payload-only compilation, formatting and the existing CI Clippy policy pass.
  • Exact clean ARM64 Linux source passes 1,392 test executions across ten metadata/read/Git/SDK/publication/checkpoint/writer suites, plus the explicit million-object all-closure test and unchanged architecture/inventory gates. Existing ignored corpus cases remain reported. No dependency, lockfile, inventory or suppression changes.
  • Official private make install passes for CLI/helper/FUSE/NFS/cache-server artifacts. CLI SHA-256: fe4d9d2b184b9e4b431560788bb4e984edbce14d289e23c73f50c800f3a0bd2a; embedded commit 6ba71789.
  • Fresh 640/640 pushes, 64 developers and 16 workers: 918 integrity checks, both historical ordinary/Xet/LFS payload versions, strict Git fsck and all 43,076 reachable raw Git objects / 46,755,448 content bytes compared with original seed/writers. No-repair fsck reports zero errors, repairs or repair failures. Overall p95 3.633s; slowest batch p95 4.327s. This qualifies only that recorded diagnostic.
  • Independent immutable acknowledgement, command-ledger and closed-client HTTP audits pass: 100,245 requests / 556,700,654 response bytes / 339,034,595 known upload bytes, zero unknown upload lengths. Capsule GETs: 54,915 / 365,403,546 bytes; conservative all-metadata responses: 476,841,324 bytes. All attempts, bucket-global metadata and foreground maintenance remain included. These are candidate totals, not matched savings. The original windows also show 194 reads of the authenticated cache range containing 65 heads.
  • The current merged base ec7a3cb07c3 retains the exact capture function from the recorded deterministic 900 ms exhausted-retry reproduction. The candidate's full Linux metadata suite passes finite-burst, bounded-stability and fresh identity/activation tests. This store-boundary proof is not a live merged-build team qualification.
  • The full-cohort controller stopped at 5,000/10,000 pushes after the partial-edit/reuse batch failed p95 at 10.704s (p99 39.017s, max 77.310s). All 5,000 pushes were acknowledged and independent prefix audits verified original timings, closed HTTP windows and 1,001 refs. Overall p95 is 7.455s; batch p95s are 7.837s / 2.824s / 0.905s / 3.617s / 10.704s. The failed fifth batch contains 1,739,480 requests / 9,653,705,678 response bytes / 1,675,168,492 known upload bytes. Original timings/seals remain immutable. Its separate cold audit now passes both recorded ordinary/Xet/LFS versions for all 1,000 developers, strict full Git fsck and 68,004 raw Git objects / 172,679,670 content bytes versus original seed/writers. No-repair fsck reports zero errors, repairs or repair failures. All 17 original sealed/report/command files remain unchanged; 11,007 audit commands and 2,428,382 verification requests are outside the original push windows. Complete 10,000-push qualification and matched savings remain open.
  • Full exact-head CI #41862776779 failed the strict HTTP-server Clippy step on two needless borrows in catalog_capture.rs; Rust tests were skipped. Both have local corrections without suppressions, pending publication and new CI. Documentation, web and both browser shards pass. This CI failure and the fifth-batch latency failure remain explicit; no final-build qualification claim.

Retained failures

  • a440daf6 acknowledged all 1,000 first-batch pushes but failed p95 at 13.539s (max 22.587s). Requests fell versus its predecessor while latency worsened; tightening checkpoint frequency alone was insufficient. The controller stopped before batch two. Its separate cold audit passes all 1,000 recorded payload histories, 48,004 raw Git objects / 104,385,543 content bytes, strict Git fsck and no-repair fsck. All original timings/seals remain unchanged.
  • Prior 7b373a97 and 4c9872a3 first-batch p95 failures remain 10.522s and 10.738s. Their prefix integrity proofs and original failures remain retained. Full exact-head a440 CI passed all nine jobs with 11,228 passing Rust test executions; it does not qualify this new revision.

Qualification notes retain the format-specific diagnostic history.

Qualification still open

The exact final revision requires 1,000 developers × 10 = 10,000 pushes, 16 workers, p95 ≤10 seconds overall and in every 1,000-push batch, all historical Git/Xet/LFS bytes, every reachable raw Git object versus original seed/writers and a no-repair audit. The matched corrected pre-coalescing control must establish ≥50% fewer capsule GETs, no increase in capsule or conservative all-metadata response bytes, with all attempts/foreground maintenance and known uploads/unknown lengths disclosed.

Per-batch acknowledgements, timings and checkpoints are immutable; planned pauses permit independent prefix audits, and historical verification runs separately from generation. Full-team/matched measurements, Kubernetes/Git-2.56 checks, new full CI and actual merged-build qualification remain open. No merge or team-readiness claim. Local RustFS results establish no AWS capacity or billing result.

@forhappy
forhappy force-pushed the codex/crab-v2-small-run-compaction branch from fc3826b to 7ebde95 Compare November 3, 2026 07:36
@forhappy forhappy changed the title perf: bound early compaction of small capsule runs perf: bound capsule metadata work and qualify team pushes Nov 3, 2026
@forhappy
forhappy marked this pull request as ready for review November 3, 2026 14:48
Reuse the final provider inventory for the next bounded attempt. Keep both identity checks, epoch filtering and fresh activation resolution. The merged implementation rejects a deterministic 900 ms burst; the extended capture succeeds and perpetual churn still errors.
try_update retains the atomic ordering and checked-update contract. Rust 1.99 deprecates the old alias; direct U64 and usize compile/runtime probes pass on the exact Rust 1.97.0 minimum. Remove the obsolete deprecation suppressions.
Seal every drained batch without replacing failed evidence. Separate push generation from historical verification using pinned inputs and new consumers for each audit attempt. Push completion alone never reports verified integrity.
Large-leaf tip reads remained serial after scalar prefetch.
Warm selected immutable point proofs in bounded waves; retain ordered
closure, removal and final membership validation.
Early folding left partially compacted sources waiting for 32 segments.
Complete the original 32-capsule tier without bypassing the small-carry
byte budget, source retention or publication CAS.
Build the workspace CLI for desktop shell-outs and select it through
CRAB_BIN. The 45-minute job expired during full-workspace test compilation;
allow 75 minutes for the same gates and cache cleanup.
Overlap independent sources; preserve publication order. Share page, object and materialization admission across bounded query lanes. Keep current capture and chunk-reuse checks.
Stable-only SemVer ranges exclude the v2 prerelease. Derive the positive fixture from the package version and protect prerelease exclusion; keep production policy unchanged.
Fold catalog suffixes under one foreground owner. Share overlapping checkpoint memberships earlier while preserving declared v1/v2 canonical bytes. Recapture verified same-epoch root replacements for current-state discovery; retain explicitly pinned replay semantics.
@forhappy
forhappy force-pushed the codex/crab-v2-small-run-compaction branch from f8c3961 to 8cdf15d Compare November 3, 2026 18:02
Use 4 KiB leaves in checkpoint visibility version 3 and retain exact v1/v2 canonical bytes. Upgrade readers before writers. Keep capsule read-ahead within two point leaves; larger edits retain selected paging.

The retained 640-push comparison failed the all-metadata byte budget despite fewer capsule requests. Focused regressions reduce scattered checkpoint proofs from 2,128,338 to 808,020 bytes and selected leaf reads from 24,785 to 8,437 bytes. Full matched qualification remains required.
Cargo loads every workspace member before selecting a package.
Copy the current desktop agent into each service builder and allow
its sources in Docker contexts. The previous paths broke image CI.
A real context export resolves all 26 locked workspace members.
Global checkpoint intake rereads every captured ref. Trigger it after four ordinary per-ref tiers while retaining generation ownership, GC admission and exact-root publication. Preserve the failed matched replay and test both deferred and eventual checkpoint publication.
Use smaller singleton visibility pages with authenticated inline routing roots to reduce repeated catalog tip-proof bytes without extra requests. The existing page decoder, cache and query admission own all validation.

New split singleton writes use visibility version 6. Upgrade readers before writers; older readers reject version 6. Declared version-3/4/5 layouts retain exact immutable bytes.
@forhappy forhappy changed the title perf: bound capsule metadata work and qualify team pushes perf: bound capsule catalog capture and proof traffic Nov 3, 2026
Bind cached bodies to origin, repository, epoch and fresh provider tokens. Keep stable inventories and resolve activations on every capture.

Emit checkpoint v11 while preserving tagged v9 history. Bound cache bytes and entries; avoid single-ref read amplification.
Current-writer fixtures must bind the exact checkpoint pointer. Keep
immutable tagged format-9 compatibility proof separate.
Record the approved base SDK and desktop contracts. Keep metadata
tests silent and bound stable SDK debug artifacts with a fresh cache.
BREAKING CHANGE: native Capsule codecs accept only the latest v1 layouts.
Republish experimental repositories into a fresh bucket; bucket-global
hints also require fresh state. Remove compatibility codecs and the
development chunk-index converter. Preserve independent Git/Xet/LFS
contracts and the released manifest graph layout.
Retain the bounded canonical records in one checked gzip member. Preserve
encoded and decoded byte limits, exact provider identities and fresh
activation checks; move encoding off the async runtime. Keep typed codec
and worker failures through CLI, protected receive and SDK boundaries.

Align unreleased capsule protected plans and prepare/evidence records
with the native schema-1 baseline while retaining strict shape admission.
Crab remains 2.0.0-alpha.1; published non-capsule contracts stay separate.

BREAKING CHANGE: native schema-1 caches require gzip. Recreate experimental
buckets with this build; no plaintext or old-development fallback reader.
The native baseline is format1. The old format9 assertion panicked while
holding GIT_DIR_MUTEX and caused four recent-LFS tests to fail on poison.
Keep the complete push/hydration/repack/cross-repository proof and check
the requested format1 without changing mutex handling or test deadlines.
Move canonical decoding off async workers. Keep CPU admission with
immutable worker inputs after caller cancellation.
Cut the pending suffix from four ref tiers to two; keep full
checkpoint validation and existing publication ownership.
New-data misses avoid unrelated visibility replay; every file, chunk
or xorb hit still completes captured authority before returning values.
Keep one query budget and retryable, cancel-safe proof initialization.

Restore the 128-capsule interval after the 64-capsule experiment
reduced requests but worsened first-batch p95 to 13.539 seconds.
@forhappy
forhappy merged commit 161f74e into crab-v2 Nov 4, 2026
51 of 61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant