Repository navigation
perf: bound capsule catalog capture and proof traffic - #645
Merged
Merged
Conversation
forhappy
force-pushed
the
codex/crab-v2-small-run-compaction
branch
from
November 3, 2026 07:36
fc3826b to
7ebde95
Compare
forhappy
marked this pull request as ready for review
November 3, 2026 14:48
Reuse the final provider inventory for the next bounded attempt. Keep both identity checks, epoch filtering and fresh activation resolution. The merged implementation rejects a deterministic 900 ms burst; the extended capture succeeds and perpetual churn still errors.
try_update retains the atomic ordering and checked-update contract. Rust 1.99 deprecates the old alias; direct U64 and usize compile/runtime probes pass on the exact Rust 1.97.0 minimum. Remove the obsolete deprecation suppressions.
Seal every drained batch without replacing failed evidence. Separate push generation from historical verification using pinned inputs and new consumers for each audit attempt. Push completion alone never reports verified integrity.
Large-leaf tip reads remained serial after scalar prefetch. Warm selected immutable point proofs in bounded waves; retain ordered closure, removal and final membership validation.
Early folding left partially compacted sources waiting for 32 segments. Complete the original 32-capsule tier without bypassing the small-carry byte budget, source retention or publication CAS.
Build the workspace CLI for desktop shell-outs and select it through CRAB_BIN. The 45-minute job expired during full-workspace test compilation; allow 75 minutes for the same gates and cache cleanup.
Overlap independent sources; preserve publication order. Share page, object and materialization admission across bounded query lanes. Keep current capture and chunk-reuse checks.
Stable-only SemVer ranges exclude the v2 prerelease. Derive the positive fixture from the package version and protect prerelease exclusion; keep production policy unchanged.
Fold catalog suffixes under one foreground owner. Share overlapping checkpoint memberships earlier while preserving declared v1/v2 canonical bytes. Recapture verified same-epoch root replacements for current-state discovery; retain explicitly pinned replay semantics.
forhappy
force-pushed
the
codex/crab-v2-small-run-compaction
branch
from
November 3, 2026 18:02
f8c3961 to
8cdf15d
Compare
Use 4 KiB leaves in checkpoint visibility version 3 and retain exact v1/v2 canonical bytes. Upgrade readers before writers. Keep capsule read-ahead within two point leaves; larger edits retain selected paging. The retained 640-push comparison failed the all-metadata byte budget despite fewer capsule requests. Focused regressions reduce scattered checkpoint proofs from 2,128,338 to 808,020 bytes and selected leaf reads from 24,785 to 8,437 bytes. Full matched qualification remains required.
Cargo loads every workspace member before selecting a package. Copy the current desktop agent into each service builder and allow its sources in Docker contexts. The previous paths broke image CI. A real context export resolves all 26 locked workspace members.
Global checkpoint intake rereads every captured ref. Trigger it after four ordinary per-ref tiers while retaining generation ownership, GC admission and exact-root publication. Preserve the failed matched replay and test both deferred and eventual checkpoint publication.
Use smaller singleton visibility pages with authenticated inline routing roots to reduce repeated catalog tip-proof bytes without extra requests. The existing page decoder, cache and query admission own all validation. New split singleton writes use visibility version 6. Upgrade readers before writers; older readers reject version 6. Declared version-3/4/5 layouts retain exact immutable bytes.
Bind cached bodies to origin, repository, epoch and fresh provider tokens. Keep stable inventories and resolve activations on every capture. Emit checkpoint v11 while preserving tagged v9 history. Bound cache bytes and entries; avoid single-ref read amplification.
Current-writer fixtures must bind the exact checkpoint pointer. Keep immutable tagged format-9 compatibility proof separate.
Record the approved base SDK and desktop contracts. Keep metadata tests silent and bound stable SDK debug artifacts with a fresh cache.
BREAKING CHANGE: native Capsule codecs accept only the latest v1 layouts. Republish experimental repositories into a fresh bucket; bucket-global hints also require fresh state. Remove compatibility codecs and the development chunk-index converter. Preserve independent Git/Xet/LFS contracts and the released manifest graph layout.
Retain the bounded canonical records in one checked gzip member. Preserve encoded and decoded byte limits, exact provider identities and fresh activation checks; move encoding off the async runtime. Keep typed codec and worker failures through CLI, protected receive and SDK boundaries. Align unreleased capsule protected plans and prepare/evidence records with the native schema-1 baseline while retaining strict shape admission. Crab remains 2.0.0-alpha.1; published non-capsule contracts stay separate. BREAKING CHANGE: native schema-1 caches require gzip. Recreate experimental buckets with this build; no plaintext or old-development fallback reader.
The native baseline is format1. The old format9 assertion panicked while holding GIT_DIR_MUTEX and caused four recent-LFS tests to fail on poison. Keep the complete push/hydration/repack/cross-repository proof and check the requested format1 without changing mutex handling or test deadlines.
Move canonical decoding off async workers. Keep CPU admission with immutable worker inputs after caller cancellation.
Cut the pending suffix from four ref tiers to two; keep full checkpoint validation and existing publication ownership.
New-data misses avoid unrelated visibility replay; every file, chunk or xorb hit still completes captured authority before returning values. Keep one query budget and retryable, cancel-safe proof initialization. Restore the 128-capsule interval after the 64-capsule experiment reduced requests but worsened first-batch p95 to 13.539 seconds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Crab/git push repeatedly reconciles repository-wide metadata before learning whether any outgoing Xet keys exist. This change defers visibility reconciliation for publication discovery, coalesces authenticated reads and reuses verified ref-head receipts. Complete capture still grows with repository state; the final team-scale gates remain open.
Implementation
PublicationCatalogLookupcaptures stable epoch-filtered heads, fresh activations, the checkpoint and authenticated run controls. File/chunk/xorb misses are advisory and return no remote values or Git visibility authority. The first hit completes ordinary ref ordering, visibility and closure reconciliation against those same immutable sources before returning values. This includes the final xorb query after residual packing.Crab remains v2 (
2.0.0-alpha.1). Git protocol-v2, thev2/namespace and published/external formats remain independently versioned. The prelaunch native reset requires fresh buckets, including bucket-global hints. Earlier experimental bytes are never rewritten or relabeled; inspect retained stores with their original binaries.Current evidence:
6ba71789ec66c25249bd838829e66af87e4f9626make installpasses for CLI/helper/FUSE/NFS/cache-server artifacts. CLI SHA-256:fe4d9d2b184b9e4b431560788bb4e984edbce14d289e23c73f50c800f3a0bd2a; embedded commit6ba71789.ec7a3cb07c3retains the exact capture function from the recorded deterministic 900 ms exhausted-retry reproduction. The candidate's full Linux metadata suite passes finite-burst, bounded-stability and fresh identity/activation tests. This store-boundary proof is not a live merged-build team qualification.catalog_capture.rs; Rust tests were skipped. Both have local corrections without suppressions, pending publication and new CI. Documentation, web and both browser shards pass. This CI failure and the fifth-batch latency failure remain explicit; no final-build qualification claim.Retained failures
a440daf6acknowledged all 1,000 first-batch pushes but failed p95 at 13.539s (max 22.587s). Requests fell versus its predecessor while latency worsened; tightening checkpoint frequency alone was insufficient. The controller stopped before batch two. Its separate cold audit passes all 1,000 recorded payload histories, 48,004 raw Git objects / 104,385,543 content bytes, strict Git fsck and no-repair fsck. All original timings/seals remain unchanged.7b373a97and4c9872a3first-batch p95 failures remain 10.522s and 10.738s. Their prefix integrity proofs and original failures remain retained. Full exact-head a440 CI passed all nine jobs with 11,228 passing Rust test executions; it does not qualify this new revision.Qualification notes retain the format-specific diagnostic history.
Qualification still open
The exact final revision requires 1,000 developers × 10 = 10,000 pushes, 16 workers, p95 ≤10 seconds overall and in every 1,000-push batch, all historical Git/Xet/LFS bytes, every reachable raw Git object versus original seed/writers and a no-repair audit. The matched corrected pre-coalescing control must establish ≥50% fewer capsule GETs, no increase in capsule or conservative all-metadata response bytes, with all attempts/foreground maintenance and known uploads/unknown lengths disclosed.
Per-batch acknowledgements, timings and checkpoints are immutable; planned pauses permit independent prefix audits, and historical verification runs separately from generation. Full-team/matched measurements, Kubernetes/Git-2.56 checks, new full CI and actual merged-build qualification remain open. No merge or team-readiness claim. Local RustFS results establish no AWS capacity or billing result.