Skip to content

fix(remote): authenticate selected capsule preparation - #596

Closed
forhappy wants to merge 1 commit into
codex/capsule-chunk-authorityfrom
codex/shared-capsule-catalog
Closed

forhappy wants to merge 1 commit into
codex/capsule-chunk-authorityfrom
codex/shared-capsule-catalog

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

Shared capsule preparation scans the complete historical Xet catalog even for an ordinary Git update. It also formats raw body digests as Xet protocol hashes, accepts scratch files changed after inspection, and never populates the shared chunk index after native-content commitment.

This change resolves only outgoing file/shard/xorb closures through the captured paged lookup. Ordinary Git and LFS-only uploads read no catalog pages. Bounded artifact reads verify the inspected raw body identity before upload. Exact create/read identities and original source descriptors survive through commitment; chunk hints are published only after proven ref visibility. Index maintenance failure preserves an already committed outcome, and uncertain publication emits no hints.

The capsule implementation moves into crates/crab-remote/src/prepare/capsule.rs, retaining the existing public API. Its ownership of the catalog until post-commit indexing avoids keeping both decoded and serialized copies across preparation. No full-scan fallback, new product configuration, or protocol alias is introduced.

Stacked on #593 (a7e2662f890); no merge requested. Both large-file audit benchmark files and the user-owned advisor plan are excluded.

Validation:

  • Four intended regressions failed before the fix; afterward all 6 public-API integration tests pass. Rewritten xorb and shard bytes both fail. Historical pointers accept the correct size and reject conflicts.
  • 8 HTTP receive/fault tests passed; the two normally ignored live cells were then explicitly executed against real RustFS.
  • 3 real-RustFS cells passed: public native-content creation/occupied-key reuse and full-file reconstruction; native Git push/fetch and browser editing through the real HTTP listener; six fault/restart scenarios. All recorded source hashes stayed unchanged. These are small correctness fixtures, not 10 GiB or latency qualification.
  • Strict all-target Clippy with publication, the minimal no-default-feature check, Rust formatting and whitespace checks passed. The UI dependency/build prerequisite for the HTTP crate passed; its unrelated lockfile rewrite was reverted.
  • The initial all-history Git fixture re-sent an old pointer; the corrected incremental pack excludes the advertised tip. That fixture failure is not claimed as a product defect. Compile-only fixture corrections also remain separate from the four observed product failures.

Review evidence map:

Boundary Evidence
Entry / caller HTTP receive::publish::publish_attempt calls the same Prepared::upload_capsule API exercised by the native-Git integration tests
Owner crab-remote retains preparation, immutable artifacts and commit outcomes; authorization remains in HTTP; metadata owns catalog lookup and candidate encoding
Callees PointerCatalogLookup::{files,shards,xorbs}, bounded inspected-file reads, Store::create_or_read_immutable, write_capsule_chunk_candidates
Lifetime / ordering Caller ref leases and global/repo GC fences cover source capture, upload, proven commitment and post-commit hint durability; writer close remains awaited
Siblings Direct CLI and protected receive use the same metadata writer and post-commit hint policy. The released manifest uploader already streams against the inspected raw size/hash and remains unchanged. Shared constructor validation retains its public inputs/errors and gains bounded reads
Dependency contract Store create/read returns the identity of the bytes verified by that operation; existing logical encodings are fully parsed and checked. SlateDB writes become durable through awaited close
Main / parent Fetched main cc2e4e25f46 has no capsule upload/catalog path. Parent #593 contains all four reproduced shared-path gaps
Release boundary v1.2.4 has PreparedContent without the new capsule catalog field; its legacy multipart path keeps raw artifact hashes. No released capsule format requires the erroneous body-digest representation
Tests / proof limits Native Git + actual public API + in-memory and real RustFS origins; real HTTP/Git and restart/fault paths. Final combined large-file/growth/provider matrix remains open

Is this the best fix? The reproduced failures share the preparation/commit ownership boundary. This uses existing authenticated lookup, immutable storage and metadata writer contracts, and removes the eager scan. Production LOC growth supplies bounded source verification and operation-bound receipts; capsule behavior is consolidated in one module. Full initiative readiness still requires the open combined-source and scale gates.

Dependency review: adds a test-only direct dependency on the already-used exact SlateDB 0.15.0 version to inspect actual published records. Cargo.lock adds only that dependency edge; no version, production dependency, patch or override changes.

API test binary SHA-256: fe8a75004fd4ef7ec0946b999d32a829b83b28f94b51317a9c34e9990e9ebf5d; HTTP test binary SHA-256: 92318648278160dd57f96f35dbcffc01f44799a75a43b84442251965c07fef51.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant