Skip to content

StorageProvider seam + private SEAL-encrypted remember/recall (public→Tatum, private→Harbor) #19

Open
Open
StorageProvider seam + private SEAL-encrypted remember/recall (public→Tatum, private→Harbor)#19
Labels
P1Important: hardens the demo and core productharborHarbor encrypted Walrus storage on Sui (Space/Bucket/File)seal@mysten/seal client-side encryption + on-chain access controlwalrusWalrus blob storage / Sites / aggregator
Milestone
Sui Overflow

Description

@harrymove-ctrl
harrymove-ctrl
opened on Jun 16, 2026

Context

Today "private" is only a D1 read-token check (CloudflareNamespaceStore.authorizeNamespace, apps/api/src/worker.ts:502) — bytes are uploaded plaintext and readable by anyone with the blobId/aggregator URL — and WalrusStorageReceipt.provider is hard-wired to the literal 'tatum' (packages/core/src/types.ts:655) with packages/walrus/src/storage.ts as the only provider. The roadmap mandates coexistence + real encryption: a StorageProvider seam routes public→Tatum (mainnet, plaintext) and private→Harbor+SEAL (testnet, encrypted), where a private namespace gets an owned Seal policy and bytes are SEAL-encrypted before upload and only recallable by decrypt via SessionKey + seal_approve.

Goal / user story

As a user with a private namespace, when I remember an artifact it goes through a StorageProvider that SEAL-encrypts it and stores ciphertext on Harbor, and recall fetches + decrypts — while public namespaces keep going to Tatum plaintext, with the chosen provider recorded honestly in the receipt.

Acceptance criteria

  • Define a StorageProvider interface { upload, status, fetch } in packages/walrus; refactor the existing Tatum functions behind a TatumStorageProvider implementing it, with no behavior change for current CLI/MCP callers.
  • Widen WalrusStorageReceipt.provider (core/types.ts:655) from 'tatum' to 'tatum' | 'harbor' and add optional Harbor fields (spaceId, bucketId, fileId, sealPolicyId, sealIdentity).
  • New packages/walrus/src/seal.ts wraps @mysten/seal: encrypt(bytes, { sealPolicyId, identity }) → ciphertext; decrypt(ciphertext, { sessionKey, sealPolicyId }) builds the seal_approve PTB.
  • A selectStorageProvider({ visibility, network }) picks Harbor+SEAL for private namespaces and Tatum for public; private writes encrypt the proof bundle with a per-file identity against the namespace's sealPolicyId, then upload ciphertext via HarborClient; the chosen provider is written into the receipt, and rememberStorageIndex/renderStorageIndex (packages/memwal/src/index.ts:164/281) carry bucketId/fileId/sealPolicyId/sealIdentity (keep back-compat parsing of the [ctxm-storage] format).
  • New GET /api/memwal/artifact (worker.ts) resolves provider + refs from D1, fetches ciphertext, decrypts via SessionKey + seal_approve, and streams plaintext — token-gated through authorizeNamespace (worker.ts:502); public namespaces fall back to the plaintext aggregator path.
  • Fail-closed: non-private writes fall back to Tatum on Harbor failure, but private writes NEVER silently fall back to plaintext; creating/revoking a namespace read token maps to granting/revoking SEAL decrypt for that policy (wired near authorizeNamespace and token create/revoke worker.ts:992/:1013).
  • Tests cover provider selection by visibility, public-only Tatum fallback, and a private round-trip (remember → recall artifact → bytes match; a request without a valid token cannot decrypt).

Implementation notes

  • Land the interface + receipt widening early (no deps); uploadProofBundle (storage.ts:183) currently always returns provider: 'tatum' — route it through selectStorageProvider.
  • SessionKey needs an Ed25519 signer (service key, or per-owner per the mapping issue's decision); execute the seal_approve PTB via SuiGrpcClient devInspect (same client as proof.ts). Per-file identity deterministic from namespace + run/chunk id so recall can reconstruct it.
  • Public share pages CANNOT do SessionKey+seal_approve — confirm public namespaces stay plaintext (Tatum) and only private go through SEAL. Network reality: Tatum=mainnet, Harbor=testnet; document that public-mainnet and private-testnet blobs live on different networks.
  • Encryption on write likely runs in the queue consumer / Node CLI+MCP path (packages/cli/src/index.ts:347, packages/mcp/src/index.ts:200) since packProofBundle is Node-only — gated by the seal Worker spike. Keep packProofBundle (tar) Node-only; the Worker private path needs an in-memory zip from R2 (separate roadmap item, out of scope here).

Sui Overflow angle

Turns "private" from a DB flag into real on-chain access control: decryption is gated by an on-chain seal_approve Move call. The headline Sui-native privacy demo — "revoke the token → recall can no longer decrypt" — lands here, shown side-by-side with a public Tatum/Walrus blob whose receipt names the provider.

Dependencies

The seal Worker spike, the HarborClient issue, and the namespace→Space/Bucket/Seal mapping + service-key issue. The interface + receipt widening have no deps and can land first.

Part of the ContextMEM roadmap (#4) • Sui Overflow build.

Activity

  1. harrymove-ctrl
    added this to the Sui Overflow milestone on Jun 16, 2026
  2. harrymove-ctrl
    added
    P1Important: hardens the demo and core product
    harborHarbor encrypted Walrus storage on Sui (Space/Bucket/File)
    walrusWalrus blob storage / Sites / aggregator
    seal@mysten/seal client-side encryption + on-chain access control
    on Jun 16, 2026

  3. harrymove-ctrl commented on Jun 19, 2026

    @harrymove-ctrl
    harrymove-ctrl
    on Jun 19, 2026
    OwnerAuthor

    Implementation Plan

    Approach
    Introduce a thin StorageProvider seam in packages/walrus ({ upload, status, fetch }) so the existing Tatum code becomes one implementation (TatumStorageProvider) with zero behavior change, and a HarborStorageProvider (built on #6's HarborClient) becomes the second. SEAL crypto lives in a standalone seal.ts (decoupled from any provider) so encrypt runs on the write path and decrypt runs on the read path independently. selectStorageProvider({ visibility, network }) routes public→Tatum (mainnet, plaintext) and private→Harbor (testnet, SEAL ciphertext). Key design decision: private encryption happens on the Node CLI/MCP write path (where packProofBundle already runs, storage.ts:231), and the Worker only does fetch+decrypt at recall time via GET /api/memwal/artifact — keeping the heavy @mysten/seal decrypt the one thing we put in the Worker, which is exactly what spike #5 must clear first. The hosted (R2→Walrus) private write path stays out of scope (deferred to the in-memory-zip roadmap item).

    Prerequisites & dependencies

    Step-by-step

    1. Define the seam (no deps). Create packages/walrus/src/provider.ts:

      • Types: StorageVisibility = "public"|"private", StorageNetwork = "mainnet"|"testnet", StorageRef = { jobId?; blobId?; fileId?; bucketId?; downloadUrls?: string[] }.
      • interface StorageProvider { readonly provider: "tatum"|"harbor"; upload(input): Promise<WalrusStorageReceipt>; status(ref): Promise<WalrusStorageJobState>; fetch(ref): Promise<Uint8Array>; }.
      • upload/status reuse the existing WalrusStorageUploadInput/WalrusStorageJobState/WaitOptions shapes from storage.ts:26-47,153.

    2. Widen the receipt (no deps). Edit packages/core/src/types.ts:655 — provider: "tatum" → provider: "tatum" | "harbor"; add optional spaceId?, bucketId?, fileId?, sealPolicyId?, sealIdentity?: string. Keep all existing fields. (This is the early, dep-free landing per the issue's implementation note.)

    3. Refactor Tatum behind TatumStorageProvider. In storage.ts, keep uploadToWalrusStorage/getWalrusStorageJob/waitForWalrusStorageCertified/uploadProofBundle/packProofBundle exported as-is (CLI/MCP import them directly — cli/src/index.ts:347, mcp/src/index.ts:200). Add class TatumStorageProvider implements StorageProvider: upload delegates to uploadProofBundle (already returns provider:"tatum", storage.ts:191); status → getWalrusStorageJob; fetch → GET ref.downloadUrls[0] (the aggregator URL) → Uint8Array. No behavior change for current callers (AC#1).

    4. SEAL wrapper (gated by Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5). Create packages/walrus/src/seal.ts wrapping @mysten/seal:

      • encrypt(bytes, { sealPolicyId, identity }) → ciphertext Uint8Array (Node write path).
      • decrypt(ciphertext, { sessionKey, sealPolicyId, identity, suiClient }) → builds the seal_approve PTB (moveCall ${HARBOR_SEAL_PACKAGE_ID}::<module>::seal_approve(identity, policyObject)) and executes it via SuiGrpcClient devInspect (same client pattern as proof.ts:12), then SEAL SessionKey-based decrypt → plaintext.
      • Helper deriveSealIdentity(namespace, runId|chunkId) = deterministic sha256(${namespace}:${id}) hex so recall reconstructs the exact identity (per implementation note). Export all from packages/walrus/src/index.ts.

    5. HarborStorageProvider (deps Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6/Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7). Create packages/walrus/src/harbor-provider.ts implementing StorageProvider over HarborClient (Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6): upload(ciphertextInput) → ensure bucket (from Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7 mapping) → uploadFile → poll status → return receipt { provider:"harbor", spaceId, bucketId, fileId, sealPolicyId, sealIdentity, blobId?, status, certified, … }; status(ref.fileId) → Harbor poll; fetch(ref.fileId) → HarborClient.download → ciphertext Uint8Array (decrypt is a separate seal.decrypt call, kept decoupled).

    6. selectStorageProvider + fail-closed write path. In provider.ts: selectStorageProvider({ visibility, network }, deps) → returns HarborStorageProvider when visibility==="private", else TatumStorageProvider. Add an orchestrator storePrivateOrFallback(...):

      • private: seal.encrypt(bundle.data, { sealPolicyId, identity }) → HarborStorageProvider.upload(ciphertext); on Harbor/SEAL failure rethrow (NEVER plaintext fallback — AC#6).
      • public/non-private: TatumStorageProvider.upload; on Harbor failure for a non-private namespace that opted into Harbor, fall back to Tatum (AC#6). Route uploadProofBundle's callers through this selector (per implementation note "route it through selectStorageProvider").

    7. Wire the write callers. Thread a visibility/network (+ sealPolicyId/bucketId from Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7) into the CLI storage push (cli/src/index.ts:341-378) and MCP upload_proof_to_walrus (mcp/src/index.ts:200) so private runs encrypt+Harbor and public stay Tatum. Default visibility=public preserves today's behavior.

    8. Carry Harbor refs through the memory index (back-compat). Extend StorageIndexInput (memwal/src/index.ts:37) and renderStorageIndex (:281) to emit bucketId=, fileId=, sealPolicyId=, sealIdentity= lines inside the existing [ctxm-storage] block; mirror in sdk.ts:123. Add a parseStorageIndex(text) that reads the new keys and still parses legacy blocks without them (AC#4). These fields also get persisted to D1 (step 9) as the authoritative recall lookup; the MemWal text is the portable mirror.

    9. D1 schema + namespace summary. Add migration apps/api/migrations/0007_harbor_seal_refs.sql: ALTER TABLE contextmem_namespaces ADD COLUMN space_id TEXT, bucket_id TEXT, seal_policy_id TEXT (and a per-version/artifact harbor_file_id + seal_identity if recall is per-artifact — store on contextmem_namespace_versions or a small contextmem_namespace_storage table keyed by namespace+version). Add the new columns to the getNamespace SELECT (worker.ts:415) and namespaceFromRow (worker.ts:4531) → HostedNamespaceSummary (sealPolicyId, bucketId, spaceId). (Mapping population is Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7's job; this issue just reads them.)

    10. GET /api/memwal/artifact (recall fetch+decrypt; gated by Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5). Register in the route block (worker.ts:~630, beside /api/memwal/recall): GET /api/memwal/artifact?namespace=…&path=….

      • await store.authorizeNamespace(namespace, readAccessToken(request)) (worker.ts:502/616) — return its {status,message} on failure (token-gated, AC#5/AC#6).
      • If auth.summary.visibility === "public" → resolve downloadUrls/aggregator from the stored receipt/index and stream plaintext (Tatum fallback path, AC#5).
      • If private → resolve bucketId/fileId/sealPolicyId/sealIdentity from D1 (step 9) → HarborStorageProvider.fetch(fileId) → ciphertext → seal.decrypt(ciphertext, { sessionKey: serviceKeySessionKey, sealPolicyId, identity, suiClient }) → stream plaintext. Build the SessionKey from HARBOR_SERVICE_PRIVATE_KEY (resolved like resolveMemwalCreds, worker.ts:3341).

    11. Token→SEAL decrypt mapping (AC#6). Wire near authorizeNamespace (:502) and token create/revoke (createNamespaceToken :992, revokeNamespaceToken :1013):

      • Primary deliverable: the Worker decrypts only after authorizeNamespace passes, so revoking the token → 403 → recall can no longer decrypt (the headline Sui-native demo) with the shared service-key SessionKey.
      • Add a grantSealDecrypt(namespace, token) / revokeSealDecrypt(namespace, tokenId) hook at those two call sites. Full on-chain per-grantee policy mutation depends on Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7's service-key/policy capability decision; land the token-gates-Worker-decrypt enforcement now and leave the on-chain grant as a hook (documented follow-up). Never silently downgrade a private namespace to plaintext.

    12. Docs. Note the network split (public-mainnet Tatum vs private-testnet Harbor blobs live on different networks) in .env.example + a short comment near selectStorageProvider, per the issue's "document" requirement.

    New / changed files

    • packages/walrus/src/provider.ts (new — interface, TatumStorageProvider re-export glue, selectStorageProvider, storePrivateOrFallback)
    • packages/walrus/src/seal.ts (new — @mysten/seal encrypt/decrypt + deriveSealIdentity)
    • packages/walrus/src/harbor-provider.ts (new — HarborStorageProvider over Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6's HarborClient)
    • packages/walrus/src/storage.ts (edit — add TatumStorageProvider; keep existing fns)
    • packages/walrus/src/index.ts (edit — export provider/seal/harbor-provider)
    • packages/walrus/package.json (edit — add @mysten/seal)
    • packages/core/src/types.ts (edit :655 — widen provider, add Harbor fields)
    • packages/memwal/src/index.ts (edit :37/:281 — Harbor fields + parseStorageIndex back-compat) and packages/memwal/src/sdk.ts (:123 mirror)
    • packages/cli/src/index.ts (edit :341-378 — visibility/seal refs) and packages/mcp/src/index.ts (edit :200)
    • apps/api/src/worker.ts (edit — WorkerEnv HARBOR_*, getNamespace/namespaceFromRow columns, GET /api/memwal/artifact, token→SEAL hooks)
    • apps/api/migrations/0007_harbor_seal_refs.sql (new)
    • apps/api/cloudflare/wrangler.jsonc + wrangler.example.jsonc + .env.example (edit — Harbor vars/secrets)

    Test plan

    • Unit (bun, packages/walrus): selectStorageProvider({visibility:"public"}) returns TatumStorageProvider and {visibility:"private"} returns HarborStorageProvider (AC#7). deriveSealIdentity is deterministic for the same (namespace, runId). parseStorageIndex reads new [ctxm-storage] blocks and a legacy block with no Harbor fields (AC#4 back-compat).
    • Unit (fallback): mock Harbor failure → non-private upload falls back to Tatum and the receipt records provider:"tatum"; a private upload with Harbor failure throws and never produces a plaintext receipt (AC#6).
    • Integration / round-trip (the proof it works): against Harbor testnet — seal.encrypt → HarborClient.uploadFile → poll certified → GET /api/memwal/artifact with a valid token → seal.decrypt → assert Buffer.equals(plaintextOut, bundle.data) (remember→recall→bytes match, AC#7). Capture the real Sui testnet tx digest from the seal_approve devInspect / bucket finalize as evidence.
    • Negative auth: GET /api/memwal/artifact for the same private namespace without a valid read token → authorizeNamespace 401/403, no ciphertext fetched, no decrypt (AC#5/AC#6/AC#7). After revokeNamespaceToken, the previously-working token returns 403 and recall can no longer decrypt — the demo assertion.
    • Manual: CLI contextmem storage push <run> --visibility private --remember produces a provider:"harbor" receipt with bucketId/fileId/sealPolicyId/sealIdentity; a public push is byte-identical to today (provider:"tatum"). Confirm CI bun-check.yml typecheck+test+build stays green.

    Risks & gotchas

    • @mysten/seal in the Worker (Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5). WebCrypto/BLS CPU + compressed-bundle cap; if decrypt busts limits, step 10's decrypt must move to a Node relayer off CONTEXTMEM_EXTRACT_QUEUE — design the recall handler so the decrypt step is swappable.
    • Network split. Tatum=mainnet, Harbor=testnet — a public blob and a private blob for "the same" namespace live on different networks; recall must pick the path from the persisted provider, never guess. Document and never cross-reference.
    • Sponsor-sig expiry / mirror_missing_grant. Bucket finalize sponsor sig expires fast and Harbor returns a ~3s post-finalize 403 — that retry belongs in Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6's HarborClient; this issue must surface (not swallow) a genuine private-write failure so it fails closed.
    • Fail-closed correctness. The single most important invariant: a private namespace must NEVER fall back to Tatum plaintext on Harbor error (AC#6). Make the private branch rethrow with no catch-all.
    • packProofBundle is Node-only (tar/child_process, storage.ts:231) — keep private encryption on the CLI/MCP/queue Node path; the Worker hosted→Walrus private write (in-memory R2 zip) is explicitly out of scope here.
    • Migration 0007. ALTER TABLE ADD COLUMN on existing contextmem_namespaces is non-destructive; ensure new SELECTs tolerate NULLs for pre-migration rows (legacy public namespaces have no Harbor refs).
    • Service-key custody (Namespace→Space/Bucket/Seal mapping + Harbor service-key management in Worker secrets #7/Decision Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6). Shared key = the Worker can decrypt everything (token gate is the only guard); per-owner = stronger isolation but needs the SessionKey signer plumbed per request. Ship shared now, leave a seam.

    Acceptance-criteria mapping

    • StorageProvider interface + TatumStorageProvider, no behavior change → Steps 1, 3.
    • Widen provider union + Harbor fields (core/types.ts:655) → Step 2.
    • packages/walrus/src/seal.ts encrypt/decrypt + seal_approve PTB → Step 4.
    • selectStorageProvider, private encrypt+Harbor upload, receipt records provider, rememberStorageIndex/renderStorageIndex carry refs + back-compat → Steps 5, 6, 7, 8.
    • GET /api/memwal/artifact fetch+decrypt, token-gated via authorizeNamespace, public→aggregator → Steps 9, 10.
    • Fail-closed (non-private Tatum fallback, private never plaintext) + token create/revoke ↔ SEAL grant/revoke → Steps 6, 11.
    • Tests: selection by visibility, public-only Tatum fallback, private round-trip + no-token can't decrypt → Test plan (unit + integration round-trip + negative auth).

    Effort
    L — ~5-8 person-days of focused work after #5/#6/#7 land. Dep-free slice (AC#1 + AC#2 + TatumStorageProvider: Steps 1-3) is ~1 day and can ship immediately; the SEAL/Harbor/recall slice (Steps 4-11) is the remaining ~4-7 days and is gated by the spike and the HarborClient/mapping issues.

    Implementation plan generated for execution (planning phase). Part of the roadmap (#4).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Important: hardens the demo and core productharborHarbor encrypted Walrus storage on Sui (Space/Bucket/File)seal@mysten/seal client-side encryption + on-chain access controlwalrusWalrus blob storage / Sites / aggregator

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions