Build HarborClient: reserve→sign→finalize bucket, multipart upload, status poll, download #6

Activity

1. harrymove-ctrl

   added this to the Sui Overflow milestone on Jun 16, 2026
2. harrymove-ctrl

   added

   P0Demo-blocking: required for a working Sui Overflow demoDemo-blocking: required for a working Sui Overflow demo

   featureUser- or agent-facing capabilityUser- or agent-facing capability

   harborHarbor encrypted Walrus storage on Sui (Space/Bucket/File)Harbor encrypted Walrus storage on Sui (Space/Bucket/File)

   walrusWalrus blob storage / Sites / aggregatorWalrus blob storage / Sites / aggregator

   on Jun 16, 2026
3. harrymove-ctrl commented on Jun 19, 2026

   

   harrymove-ctrl

   on Jun 19, 2026

   OwnerAuthor

   More actions

   Implementation Plan

   Approach
   Add a self-contained, runtime-agnostic packages/walrus/src/harbor.ts exporting a HarborClient built on pure fetch (Worker + Node) plus @mysten/sui for signing only (no @mysten/seal import here — encryption lives in the sibling seal.ts issue). The one hard design constraint is that the Enoki sponsor signature expires fast, so createBucket performs RESERVE → (synchronous CPU-only) SIGN → FINALIZE as one tight back-to-back sequence with zero unrelated awaits in between. Everything else (config injection, defensive JSON parsing, the status-poll loop) is mirrored from the proven Tatum code in storage.ts so the later StorageProvider seam can wrap it cleanly.

   Prerequisites & dependencies

   • Informs but does NOT block this build — Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5 (@mysten/seal Worker spike). It decides where sign/upload eventually runs (Worker vs queue vs Node), but HarborClient is runtime-agnostic by design and uses only @mysten/sui crypto (BLS-free Ed25519 signing), so it can be authored and unit-tested now. Do not import @mysten/seal from harbor.ts.
   • Consumer issues (not blockers): the namespace→Space/Bucket/Seal-mapping issue supplies serviceKey + spaceId config this client consumes; the StorageProvider-abstraction/receipt-widening issue (which widens WalrusStorageReceipt.provider from the literal "tatum" at packages/core/src/types.ts:656) wraps this client. harbor.ts stays decoupled from both — it returns plain typed shapes.
   • External creds/config to inject (constructor, mirroring TatumStorageConfig at storage.ts:21): apiKey (hbr_… Bearer), serviceKey (suiprivkey1…), spaceId, baseUrl, optional fetch. A Sui testnet service key is required, but it needs no SUI balance (Enoki sponsors bucket-creation gas). Worker-secret wiring (HARBOR_API_KEY/HARBOR_SERVICE_PRIVATE_KEY/HARBOR_SPACE_ID into WorkerEnv at apps/api/src/worker.ts:32) is the separate key-mgmt issue — out of scope here.
   • OPEN DECISION (single biggest unknown) — exact Harbor alpha API surface. The precise endpoint paths and request/response field names for RESERVE / FINALIZE / upload / status / download are not in the repo. Before coding, confirm them against Harbor docs or a live curl (capture one real RESERVE response). Mitigation baked into the plan: parse defensively (port pickString/pickUrls from storage.ts:74-101) so minor field-name drift doesn't break the client.
   • @mysten/sui is already a workspace dep at ^2.9.1 (root package.json:31, apps/api/package.json:20), so the version is known-good under nodejs_compat.

   Step-by-step

   1. Add deps + re-export. In packages/walrus/package.json (currently only fast-xml-parser + p-limit, lines 13-17) add "@mysten/sui": "^2.9.1" and "@mysten/seal": "^0.x" (seal added per issue note for the sibling seal.ts; harbor.ts itself must not import it). Add export * from "./harbor.js"; to packages/walrus/src/index.ts (after line 6). Run bun install.
   2. Config + types in new packages/walrus/src/harbor.ts. Define HarborClientConfig = { apiKey, serviceKey, spaceId, baseUrl, fetch? } and a private resolveFetch/resolveConfig (mirror storage.ts:49, but read no process.env so it stays Worker-clean — caller injects everything). Define result types: HarborBucket = { bucketId, sealPolicyId, txDigest? }, HarborFileAccepted = { fileId, status? }, HarborFileState = { fileId, status, ready, failed, raw } (shape parallels WalrusStorageJobState at storage.ts:39). Port readBody/pickString/pickUrls (storage.ts:64-101) as module-private helpers. No node:fs/node:child_process/node:os/node:path imports (AC Walrus storage + Memory via Tatum, plus CI deploy #1).
   3. createBucket(spaceId, name) — RESERVE → SIGN → FINALIZE, back-to-back:
      • Build the signer once: import { Ed25519Keypair } from "@mysten/sui/keypairs/ed25519" + import { decodeSuiPrivateKey } from "@mysten/sui/cryptography"; const keypair = Ed25519Keypair.fromSecretKey(decodeSuiPrivateKey(serviceKey).secretKey); (memoize on the instance).
      • RESERVE: POST {baseUrl}/…/buckets/reserve with Authorization: Bearer hbr_…, body { spaceId, name }. Response carries the Enoki-sponsored unsigned tx: { txBytes (base64 BCS TransactionData), sponsorSignature, reservationId } — parse via pickString with candidate keys (txBytes/transactionBytes/bytes, reservationId/id).
      • SIGN (synchronous, no network/await of unrelated work): const { signature } = await keypair.signTransaction(fromBase64(txBytes)); (the single unavoidable await; nothing else between reserve and finalize — this is the sponsor-sig-expiry guard, issue note line 17).
      • FINALIZE: immediately POST {baseUrl}/…/buckets/finalize body { reservationId, signature } (Harbor already holds txBytes+sponsorSignature from reserve). Parse { bucketId, sealPolicyId } from the response (candidate keys bucketId/bucket_id/id, sealPolicyId/seal_policy_id/policyId). Return HarborBucket.
   4. uploadFile(bucketId, { name, data, contentType }) + download(fileId):
      • Upload via multipart exactly like storage.ts:109-112: new FormData() + new Blob([data], { type: contentType ?? "application/octet-stream" }), form.append("file", blob, name), POST {baseUrl}/…/buckets/{bucketId}/files with the Bearer header (no Content-Type — let fetch set the boundary). Wrap this POST in the mirror-grant retry (Step 5). Return { fileId }. The client takes ciphertext only (issue note line 19) — no encryption here.
      • download(fileId): GET {baseUrl}/…/files/{fileId}/download with Bearer → new Uint8Array(await res.arrayBuffer()) (ciphertext bytes) (AC Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5).
   5. mirror_missing_grant retry. Add a private withMirrorGrantRetry(fn) helper: on a 403 whose body contains mirror_missing_grant, retry with backoff covering the ~3s window (e.g. delays 400, 800, 1200, 1600, 2000 ms via injectable sleep, ~6 attempts ≈ 6s of cushion); any other non-OK status throws immediately with the parsed body (mirror the error style of storage.ts:115). Wrap uploadFile's POST and the first getFileStatus call with it (AC docs: add product roadmap (Harbor storage + Talus on-chain direction) #4).
   6. getFileStatus(fileId) + waitForFileReady. getFileStatus: GET {baseUrl}/…/files/{fileId}, normalize status to { ready, failed } (uppercase compare for ready states like READY/CERTIFIED/STORED and terminal FAILED/ERROR) — same normalization shape as getWalrusStorageJob (storage.ts:140-150). waitForFileReady(fileId, opts): deadline loop with pollIntervalMs/timeoutMs/onPoll, return on ready, throw on failed/timeout — a near-copy of waitForWalrusStorageCertified (storage.ts:160-177) so the receipt/poll shape matches Tatum (AC fix(ci): force bun + pin wrangler in Deploy workflow #3).
   7. Unit test packages/walrus/src/harbor.test.ts (vitest — import { describe, expect, it, vi } from "vitest", matching quilt.test.ts:1). Inject a fake fetch via config and a use a deterministic test suiprivkey1… testnet key. Cover: (a) reserve→sign→finalize ordering — assert RESERVE called before FINALIZE, FINALIZE body carries a real signature produced by signTransaction over the reserve txBytes, and { bucketId, sealPolicyId } returned; (b) mirror-grant retry — fake fetch returns 403 {"error":"mirror_missing_grant"} twice then 200, assert upload eventually succeeds and call count == 3 (inject a no-op sleep); (c) upload+poll happy path — uploadFile → getFileStatus/waitForFileReady transitions PENDING→READY; (d) download returns the exact ciphertext bytes the fake fetch served (round-trip assert). Add a static guard test (or rely on AC#1 review) that harbor.ts source contains no node: import.

   New / changed files

   • packages/walrus/src/harbor.ts (new — HarborClient + types + helpers)
   • packages/walrus/src/harbor.test.ts (new — vitest, mocked fetch)
   • packages/walrus/src/index.ts (edit — add export * from "./harbor.js";)
   • packages/walrus/package.json (edit — add @mysten/sui + @mysten/seal)

   Test plan

   • Unit (CI, no network): the four cases in Step 7, all on a mocked fetch + injected sleep so they run in bun run check/vitest run. The mirror-retry and reserve-before-finalize assertions are the load-bearing ones.
   • Integration / manual (proves it really works): a bun-run scratch script using a real HARBOR_API_KEY + testnet suiprivkey1 against the live alpha: createBucket → log the returned bucketId/sealPolicyId and the Sui tx digest, then verify the Bucket + Seal-policy objects on a testnet explorer (this is ContextMEM's first real signed Sui tx — the concrete "where's the contract" artifact, issue "Sui Overflow angle"). Then uploadFile a small ciphertext blob → waitForFileReady to ready → download → assert bytes are byte-identical to what was uploaded (full round-trip). Exercise the upload immediately after finalize to confirm the mirror_missing_grant path fires on real infra.
   • Acceptance: a captured testnet tx digest + a green round-trip assert = done.

   Risks & gotchas

   • Sponsor-sig expiry (highest): any await (DB write, logging round-trip, extra fetch) between RESERVE and FINALIZE can let the Enoki sponsor signature expire → finalize fails. Keep createBucket a single straight-line method; the only await between reserve and finalize is signTransaction. Surface a clear error if finalize returns an expiry/sponsor error so callers retry the whole reserve→finalize sequence (never reuse stale txBytes).
   • API-shape uncertainty: Harbor is testnet alpha; endpoint paths/field names may differ from assumptions. Defensive pickString parsing + centralizing all paths/keys in one const block at the top of harbor.ts keeps adaptation to a one-spot edit once a real response is captured.
   • Worker limits: harbor.ts deliberately avoids @mysten/seal (the BLS/CPU/bundle risk gated by Spike: run @mysten/seal encrypt+decrypt inside a Cloudflare Worker #5); @mysten/sui Ed25519 signing + fetch are Worker-safe under nodejs_compat. Multipart FormData/Blob upload already works in the Worker (proven by Tatum storage.ts).
   • signTransaction input format: it expects raw BCS TransactionData bytes, not a serialized Transaction object. Decode the reserve txBytes with fromBase64 (or fromB64, version-dependent) before signing; pin against the installed @mysten/sui@2.9.1 API.
   • Network split: these are testnet objects while Tatum stays mainnet — do not cross-reference Harbor fileId/bucketId with mainnet Tatum blobIds. Receipt/provider reconciliation is the StorageProvider issue's job.
   • Mirror window tuning: ~3s is a guideline; size the retry budget to ~6s and make sleep injectable so tests stay instant.

   Acceptance-criteria mapping

   • New harbor.ts exports HarborClient, no node:fs/child_process/os imports → Steps 1, 2 (and the static no-node: guard in Step 7).
   • createBucket(spaceId,name) RESERVE→sign(Ed25519Keypair+decodeSuiPrivateKey)→FINALIZE back-to-back, returns {bucketId, sealPolicyId} → Step 3.
   • uploadFile multipart → fileId; getFileStatus polls mirroring waitForWalrusStorageCertified (storage.ts:160) → Steps 4, 6.
   • Post-finalize 403 mirror_missing_grant retried with backoff over ~3s → Step 5.
   • download(fileId) returns ciphertext bytes → Step 4.
   • Unit test (mocked fetch) covers reserve→sign→finalize ordering, mirror retry, upload+poll happy path → Step 7.

   Effort
   L — ~3-4 person-days. ~1 day client skeleton (config/types/helpers/upload/status/download), ~1 day the reserve→sign→finalize signing path + mirror-grant retry, ~0.5 day tests, ~0.5-1 day reconciling the real Harbor alpha API shape from a live capture. Add a few hours of buffer if the alpha endpoint contract differs materially from the assumed shape.

   ---

   Implementation plan generated for execution (planning phase). Part of the roadmap (#4).
4. harrymove-ctrl

   added 2 commits that reference this issue on Jun 20, 2026

   fix(walrus): declare @mysten/sui dependency

   ...

   0f6fe00

   Merge pull request #29 from harrymove-ctrl/feat/harbor-seal-storage

   ...

   4dc3c48