Crawl Walrus Sites via on-chain resource enumeration (Site object + blobIds) #22

Activity

1. harrymove-ctrl

   added this to the Sui Overflow milestone on Jun 16, 2026
2. harrymove-ctrl

   added

   P1Important: hardens the demo and core productImportant: hardens the demo and core product

   walrusWalrus blob storage / Sites / aggregatorWalrus blob storage / Sites / aggregator

   suiSui chain: tx signing, objects, wallet, zkLogin, explorerSui chain: tx signing, objects, wallet, zkLogin, explorer

   crawlingWeb/Walrus crawling and context-extraction qualityWeb/Walrus crawling and context-extraction quality

   on Jun 16, 2026
3. harrymove-ctrl commented on Jun 19, 2026

   

   harrymove-ctrl

   on Jun 19, 2026

   OwnerAuthor

   More actions

   Implementation Plan

   Approach
   Wire the already-real on-chain enumeration (packages/walrus resolveWalrusTarget + listWalrusResources) into the production Worker's per-source extractor (apps/api/src/worker.ts extractTargetContext, ~L2452). For .wal.app / SuiNS / 0x targets we resolve a WalrusSiteContext, enumerate resource paths + blobIds from the Sui Site object, fetch HTML/markdown bodies through the Walrus aggregator (verifying blobHash), and emit manifest pages whose source carries blobId/resourcePath. The key design decision is to avoid the @contextmem/walrus barrel (it re-exports storage.ts's child_process/tar and resources.ts's node:http.createServer preview server, which break the Worker bundle) by adding a Worker-safe subpath @contextmem/walrus/enumerate, and to thread Walrus provenance all the way into chunks/facts so FactSourceRef.blobId resolves. On any RPC/grpc failure the existing HTTP-portal crawl runs unchanged.

   Prerequisites & dependencies

   • GATING SPIKE (do first): confirm @mysten/sui/grpc SuiGrpcClient (getObject, listDynamicFields, nameService.lookupName) runs in the Worker under nodejs_compat — bundle size, CPU, and whether the Connect/gRPC-over-fetch transport works on Workers. This is the same SuiGrpcClient-in-Worker spike the roadmap calls a shared dependency with the on-chain receipt work (roadmap §"Dependencies"). If it busts limits, the enumeration still runs inside processExtractionJob, which already executes in the queue consumer (worker.ts:553) with a larger CPU budget than a request handler — note this as the fallback host, not the Node CLI.
   • No new creds/bindings. Enumeration is read-only against public mainnet Sui RPC (https://fullnode.mainnet.sui.io) + the public aggregator (https://aggregator.walrus-mainnet.walrus.space). No HARBOR_*, no Vectorize, no wallet/service key. (This zero-secret, verifiable read is the Sui-native demo angle.)
   • @mysten/sui ^2.9.1 is already a dep of apps/api (apps/api/package.json:20) and @contextmem/walrus is already a workspace dep (line 19). p-limit is a pure-JS walrus dep and bundles fine.
   • Network is fixed to mainnet for this issue: resolveWalrusTarget forces network:"mainnet" for .wal.app (resolve.ts:82-90) and defaults mainnet for 0x ids (resolve.ts:16). No mainnet/testnet split to reconcile here.

   Step-by-step

   1. SPIKE: SuiGrpcClient in the Worker. Add a temporary route or a tiny test Worker that calls resolveWalrusTarget("https://seal-docs.wal.app/") (the configured CONTEXTMEM_DEMO_SAMPLE_TARGET, wrangler.jsonc:8) then listWalrusResources(site) and logs resource count + first blobId. Verify it returns non-empty, the bundle builds with wrangler deploy --dry-run, and CPU stays in budget. Decide Worker vs queue-consumer host. Delete the spike route after.

   2. Create the Worker-safe enumeration module. New packages/walrus/src/enumerate.ts exporting resolveWalrusTarget (re-export from ./resolve.js) plus Worker-safe listWalrusResources and fetchWalrusResource extracted from resources.ts — same Sui/BCS logic (SuiGrpcClient.listDynamicFields + getObject + DynamicFieldStruct(ResourcePathStruct, ResourceStruct), bcs.ts), blobAggregatorEndpoint/quiltAggregatorEndpoint (quilt.ts), sha256Base64 verify — but with no node:fs cache, no node:http, no materializeWalrusSite/startWalrusPreview. Refactor resources.ts to import the shared core from enumerate.ts (single source of truth; CLI keeps working via the barrel). Add to packages/walrus/package.json exports: "./enumerate": "./src/enumerate.ts". (Note: resolve.ts's top-level import fs from "node:fs/promises" is only used in the .json branch, never hit for .wal.app/0x, and resolves harmlessly under nodejs_compat.)

   3. Add the Walrus enumeration branch in extractTargetContext (worker.ts:2452-2607). Detection already exists via target.hostname.endsWith(".wal.app") (used at 2462/2475/2554/2568). For Walrus targets, before the current header-scrape path:

      • import { resolveWalrusTarget, listWalrusResources, fetchWalrusResource } from "@contextmem/walrus/enumerate";
      • const site = await resolveWalrusTarget(target.toString());
      • const records = await listWalrusResources(site, { concurrency: 5 });
      • Cap to ~60-80 records; select HTML (/html/i.test(contentType) or .html?$) + markdown (.mdx?$) resources to fetch as pages; enumerate (but do not fetch bodies for) non-HTML assets — they still get blobId provenance in the structure.
      • For each page resource, await fetchWalrusResource(site, record) (omit cacheDir), respecting a per-resource byte cap aligned with the existing 1.5MB guard (worker.ts:2813/2816): skip/flag any resource > ~1.5MB instead of loading it. Convert HTML→markdown with the Worker's existing htmlToText/markdown path; verify sha256Base64(body) === record.blobHash (on mismatch, skip the page, don't fail the run).
      • Wrap the whole block in try/catch (Step 7).

   4. Build manifest pages + walrus.resources with real provenance. Replace the fake walrusResources (worker.ts:2479-2485, blobId "") with the real WalrusResourceRecord[]. Each manifest page built from a Walrus resource must set source: { kind: "walrus", resourcePath: record.path, blobId: record.blobId, blobHash: record.blobHash, quiltPatchId: record.quiltPatchId } (matches PageArtifact.source, types.ts:335). Set manifest.walrus = { site: { network: site.network, siteObjectId: site.siteObjectId, aggregatorUrl: site.aggregatorUrl }, resources: records } (replacing the headers-derived block at 2568-2576). Keep mode: "walrus" (2554).

   5. Populate the Walrus Provenance group via core buildSiteStructure. For Walrus runs, call buildSiteStructure({ target, pages, walrus: { resources: records } }) (site-structure.ts:16; already exported from core, index.ts:10) instead of the hand-rolled siteStructure object (worker.ts:2581-2607) — its walrus-provenance group (site-structure.ts:148) is built directly from input.walrus.resources. Because the final namespace manifest is assembled by buildCombinedNamespaceBundle (worker.ts:2125), also: (a) make sourceFromSingleExtraction (worker.ts:2069) carry the real WalrusResourceRecord[] for walrus sources so they flow through ExtractedNamespaceSource.resources into buildCombinedWalrusSummary (worker.ts:2244); and (b) in buildCombinedNamespaceBundle, when any source is walrus, set manifest.siteStructure = buildSiteStructure({ target: job.target, pages, walrus: { resources: combinedWalrusRecords } }) (replacing buildCombinedSiteStructure at 2166 for walrus runs; keep it for pure-web).

   6. Thread blobId provenance into chunks → facts (AC4).

      • Extend ContextChunk (types.ts:726) with optional blobId?, resourcePath?, quiltPatchId?.
      • In buildChunks (chunks.ts:13-43) copy page.source?.blobId/resourcePath/quiltPatchId onto each emitted chunk. chunkGraphDigest (chunks.ts:59) only hashes chunkId:contentHash, so adding these fields does NOT change the digest — planMemoryWrite diffing stays stable (call this out in the PR).
      • In sourceFromChunk (facts.ts) add blobId: chunk.blobId, resourcePath: chunk.resourcePath to the returned FactSourceRef (the fields already exist on FactSourceRef, facts.ts:25-32 — just unpopulated today).
      • In the Worker, ensure factsPages (worker.ts:2660-2671) preserves each page's source (currently dropped) so buildSiteFacts/buildChunks emit blobId-backed sources. Verify source survives sourceFromSingleExtraction (...page spread, worker.ts:2080) and the combined buildChunks(pages…) (worker.ts:2186) so /context/chunks.ndjson and /context/facts.json both carry blobIds.

   7. Fallback (AC5). Wrap Steps 3-4 in try/catch. On any error (SuiNS lookup failure, RPC/grpc error, empty enumeration), console.warn and fall through to the existing HTTP-portal crawl (the current extractWalrusHeaders + link-discovery path, worker.ts:2475-2606) so the run still completes with mode:"walrus".

   New / changed files

   • packages/walrus/src/enumerate.ts — NEW Worker-safe resolveWalrusTarget re-export + listWalrusResources/fetchWalrusResource (no fs/http/preview/cache).
   • packages/walrus/package.json — add "./enumerate" to exports.
   • packages/walrus/src/resources.ts — import shared enumeration core from enumerate.ts (de-dupe; keep Node caching/materialize wrappers here).
   • packages/core/src/types.ts — ContextChunk += blobId?/resourcePath?/quiltPatchId? (L726).
   • packages/core/src/chunks.ts — buildChunks copies page.source provenance (L13-43).
   • packages/core/src/facts.ts — sourceFromChunk emits blobId/resourcePath.
   • apps/api/src/worker.ts — Walrus enumeration branch + provenance in extractTargetContext (~L2452-2607); real walrus records through sourceFromSingleExtraction (L2069) + buildCombinedWalrusSummary (L2233); buildSiteStructure for walrus runs (replace L2166/L2581-2607); preserve source in factsPages (L2660).

   Test plan

   • Unit (bun test in packages/core): buildChunks on a PageArtifact with source.blobId emits chunks carrying that blobId; sourceFromChunk round-trips it into FactSourceRef.blobId; assert chunkGraphDigest is byte-identical before/after the field addition (proves diff stability).
   • Integration (packages/walrus, network): against seal-docs.wal.app, resolveWalrusTarget returns a real siteObjectId; listWalrusResources returns >0 records each with a non-empty blobId; fetchWalrusResource body passes sha256Base64 === blobHash (the round-trip provenance assert). This is the verifiable, tamper-evident proof a normal crawler can't produce.
   • Manual end-to-end: deploy to staging, enqueue a build for https://seal-docs.wal.app/, then inspect the run's /context/manifest.json (walrus.resources[].blobId populated, not ""), /context/site-structure.json (non-empty Walrus Provenance group), /context/facts.json + /context/chunks.ndjson (source.blobId/chunk blobId present). Record the resolved siteObjectId and one blobId in the issue as evidence.
   • Fallback: point at a .wal.app host while forcing an RPC error (bad rpcUrl in the spike, or block the fullnode) and confirm the run still completes via the HTTP portal path.

   Risks & gotchas

   • Worker bundle/CPU from @mysten/sui/grpc — the gating spike. If over budget, keep enumeration in the queue consumer (already the host for processExtractionJob); do not regress to Node-only.
   • Never import the @contextmem/walrus barrel in the Worker — it pulls storage.ts (node:child_process/spawn, node:os) and resources.ts (node:http.createServer), which fail under nodejs_compat. The ./enumerate subpath is mandatory.
   • Aggregator latency / large sites: cap resource count, concurrency (~5), and per-resource size (1.5MB guard) so a big site can't blow the Worker time/memory limits; enumerate assets without fetching their bodies.
   • Mainnet RPC rate limits / SuiNS lookup miss (.wal.app → nameService.lookupName, resolve.ts:109) — both are covered by the Step 7 try/catch fallback.
   • blobHash mismatch: skip the offending page rather than fail the whole run.
   • Combined-manifest plumbing: the final manifest is rebuilt by buildCombinedNamespaceBundle; if source/walrus records aren't threaded through sourceFromSingleExtraction, AC3/AC4 will pass per-source but vanish from the run output. Test the combined output, not just per-source.

   Acceptance-criteria mapping

   • Resolve WalrusSiteContext + enumerate paths/blobIds from the on-chain Site object (not HTML discovery) → Steps 2, 3.
   • Each resource fetched via the aggregator (quilt.ts blobAggregatorEndpoint) and mapped into manifest pages/resources carrying source.blobId/resourcePath → Steps 3, 4.
   • buildSiteStructure populates its existing Walrus Provenance group from input.walrus.resources → Step 5.
   • Chunks/facts carry blobId-backed FactSourceRef provenance → Step 6.
   • On-chain read failure falls back to the HTTP portal crawl and the run still completes → Step 7.

   Effort
   L — ~4-6 person-days. ~1 day gating spike; ~2-3 days Worker wiring + walrus subpath + combined-manifest plumbing; ~1-1.5 days core chunks/facts provenance + tests + staging verification.

   ---

   Implementation plan generated for execution (planning phase). Part of the roadmap (#4).