Local hybrid retrieval index (Vectorize + BM25) over extracted chunks #14

Activity

1. harrymove-ctrl

   added this to the Sui Overflow milestone on Jun 16, 2026
2. harrymove-ctrl

   added

   P0Demo-blocking: required for a working Sui Overflow demoDemo-blocking: required for a working Sui Overflow demo

   featureUser- or agent-facing capabilityUser- or agent-facing capability

   crawlingWeb/Walrus crawling and context-extraction qualityWeb/Walrus crawling and context-extraction quality

   platformBackend platform plumbing: Worker, D1, queues, secrets, meteringBackend platform plumbing: Worker, D1, queues, secrets, metering

   on Jun 16, 2026
3. harrymove-ctrl commented on Jun 19, 2026

   

   harrymove-ctrl

   on Jun 19, 2026

   OwnerAuthor

   More actions

   Implementation Plan

   Approach
   Add a Cloudflare Vectorize index (contextmem-chunks, 768-dim cosine, model @cf/baai/bge-base-en-v1.5) and embed every chunk at build time inside the queue consumer, where buildChunks already runs and storeNamespaceImport already persists /context/chunks.ndjson to R2. The key design decision is to not duplicate chunk text in a new D1 table — chunk text is already stored verbatim in the R2 /context/chunks.ndjson artifact, so Vectorize stores only the embedding + tiny metadata {namespace, chunkId, routePath, url}, and retrieval hydrates full text from that R2 file. memwalChat then does hybrid retrieval (Vectorize ANN + in-memory BM25 over the loaded ndjson → RRF → MMR de-dup) and only falls back to the MemWal relayer when the local index returns zero hits. All hybrid/BM25/RRF/MMR logic lands in a new pure-function module so it's unit-testable without bindings.

   Prerequisites & dependencies

   • Vectorize binding + index (ops, blocking). Create the index and its metadata filter index before deploy:
     • npx wrangler vectorize create contextmem-chunks --dimensions=768 --metric=cosine
     • npx wrangler vectorize create-metadata-index contextmem-chunks --property-name=namespace --type=string (required — namespace filtering fails without it)
     • Repeat both for staging (contextmem-chunks-staging).
   • Workers AI binding — already present (env.AI, wrangler.jsonc:36-38; used at worker.ts:3582). No new secret.
   • Soft dependency on "Token-aware chunking with overlap" (roadmap §3 P0): better chunks → better embeddings. Not blocking — current buildChunks (packages/core/src/chunks.ts:13, MAX_CHUNK_CHARS=4000) already emits stable chunkIds we key on. Ship independently; quality improves when chunking lands.
   • Roadmap decision (Token-aware chunking with overlap and structure-safe splits #12 in roadmap): "Commit to Cloudflare Vectorize binding/cost vs. continue delegating recall to MemWal." This issue assumes commit to Vectorize — confirm before starting (Vectorize has a free tier; cost note in Risks).
   • No Sui/Harbor/SEAL involvement — this is off-chain edge infra (verifiability stays with receipt + chunkGraphDigest + stored ciphertext, per issue notes).

   Step-by-step

   1. Add the Vectorize binding + types.

      • apps/api/cloudflare/wrangler.jsonc: add a top-level "vectorize": [{ "binding": "VECTORIZE", "index_name": "contextmem-chunks" }] (next to ai at line 36-38) and a mirrored block in the env.staging section (after the staging ai binding, ~line 86) pointing at contextmem-chunks-staging.
      • apps/api/src/worker.ts: extend WorkerEnv (type at :32-55) with VECTORIZE?: VectorizeIndexLike; after AI? (:47). Define a local VectorizeIndexLike interface (mirroring the existing *Like pattern at :70/:82/:99) with upsert(vectors: Array<{id:string; values:number[]; metadata?:Record<string,string>}>): Promise<unknown> and query(values:number[], opts:{topK:number; filter?:Record<string,unknown>; returnMetadata?:boolean|"all"}): Promise<{matches:Array<{id:string; score:number; metadata?:Record<string,string>}>}>.
      • Widen the WorkersAiBinding type (:57-59) with an embeddings overload: run(model:string, options:{text:string|string[]}): Promise<{data:number[][]}> (current signature only allows the chat messages shape).

   2. Create apps/api/src/retrieval.ts (new, pure functions — no bindings). Houses the testable core:

      • parseChunksNdjson(text:string): ContextChunk[] — split on newlines, JSON.parse each (inverse of renderChunksNdjson, packages/core/src/chunks.ts:45).
      • bm25(query:string, chunks:ContextChunk[]): Array<{chunkId:string; score:number}> — tokenize, compute IDF + BM25 (k1=1.2, b=0.75) over the in-memory candidate set (a namespace is ~hundreds of chunks, fine in-memory; no inverted index needed per issue note).
      • reciprocalRankFusion(lists: Array<Array<{chunkId:string}>>, k=60): Array<{chunkId:string; score:number}> — 1/(k+rank) summed across the vector list and BM25 list.
      • mmr(ranked: ContextChunk[], queryTerms, lambda=0.7, topK): ContextChunk[] — token-overlap (Jaccard) diversity de-dup so near-identical chunks don't dominate.
      • vectorIdFor(namespace, chunkId) helper (see Risks: hash to respect the 64-byte id cap).

   3. Embed + upsert at build time. Add async function indexNamespaceChunks(env, namespace, chunks: ContextChunk[]): Promise<void> in worker.ts:

      • Guard if (!env.VECTORIZE || !env.AI) return; (best-effort — never fail a build on index errors).
      • Batch chunk text into groups of ~96, call env.AI.run("@cf/baai/bge-base-en-v1.5", { text: batch }) → data: number[][].
      • Build vectors { id: vectorIdFor(namespace, chunk.chunkId), values, metadata: { namespace, chunkId: chunk.chunkId, routePath: chunk.routePath, url: chunk.url ?? "" } } and await env.VECTORIZE.upsert(...) in batches (≤1000/call).
      • Call it from processExtractionJob (worker.ts:1951) immediately after storeNamespaceImport returns (~:1977), wrapped in try/catch and routed through requestContext.waitUntil?.(...) so indexing doesn't block job completion and a Vectorize/AI outage can't fail the extraction. Recompute chunks there via buildChunks(...) on the same pages (or thread the already-built chunks out of extractNamespaceContext to avoid recompute — both build paths already call buildChunks at :2186 and :2790). The chunk text needs no extra persistence: /context/chunks.ndjson is already written to R2 by storeNamespaceImport (:1786).

   4. Hybrid retrieval in memwalChat (worker.ts:3498). Replace the relayer-first grounding block (:3521-3535) with a local-first path. Add async function hybridRecall(env, namespace, query, topK): Promise<ContextChunk[]>:

      1. Load /context/chunks.ndjson via new CloudflareNamespaceStore(env).readArtifact(namespace, "/context/chunks.ndjson") (:454) → parseChunksNdjson → Map<chunkId, ContextChunk>. If absent/empty, return [].
      2. Vector pass: env.AI.run("@cf/baai/bge-base-en-v1.5", { text: query }) → env.VECTORIZE.query(vec, { topK: topK*3, filter: { namespace }, returnMetadata: "all" }) → ordered chunkIds (from match.metadata.chunkId). Drop matches whose chunkId isn't in the current ndjson map (handles orphan vectors from prior versions).
      3. BM25 pass over the in-memory chunk set.
      4. reciprocalRankFusion([vectorList, bm25List]) → take top candidates → mmr(...) → first topK.
      • In memwalChat, read topK from the request body (default 6 to match today), call hybridRecall, and map results into the existing recallHits shape ({ text, distance? }).

   5. Preserve sources[] shape. Where memwalChat builds sources (:3602-3604), emit from the hydrated chunks: { url: chunk.url ?? "", routePath: chunk.routePath, quote: chunk.text.slice(0, 280), blobId: undefined }. This keeps the exact {url, routePath, quote, blobId} contract the web UI renders. Also set source to a clear local label (e.g. "local-index" / keep "mixed" when facts also contribute) and surface a usedProvider like local-hybrid:bge-base+bm25 for demo/debug visibility.

   6. Demote MemWal to fallback-only. Keep the existing MemWalMcpClient.recallSiteContext call (:3525) but only invoke it when hybridRecall returns zero hits (and a delegate is present via resolveMemwalCreds, :3341). This removes the flaky relayer from the hot path while preserving graceful degradation for namespaces not yet indexed.

   7. Backfill note (optional, P2). Existing namespaces built before this lands have no vectors. Add a tiny authed POST /api/hosted/namespaces/:ns/reindex (or a CLI one-shot) that loads /context/chunks.ndjson and runs indexNamespaceChunks. Not required for acceptance; the demo namespace can simply be rebuilt once.

   New / changed files

   • apps/api/cloudflare/wrangler.jsonc — add vectorize binding (top-level + staging env).
   • apps/api/src/worker.ts — WorkerEnv + VectorizeIndexLike + WorkersAiBinding embedding overload; indexNamespaceChunks + call site in processExtractionJob (:1951/~:1977); rewrite of memwalChat grounding/sources (:3498, :3521-3535, :3602-3604); hybridRecall.
   • apps/api/src/retrieval.ts — new: parseChunksNdjson, bm25, reciprocalRankFusion, mmr, vectorIdFor.
   • apps/api/src/retrieval.test.ts — new: unit tests for the pure functions.
   • (Optional) apps/api/src/worker.ts reindex route + tiny handler.
   • No D1 migration (text reuses R2 /context/chunks.ndjson). If the team prefers D1, add apps/api/migrations/0007_chunk_index.sql with contextmem_chunks(namespace, chunk_id, route_path, url, text, version_id) — documented as the alternative.

   Test plan

   • Unit (bun test, runs in bun-check.yml): deterministic asserts on bm25 (ranking order for a known doc set), reciprocalRankFusion (fused order from two synthetic rank lists), mmr (de-dups two near-identical chunks), parseChunksNdjson round-trips renderChunksNdjson(buildChunks(...)), and vectorIdFor stays ≤64 bytes for a long walrus:testnet:0x… namespace.
   • Integration (hybridRecall/indexNamespaceChunks): inject fake VectorizeIndexLike + WorkersAiBinding (return canned embeddings) + an in-memory R2 stub; assert a build upserts N vectors with correct ids/metadata, and a query returns the expected sources[] hydrated from ndjson. (Note: Miniflare/wrangler dev does not emulate Vectorize, so CI must use mocks — see Risks.)
   • Manual on staging (the proof): create the staging Vectorize index + metadata index, deploy the Worker (manual wrangler deploy --env staging — CI doesn't ship the Worker), build the seal-docs.wal.app demo namespace, wait for eventual consistency, then POST /api/memwal/chat. Pass criteria: (a) usedProvider reports local-hybrid…, (b) sources[] are populated with real url/routePath/quote from chunks, (c) with MemWal creds unset/removed, chat is still grounded — proving the relayer is off the hot path, (d) a namespace with no index falls back to MemWal/facts without erroring.

   Risks & gotchas

   • Vectorize eventual consistency — never query right after upsert in the same run; the build-time upsert and chat-time query are separate invocations, so fine in practice, but the integration test must not assert query-after-upsert without a delay.
   • Metadata filter index is mandatory — query({filter:{namespace}}) silently returns cross-namespace or empty results unless create-metadata-index --property-name=namespace was run. Per-namespace isolation depends on it.
   • Vector id 64-byte cap — ${namespace}:${chunkId} overflows for long walrus/target namespaces. vectorIdFor must hash: e.g. ${sha256Hex(namespace).slice(0,24)}:${chunkId} (keep real namespace+chunkId in metadata for filtering/hydration).
   • Don't store chunk text in Vectorize metadata — Vectorize caps metadata at ~10KiB/vector and chunks can be ~4KB; we hydrate text from R2 instead, which sidesteps the cap entirely.
   • Orphan vectors on rebuild — chunkIds are location-stable so upserts overwrite, but deleted pages leave stale vectors. Retrieval already drops any matched chunkId not present in the current ndjson, so stale vectors are inert; periodic cleanup is optional (P2).
   • Workers AI / subrequest limits — embedding + Vectorize calls are subrequests (free plan caps at ~50/invocation; paid ~1000). Batch embeddings (~96 texts/call) and upserts (≤1000 vectors/call) to stay well under. Indexing is best-effort + waitUntil so limits/outages never fail a build.
   • Cost — Workers AI embeddings + Vectorize queries/storage are billed per build/query; acceptable for alpha, but flag for the team (ties to roadmap decision Token-aware chunking with overlap and structure-safe splits #12).
   • Model/dim lock-in — the index --dimensions must match the model (768 for bge-base-en-v1.5). Switching to bge-small (384) later requires a new index + full re-embed; pick bge-base now for quality.
   • Two build paths — buildChunks is called in both the combined (:2186) and single-target (:2790) builders; ensure indexNamespaceChunks runs once per job from processExtractionJob, not per builder, to avoid double work.

   Acceptance-criteria mapping

   • Vectorize binding added to wrangler/WorkerEnv, dim matches model → Step 1 (+ Prereqs: index creation).
   • Build-time embed via Workers AI bge, upsert keyed ${namespace}:${chunkId} with {namespace,chunkId,routePath,url} metadata; chunk text persisted for hydration → Step 3 (text reuse of R2 /context/chunks.ndjson; id hashed per Risks but metadata carries chunkId).
   • memwalChat hybrid retrieval: embed query → Vectorize topK (namespace-filtered) fused with BM25 via RRF, then MMR de-dup, configurable topK → Steps 2 + 4.
   • MemWal relayer recall becomes fallback used only when local index returns nothing → Step 6.
   • sources[] shape (url/routePath/quote/blobId) preserved for the UI → Step 5.

   Effort
   L — ~3-4 person-days. ~0.5d binding/types/index setup; ~1d retrieval.ts + unit tests; ~1d build-time indexing wiring; ~1d memwalChat rewrite + staging round-trip verification.

   ---

   Implementation plan generated for execution (planning phase). Part of the roadmap (#4).