Cloudflare Browser Rendering fallback for JS/SPA pages #13

Activity

1. harrymove-ctrl

   added this to the Sui Overflow milestone on Jun 16, 2026
2. harrymove-ctrl

   added

   P0Demo-blocking: required for a working Sui Overflow demoDemo-blocking: required for a working Sui Overflow demo

   featureUser- or agent-facing capabilityUser- or agent-facing capability

   crawlingWeb/Walrus crawling and context-extraction qualityWeb/Walrus crawling and context-extraction quality

   platformBackend platform plumbing: Worker, D1, queues, secrets, meteringBackend platform plumbing: Worker, D1, queues, secrets, metering

   on Jun 16, 2026
3. harrymove-ctrl commented on Jun 19, 2026

   

   harrymove-ctrl

   on Jun 19, 2026

   OwnerAuthor

   More actions

   Implementation Plan

   Approach
   Add a Cloudflare Browser Rendering binding (BROWSER) and @cloudflare/puppeteer to the Worker, then insert a headless-render fallback into the production extractor fetchPageContent() (apps/api/src/worker.ts:2884). The render path is gated by an empty-shell heuristic: we still raw-fetch first (cheap, gives us the HTML to inspect + Walrus headers), and only spin up the browser when the fetched HTML looks like an un-hydrated SPA. Because fetchPageContent is reached only via extractTargetContext (:2452) → processExtractionJob (:1951) → async queue consumer (:545), the render path is already confined to the queue consumer and never touches the request path — no extra guard needed. Key design decision: render lazily and reuse one browser per extraction with a small concurrency cap, and degrade to the existing raw-fetch + htmlToText result on any render failure so a run never fails because of rendering.

   Prerequisites & dependencies

   • No blocking issues. Issue states "Dependencies: None"; independent of the chunking/Vectorize/Harbor tracks.
   • Cloudflare account binding/quota: Browser Rendering must be enabled on the account for both contextmem-backend and contextmem-backend-staging. Browser Rendering has a free tier (limited minutes/day) but real usage typically needs a Workers Paid plan; confirm before relying on it in the demo. There is no --remote-free local browser — wrangler dev must use --remote to exercise it.
   • New dep: @cloudflare/puppeteer in apps/api/package.json (Worker-native; do not confuse with the playwright stub already aliased in wrangler.jsonc).
   • Roadmap open question to answer in the PR (cite issue): "is Firecrawl permanent or does Browser Rendering become the default renderer?" — this plan keeps Firecrawl first and Browser Rendering as the empty-shell fallback (recommended for cost); note the decision in the PR.

   Step-by-step

   1. Add the binding to apps/api/cloudflare/wrangler.jsonc. Add a top-level "browser": { "binding": "BROWSER" } and also add the same block under env.staging — bindings are NOT inherited in this config (see the comment block in the file; ai/queues/d1/r2 are redefined per-env).
   2. Add the dependency: cd apps/api && bun add @cloudflare/puppeteer (adds to apps/api/package.json deps alongside @mysten/sui).
   3. Extend WorkerEnv (apps/api/src/worker.ts:32): add BROWSER?: BrowserWorkerLike;. Define a minimal hand-rolled type next to the other *Like types (R2BucketLike:82, QueueLike:99) — the repo uses hand-rolled binding types, not @cloudflare/workers-types: type BrowserWorkerLike = { fetch: typeof fetch }; (puppeteer's launch() accepts a Fetcher-shaped binding; cast at the call site).
   4. Widen the engine unions to add "browser": FetchedText at :298 (engine?: "fetch" | "firecrawl" | "browser") and the source-row type at :253 (engine?: "firecrawl" | "fetch" | "browser").
   5. Add an empty-shell heuristic (new helper near crawlContentHash, :2904): function isEmptySpaShell(html: string, text: string): boolean. Return true when the stripped main-content length (reuse htmlToText(html).trim().length, :5217) is below a threshold (~300 chars) AND the HTML shows SPA markers: /id=["']root["']/, __next / id=["']__next["'], id=["']app(-root)?["'], ng-version, data-reactroot, OR a high script-to-prose ratio (sum of <script> payload length ≫ stripped-text length). Keep it conservative to avoid rendering normal pages.
   6. Add the render helper (near firecrawlScrape, :2830): async function renderWithBrowser(url, env, opts): Promise<FetchedText>:
      • import puppeteer from "@cloudflare/puppeteer"; (top of file).
      • const browser = await puppeteer.launch(env.BROWSER as any); then const page = await browser.newPage();
      • await page.goto(url, { waitUntil: "networkidle0", timeout: opts.timeoutMs ?? 20000 });
      • Honor a wait: if opts.waitForMs (map from CrawlOptions.waitForMs, packages/core/src/types.ts:66) await new Promise(r => setTimeout(r, waitForMs)).
      • const html = await page.content(); — enforce the same 1.5 MB guard as fetchText (:2813/2816): if html.length > 1_500_000 truncate/throw.
      • Return { text: html, contentType: "text/html; charset=utf-8", headers: {}, engine: "browser", markdown: htmlToText(html) }.
      • Wrap in try/finally so the page/browser always close; never let a render error bubble — caller falls back.
   7. Rewire fetchPageContent order (:2884) to firecrawl (if key) → Browser Rendering (if empty-shell) → raw fetch + htmlToText:
      • If env.FIRECRAWL_API_KEY: try firecrawlScrape (unchanged) — Firecrawl already executes JS, so on success return it.
      • const raw = await fetchText(url); (we need the HTML to run the heuristic and to keep Walrus headers).
      • if (env.BROWSER && isEmptySpaShell(raw.text, htmlToText(raw.text))) { try { return await renderWithBrowser(url, env, {...}); } catch (e) { console.warn("[browser-render] fallback to raw fetch", ...); } }
      • raw.markdown = htmlToText(raw.text); return raw; (unchanged final fallback).
   8. Record the engine on the artifact (real fix, not the brittle string match). Today sourceFromSingleExtraction derives engine from sourceText.includes("firecrawl") (:2111) and manifest.source is "firecrawl-v2" | "cloudflare-fetch-extractor" (:2556). Add engine: home.engine to the manifest object in extractTargetContext (:2549), and change :2111 to engine: (manifest.engine as ... ) ?? (sourceText.includes("firecrawl") ? "firecrawl" : "fetch"). This makes FetchedText.engine (incl. "browser") flow into sources.json per AC.
   9. Browser reuse + concurrency cap (cost control). The crawl loop fans out with Promise.allSettled over fetchPageContent (:2524) with no throttle. Add a tiny semaphore (cap ~2 concurrent renders) around the render branch, and lazily launch one browser per extractTargetContext invocation reused across that run's empty-shell pages, closed in a finally at the end of extractTargetContext (:2452). Simplest threading: a small RenderContext object (holds the lazily-launched browser + semaphore) created in extractTargetContext and passed into fetchPageContent/renderWithBrowser as an extra optional arg (default undefined → launch-per-call). Optionally use puppeteer.sessions(env.BROWSER) + puppeteer.connect() to reuse a warm session across batches (nice-to-have; document the cap).
   10. Confirm downstream shape is unchanged (AC5). renderWithBrowser returns the same FetchedText shape with .text = rendered HTML, so the existing markdown/headings/brand path (home.markdown ?? htmlToText(home.text) at :2561/2781, brand/design enrichment that reads home.text) is untouched. No changes to chunk/fact/design-system code.

   New / changed files

   • apps/api/cloudflare/wrangler.jsonc — add browser binding (top-level + env.staging).
   • apps/api/package.json — add @cloudflare/puppeteer.
   • apps/api/src/worker.ts — import puppeteer; BrowserWorkerLike type; WorkerEnv.BROWSER (:32); widen engine unions (:253, :298); isEmptySpaShell() + renderWithBrowser() helpers; rewire fetchPageContent() (:2884); thread engine into manifest (:2549) + read it in sourceFromSingleExtraction (:2111); render context/semaphore in extractTargetContext (:2452).
   • apps/api/src/worker.test.ts — new tests (below).

   Test plan

   • Unit (worker.test.ts): Mirror the existing test "uses Firecrawl when configured and falls back to raw fetch when it fails" (:454) and the fetch-stub helper at :1094. Add:
     • vi.mock("@cloudflare/puppeteer", () => ({ default: { launch: async () => fakeBrowser } })) where fakeBrowser.newPage() returns a page whose content() resolves to fully-rendered HTML.
     • Stub fetch so the source URL returns an SPA shell (<div id="root"></div> + a big inline <script>, near-empty prose). Provide an env with BROWSER: {} and no FIRECRAWL_API_KEY, run an extraction job, then assert sources.json contains "engine": "browser" and the rendered text landed in /site/index.md (read via CloudflareNamespaceStore.readArtifact, same as :478/500).
     • Negative test: a content-rich page (no SPA markers) → isEmptySpaShell false → engine: "fetch", browser never launched.
     • Graceful-degrade test: launch()/goto throws → result still engine: "fetch" with raw htmlToText content, run does not fail.
   • Integration / manual: bunx wrangler dev --remote --config apps/api/cloudflare/wrangler.jsonc, POST an extraction for a known JS-rendered SPA (e.g. a *.wal.app portal or a Next.js docs site) with no Firecrawl key set; verify the run completes, sources.json shows "engine": "browser", and /site/index.md has real prose (not an empty shell). Capture the before/after content length as proof.
   • CI: bun run check (typecheck + vitest + build) — confirm the worker still bundles under the Worker size limit with @cloudflare/puppeteer added (see Risks).

   Risks & gotchas

   • Worker bundle size / build: @cloudflare/puppeteer is sizeable; verify the deployed bundle stays under the gzip limit in bun-check.yml's build step. If it busts, lazy-import() puppeteer inside renderWithBrowser so it only loads when the binding exists.
   • No local browser: Browser Rendering only works with wrangler dev --remote / deployed — unit tests must mock puppeteer; don't expect it under plain vitest.
   • Cost / concurrency caps: Browser Rendering bills per session and has concurrent-session limits. The empty-shell gate + one-browser-per-run + semaphore (step 9) are the mitigations; do not render every page unconditionally.
   • Hangs: networkidle0 can stall on pages with long-polling/analytics — the hard timeout in goto + try/finally close + graceful fallback (AC4) are mandatory.
   • Heuristic tuning: too-aggressive thresholds will render normal pages (cost) or miss shells (empty extraction). Keep conservative defaults and log when the browser path triggers so it can be tuned from real runs.
   • Network split: unrelated to mainnet/testnet storage — this is pure HTTP fetch; no Sui/Walrus tx involved.
   • Engine-recording bug: the current sources.json engine field is derived from a string match on manifest.source (:2111), not FetchedText.engine — step 8 fixes this so "browser" is actually recorded; without it AC2 silently fails.

   Acceptance-criteria mapping

   • browser binding in wrangler.jsonc + WorkerEnv, @cloudflare/puppeteer wired in the queue consumer → Steps 1, 2, 3, 6 (render runs only via extractTargetContext→queue, already confined).
   • Order firecrawl → Browser Rendering (if empty-shell) → raw fetch, engine recorded on FetchedText.engine → Steps 4, 7, 8.
   • Empty-shell heuristic (root/__next/app-root/large script payload + low prose) → Step 5.
   • Honors wait (waitForMs/networkidle) + hard timeout; failures degrade to raw fetch → Steps 6, 7 (try/catch fallback), 9.
   • Rendered HTML flows through existing markdown/headings/brand path, downstream artifacts unchanged in shape → Steps 6 (returns standard FetchedText), 10; verified by the unit test reading /site/index.md + chunks.

   Effort: M, ~2-3 person-days (1 day binding+wiring+heuristic, 0.5 day browser reuse/semaphore, 1 day tests + remote verification + threshold tuning).

   ---

   Implementation plan generated for execution (planning phase). Part of the roadmap (#4).