fix(platform): render a crawled page again only when it changed

[larryro]

Summary

The crawler's one way to learn whether a page had changed was to do everything again: fetch it, render it in a browser with every image, script and stylesheet the render loads, and compare content hashes afterwards. A 700-page site scanned every six hours answered about 8,000 requests and 2 GiB per scan for pages unchanged in months.

A scan now asks for each page once and opens the browser only for a page that changed.

What the one request brings back	What the scan does
304 Not Modified to the validators of the last visit (ETag, Last-Modified)	leaves the page as it is; nothing is downloaded
the page, and its plain HTML reads as it did at the last visit	leaves the page as it is; no browser
the page, with other text	renders it in the browser and indexes it again
a page whose content only its JavaScript draws	renders it on every scan, as before

Why the text and not a date. A server that builds its pages on request has no modification time to give. On the site this was measured on, a HEAD returns no ETag, no Last-Modified and no Content-Length, a conditional GET is answered 200 with the whole page, and two responses for one page differ in their bytes (a token, a nonce) while the text read out of them is identical. A static file on the same host carries both validators. The sitemap is no better witness: it dates its blog index 2023 while that page lists articles from this month.

Which pages can be judged by one request. When a page is rendered, the text of its plain HTML is compared with what the browser showed. If the plain HTML carried at least nine tenths of the words, the page is server-rendered and keeps its validators and the hash of its plain text; otherwise it keeps nothing and is rendered every time. Measured on real pages:

Page	Share of the rendered words in the plain HTML
three pages of the measured site	0.997, 0.949, 0.999
a prerendered docs page	0.990
a heavily hydrated React page	0.874 (rendered every time)
an Angular single-page application	0.007 (rendered every time)

Scan now runs the same scan as the schedule; there is no full mode.

What is stored

Knowledge-db migration 13 adds public_web.website_urls.probe_hash and puts the baseline's etag and last_modified back to work. The two went unused when the render lane arrived, so whatever they still hold is cleared once: a value written before then may belong to a JavaScript shell. The three columns are written only when a page is stored and indexed, and cleared when its content is purged.

Rollout

The first scan after the upgrade still renders every HTML page: no page has a plain-text hash yet. From the second scan on, unchanged pages cost one request. On the measured site that is about 700 requests of compressed HTML (55 to 91 KB each) in place of about 8,000 requests and 2 GiB.

Replaces the first approach on this branch

The first commit skipped pages by their sitemap <lastmod> and read every page again weekly. It is gone: a sitemap dates a listing page by its own last edit, not by what it lists, and a clock-based re-read is the load this change removes.

Not in this PR

• A changed server-rendered page is still fetched twice (probe, then browser). The debt ledger entry is updated.
• Blocking images, media and fonts in the render lane.

Checklist

• bun run check: every task of this PR's own passes locally on the merged tree (format, conflict markers, links, lint:manual, platform lint, platform typecheck, 82,556 platform tests, docs suite). Three tasks fail on this machine for reasons outside the diff and pass in CI: @tale/cli#test (a stale local install: tools/cli/node_modules/@inquirer/core resolves signal-exit to the hoisted 3.x), @tale/platform#test:ui (one tasks search test under load; 12 of 12 alone), @tale/sandbox-runtime-daemon#test (timeouts that vary run to run beside a running dev stack).
• bun run lint:sast passes (Opengrep): 0 findings.
• Translations: N/A, no UI string changed.
• Docs updated in docs/{en,de,fr}/platform/knowledge/crawling.md.
• READMEs: N/A, no setup, env var or command changed.

Test plan

• bunx vitest --run --project server backend/core/knowledge lib/knowledge in services/platform: the conditional request, both ways a page is found unchanged, the gates an unchanged page still passes (alias, X-Robots-Tag, robots meta tag), the probe handed to the render batch, what a rendered page keeps to be checked by, and the coverage measure ([KNOW-R16]).
• ITEST_LANES=checkFiles,checkWebsitesCrawl bun run backend:integration against a tale-db image built from this branch: 48/48. The three websites change check lanes run a fixture site whose server answers conditional requests and counts what it sent:
  • a page the server vouches for is answered 304 and not downloaded again, and is stamped as visited;
  • a changed page is downloaded and indexed again, and an HTML page with a new token in every response and the same text ends the scan completed without a render session, which the harness could not open;
  • Scan now is answered exactly as the scheduled scan was.
• Not observed in a running dev stack.

[yannickmonney]

Independent review: PR #4439

Verdict: FINDINGS

CODE

Change detection is a conditional GET using stored ETag/Last-Modified, with a fallback SHA/content hash of htmlToText output. Values are kept on public_web.website_urls (etag, last_modified, probe_hash) and written only after storing/indexing; purge clears them. Fetch/lifecycle errors are recorded on the URL row, and the scan/domain claim owns scan-level errors.

DATA

Migration 13 is additive/idempotent (ADD COLUMN IF NOT EXISTS) and clears legacy validators once. It is rolling-safe for old readers. The per-page URL row lock/claim prevents two scans of a domain, while the indexer transaction deletes/inserts chunks atomically. The migration down is intentionally empty, which is operationally safe but leaves rollback dependent on a later reviewed migration.

REL-MANUAL

304 and same-text probes leave indexed content untouched; changed HTML is rendered and re-indexed; failed probes/renders leave the page due and record an error. Robots meta/header checks still run on the probe path. Fetch and render timeouts are classified and retained. Finding: the coverage classifier can treat a page as server-rendered when only 90% of rendered words are present in plain HTML; a later JS change in the uncovered portion is skipped as unchanged (see below).

PERFORMANCE / RESOURCES

The optimization removes browser work for unchanged pages and bounds probe body reads by the render-size cap. Hashing is linear in the bounded response; validator strings are capped at 512 chars. Storage adds three nullable columns per URL and does not grow per scan.

SECURITY

The existing safe-fetch path is retained: HTTPS, host allowlist, redirect revalidation, private-address checks, response-size and timeout limits. Conditional headers contain only origin-provided validator values (trimmed/capped) and introduce no new egress or SSRF path.

TESTS

The new unit/integration coverage exercises 304, same-text token churn, changed text, JS-only pages, robots meta withdrawal, charset handling, render settlement, and scan claims. Those tests would fail against the old always-render behavior. Missing regression: a page with 90% plain-text coverage whose JS changes a visible 10% segment, and a relative-link change with unchanged visible text. Main checks run 37482586587 passed; no local test was run under the optional guard.

Finding

• MEDIUM — services/platform/backend/core/knowledge/crawl_action.ts:1563-1588: The unchanged fast path accepts any page whose initial render had plainTextCoverage >= 0.9 (services/platform/lib/knowledge/crawl-parse.ts:563-588). On later scans it compares only plain.hash and returns not_modified, so JavaScript-rendered changes in the uncovered portion are never observed or indexed. For example, a server-rendered article with a 95%-covered body and a JS-injected 5% price/status/banner can change that injected text while the probe hash stays identical; the crawler serves stale content indefinitely. Either require exact coverage for the one-request path, or retain a render/check signal that detects changes in the uncovered DOM, and add a regression for this boundary.

Related PR

Open PR #4445 reworks the same crawl/indexing area (vector storage and PageIndexer behavior). It is not part of this verdict; its diff changes crawl_action.ts and should carry this finding/regression forward when reviewed.

UI, LOCALE and A11Y are not applicable: this PR changes crawler/backend/migration behavior and documentation only, with no user-facing UI controls or locale keys.