diff --git a/.agents/repo.md b/.agents/repo.md index b3e4a5649a..2b178110f9 100644 --- a/.agents/repo.md +++ b/.agents/repo.md @@ -615,11 +615,14 @@ xlsx,odt}.ts`) still reach the catch-all as `failed` + `indexer_error` and are r refusing every delivery reads as never called (2026-09, round i). The reference says so. Paying it down means a `delivery_refused` skip reason stamped from the webhook door beside `start_refused`. -- **A page is fetched three to four times a scan** — the plain probe, the render pass and, for - the homepage, a create-time metadata probe. The robots parser now honours group selection by - product token (`User-agent: TaleBot` > `*`), `Allow` with longest-match precedence, `$` - end-anchors and `Crawl-delay` (`lib/knowledge/crawl-parse.ts`, 2026-09 round J); paying down the - remaining fetch count means one fetch per page per scan. +- **A changed page is fetched twice a scan** — the plain probe and the render pass — and the + homepage once more at create, for its metadata. A page that did not change is asked for once + and not rendered (2026-10: the probe's validators and the text of its plain HTML decide, + `core/knowledge/crawl_action.ts`); a page built by its JavaScript is still rendered on every + scan. The robots parser honours group selection by product token (`User-agent: TaleBot` > + `*`), `Allow` with longest-match precedence, `$` end-anchors and `Crawl-delay` + (`lib/knowledge/crawl-parse.ts`, 2026-09 round J). Paying down the rest means one fetch per + changed page: a server-rendered page read from the probe's own bytes, without the browser. - **A cancelled run answers `trace: null` and `effects: null`** where a failed run answers both, so what a cancel did not undo is readable only through `checkpoints` (2026-09, round i). Paying it down means keeping the partial trace the way the failed path does. diff --git a/docs/de/platform/knowledge/crawling.md b/docs/de/platform/knowledge/crawling.md index 07c6d4194d..f2860aace2 100644 --- a/docs/de/platform/knowledge/crawling.md +++ b/docs/de/platform/knowledge/crawling.md @@ -42,6 +42,8 @@ Bei einer ganzen Website nutzt der Crawler Startseite und veröffentlichte Sitem Scans arbeiten schrittweise: Unveränderte Inhalte werden übersprungen, geänderte erneut indexiert, neue Seiten hinzugefügt und entfernte aus dem Index genommen — ebenso Seiten, die die `robots.txt` inzwischen verbietet. Eine Seite, die auf eine andere Adresse derselben Website weiterleitet, wird nur einmal indexiert: unter der Adresse, auf der sie landet. Die Seitenzähler der Zeile folgen dem Scan, während Seiten landen, nach dem Entdecken und nach jedem gespeicherten Batch, die Tabelle bewegt sich also, während ein Scan läuft. Eine URL-Liste aktualisiert ihre feste Auswahl nach demselben Zeitplan. Nach erfolgreicher Indexierung ist keine gesonderte Veröffentlichung nötig. +Ein Scan fragt jede Seite einmal ab. Dabei schickt er mit, was der Server der Website beim letzten Besuch über die Seite angegeben hat; antwortet der Server mit `304 Not Modified`, wird die Seite nicht erneut übertragen. Viele Server erzeugen ihre Seiten bei jeder Anfrage neu und nennen keinen Änderungszeitpunkt; dann vergleicht der Crawler den Text der gelieferten Seite. In beiden Fällen wird eine unveränderte Seite nicht im Browser geöffnet, ihre Bilder, Skripte und Stylesheets werden also nicht geladen. Die Ausnahme ist eine Seite, deren Inhalt erst ihr eigenes JavaScript aufbaut: Nur ein Browser kann sie lesen, deshalb wird sie bei jedem Scan gerendert. + Der Crawler besucht die Seiten ohne Anmeldung. Eine URL macht private Inhalte nicht zugänglich. Bei jeder Anfrage stellt er sich als `TaleBot/ (+https://docs.tale.dev/platform/knowledge/crawling)` vor, sodass eine `robots.txt`-Gruppe ihn beim Namen nennen kann — `User-agent: TaleBot` —, um allein ihn zu erlauben, zu drosseln oder abzuweisen. Der Crawler hält sich an die `Disallow`-Regeln der `robots.txt` für den Agenten `*` auf jedem Weg, über den eine URL hereinkommen kann — die Sitemaps, der Linklauf und die Links, die eine gerenderte JavaScript-Seite preisgibt — und noch einmal vor jedem Abruf: Eine Seite, die eine Regel abdeckt, wird nie geholt, und eine Seite, die eine später hinzugekommene Regel abdeckt, verlässt den Index beim nächsten Scan. Ausdrücklich angegebene URLs filtern die Regeln nicht: Eine gelistete Adresse ist deine Anweisung. Liefert ein Abruf den HTTP-Header `X-Robots-Tag: noindex` oder `none` oder trägt die Seite ein HTML-Tag ``, wird der Inhalt nicht indexiert — auch bei einer URL-Liste —, und was ein früherer Scan von der Seite gespeichert hat, fällt weg. Diese Regeln sind Höflichkeit, kein Zugriffsschutz: Wenn du die Quellwebsite verwaltest, verlass dich nicht auf den Crawler als Zugangskontrolle. diff --git a/docs/en/platform/knowledge/crawling.md b/docs/en/platform/knowledge/crawling.md index 7319e2822a..ef453b421c 100644 --- a/docs/en/platform/knowledge/crawling.md +++ b/docs/en/platform/knowledge/crawling.md @@ -42,6 +42,8 @@ For a whole website, the crawler uses the homepage and published sitemaps, inclu Scans are incremental. Unchanged content is skipped, changed content is indexed again, new pages are added, and removed pages leave the index — and so do pages `robots.txt` has come to disallow. A page that redirects to another address of the same site is indexed once, under the address it lands on. The row's page counts follow the scan as pages land, after discovery and after every stored batch, so the table moves while a scan runs. A URL list follows the same refresh schedule with its fixed selection. There is no separate publish step after successful indexing. +A scan asks for each page once. It sends along what the site's server said about the page at the last visit, and a server that answers `304 Not Modified` is not asked for the page again. Many servers build their pages on request and give no modification time; for those the crawler compares the text of the page it receives. Either way, a page that did not change is not opened in the browser, so none of its images, scripts and stylesheets are loaded. A page whose content is drawn by its own JavaScript is the exception: only a browser can read it, so it is rendered on every scan. + The crawler visits as an anonymous reader. Content that depends on a private session is not made accessible by adding its URL. It identifies itself on every request as `TaleBot/ (+https://docs.tale.dev/platform/knowledge/crawling)`, so a `robots.txt` group can address it by name — `User-agent: TaleBot` — to allow, throttle or refuse it alone. The crawler applies `robots.txt` `Disallow` rules for the `*` agent on every path a URL can enter by — the sitemaps, the link walk and the links a rendered JavaScript page reveals — and again before every fetch: a page a rule covers is never fetched, and a page a rule added later covers leaves the index on the next scan. The rules do not filter an explicit URL list: a listed address is your instruction. On each content fetch, an HTTP `X-Robots-Tag: noindex` or `none`, or an HTML `` tag, prevents indexing, including for listed URLs, and drops whatever an earlier scan stored of that page. These rules are courtesy, not access control: do not rely on Tale's crawler as an access-control mechanism. diff --git a/docs/fr/platform/knowledge/crawling.md b/docs/fr/platform/knowledge/crawling.md index 7dd12e5d9d..08b4130c5d 100644 --- a/docs/fr/platform/knowledge/crawling.md +++ b/docs/fr/platform/knowledge/crawling.md @@ -42,6 +42,8 @@ Pour un site entier, le crawler utilise l’accueil et les sitemaps publiés, y Les scans sont incrémentaux : les contenus inchangés sont ignorés, les contenus modifiés réindexés, les nouvelles pages ajoutées et les pages retirées supprimées de l’index — comme les pages que le `robots.txt` en est venu à interdire. Une page qui redirige vers une autre adresse du même site n’est indexée qu’une fois, sous l’adresse où elle aboutit. Les compteurs de pages de la ligne suivent le scan à mesure que les pages arrivent, après la découverte et après chaque lot stocké, la table bouge donc pendant qu’un scan tourne. Une liste d’URL actualise sa sélection fixe au même rythme. Aucune publication séparée n’est nécessaire après l’indexation. +Un scan demande chaque page une fois. Il joint à sa requête ce que le serveur du site avait indiqué sur la page lors de la dernière visite, et un serveur qui répond `304 Not Modified` n’a pas à renvoyer la page. Beaucoup de serveurs construisent leurs pages à chaque requête et ne donnent aucune date de modification ; le crawler compare alors le texte de la page reçue. Dans les deux cas, une page inchangée n’est pas ouverte dans le navigateur, et ses images, scripts et feuilles de style ne sont donc pas chargés. L’exception est une page dont le contenu est construit par son propre JavaScript : seul un navigateur peut la lire, elle est donc rendue à chaque scan. + Le crawler visite en lecteur anonyme. Ajouter une URL ne rend pas accessible un contenu privé. À chaque requête il se présente comme `TaleBot/ (+https://docs.tale.dev/platform/knowledge/crawling)`, si bien qu’un groupe de `robots.txt` peut le désigner par son nom — `User-agent: TaleBot` — pour l’autoriser, le ralentir ou le refuser seul. Le crawler applique les règles `Disallow` de `robots.txt` destinées à l’agent `*` sur chaque chemin par lequel une URL peut entrer — les sitemaps, le parcours de liens et les liens qu’une page JavaScript révèle après rendu — et de nouveau avant chaque récupération : une page qu’une règle couvre n’est jamais récupérée, et une page qu’une règle ajoutée plus tard couvre quitte l’index au scan suivant. Les règles ne filtrent pas une liste d’URL explicite : une adresse listée est ta consigne. À chaque récupération, un en-tête HTTP `X-Robots-Tag: noindex` ou `none`, ou une balise HTML ``, empêche l’indexation, y compris pour une URL de liste, et retire ce qu’un scan antérieur avait stocké de la page. Ces règles sont une courtoisie, pas un contrôle d’accès : si tu administres le site source, ne compte pas sur le crawler comme mécanisme de contrôle d’accès. diff --git a/services/db/migrations/knowledge-db/public_web/00000000000013_website_url_change_check.sql b/services/db/migrations/knowledge-db/public_web/00000000000013_website_url_change_check.sql new file mode 100644 index 0000000000..e130f95e96 --- /dev/null +++ b/services/db/migrations/knowledge-db/public_web/00000000000013_website_url_change_check.sql @@ -0,0 +1,53 @@ +-- migrate:up +-- +-- What a scan checks a page by, before it spends a browser on it. Until +-- here the only way the crawler could tell whether a page had changed was +-- to do everything again: fetch it, render it in a browser with every image, +-- script and stylesheet the render loads, and compare content hashes +-- afterwards. A 700-page site scanned every six hours answered some 8,000 +-- requests and two gigabytes a scan for pages that had not changed in +-- months (2026-10-06). +-- +-- A scan now asks for each page once, and opens the browser only for a page +-- that did change. The row keeps what that one request is judged by: +-- +-- etag the `ETag` of the response the stored text came from +-- last_modified its `Last-Modified`; the two are sent back as +-- `If-None-Match` / `If-Modified-Since`, and a server that +-- answers 304 is not asked for the page at all +-- probe_hash the hash of the text read out of the page's plain HTML, +-- for the servers that send neither: the same text again +-- is the same page +-- +-- All three describe the response whose text the row stores AND has +-- indexed: they are written when a page settles and cleared when its +-- content is purged. They stay NULL for a page whose plain HTML does not +-- carry what a browser shows — a page built by its JavaScript — because a +-- 304 on such a shell, or the same shell text, says nothing about the +-- content; that page is rendered on every scan, as every page was. +-- +-- `etag` and `last_modified` are in the baseline and went unused when the +-- render lane arrived; they are named here too so a corpus that was +-- bootstrapped without them converges. ADD COLUMN IF NOT EXISTS is +-- metadata-only, idempotent and rolling-safe — the previous image neither +-- reads nor writes the columns. +-- +-- What the two columns still hold was written before the render lane, for +-- every page alike: a value there may belong to a JavaScript shell, and +-- sent back it would have that page answered 304 and never rendered again. +-- It is cleared once, here. Applied again (a corpus without a migration +-- ledger re-applies every file) it only costs each page one full visit. + +ALTER TABLE public_web.website_urls + ADD COLUMN IF NOT EXISTS etag TEXT, + ADD COLUMN IF NOT EXISTS last_modified TEXT, + ADD COLUMN IF NOT EXISTS probe_hash TEXT; + +UPDATE public_web.website_urls + SET etag = NULL, last_modified = NULL + WHERE etag IS NOT NULL OR last_modified IS NOT NULL; + +-- migrate:down +-- Deliberately empty, like the baseline: dropping the columns would only +-- send every scan back to rendering every page. Remove with an explicit, +-- reviewed migration instead. diff --git a/services/platform/backend/core/knowledge/crawl_action.change-check.test.ts b/services/platform/backend/core/knowledge/crawl_action.change-check.test.ts new file mode 100644 index 0000000000..32e26d3452 --- /dev/null +++ b/services/platform/backend/core/knowledge/crawl_action.change-check.test.ts @@ -0,0 +1,593 @@ +// @vitest-environment node + +import { computeContentHash } from '@tale/shared/utils/hashing'; +import type { Sql } from 'postgres'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { EMPTY_ROBOTS_POLICY } from '../../../lib/knowledge/crawl-parse'; +import { htmlToText } from '../../../lib/knowledge/html-to-text'; +import { safeFetchBinary } from '../../../lib/net/safe-fetch'; +import type { ActionCtx } from '../lib/ctx'; +import { renderUrlsInSandbox } from '../node_only/sandbox/render_fetch'; +import { + checkAfterRender, + fetchAndStorePage, + scanWebsiteImpl, +} from './crawl_action'; +import { assertCorpusWritable } from './index_health'; +import { getKnowledgePoolForOrg } from './pool'; + +vi.mock('../../../lib/net/safe-fetch', async (importOriginal) => ({ + ...(await importOriginal()), + safeFetchBinary: vi.fn(), +})); +vi.mock('../node_only/sandbox/render_fetch', async (importOriginal) => ({ + ...(await importOriginal< + typeof import('../node_only/sandbox/render_fetch') + >()), + renderUrlsInSandbox: vi.fn(), +})); +vi.mock('./pool', async (importOriginal) => ({ + ...(await importOriginal()), + getKnowledgePoolForOrg: vi.fn(), + resolveOrgUrl: vi.fn(async () => 'postgresql://corpus.example/tale'), +})); +// The indexer's gate on a corpus under repair: these tests store text the +// index already holds, so it must never be asked — and never reach for a +// real database when one is. +vi.mock('./index_health', () => ({ assertCorpusWritable: vi.fn() })); + +/** + * A scan asks for each page once and opens the browser only for a page that + * changed. The regression: the crawler's one way to learn whether a page + * had changed was to do everything again — fetch it, render it with every + * image, script and stylesheet the render loads, and compare content hashes + * afterwards. A 700-page site scanned every six hours answered some 8,000 + * requests and two gigabytes a scan for pages unchanged in months. + * + * The request now carries the validators of the last visit, and a 304 ends + * it. A server that builds its pages on request has no modification time to + * give — no `ETag`, no `Last-Modified`, a new token in every response — so + * the text read out of the plain HTML is compared too. A page whose plain + * HTML does not carry what a browser shows keeps nothing to be checked by: + * a 304 on such a shell proves nothing, and that page is rendered again. + * + * Pinned on recording doubles; what the statements do to real rows rides + * the integration check. + */ + +const DOMAIN = 'docs.example'; +const URL = 'https://docs.example/guide'; + +const body = (words: string, token: string): string => + `Guide

Guide

${words}

`; +const TEXT_V1 = + 'Tale reads the public pages of a website and answers questions from them.'; +const TEXT_V2 = + 'Tale reads the public pages of a website, and every changed page again.'; +const hashOf = (html: string): string => computeContentHash(htmlToText(html)); + +interface Statement { + text: string; + params: unknown[]; +} + +function corpus(): { sql: Sql; statements: Statement[] } { + const statements: Statement[] = []; + const unsafe = (text: string, params: unknown[] = []): Promise => { + statements.push({ text: text.replace(/\s+/g, ' ').trim(), params }); + return Promise.resolve(text.includes('RETURNING u.url') ? [{}] : []); + }; + const sql = { + unsafe, + begin: async (run: (tx: { unsafe: typeof unsafe }) => Promise) => + run({ unsafe }), + }; + // oxlint-disable-next-line typescript/no-unsafe-type-assertion -- test double + return { sql: sql as unknown as Sql, statements }; +} + +interface Answer { + status?: number; + type?: string; + body?: string | Uint8Array; + headers?: Record; + finalUrl?: string; +} + +const answers = (answer: Answer): void => { + vi.mocked(safeFetchBinary).mockResolvedValue({ + status: answer.status ?? 200, + statusText: answer.status === 304 ? 'Not Modified' : 'OK', + headers: new Headers({ + 'content-type': answer.type ?? 'text/html; charset=utf-8', + ...answer.headers, + }), + body: new Blob([answer.body ?? '']), + finalUrl: answer.finalUrl ?? URL, + }); +}; + +/** A page a past scan stored, indexed and left something to check it by. */ +const settled = (check: { + etag?: string | null; + last_modified?: string | null; + probe_hash?: string | null; +}) => ({ + url: URL, + content_hash: 'stored-text-hash', + listed: false, + etag: null, + last_modified: null, + probe_hash: null, + indexed: true, + ...check, +}); + +const sentHeaders = (): Record => { + const options = vi.mocked(safeFetchBinary).mock.calls[0]?.[1]; + return { ...options?.headers }; +}; +const stamps = (statements: Statement[]) => + statements.filter((statement) => + statement.text.includes('last_crawled_at = NOW()'), + ); +const probe = (page: Parameters[2]) => + fetchAndStorePage(corpus().sql, DOMAIN, page, EMPTY_ROBOTS_POLICY); + +beforeEach(() => { + vi.mocked(safeFetchBinary).mockReset(); + vi.mocked(renderUrlsInSandbox).mockReset(); + vi.mocked(getKnowledgePoolForOrg).mockReset(); + vi.mocked(assertCorpusWritable).mockClear(); + // Cleared: a spy on a method that is already spied on keeps its calls. + vi.spyOn(console, 'log') + .mockClear() + .mockImplementation(() => undefined); + vi.spyOn(console, 'warn').mockImplementation(() => undefined); +}); + +describe('fetchAndStorePage — asking whether a page changed', () => { + it('sends the validators of the last settled visit', async () => { + answers({ status: 304 }); + + await probe( + settled({ + etag: '"v1"', + last_modified: 'Fri, 10 Jul 2026 09:11:16 GMT', + }), + ); + + expect(sentHeaders()).toMatchObject({ + 'If-None-Match': '"v1"', + 'If-Modified-Since': 'Fri, 10 Jul 2026 09:11:16 GMT', + }); + expect(sentHeaders()['User-Agent']).toContain('TaleBot/'); + }); + + // "Unchanged" would leave a page without text, or with the chunks of its + // earlier text, exactly as it is. + it.each([ + ['holds no text', { content_hash: null }], + ['holds text the index does not have yet', { indexed: false }], + ])('asks unconditionally for a page whose row %s', async (_case, state) => { + answers({ body: body(TEXT_V1, 'a') }); + + const outcome = await probe({ + ...settled({ etag: '"v1"', probe_hash: hashOf(body(TEXT_V1, 'a')) }), + ...state, + }); + + expect(sentHeaders()).not.toHaveProperty('If-None-Match'); + expect(outcome).toMatchObject({ kind: 'render' }); + }); + + it('leaves a page as it is when the server answers 304 [KNOW-R16]', async () => { + answers({ status: 304 }); + const { sql, statements } = corpus(); + + await expect( + fetchAndStorePage( + sql, + DOMAIN, + settled({ etag: '"v1"' }), + EMPTY_ROBOTS_POLICY, + ), + ).resolves.toBe('not_modified'); + + // One write: the visit, and the end of whatever failure stood before. + expect(statements).toHaveLength(1); + expect(statements[0]?.text).toContain('last_crawled_at = NOW()'); + expect(statements[0]?.text).toContain('fail_count = 0'); + expect(statements[0]?.text).toContain('last_error = NULL'); + // A 304 brings no validators: the row keeps its own. + expect(statements[0]?.text).not.toContain('etag'); + expect(statements[0]?.params).toEqual([DOMAIN, URL]); + }); + + it('reads a 304 nobody asked for as the error it is', async () => { + answers({ status: 304 }); + const { sql, statements } = corpus(); + + await expect( + fetchAndStorePage(sql, DOMAIN, settled({}), EMPTY_ROBOTS_POLICY), + ).resolves.toBe('failed'); + + expect(statements[0]?.params).toContain('http_error'); + }); + + // A page that is unchanged is still a page: the gates every response + // passes come first. + it('retires an alias and honours X-Robots-Tag on a 304 too', async () => { + answers({ status: 304, finalUrl: 'https://docs.example/manual' }); + await expect(probe(settled({ etag: '"v1"' }))).resolves.toBe('unchanged'); + + answers({ status: 304, headers: { 'x-robots-tag': 'noindex' } }); + await expect(probe(settled({ etag: '"v1"' }))).resolves.toBe('failed'); + }); +}); + +describe('fetchAndStorePage — a server that gives no modification time', () => { + // Every response of such a server differs in its bytes: a token in a meta + // tag, a nonce in a script. The text a reader sees is what is compared. + it('leaves a page as it is when its plain HTML reads as it did [KNOW-R16]', async () => { + const stored = body(TEXT_V1, 'token-of-the-last-visit'); + const now = body(TEXT_V1, 'token-of-this-visit'); + expect(now).not.toBe(stored); + answers({ body: now, headers: { etag: '"regenerated"' } }); + const { sql, statements } = corpus(); + + await expect( + fetchAndStorePage( + sql, + DOMAIN, + settled({ probe_hash: hashOf(stored) }), + EMPTY_ROBOTS_POLICY, + ), + ).resolves.toBe('not_modified'); + + expect(statements).toHaveLength(1); + expect(statements[0]?.text).toContain('last_crawled_at = NOW()'); + // The validators of THIS response replace the row's. + expect(statements[0]?.params).toEqual([ + DOMAIN, + URL, + '"regenerated"', + null, + hashOf(stored), + ]); + }); + + it('sends a page whose text changed to the browser, with what the request found', async () => { + const now = body(TEXT_V2, 'b'); + answers({ + body: now, + headers: { 'last-modified': 'Tue, 06 Oct 2026 08:00:00 GMT' }, + }); + const { sql, statements } = corpus(); + + await expect( + fetchAndStorePage( + sql, + DOMAIN, + settled({ probe_hash: hashOf(body(TEXT_V1, 'a')) }), + EMPTY_ROBOTS_POLICY, + ), + ).resolves.toEqual({ + kind: 'render', + probe: { + text: htmlToText(now), + hash: hashOf(now), + etag: null, + lastModified: 'Tue, 06 Oct 2026 08:00:00 GMT', + }, + }); + + // The row stays unmarked until the render batch settles it. + expect(statements).toEqual([]); + }); + + // A page built by its JavaScript keeps nothing to be checked by: the same + // shell says nothing about what the page shows. + it('renders a page only a browser can read on every scan [KNOW-R16]', async () => { + const shell = body('Loading…', 'a'); + answers({ body: shell, headers: { etag: '"shell"' } }); + + await expect(probe(settled({}))).resolves.toMatchObject({ + kind: 'render', + }); + expect(sentHeaders()).not.toHaveProperty('If-None-Match'); + }); + + // The tag can change without a word of the text moving. + it('still honours a robots meta tag on a page whose text did not change', async () => { + const stored = body(TEXT_V1, 'a'); + const withdrawn = stored.replace( + '', + '<meta name="robots" content="noindex"><title>', + ); + expect(hashOf(withdrawn)).toBe(hashOf(stored)); + answers({ body: withdrawn }); + const { sql, statements } = corpus(); + + await expect( + fetchAndStorePage( + sql, + DOMAIN, + settled({ probe_hash: hashOf(stored) }), + EMPTY_ROBOTS_POLICY, + ), + ).resolves.toBe('failed'); + + expect( + statements.some((statement) => + statement.params.includes('robots_noindex'), + ), + ).toBe(true); + // What it was checked by goes with its content. + const purge = statements.find((statement) => + statement.text.includes('SET content = NULL'), + ); + expect(purge?.text).toContain( + 'etag = NULL, last_modified = NULL, probe_hash = NULL', + ); + }); + + // Read as UTF-8, every accented word of such a page comes out broken and + // would never match what the browser shows. + it('reads a page in the charset its content type names', async () => { + const html = body('Über die Größe der Änderung', 'a'); + answers({ + type: 'text/html; charset=ISO-8859-1', + body: Uint8Array.from(Buffer.from(html, 'latin1')), + }); + + await expect(probe(settled({}))).resolves.toMatchObject({ + probe: { hash: hashOf(html) }, + }); + }); +}); + +describe('fetchAndStorePage — a document', () => { + // A document's bytes are its content, so its validators always hold. + it('hands the validators of a stored document on, to be kept once it is indexed', async () => { + answers({ + type: 'text/plain', + body: 'Release notes. Enough words to be stored as the text of this page.', + headers: { + etag: '"doc-1"', + 'last-modified': 'Fri, 10 Jul 2026 09:11:16 GMT', + }, + }); + const { sql, statements } = corpus(); + + await expect( + fetchAndStorePage( + sql, + DOMAIN, + { url: URL, content_hash: null, listed: false }, + EMPTY_ROBOTS_POLICY, + ), + ).resolves.toEqual({ + kind: 'stored', + outcome: 'changed', + check: { + etag: '"doc-1"', + lastModified: 'Fri, 10 Jul 2026 09:11:16 GMT', + probeHash: null, + }, + }); + + // Stored, not yet stamped: the scan indexes it first. + expect(stamps(statements)).toEqual([]); + }); + + it('is not downloaded again when the server answers 304', async () => { + answers({ status: 304, type: 'application/pdf' }); + + await expect(probe(settled({ etag: '"doc-1"' }))).resolves.toBe( + 'not_modified', + ); + }); +}); + +describe('checkAfterRender', () => { + const plain = htmlToText(body(TEXT_V1, 'a')); + const found = { + text: plain, + hash: computeContentHash(plain), + etag: '"v2"', + lastModified: null, + }; + + it('keeps what the request found when the plain HTML carried what the browser showed', () => { + expect(checkAfterRender(found, plain)).toEqual({ + etag: '"v2"', + lastModified: null, + probeHash: found.hash, + }); + }); + + it.each([ + [ + 'the browser showed a page its plain HTML does not carry', + found, + `${TEXT_V2} ${TEXT_V2} A long article that only the page's own scripts fetched and rendered for the reader.`, + ], + [ + 'the plain HTML was too large to read', + { ...found, text: null, hash: null }, + plain, + ], + ])('keeps nothing when %s', (_case, probed, rendered) => { + expect(checkAfterRender(probed, rendered)).toEqual({ + etag: null, + lastModified: null, + probeHash: null, + }); + }); +}); + +/** + * One link of a scan, on a later link (no discovery): the frontier hands it + * one page, whose request says it changed. + */ +describe('scanWebsiteImpl — a page that changed', () => { + const SCAN = { + domain: DOMAIN, + orgSlug: 'acme', + organizationId: 'org-1', + continuation: 1, + scanStartedAt: '2026-10-06T13:00:00.000Z', + }; + + /** The frontier holds `page` once; its stored text is already what the + * browser shows, so storing it changes nothing in the index. */ + function scanCorpus(page: Record<string, unknown>): { + sql: Sql; + statements: Statement[]; + } { + const statements: Statement[] = []; + let handed = false; + const unsafe = ( + text: string, + params: unknown[] = [], + ): Promise<unknown[]> => { + const statement = text.replace(/\s+/g, ' ').trim(); + statements.push({ text: statement, params }); + if (statement.startsWith('SELECT kind')) { + return Promise.resolve([{ kind: 'site', robots_disallow: null }]); + } + if (statement.startsWith('SELECT url, content_hash, listed')) { + if (handed) return Promise.resolve([]); + handed = true; + return Promise.resolve([page]); + } + if (statement.includes('bool_or(embedding IS NULL)')) { + return Promise.resolve([ + { present: true, current: true, vectorless: false }, + ]); + } + if (statement.includes('FILTER (WHERE u.status')) { + return Promise.resolve([ + { stored: '1', attempted: '1', failed: '0', skipped: '0' }, + ]); + } + return Promise.resolve([]); + }; + const sql = { + unsafe, + begin: async (run: (tx: { unsafe: typeof unsafe }) => Promise<void>) => + run({ unsafe }), + }; + // oxlint-disable-next-line typescript/no-unsafe-type-assertion -- test double + return { sql: sql as unknown as Sql, statements }; + } + + function engineCtx(): ActionCtx { + const ctx = { + runMutation: vi.fn(async () => ({ paused: false })), + runQuery: vi.fn(), + scheduler: { runAfter: vi.fn() }, + }; + // oxlint-disable-next-line typescript/no-unsafe-type-assertion -- only what the link reaches + return ctx as unknown as ActionCtx; + } + + const shows = (html: string): void => { + vi.mocked(renderUrlsInSandbox).mockResolvedValue({ + outcomes: new Map([ + [URL, { kind: 'ok', status: 200, finalUrl: URL, html }], + ]), + halted: null, + }); + }; + const settledWrite = (statements: Statement[]) => + statements.find( + (statement) => + statement.text.includes('last_crawled_at = NOW()') && + statement.text.includes('probe_hash = $5'), + ); + + it('reads what a page is checked by out of the frontier, with whether its text is indexed', async () => { + const { sql, statements } = scanCorpus(settled({})); + vi.mocked(getKnowledgePoolForOrg).mockResolvedValue(sql); + answers({ status: 304 }); + + await scanWebsiteImpl(engineCtx(), SCAN); + + const frontier = statements.find((statement) => + statement.text.startsWith('SELECT url, content_hash, listed'), + ); + expect(frontier?.text).toContain('etag, last_modified, probe_hash'); + expect(frontier?.text).toContain( + 'c.content_hash IS DISTINCT FROM website_urls.content_hash ) AS indexed', + ); + }); + + it('opens no browser for a page that did not change [KNOW-R16]', async () => { + const { sql } = scanCorpus(settled({ etag: '"v1"' })); + vi.mocked(getKnowledgePoolForOrg).mockResolvedValue(sql); + answers({ status: 304 }); + + await scanWebsiteImpl(engineCtx(), SCAN); + + expect(renderUrlsInSandbox).not.toHaveBeenCalled(); + expect(vi.mocked(console.log).mock.calls.flat().join('\n')).toContain( + '1 page(s) requested, 1 unchanged, 0 rendered', + ); + }); + + it('renders a changed page and keeps what to check it by next time', async () => { + const now = body(TEXT_V2, 'b'); + const { sql, statements } = scanCorpus({ + ...settled({ probe_hash: hashOf(body(TEXT_V1, 'a')) }), + // The row already stores what the browser is about to show (an + // earlier scan stored it and stopped before stamping the visit), so + // the store finds nothing to index. + content_hash: hashOf(now), + }); + vi.mocked(getKnowledgePoolForOrg).mockResolvedValue(sql); + answers({ body: now, headers: { etag: '"v2"' } }); + // The browser shows what the plain HTML already said. + shows(now); + + await scanWebsiteImpl(engineCtx(), SCAN); + + expect(renderUrlsInSandbox).toHaveBeenCalledTimes(1); + expect(assertCorpusWritable).not.toHaveBeenCalled(); + expect(settledWrite(statements)?.params).toEqual([ + DOMAIN, + URL, + '"v2"', + null, + hashOf(now), + ]); + }); + + it('keeps nothing to check a page by when only the browser could read it', async () => { + const shell = body('Loading…', 'a'); + const shown = body( + `${TEXT_V1} ${TEXT_V2} A long article that only the page's own scripts fetched and rendered for the reader.`, + 'a', + ); + const { sql, statements } = scanCorpus({ + ...settled({}), + content_hash: hashOf(shown), + }); + vi.mocked(getKnowledgePoolForOrg).mockResolvedValue(sql); + answers({ body: shell, headers: { etag: '"shell"' } }); + shows(shown); + + await scanWebsiteImpl(engineCtx(), SCAN); + + expect(assertCorpusWritable).not.toHaveBeenCalled(); + expect(settledWrite(statements)?.params).toEqual([ + DOMAIN, + URL, + null, + null, + null, + ]); + }); +}); diff --git a/services/platform/backend/core/knowledge/crawl_action.redirects.test.ts b/services/platform/backend/core/knowledge/crawl_action.redirects.test.ts index 2f601f2068..bc374df4d9 100644 --- a/services/platform/backend/core/knowledge/crawl_action.redirects.test.ts +++ b/services/platform/backend/core/knowledge/crawl_action.redirects.test.ts @@ -176,7 +176,7 @@ describe('fetchAndStorePage — a redirect inside the site', () => { await expect( fetchAndStorePage(sql, DOMAIN, page(from, listed), EMPTY_ROBOTS_POLICY), - ).resolves.toBe('render'); + ).resolves.toMatchObject({ kind: 'render' }); expect(retired(statements)).toEqual([]); expect(admitted(statements)).toEqual([]); @@ -221,7 +221,7 @@ describe('fetchAndStorePage — a page the crawler does not request', () => { await expect( fetchAndStorePage(sql, DOMAIN, page(url, true), PRIVATE_RULE), - ).resolves.toBe('render'); + ).resolves.toMatchObject({ kind: 'render' }); expect(safeFetchBinary).toHaveBeenCalledTimes(1); expect(retired(statements)).toEqual([]); @@ -258,7 +258,7 @@ describe('fetchAndStorePage — a page the crawler does not request', () => { await expect( fetchAndStorePage(sql, 'intranet.corp', page(url), EMPTY_ROBOTS_POLICY), - ).resolves.toBe('render'); + ).resolves.toMatchObject({ kind: 'render' }); expect(safeFetchBinary).toHaveBeenCalledWith( url, diff --git a/services/platform/backend/core/knowledge/crawl_action.ts b/services/platform/backend/core/knowledge/crawl_action.ts index 70a1feb35d..bc579f7db8 100644 --- a/services/platform/backend/core/knowledge/crawl_action.ts +++ b/services/platform/backend/core/knowledge/crawl_action.ts @@ -12,8 +12,13 @@ * in-process; HTML is rendered in batches by a sandboxed browser * (`renderUrlsInSandbox`) so JS-rendered sites yield their real content. * The in-process probe stays the authority on page lifecycle — status - * codes, deletes, size caps, SSRF guards — and content-hash comparison is - * the only change detection. + * codes, deletes, size caps, SSRF guards — and it is also where a scan + * finds out, with that one request, whether a page changed at all: a + * server that answers 304 to the validators of the last visit, or plain + * HTML that reads the same as it did then, is a page left as it is. Only a + * page that did change — or one whose plain HTML does not carry what a + * browser shows — is rendered (`PageCheck`). Content-hash comparison then + * decides what a stored page means for the index. * * A scan is a CONTINUATION CHAIN, not one long action: a Convex node action * is hard-killed near ten minutes without running its catch, so each link @@ -51,6 +56,7 @@ import { paragraphsForHashing, parseRobots, parseSitemapLocs, + plainTextCarriesRendered, publicPageError, renderLaneHaltMessage, ROBOTS_TXT_MAX_BYTES, @@ -97,6 +103,7 @@ import { orgSlugFromIdOrNull } from '../lib/helpers/org_slug'; import { extractText } from '../lib/knowledge/extraction/router'; import { sniffDocumentExtension } from '../lib/knowledge/extraction/sniff'; import { + RENDER_MAX_HTML_BYTES, RenderCapacityError, renderCapacityPollMs, renderUrlsInSandbox, @@ -371,8 +378,12 @@ export async function scanWebsiteImpl( const hardWall = actionStartedAt + ACTION_HARD_WALL_MS; const deadline = Math.min(linkStartedAt + SCAN_BUDGET_MS, hardWall); const indexer = new PageIndexer(ctx, sql, identity); - const renderQueue: DuePage[] = []; + const renderQueue: { page: DuePage; probe: PageProbe }[] = []; let renderBatchCounter = 0; + // What this link's requests found, for the scan's log: how many pages + // it asked for, how many of them had not changed, how many it had to + // open a browser for. + const seen = { requested: 0, unchanged: 0, rendered: 0 }; // Set once the organization's render sessions stayed spent for the // rest of this link's window: the link stops fetching (every HTML // page it probed would only queue behind the same wait) and hands the @@ -427,7 +438,7 @@ export async function scanWebsiteImpl( } const batch = renderQueue.splice(0); renderBatchCounter += 1; - const results = await renderBatch(batch); + const results = await renderBatch(batch.map((entry) => entry.page)); if (results === null) { // The batch's rows are unmarked and stay due; the next link // renders them once a session is free. @@ -441,7 +452,7 @@ export async function scanWebsiteImpl( // A navigation the browser lost twice is the row's to show, without // a strike; the lane's own fault (`halted`) fails the scan below, // once what rendered is stored. - for (const page of batch) { + for (const { page, probe } of batch) { const outcome = results.outcomes.get(page.url) ?? { kind: 'not_attempted' as const, }; @@ -472,15 +483,25 @@ export async function scanWebsiteImpl( }); continue; } + const renderedText = htmlToText(outcome.html); const stored = await storePageText( sql, args.domain, page, htmlTitle(outcome.html), - htmlToText(outcome.html), + renderedText, ); await indexer.settle(page.url, stored); - await markPageCrawled(sql, args.domain, page.url); + // Whether the next scan can judge this page by one request is + // decided here, from the two texts of this visit: what the plain + // HTML said and what the browser showed. + await markPageCrawled( + sql, + args.domain, + page.url, + checkAfterRender(probe, renderedText), + ); + seen.rendered += 1; if (kind === 'site') { await admitRenderedLinks( sql, @@ -520,11 +541,22 @@ export async function scanWebsiteImpl( policy, scanStartedAt, ); - if (outcome === 'render') renderQueue.push(page); - else if (outcome !== 'failed') { - await indexer.settle(page.url, outcome); - await markPageCrawled(sql, args.domain, page.url); + seen.requested += 1; + if (outcome === 'not_modified') { + seen.unchanged += 1; + } else if (typeof outcome === 'object') { + if (outcome.kind === 'render') { + renderQueue.push({ page, probe: outcome.probe }); + } else { + // A document without validators is downloaded to be judged: + // the same text is the same document. + if (outcome.outcome === 'unchanged') seen.unchanged += 1; + await indexer.settle(page.url, outcome.outcome); + await markPageCrawled(sql, args.domain, page.url, outcome.check); + } } + // Any other outcome wrote its own row: a retired page, a failed + // attempt. await sleep(fetchDelayMs(policy)); if (renderQueue.length >= RENDER_BATCH_SIZE) await flushRenderBatch(); } @@ -536,6 +568,11 @@ export async function scanWebsiteImpl( // of at the link's end (2026-09-14 evaluation, h5). await fanOutRowSync(ctx, sql, args.domain); } + if (seen.requested > 0) { + console.log( + `[crawl] ${args.domain}: ${seen.requested} page(s) requested, ${seen.unchanged} unchanged, ${seen.rendered} rendered`, + ); + } // What was stored without vectors — by this scan before an admin saved // a model, or by an earlier one — is embedded from its stored text // once a model can, in the time the link has left; what is left keeps @@ -1100,6 +1137,171 @@ interface DuePage { /** An operator-listed URL: the robots rules do not govern it, and a 404 * keeps its row. */ readonly listed: boolean; + /** What the page's last settled visit left to check it by + * ({@link PageCheck}); absent or null when it left nothing. */ + readonly etag?: string | null; + readonly last_modified?: string | null; + readonly probe_hash?: string | null; + /** False while a chunk of the page is cut from other text than the row + * stores — a scan stored the new text and stopped before indexing it. */ + readonly indexed?: boolean; +} + +/** + * What one request can check a page by, kept from the response its stored + * text came from. + * + * `etag` and `lastModified` are that response's validators: sent back as + * `If-None-Match` / `If-Modified-Since`, they let a server that tracks its + * pages answer 304 without sending the page. Many servers that build their + * pages on request send neither — the page has no modification time to + * give — so `probeHash` is the hash of the text read out of the plain, + * unrendered HTML: the same text again is the same page, whatever tokens + * and nonces the markup around it carries. + * + * The three describe one response, and only a response whose text is + * stored AND indexed: they are written when a page settles + * ({@link markPageCrawled}) and cleared when its content is purged. A page + * whose plain HTML does not carry what a browser shows keeps none of them + * ({@link checkAfterRender}): a 304 on a JavaScript shell, or the same + * shell text, proves nothing about the content, which is why the render + * lane dropped conditional requests for every page when it arrived. That + * page is rendered on every scan; every other page is rendered only when + * its one request says it changed. + */ +interface PageCheck { + readonly etag: string | null; + readonly lastModified: string | null; + readonly probeHash: string | null; +} + +const NO_PAGE_CHECK: PageCheck = { + etag: null, + lastModified: null, + probeHash: null, +}; + +/** What a row offers this scan to check its page by: nothing unless it + * holds the page's text and that text is indexed — "unchanged" would + * otherwise leave a page without text, or with chunks of its earlier text, + * exactly as it is. */ +function storedCheck(page: DuePage): PageCheck { + if (page.content_hash === null || page.indexed === false) { + return NO_PAGE_CHECK; + } + return { + etag: page.etag ?? null, + lastModified: page.last_modified ?? null, + probeHash: page.probe_hash ?? null, + }; +} + +/** A validator longer than this is not kept: it travels back in a request + * header on every later scan. */ +const VALIDATOR_MAX_CHARS = 512; + +function validatorOf(value: string | null): string | null { + const trimmed = value?.trim() ?? ''; + return trimmed === '' || trimmed.length > VALIDATOR_MAX_CHARS + ? null + : trimmed; +} + +/** The validators a response carries, to check its page by next time. */ +function responseValidators( + headers: Headers, +): Pick<PageCheck, 'etag' | 'lastModified'> { + return { + etag: validatorOf(headers.get('etag')), + lastModified: validatorOf(headers.get('last-modified')), + }; +} + +/** The conditional headers that ask "has it changed since?" — none when + * the row holds no validator. */ +function conditionalHeaders(check: PageCheck): Record<string, string> { + const headers: Record<string, string> = {}; + if (check.etag !== null) headers['If-None-Match'] = check.etag; + if (check.lastModified !== null) { + headers['If-Modified-Since'] = check.lastModified; + } + return headers; +} + +/** The plain HTML of a probed page, and the text and hash a later scan + * compares against. */ +interface PlainHtml { + readonly html: string; + readonly text: string; + readonly hash: string; +} + +/** What the probe of an HTML page hands the render batch: the text of its + * plain HTML (null when the page is too large to read twice) and the + * validators of the response, for {@link checkAfterRender} to judge once + * the browser has shown the page. */ +interface PageProbe { + readonly text: string | null; + readonly hash: string | null; + readonly etag: string | null; + readonly lastModified: string | null; +} + +/** Decode a page under the charset its `Content-Type` names — a page in + * another encoding read as UTF-8 comes out with every accented word + * broken, and would never match what the browser shows. UTF-8 without one. */ +function decodeHtml(bytes: Uint8Array, contentType: string): string { + const label = /charset\s*=\s*"?([^";\s]+)/i.exec(contentType)?.[1]; + if (label !== undefined && label.toLowerCase() !== 'utf-8') { + try { + return new TextDecoder(label).decode(bytes); + } catch (error) { + console.warn( + `[crawl] unknown charset "${label}", reading the page as UTF-8:`, + error instanceof Error ? error.message : error, + ); + } + } + return new TextDecoder().decode(bytes); +} + +/** Read the text out of a probed page's plain HTML — null for a page over + * the bound a rendered page has, which is rendered without the comparison. */ +async function readPlainHtml( + body: Blob, + contentType: string, +): Promise<PlainHtml | null> { + if (body.size > RENDER_MAX_HTML_BYTES) return null; + const html = decodeHtml( + new Uint8Array(await body.arrayBuffer()), + contentType, + ); + const text = htmlToText(html); + return { html, text, hash: computeContentHash(text) }; +} + +/** + * What a rendered page leaves to check it by next time: its probe's + * validators and plain-text hash when the plain HTML carried what the + * browser showed ({@link plainTextCarriesRendered}), nothing when it did + * not — that page is rendered again on every scan. Exported for tests only. + */ +export function checkAfterRender( + probe: PageProbe, + renderedText: string, +): PageCheck { + if ( + probe.text === null || + probe.hash === null || + !plainTextCarriesRendered(probe.text, renderedText) + ) { + return NO_PAGE_CHECK; + } + return { + etag: probe.etag, + lastModified: probe.lastModified, + probeHash: probe.hash, + }; } const DUE_PAGE_PREDICATE = ` @@ -1118,7 +1320,13 @@ async function nextDuePages( limit: number, ): Promise<DuePage[]> { return await sql.unsafe<DuePage[]>( - `SELECT url, content_hash, listed + `SELECT url, content_hash, listed, etag, last_modified, probe_hash, + NOT EXISTS ( + SELECT 1 FROM ${PUBLIC_WEB_SCHEMA}.chunks c + WHERE c.domain = website_urls.domain + AND c.url = website_urls.url + AND c.content_hash IS DISTINCT FROM website_urls.content_hash + ) AS indexed FROM ${PUBLIC_WEB_SCHEMA}.website_urls WHERE ${DUE_PAGE_PREDICATE} ORDER BY last_crawled_at ASC NULLS FIRST, url ASC @@ -1140,7 +1348,30 @@ async function countDuePages( return Number(rows[0]?.n ?? 0); } -type FetchOutcome = StoreOutcome | 'failed' | 'render'; +/** + * What a probe made of a page: + * + * - `not_modified` — requested and found as it was: the server answered + * 304, or the plain HTML reads as it did. The row is stamped; nothing is + * rendered, stored or indexed. + * - `unchanged` — the row was retired (gone, disallowed, an alias) and + * nothing is left to do for it. + * - `failed` — the attempt is on the row, with its reason. + * - `stored` — a document or plain text was stored in-process; the scan + * indexes it and then stamps it with what to check it by next time. + * - `render` — an HTML page that changed, or that only a browser can + * read: it joins the render batch with what its probe found. + */ +export type FetchOutcome = + | 'not_modified' + | 'unchanged' + | 'failed' + | { + readonly kind: 'stored'; + readonly outcome: StoreOutcome; + readonly check: PageCheck; + } + | { readonly kind: 'render'; readonly probe: PageProbe }; /** Whether a redirect landed on another host or path of the site — not * merely on the same address with another query string. */ @@ -1153,14 +1384,22 @@ function isAnotherAddress(from: string, to: string): boolean { } /** - * Probe one page and dispatch on its content type: binaries and plain text - * are extracted and stored in-process; HTML reports `render` (body - * discarded, row left unmarked) so the caller batches it through the - * sandboxed browser. The probe is the sole authority on page LIFECYCLE — - * status codes, deletes, size caps, and the SSRF guard for every byte - * download. Change detection is the stored content hash alone: a 304 on an - * SPA shell proves nothing about rendered content, so no conditional - * validators are sent. Exported for tests only. + * Probe one page — the one request a scan makes for it — and say what is + * left to do ({@link FetchOutcome}). + * + * The request carries the validators of the page's last settled visit, when + * the row holds any ({@link storedCheck}); a 304 answer is a page left as + * it is. Otherwise the response is dispatched on its content type: + * binaries and plain text are extracted and stored in-process; an HTML + * page whose plain text reads as it did at that visit is left as it is + * too, and any other HTML page reports `render`, row unmarked, so the + * caller batches it through the sandboxed browser. + * + * The probe is the sole authority on page LIFECYCLE — status codes, + * deletes, size caps, and the SSRF guard for every byte download — and a + * page found unchanged passes the same gates first: a redirect that makes + * it an alias, an `X-Robots-Tag` or a robots meta tag that withdraws it. + * Exported for tests only. */ export async function fetchAndStorePage( sql: Sql, @@ -1180,6 +1419,8 @@ export async function fetchAndStorePage( return 'unchanged'; } + const check = storedCheck(page); + const conditional = conditionalHeaders(check); let response; try { assertCrawlableUrl(page.url); @@ -1189,7 +1430,7 @@ export async function fetchAndStorePage( allowedHosts: [...hosts], allowPrivateAddresses: privateCrawlHostsAllowed(), httpsOnly: true, - headers: crawlerRequestHeaders(), + headers: { ...crawlerRequestHeaders(), ...conditional }, }); } catch (error) { const cause = error instanceof Error ? error.message : String(error); @@ -1239,7 +1480,11 @@ export async function fetchAndStorePage( await retirePage(sql, domain, page.url); return 'unchanged'; } - if (response.status < 200 || response.status >= 300) { + // 304 answers the validators this request carried: the page is what the + // row stores. Without them it is an error like any other status. + const notModified = + response.status === 304 && Object.keys(conditional).length > 0; + if (!notModified && (response.status < 200 || response.status >= 300)) { await recordPageFailure(sql, domain, page.url, { kind: 'http_error', message: `The page answered HTTP ${response.status}${ @@ -1287,6 +1532,12 @@ export async function fetchAndStorePage( }); return 'failed'; } + if (notModified) { + // The server vouches for the page as the row stores it: no body came, + // and none is asked for. The validators the row holds stay. + await markPageUnchanged(sql, domain, page.url); + return 'not_modified'; + } const contentType = response.headers.get('content-type') ?? ''; let dispatch = classifyContentType( contentType, @@ -1307,10 +1558,48 @@ export async function fetchAndStorePage( return 'failed'; } if (dispatch.kind === 'html') { - // Content comes from the rendered DOM, not this probe body — the page - // joins the render batch and its row stays unmarked until the batch - // settles it. - return 'render'; + const plain = await readPlainHtml(response.body, contentType); + const validators = responseValidators(response.headers); + if ( + plain !== null && + check.probeHash !== null && + plain.hash === check.probeHash + ) { + // The plain HTML reads as it did when the page was last stored, and + // at that visit it carried what the browser showed: the page has not + // changed, and no browser is opened to learn that. Its links were + // admitted when it was last rendered, and text that reads the same + // links to the same pages. Its own wish can have changed without a + // word of its text moving, so the robots tag is still read — off + // this body, the only one there is. + const noindex = robotsMetaNoindexDirective(plain.html); + if (noindex !== null) { + await purgePageContent(sql, domain, page.url); + await recordPageFailure(sql, domain, page.url, { + kind: 'robots_noindex', + message: `The origin asked not to index this page (<meta name="robots" content="${noindex}">)`, + }); + return 'failed'; + } + await markPageUnchanged(sql, domain, page.url, { + ...validators, + probeHash: plain.hash, + }); + return 'not_modified'; + } + // The page changed, was never stored, or is one only a browser can + // read: content comes from the rendered DOM, so it joins the render + // batch and its row stays unmarked until the batch settles it. What + // this request found travels with it, to be judged against what the + // browser shows. + return { + kind: 'render', + probe: { + text: plain?.text ?? null, + hash: plain?.hash ?? null, + ...validators, + }, + }; } const bytes = new Uint8Array(await response.body.arrayBuffer()); if (dispatch.kind === 'sniff') { @@ -1366,7 +1655,13 @@ export async function fetchAndStorePage( } title = name; } - return await storePageText(sql, domain, page, title, text); + // A document's bytes are its content, so its validators always hold: a + // 304 next time spares the download and the extraction. + return { + kind: 'stored', + outcome: await storePageText(sql, domain, page, title, text), + check: { ...responseValidators(response.headers), probeHash: null }, + }; } /** @@ -1448,16 +1743,54 @@ export async function storePageText( return 'changed'; } -/** Stamp a page as visited by this scan, once it is stored and indexed. */ +/** Stamp a page as visited by this scan, once it is stored and indexed, and + * record what the next scan checks it by ({@link PageCheck}) — in the same + * write, so the check never describes text the index does not hold yet. */ async function markPageCrawled( sql: Sql, domain: string, url: string, + check: PageCheck, ): Promise<void> { await sql.unsafe( - `UPDATE ${PUBLIC_WEB_SCHEMA}.website_urls SET last_crawled_at = NOW() + `UPDATE ${PUBLIC_WEB_SCHEMA}.website_urls + SET last_crawled_at = NOW(), + etag = $3, last_modified = $4, probe_hash = $5 WHERE domain = $1 AND url = $2`, - [domain, url], + [domain, url, check.etag, check.lastModified, check.probeHash], + ); +} + +/** + * Stamp a page this scan requested and found as the row stores it. The + * request reached the page, so a failure an earlier scan recorded is over: + * its count and its reason are cleared, as a store clears them. `check` + * replaces what the row is checked by — the validators of a response whose + * text read the same; a 304 brings none and the row keeps its own. + */ +async function markPageUnchanged( + sql: Sql, + domain: string, + url: string, + check?: PageCheck, +): Promise<void> { + const cleared = `last_crawled_at = NOW(), fail_count = 0, + last_error = NULL, last_error_kind = NULL, last_error_at = NULL`; + if (check === undefined) { + await sql.unsafe( + `UPDATE ${PUBLIC_WEB_SCHEMA}.website_urls + SET ${cleared} + WHERE domain = $1 AND url = $2`, + [domain, url], + ); + return; + } + await sql.unsafe( + `UPDATE ${PUBLIC_WEB_SCHEMA}.website_urls + SET ${cleared}, + etag = $3, last_modified = $4, probe_hash = $5 + WHERE domain = $1 AND url = $2`, + [domain, url, check.etag, check.lastModified, check.probeHash], ); } @@ -1484,6 +1817,7 @@ async function purgePageContentIn( await tx.unsafe( `UPDATE ${PUBLIC_WEB_SCHEMA}.website_urls SET content = NULL, content_hash = NULL, word_count = 0, + etag = NULL, last_modified = NULL, probe_hash = NULL, status = CASE WHEN status = 'deleted' THEN status ELSE 'discovered' END WHERE domain = $1 AND url = $2`, [domain, url], diff --git a/services/platform/backend/core/node_only/sandbox/render_fetch.ts b/services/platform/backend/core/node_only/sandbox/render_fetch.ts index 84b69f4db6..83d4d2ab59 100644 --- a/services/platform/backend/core/node_only/sandbox/render_fetch.ts +++ b/services/platform/backend/core/node_only/sandbox/render_fetch.ts @@ -73,7 +73,7 @@ const WORKER_EXIT_MARGIN_MS = 20_000; * back fails the batch deterministically (the crawl would retry the same * batch forever), so the worker bounds the serialized file before it admits * each page. */ -const RENDER_MAX_HTML_BYTES = 6 * 1024 * 1024; +export const RENDER_MAX_HTML_BYTES = 6 * 1024 * 1024; const RENDER_MAX_TOTAL_BYTES = 15 * 1024 * 1024; export type RenderPageOutcome = diff --git a/services/platform/backend/domains/knowledge/spec.md b/services/platform/backend/domains/knowledge/spec.md index 0a42a37bda..6fd373e042 100644 --- a/services/platform/backend/domains/knowledge/spec.md +++ b/services/platform/backend/domains/knowledge/spec.md @@ -269,6 +269,23 @@ networks for the crawler. networks, and the operator then turns that off → the next scan requests none of its pages and shows each as failed, with a private address as the reason. +### KNOW-R16 · The crawler opens a page in the browser again only when it changed + +A scan asks for each page once. The page is left as it is when the site's server answers +that it has not changed, or, for a server that gives no modification time, when the text of +the page reads as it did at the last visit. Only a page that changed is opened in the +browser and indexed again. A page whose content only its own JavaScript draws is the +exception: it is opened on every scan. **Scan now** asks the same way as a scheduled scan. + +- **Example**: `docs.example` answers `304 Not Modified` for `/pricing` → the page is not + downloaded, and no browser is opened for it. +- **Example**: `/blog` is built on every request, with a new token in each response around + the same text → the scan reads it once and leaves it as it is. +- **Example**: The site adds an article to `/blog` → the next scan finds other text, opens + the page in the browser and indexes it again. +- **Example**: Noah selects **Scan now** → every page is asked once, and only the changed + ones are opened in the browser. + ## Not yet - **The knowledge settings**: who can change the embedding model and the knowledge database, @@ -296,7 +313,7 @@ networks for the crawler. `core/knowledge/crawl_limits.ts`, `lib/knowledge/crawl-parse.ts`). Of the limits, only the size limit for one document has a test, and that test holds the setting, not the refusal. The contract debt ledger in [`.agents/repo.md`](../../../../../.agents/repo.md) records that - the crawler's clocks and limits are not on the API, and that a page is fetched several times + the crawler's clocks and limits are not on the API, and that a changed page is fetched twice a scan. - **`KNOW-R15` covers a page's own address.** A public name that leads to a private address, a redirect into a private network, and the cloud provider addresses that the operator's diff --git a/services/platform/backend/domains/websites/spec.md b/services/platform/backend/domains/websites/spec.md index d7744e756c..ab1965c97f 100644 --- a/services/platform/backend/domains/websites/spec.md +++ b/services/platform/backend/domains/websites/spec.md @@ -103,6 +103,6 @@ can call it. Without either, the websites page says that chat cannot search the - **Websites over the REST API**: the limits of its lists and searches, and the shape of its answers (`rest/v1-websites.ts`). Its tests hold `WEB-R2` to `WEB-R4`. - **The length of the waiting time in `WEB-R6` and the number of pickups in `WEB-R7`.** -- **The crawler's clocks and limits are not on the API**, and a page is fetched several times +- **The crawler's clocks and limits are not on the API**, and a changed page is fetched twice a scan; the contract debt ledger in [`.agents/repo.md`](../../../../../.agents/repo.md) records both. diff --git a/services/platform/backend/integration-check.ts b/services/platform/backend/integration-check.ts index 47077c91dd..f79e1d4885 100644 --- a/services/platform/backend/integration-check.ts +++ b/services/platform/backend/integration-check.ts @@ -34,12 +34,14 @@ import path from 'node:path'; import { serve } from '@hono/node-server'; import { transactSerializable } from '@tale/shared/db/serializable'; +import { computeContentHash } from '@tale/shared/utils/hashing'; import type { PgBoss } from 'pg-boss'; import type { Sql, TransactionSql } from 'postgres'; import * as XLSX from 'xlsx'; import { z } from 'zod'; import { robotsPolicyFromStored } from '../lib/knowledge/crawl-parse.ts'; +import { htmlToText } from '../lib/knowledge/html-to-text.ts'; import { setSafeFetchResolverForTests } from '../lib/net/safe-fetch.ts'; import { objectStorageConnectionFileSchema } from '../lib/shared/schemas/object_storage.ts'; import { createApp } from './app.ts'; @@ -33831,6 +33833,37 @@ async function checkWebsitesCrawl( // page, no robots.txt and no sitemap, so the scan that restores the // registration finishes on the homepage. const HEAL_DOMAIN = 'itest-heal.example'; + // A site for the change-check lanes (2026-10-06). Its fake server answers + // a conditional request the way a real one does — 304 when the validator + // it is asked about still stands — and counts what it sent for each page: + // the page, or a 304 in its place. `/page.html` is the page of a server + // that gives no modification time: no validator, and a new token in every + // response, around text that does not change. + const CHECK_DOMAIN = 'itest-check.example'; + const checkPage = (word: string, version: string): string => + `Change-check fixture ${word} page ${version}. Enough words about the ${word} subsystem to survive the chunking thresholds of the pipeline.`; + const checkSite = new Map< + string, + { body: string; etag?: string; lastModified?: string } + >([ + ['/', { body: checkPage('home', 'v1') }], + ['/tagged.txt', { body: checkPage('tagged', 'v1'), etag: '"t1"' }], + [ + '/dated.txt', + { + body: checkPage('dated', 'v1'), + lastModified: 'Fri, 10 Jul 2026 09:11:16 GMT', + }, + ], + ['/plain.txt', { body: checkPage('plain', 'v1') }], + ]); + const checkSitemapRoutes = [...checkSite.keys()]; + const checkSent = new Map<string, { bodies: number; notModified: number }>(); + let checkToken = 0; + const checkHtml = (): string => { + checkToken += 1; + return `<html><head><meta name="csrf-token" content="token-${checkToken}"><title>Check fixture

${checkPage('html', 'v1')}

`; + }; const FAKE_HOSTS = new Set([ DOMAIN, `www.${DOMAIN}`, @@ -33901,6 +33934,66 @@ async function checkWebsitesCrawl( }, }); } + if ( + url.hostname === CHECK_DOMAIN || + url.hostname === `www.${CHECK_DOMAIN}` + ) { + const answer = (body: string, type: string): Response => + new Response(body, { + headers: { + 'content-type': type, + 'content-length': String(body.length), + }, + }); + if (url.pathname === '/robots.txt') { + return answer( + `User-agent: *\nSitemap: https://${CHECK_DOMAIN}/sitemap.xml\n`, + 'text/plain', + ); + } + if (url.pathname === '/sitemap.xml') { + return answer( + `${checkSitemapRoutes + .map( + (route) => + `https://${CHECK_DOMAIN}${route}`, + ) + .join('')}`, + 'application/xml', + ); + } + const sent = checkSent.get(url.pathname) ?? { bodies: 0, notModified: 0 }; + checkSent.set(url.pathname, sent); + if (url.pathname === '/page.html') { + sent.bodies += 1; + return answer(checkHtml(), 'text/html; charset=utf-8'); + } + const page = checkSite.get(url.pathname); + if (!page) return new Response('gone', { status: 404 }); + const validators = { + ...(page.etag === undefined ? {} : { etag: page.etag }), + ...(page.lastModified === undefined + ? {} + : { 'last-modified': page.lastModified }), + }; + const asked = new Headers(init?.headers); + if ( + (page.etag !== undefined && asked.get('if-none-match') === page.etag) || + (page.lastModified !== undefined && + asked.get('if-modified-since') === page.lastModified) + ) { + sent.notModified += 1; + return new Response(null, { status: 304, headers: validators }); + } + sent.bodies += 1; + return new Response(page.body, { + headers: { + 'content-type': 'text/plain', + 'content-length': String(page.body.length), + ...validators, + }, + }); + } if (url.hostname === HEAL_DOMAIN || url.hostname === `www.${HEAL_DOMAIN}`) { if (url.pathname !== '/') return new Response('gone', { status: 404 }); const body = @@ -35189,6 +35282,182 @@ async function checkWebsitesCrawl( await v1(`/websites/${robotsId}`, { method: 'DELETE' }); } + // 4h. A scan asks for each page once, and a page that did not change is + // neither downloaded nor rendered again (2026-10-06). The crawler's + // one way to learn whether a page had changed was to do everything + // again: fetch it, render it with every asset a render loads, and + // compare content hashes afterwards — some 8,000 requests a scan on + // a 700-page site whose pages had not changed in months. The request + // now carries the validators of the last visit, a 304 ends it, and + // for a server that gives no modification time the text of the + // plain HTML is compared. "Scan now" asks the same way. + const checkCreated = z.looseObject({ id: z.string() }).safeParse( + await ( + await v1('/websites', { + body: { domain: CHECK_DOMAIN, scanInterval: '6h' }, + }) + ).json(), + ); + const checkId = checkCreated.success ? checkCreated.data.id : ''; + await drainCrawlJobs(); + const checkUrl = (route: string) => `https://${CHECK_DOMAIN}${route}`; + const checkRoutes = [ + '/tagged.txt', + '/dated.txt', + '/plain.txt', + '/page.html', + ]; + const checkSnapshot = () => + new Map([...checkSent].map(([route, sent]) => [route, { ...sent }])); + /** What the site sent for each page since `before`, as + * `route:bodies/304s`. */ + const sentSince = (before: ReturnType): string => + checkRoutes + .map((route) => { + const now = checkSent.get(route) ?? { bodies: 0, notModified: 0 }; + const then = before.get(route) ?? { bodies: 0, notModified: 0 }; + return `${route}:${now.bodies - then.bodies}/${now.notModified - then.notModified}`; + }) + .join(' '); + /** Every page reads as last visited two days ago, so a fresh stamp is + * this scan's. */ + const checkRescan = async (): Promise => { + await pool` + UPDATE public_web.website_urls + SET last_crawled_at = now() - interval '2 days' + WHERE domain = ${CHECK_DOMAIN} + `; + const before = checkSnapshot(); + await websites.runWebsitesScan(sql, { + domain: CHECK_DOMAIN, + orgSlug, + organizationId: orgId, + }); + await drainCrawlJobs(); + return sentSince(before); + }; + const checkRow = async (route: string) => + ( + await pool< + { + status: string; + etag: string | null; + lastModified: string | null; + probeHash: string | null; + visited: boolean; + chunks: string; + text: string | null; + }[] + >` + SELECT u.status, u.etag, u.last_modified AS "lastModified", + u.probe_hash AS "probeHash", + u.last_crawled_at > now() - interval '1 hour' AS visited, + (SELECT count(*)::text FROM public_web.chunks c + WHERE c.domain = u.domain AND c.url = u.url) AS chunks, + (SELECT string_agg(c.chunk_content, ' ') FROM public_web.chunks c + WHERE c.domain = u.domain AND c.url = u.url) AS text + FROM public_web.website_urls u + WHERE u.domain = ${CHECK_DOMAIN} AND u.url = ${checkUrl(route)} + ` + )[0]; + + // The first scan stored every page and kept what each response gave to + // check it by. The rescan: the two pages the server vouches for are + // not sent again; the one without a validator is. + const taggedFirst = await checkRow('/tagged.txt'); + const datedFirst = await checkRow('/dated.txt'); + const checkSecond = await checkRescan(); + const taggedSecond = await checkRow('/tagged.txt'); + const datedSecond = await checkRow('/dated.txt'); + record( + 'websites change check: a page the server vouches for is not downloaded again, and is stamped as visited', + checkCreated.success && + taggedFirst?.etag === '"t1"' && + datedFirst?.lastModified === 'Fri, 10 Jul 2026 09:11:16 GMT' && + checkSecond === + '/tagged.txt:0/1 /dated.txt:0/1 /plain.txt:1/0 /page.html:0/0' && + (taggedSecond?.visited ?? false) && + (datedSecond?.visited ?? false) && + taggedSecond?.status === 'active' && + taggedSecond.chunks === taggedFirst.chunks && + Number(taggedSecond.chunks) >= 1, + `created=${checkCreated.success} first scan kept etag=${taggedFirst?.etag ?? 'none'}/"t1" lastModified=${datedFirst?.lastModified ?? 'none'}, rescan sent bodies/304s [${checkSecond}] (want tagged 0/1, dated 0/1, plain 1/0), stamped tagged=${taggedSecond?.visited ?? '?'} dated=${datedSecond?.visited ?? '?'}, chunks ${taggedFirst?.chunks ?? '?'}→${taggedSecond?.chunks ?? '?'}`, + ); + + // A page that changed is sent and indexed again. And an HTML page of a + // server without validators — a new token in every response, the same + // text — is found unchanged by that text: the scan ends without a + // render session, which this harness could not open. + checkSite.set('/tagged.txt', { + body: checkPage('tagged', 'v2'), + etag: '"t2"', + }); + const htmlText = htmlToText(checkHtml()); + const htmlHash = computeContentHash(htmlText); + await pool` + INSERT INTO public_web.website_urls + (domain, url, status, discovered_at, listed, last_crawled_at, + word_count, content, content_hash, probe_hash) + VALUES (${CHECK_DOMAIN}, ${checkUrl('/page.html')}, 'active', NOW(), FALSE, + NOW() - INTERVAL '2 days', 20, ${htmlText}, ${htmlHash}, ${htmlHash}) + ON CONFLICT (domain, url) DO NOTHING + `; + await pool` + INSERT INTO public_web.chunks + (domain, url, title, content_hash, chunk_index, chunk_content) + VALUES (${CHECK_DOMAIN}, ${checkUrl('/page.html')}, 'Check fixture', ${htmlHash}, 0, ${htmlText}) + ON CONFLICT DO NOTHING + `; + checkSitemapRoutes.push('/page.html'); + const checkThird = await checkRescan(); + const taggedThird = await checkRow('/tagged.txt'); + const htmlThird = await checkRow('/page.html'); + const checkCorpus = await pool<{ status: string; error: string | null }[]>` + SELECT status, error FROM public_web.websites WHERE domain = ${CHECK_DOMAIN} + `; + record( + 'websites change check: a changed page is downloaded and indexed again, and an HTML page whose text reads the same ends the scan without a browser', + checkThird === + '/tagged.txt:1/0 /dated.txt:0/1 /plain.txt:1/0 /page.html:1/0' && + taggedThird?.etag === '"t2"' && + (taggedThird.text ?? '').includes('v2') && + !(taggedThird.text ?? '').includes('v1') && + htmlThird?.status === 'active' && + htmlThird.visited && + htmlThird.probeHash === htmlHash && + htmlThird.chunks === '1' && + checkCorpus[0]?.status === 'completed' && + checkCorpus[0].error === null, + `rescan sent bodies/304s [${checkThird}] (want tagged 1/0, dated 0/1, plain 1/0, html 1/0), tagged etag=${taggedThird?.etag ?? 'none'}/"t2" reindexed=${(taggedThird?.text ?? '').includes('v2')}, html=${htmlThird?.status ?? 'MISSING'}/active stamped=${htmlThird?.visited ?? '?'} hashKept=${htmlThird?.probeHash === htmlHash} chunks=${htmlThird?.chunks ?? '?'}/1, scan=${checkCorpus[0]?.status ?? '?'}/completed error=${checkCorpus[0]?.error ?? 'null'}`, + ); + + // "Scan now" is the same scan, started by a person: it asks each page + // the same way and is answered the same way. + await pool` + UPDATE public_web.website_urls + SET last_crawled_at = now() - interval '2 days' + WHERE domain = ${CHECK_DOMAIN} + `; + const beforeScanNow = checkSnapshot(); + const scanNow = z + .looseObject({ queued: z.boolean() }) + .safeParse(await (await post(`/${checkId}/scan`, {})).json()); + await drainCrawlJobs(); + const checkScanNow = sentSince(beforeScanNow); + const htmlScanNow = await checkRow('/page.html'); + record( + 'websites change check: Scan now asks each page the same way as a scheduled scan', + scanNow.success && + scanNow.data.queued && + checkScanNow === + '/tagged.txt:0/1 /dated.txt:0/1 /plain.txt:1/0 /page.html:1/0' && + (htmlScanNow?.visited ?? false), + `scanNow queued=${scanNow.success ? scanNow.data.queued : 'BAD SHAPE'} sent bodies/304s [${checkScanNow}] (want tagged 0/1, dated 0/1, plain 1/0, html 1/0) htmlStamped=${htmlScanNow?.visited ?? '?'}`, + ); + if (checkId !== '') { + await v1(`/websites/${checkId}`, { method: 'DELETE' }); + } + // 4f. Round g, g4-5: a site whose every page failed is not "a successful // scan". The homepage is always admitted, its 503 is one attempted // page and nothing stored, so the scan ends on the documented diff --git a/services/platform/lib/knowledge/crawl-parse.test.ts b/services/platform/lib/knowledge/crawl-parse.test.ts index 97c9d3a871..2b024a9f43 100644 --- a/services/platform/lib/knowledge/crawl-parse.test.ts +++ b/services/platform/lib/knowledge/crawl-parse.test.ts @@ -27,6 +27,8 @@ import { paragraphsForHashing, parseRobots, parseSitemapLocs, + plainTextCarriesRendered, + plainTextCoverage, robotsHeaderForbidsIndexing, siteHosts, stripBoilerplate, @@ -483,6 +485,55 @@ describe('boilerplate', () => { }); }); +/** + * One plain request can tell a scan whether a page changed only when the + * plain HTML carries what a browser shows. The crawler used to render every + * page to find that out — the browser, and every image, script and + * stylesheet the render loads, for pages that had not changed in months. + * The share is measured on the two texts of one visit. + */ +describe('plainTextCoverage', () => { + const article = + 'Tale indexes the public pages of the websites an organization added and answers questions from them.'; + + it('is whole for a page whose plain HTML already reads as the browser shows it', () => { + expect(plainTextCoverage(article, article)).toBe(1); + // Layout only moves the line breaks. + expect(plainTextCoverage(article, article.replaceAll(' ', '\n'))).toBe(1); + expect(plainTextCarriesRendered(article, article)).toBe(true); + }); + + it('counts what the scripts of a page add, and calls the page server-rendered while that stays a small part', () => { + const rendered = `${article} We use cookies.`; + // 16 of the 19 rendered words are in the plain HTML. + expect(plainTextCoverage(article, rendered)).toBeCloseTo(16 / 19, 5); + expect(plainTextCarriesRendered(article, rendered)).toBe(false); + const longer = `${article} ${article} ${article} We use cookies.`; + expect( + plainTextCarriesRendered(`${article} ${article} ${article}`, longer), + ).toBe(true); + }); + + it('is next to nothing for a page built by its JavaScript', () => { + const shell = 'Loading… You need to enable JavaScript to run this app.'; + const rendered = `${article} ${article} ${article}`; + expect(plainTextCoverage(shell, rendered)).toBeLessThan(0.1); + expect(plainTextCarriesRendered(shell, rendered)).toBe(false); + }); + + // A word the plain HTML holds once does not vouch for it shown five times. + it('counts a word as often as it occurs', () => { + expect(plainTextCoverage('alpha beta', 'alpha alpha alpha alpha')).toBe( + 0.25, + ); + }); + + it('is whole for a page that shows no text', () => { + expect(plainTextCoverage('anything', '')).toBe(1); + expect(plainTextCarriesRendered('', ' \n ')).toBe(true); + }); +}); + describe('metaDescription', () => { it('prefers name=description and falls back to og:description', () => { expect( diff --git a/services/platform/lib/knowledge/crawl-parse.ts b/services/platform/lib/knowledge/crawl-parse.ts index 9632bec201..cf516eaf2e 100644 --- a/services/platform/lib/knowledge/crawl-parse.ts +++ b/services/platform/lib/knowledge/crawl-parse.ts @@ -531,6 +531,63 @@ export function siteHosts(domain: string): Set { return hosts; } +const wordsOf = (text: string): string[] => + text.split(/\s+/).filter((word) => word !== ''); + +/** + * How much of the text a browser shows for a page its plain HTML already + * carries: the share of the rendered text's words found in the text read + * out of the unrendered HTML, each word counted as often as it occurs. `1` + * for a page whose rendered text is empty — nothing it shows is missing. + */ +export function plainTextCoverage( + plainText: string, + renderedText: string, +): number { + const rendered = wordsOf(renderedText); + if (rendered.length === 0) return 1; + const carried = new Map(); + for (const word of wordsOf(plainText)) { + carried.set(word, (carried.get(word) ?? 0) + 1); + } + let found = 0; + for (const word of rendered) { + const left = carried.get(word) ?? 0; + if (left === 0) continue; + carried.set(word, left - 1); + found += 1; + } + return found / rendered.length; +} + +/** The share of a rendered page's words its plain HTML has to carry for the + * page to count as server-rendered. Measured, 2026-10-06: server-rendered + * pages sit between 0.87 and 1.00 (the rest is what their scripts add — a + * consent banner, a counter, a hydrated widget), a page built by its + * JavaScript at 0.01. Nine in ten keeps the first kind and can never admit + * the second; a page in between is rendered, which is only slower. */ +const PLAIN_TEXT_COVERAGE_FLOOR = 0.9; + +/** + * Whether a page's plain HTML carries what a browser shows for it — the + * page is server-rendered, so one plain request can tell a later scan + * whether it changed: the same text read out of the plain HTML is the same + * page. False for a page built by its JavaScript, whose plain HTML is a + * shell that reads the same whatever the page shows; such a page has to be + * rendered to be judged. + * + * Decided per page from the two texts of one visit, never from the site, + * its framework or its markup. + */ +export function plainTextCarriesRendered( + plainText: string, + renderedText: string, +): boolean { + return ( + plainTextCoverage(plainText, renderedText) >= PLAIN_TEXT_COVERAGE_FLOOR + ); +} + /** Split page text into the paragraphs the boilerplate ledger hashes. Short * fragments (menu items, button labels) are ignored — hashing them would * blocklist ordinary words. */ diff --git a/services/platform/tests/manual/reference/automation.md b/services/platform/tests/manual/reference/automation.md index 458f415754..1468547154 100644 --- a/services/platform/tests/manual/reference/automation.md +++ b/services/platform/tests/manual/reference/automation.md @@ -396,6 +396,7 @@ Legend: ✅ fully automated · 🔶 partially automated · ⛔ manual-only (no s | [knowledge](../suites/knowledge.md) | Website sources take the knowledge write role on the server too: a read-only Member is refused `RBAC_FORBIDDEN` (403) on add, edit, delete, **Resume scanning** and **Scan now** (`POST /api/app/websites`, `PATCH` and `DELETE …/{websiteId}`, `…/{websiteId}/resume`, `…/{websiteId}/scan`) before the service is reached, where every member of the organization could call them before; Editor, Developer, Admin and Owner pass, and the list, a row, the status sync and the search fact stay open to every member | ✅ route | `backend/domains/websites/routes.test.ts` | | [knowledge](../suites/knowledge.md) | A page is stamped as visited only once it is stored and indexed, so a link cut off in between leaves it due for the resumed scan; unchanged text whose chunks were cut from other text is indexed again instead of serving the old text; a row the running scan retired (an alias it followed, a page that answered 404) is not revived by the links of the next page, while discovery still revives what an earlier scan retired | ✅ unit + real Postgres | `backend/core/knowledge/crawl_action.vectors.test.ts`, `backend/core/knowledge/crawl.test.ts`, `backend/core/knowledge/crawl_action.redirects.test.ts`, `backend/integration-check.ts` (`checkWebsitesCrawl`) | | [knowledge](../suites/knowledge.md) | A page that redirects onto another host or path of its site (`www.` onto the apex, a missing trailing slash, a removed page sent to the homepage) is retired as an alias and its target tracked instead, unless robots.txt disallows the target; a listed URL and a redirect that only rewrites the query keep their row | ✅ unit | `backend/core/knowledge/crawl_action.redirects.test.ts` | +| [knowledge](../suites/knowledge.md) | A scan asks for each page once and opens the browser only for a page that changed: the request carries the validators of the page's last settled visit (`ETag`, `Last-Modified`), and a 304 leaves the page as it is, stamped as visited with its failure record cleared; for a server that sends neither, an HTML page whose plain text reads as it did is left as it is too, although every response differs in its bytes (a token, a nonce), and a robots meta tag on it is still honoured; a changed page is rendered, and keeps what to check it by only when its plain HTML carried at least nine tenths of the words the browser showed, so a page built by its JavaScript is rendered on every scan; a row without stored text, or whose text is not indexed yet, is asked unconditionally; a document's 304 spares its download; purging a page's content clears what it was checked by; **Scan now** asks the same way as a scheduled scan | ✅ unit + real Postgres | `lib/knowledge/crawl-parse.test.ts`, `backend/core/knowledge/crawl_action.change-check.test.ts`, `backend/core/knowledge/crawl_action.redirects.test.ts`, `backend/integration-check.ts` (`checkWebsitesCrawl`: the `websites change check` lanes) | | [knowledge](../suites/knowledge.md) / [settings](../suites/settings.md) | The crawler's browser session has room to run: the `default` session profile allows 512 tasks (a headless Chromium needs 160 and more; under 128 every navigation after the first two or three died `net::ERR_ABORTED`), and a browser that dies mid-batch ends the batch with the untouched pages handed back, not charged | ✅ unit | `services/sandbox/src/session/docker-session-args.test.ts`, `backend/core/node_only/sandbox/render_fetch.test.ts` | | [settings](../suites/settings.md) | `SET-F33` (MCP endpoint) — the wire contract behind the page: envelope validation, version negotiation, batches, tool-argument validation against the advertised schema (document tools included), `params` typing, the unissued `tools/list` cursor, 405 + `Allow` on other verbs, `isError` on failure-shaped results, `get_docs` serving a reference and never the builder prompt | ✅ automated | `backend/core/automations_builder/mcp_http.test.ts`, `lib/engine/selftest/dispatch.test.ts` | | [projects](../suites/projects.md) | REST chat lifecycle: door-side model resolution (`CHAT_MODEL_UNKNOWN` / `CHAT_MODEL_AMBIGUOUS`), model facts and the default marker on `GET /models`, message `status` (a stop settles as `cancelled`) and `usage` (`costEstimateCents` included), the 202’s pre-minted `messageId` and the `queued` poll state until the worker opens the turn, streaming progress on the poll (`since` and `reasoningSince` slice the text and the reasoning independently, each with its offset/length pair and the reset-to-0 rule), `reasoningEffort` / `maxOutputTokens` on a send (a cap above the model’s ceiling refused; the cap a per-turn budget the tool rounds share — a later round gets what the earlier ones left, a round with nothing left is not run, a cut round marks the turn `finishReason: length` and its calls settle `invalid_args` unrun), `usage.costEstimateCents` rounded to a millionth of a cent, a named `locale` asking for the reply language on the system prompt and on the message (`localeFixed` through the job into the runtime directive and the wire-only notice; the app lane untouched), archive with `archivedAt` echoed from the domain’s own stamp / rename / delete / cancel routes, the stop of a send still queued (202 `cancelling` → a `cancelled` reply without a model call) and the 404’s `data.lastMessageId` / `data.lastStatus`, hidden regenerate-siblings invisible by id, blank-prompt and locale refusals | ✅ automated | `backend/rest/v1-threads.test.ts`, `backend/rest/v1-threads.contract.test.ts`, `backend/rest/v1-threads.scope.test.ts`, `backend/domains/chat/rest-turn.test.ts`, `backend/domains/chat/store.test.ts`, `lib/chat/turn.test.ts`, `lib/chat/context.test.ts`, `lib/chat/effort.test.ts`, `lib/chat/tools.test.ts`, `lib/shared/utils/language-name.test.ts`, `scripts/openapi/spec.test.ts`, the `REST resources` lane in `backend:integration` (the queued stop and the 404’s data on the real schema) |