fix(platform): prevent unsafe crawl fast-path skips

[yannickmonney]

Fixes #4439. Follow-up to review comment 6019728045.

Require complete plain-text coverage before storing a probe hash for the unchanged-page fast path; partial coverage can miss script-rendered changes. Adds regression coverage.

Related: #4445 (nearby crawl/indexing changes should rebase cleanly).

[yannickmonney]

Independent review (TALE-1016): REQUEST CHANGES at 381ba022b142c8366195e16723cf0454e09e6b78.

• DATA — BLOCKING, MEDIUM: Previously saved partial-coverage checks remain trusted. storedCheck (crawl_action.ts:1188–1196) accepts existing probe_hash, etag and last_modified whenever content is present/indexed. An unchanged plain response matches the old hash at lines 1567–1593, calls markPageUnchanged, writes the same hash back, and returns before checkAfterRender. A 304 likewise skips rendering and retains the old check. Thus a page previously admitted at 95% coverage can change its script-rendered 5% forever without refreshing its index. The two-file patch has no invalidation/versioning. Migration 13 clears pre-render-era validators, not already-written probe hashes, and is unchanged here; it does not repair this case. Invalidate/version all three unsafe check fields through a rollout-safe mechanism (including old writers during rolling deployment), or prove they could never have been populated in deployed corpora. Add a scan regression seeded with an old partial-coverage check that forces a render and observes the changed widget.
• REL-MANUAL / CODE — PARTIAL; regression requirement FAIL: The new comparison rejects coverage below 1 for future render settlements and clears all three fields together. Complete coverage still returns the probe hash/validators; the existing complete-coverage and same-text fast-path tests cover preservation. However the new test at crawl_action.change-check.test.ts:411 does not fail without the source change: htmlToText drops the head/scripts and produces # Guide plus the 13-word sentence (15 tokens); appending widget update gives 17 tokens. Coverage is 15/17 = 0.88235, already below the old 0.9 threshold, so both versions return NO_PAGE_CHECK. Use a longer plain fixture with coverage >=0.9 and <1 (for example append the same two missing words to at least 18 plain tokens). Source reasoning, not a claimed executed test.
• Writer/reader sweep: checkAfterRender feeds rendered settlement; markPageCrawled persists it after indexing. Non-HTML settlement keeps no probe hash. The unchanged HTML writer in markPageUnchanged still renews legacy hashes, and 304 preserves them. purgePageContentIn clears all three. Frontier selection and storedCheck read them; no freshness/provenance check exists. Integration fixtures also directly seed/read hashes. The fix therefore covers new rendered writes, but not all trust paths.
• PERFORMANCE — acceptable tradeoff, unmeasured: After safe invalidation, pages with 90%–<100% word coverage lose conditional/hash skips and pay the existing plain probe plus browser render each scan, including script/style/subresource traffic. Pages already below 90% have no additional steady-state cost; complete-coverage pages retain the optimization. Coverage calculation remains linear with a word-count map. No benchmark run or numerical cost claim.
• SECURITY — NOT_APPLICABLE to a new security surface: This patch changes check eligibility only; safe-fetch/SSRF, redirect, tenant, credential and response-limit boundaries are unchanged. Clearing validators reduces conditional-header reuse.
• A11Y — NOT_APPLICABLE: No controls, markup, keyboard or focus behavior changes.
• UI-RESPONSIVE — NOT_APPLICABLE: Backend crawler and unit fixture only; no layouts or breakpoints.
• LOCALE — NOT_APPLICABLE: No user-facing strings or locale keys.
• DOCS — NOT_APPLICABLE to product docs: Internal correctness repair, no public API/configuration or documented user workflow change. Minor source-doc cleanup: checkAfterRender still links plainTextCarriesRendered although the import was replaced; reference plainTextCoverage instead.
• feat(platform): store knowledge vectors in a table per width #4445 coordination: Open at d519cccd93144d3000e345ff202ae6ddbc803b5b, currently MERGEABLE/UNSTABLE; the task records prior REQUEST CHANGES, while native reviewDecision is empty. It modifies both repair files and nearby frontier/indexing/vector behavior. Its inspected crawl hunks do not replace this coverage condition, so there is conceptual/file overlap but no demonstrated direct hunk conflict with this repair. Carry the strict coverage rule, corrected regression and legacy invalidation through its review; standalone MERGEABLE does not prove the two PRs compose safely.

Verification: Local test NOT_RUN; the permitted diff-reasoning route suffices to establish both blockers. No install/build/stack, source edits, commits, branch updates, pushes or reruns. Native PR snapshot: 57 SUCCESS, 24 SKIPPED, 1 NEUTRAL, zero failing/pending; all seven CI-ready aggregators SUCCESS. Green CI does not establish a regression that already passes the old code. Exact head remains as above, OPEN/MERGEABLE/CLEAN.

Merge: Not performed: REQUEST CHANGES, and the 19:20Z authorized merge cutoff had already passed before this run (started after 20:32Z). Fleet owns subsequent repair/review/merge routing; this comment grants no later merge authorization.

[yannickmonney]

TALE-1023 repair of C7's REQUEST CHANGES, now at 7781964c8641ff5eda7df7a2906cdc788a1bb0e2 (added atop 381ba022b142c8366195e16723cf0454e09e6b78 on the existing branch).

1. Legacy checks / rolling deployment: knowledge migration 14 introduces separate nullable etag_v2, last_modified_v2, and probe_hash_v2 fields without copying legacy values. The crawler reads/writes only v2. The migration clears all three legacy fields and installs a persistent trigger that discards old inserts/updates and invalidates v2 on legacy or content writes, including NULL-to-NULL updates. Old readers therefore see no persisted checks; old writers remain compatible but cannot renew them. New instances regain skips only after strict complete-coverage settlement and indexing. An old request already holding a pre-migration snapshot can finish once, but cannot perpetuate its unsafe check. The migration is forward-only and idempotent; reapplication preserves fresh v2 checks. Real-Postgres integration probes cover preexisting checks, reapplication, old inserts/updates, NULL-to-NULL purge and content writes; existing scan consumers now seed/read v2.
2. Regression fixture: the longer plain fixture explicitly asserts coverage >=0.9 and <1. Added scan regressions seed the legacy hash and validators, keep the plain response unchanged, change the rendered widget, and assert a render, updated stored text/hash, no unsafe conditional headers, and no subsequent fast-path check.
3. Doc nit: checkAfterRender now links plainTextCoverage.
4. feat(platform): store knowledge vectors in a table per width #4445 coordination: its current head remains d519cccd93144d3000e345ff202ae6ddbc803b5b; this repair keeps coverage strictly complete. Re-review should assess the frontier/settlement overlap and v2 fields when composing the PRs; standalone mergeability is not composition proof.

Observed verification: normal commit hooks, scoped oxlint/oxfmt, and git diff --check passed. OLD used the base implementation with the pre-fix 0.9 threshold restored and selected the three regression cases: all three failed as expected (20 unrelated cases skipped). NEW ran the complete targeted file with one worker: 23/23 passed. Both runs used /opt/node/bin/node and fresh two-reading resource guards. Logs, exact OLD source, complete patch, tree and source hashes are retained in TALE-1023's native delivery artifacts. No install/build/local PostgreSQL stack or CI rerun. Backend integration and full typecheck/suites remain CI's oracle; results are not claimed yet.

Pushed once, no force-push/new PR. Handoff to Fleet/C7 for independent re-review of the exact new head. No merge: the re-review and #5's v0.5.75 candidate record remain required. Fleet owns the next CI check/review checkpoint; please inspect the native workflows for this head after this initial dispatch completes.

[yannickmonney]

Independent re-review at exact head 7781964c8641ff5eda7df7a2906cdc788a1bb0e2 — ACCEPT.

• DATA blocker from 6024950825: CLOSED. Migration 14 is forward-only and idempotent: it adds nullable etag_v2, last_modified_v2, and probe_hash_v2 without backfilling unsafe legacy provenance. Reapplying it preserves fresh v2 checks. The persistent BEFORE INSERT OR UPDATE OF etag, last_modified, probe_hash, content, content_hash trigger clears legacy fields and invalidates v2 on legacy writes (including NULL-to-NULL updates) and content writes. During a rolling deploy, old readers see NULL legacy checks; old writers cannot renew checks; an in-flight old request may complete from its snapshot, but cannot persist a trusted legacy check. Intended invalidation is the only loss.
• Regression fixture from 6024950825: CLOSED. The fixture now has plain-text coverage >=0.9 and <1, and the scan regressions seed legacy hash/validators, force render, verify changed rendered text/hash, suppress conditional headers, and verify no unsafe fast path. The doc nit is also closed: checkAfterRender links plainTextCoverage.
• Backend integration: Backend integration succeeded in workflow run 37532029440, job 112503597724 (real PostgreSQL), completed 2026-10-06T21:18:21Z.
• CODE / PERFORMANCE: frontier selection, storedCheck, settlement, unchanged-page writes, purge, and integration fixtures use v2 fields. Migration 14 adds three nullable columns and a row-level trigger on hot writes; no indexes are added or changed. The trigger adds small per-row PL/pgSQL work and intentionally invalidates checks on content writes; no benchmark isolates that cost, while the performance check passed.
• feat(platform): store knowledge vectors in a table per width #4445 composition: migration number is 14 and the v2 field names are etag_v2, last_modified_v2, probe_hash_v2. Open feat(platform): store knowledge vectors in a table per width #4445 (d519cccd93144d3000e345ff202ae6ddbc803b5b) also touches crawler/knowledge migrations, so retain migration-number uniqueness and these exact names when composing; standalone mergeability is not composition proof.
• Part A: SECURITY, A11Y, UI-RESPONSIVE, and LOCALE are NOT_APPLICABLE: this is backend crawler/database behavior with no new security surface, controls/markup, layout, or user-facing locale strings. Product DOCS is NOT_APPLICABLE; the source-doc link correction is included.

Checks read at 2026-10-06T21:xxZ: 58 SUCCESS, 24 SKIPPED, 1 NEUTRAL, 0 failing, 0 pending (native PR snapshot); mergeable/clean. No local test, install, build, stack, push, branch update, or merge performed. Merge remains Fleet-controlled after the v0.5.75 candidate record.

[yannickmonney]

Merge-lane coverage dispositions at 7781964c8641ff5eda7df7a2906cdc788a1bb0e2. From agent #2 (TALE-359 run 0d0927b5), the merger.

• Implementer: C3 4c7208dc (TALE-1007, TALE-1023).
• Independent ACCEPT at this exact head: C7 ecfaae72 (re-review TALE-1024, after REQUEST CHANGES 6024950825), comment 6025882356.
• Independence: I am neither the author nor a reviewer of this PR.
• Composition: Composition: the seven merge-ready PRs compose cleanly in sequence on main 5b6ab36b (fix(platform): stage re-delivered outputs by last write #4452, fix(platform): prevent unsafe crawl fast-path skips #4451, fix(platform): align agent details locale terminology #4461, fix(platform): retain comment edits across log window threshold #4460, docs(platform): explain project environment credentials #4463, fix(platform): contain optional lazy chunk failures #4458, fix: localize governance model access labels #4462). They share no files, and the final composed tree is 5ef5a8c6. This PR's patch-id is the same on main and against its own merge base (432a4ebe).

Topic	Disposition	Basis
DATA	PASS (real PostgreSQL in CI)	C7: migration public_web/00000000000014_website_url_strict_change_check.sql is forward-only and idempotent: three nullable v2 columns and no unsafe backfill. A BEFORE INSERT OR UPDATE trigger invalidates checks on legacy or content writes. Backend integration (real PostgreSQL) succeeded in run 37532029440. Number 14 is unique on main (public_web has 09/10/11/13). Open #4445 uses public_web/15 and private_knowledge/14, so there is no collision.
CODE	PASS	Frontier selection, the stored check, settlement, unchanged-page writes, purge and the integration fixtures all use the v2 fields (C7).
REL-MANUAL	PASS (CI); old-fails by reasoning	The repaired fixture has plain-text coverage in [0.9, 1). The scan regressions force a render and verify there is no unsafe fast path. The first review's blocker (6024950825) is closed. NEW executed in CI.
SECURITY	NOT_APPLICABLE	No new trust boundary (C7).
PERFORMANCE, RESOURCES	PASS (source); benchmark NOT_RUN	Three nullable columns and a row-level trigger on hot writes, with no index change. The Performance check passed; there is no isolated benchmark.
DEPLOYMENT, RECOVERY	PASS (source); live upgrade UNAVAILABLE	Rolling deploy: old readers see NULL legacy checks, and old writers cannot renew checks; the only effect is the intended invalidation. Forward-only: a rollback image re-renders more but loses no content. There is no disposable upgrade proof and no Ops evidence.
UX-SIMPLICITY, UX-CLARITY, A11Y, UI-RESPONSIVE, LOCALE, DOCS	NOT_APPLICABLE	Backend crawler and database only. The source-doc link fix is included (C7).
WORKFLOW, PROVIDERS, OBSERVABILITY	NOT_APPLICABLE	No cycle-workflow, provider or telemetry change.
CICD	PASS	83 checks: 58 success, 24 by-design skips, 1 neutral (Trivy). CI started at 21:11Z, so it tested current main 5b6ab36b.

Merge-gate dry run at 01:55:07Z: open, CLEAN, head matches; no required contexts. checks total=83 not-completed=0 failing=0 [neutral=1 skipped=24 success=58] neutral=[Trivy[github-advanced-security]] statuses=[pending total=0]. Every skip is on the by-design allowlist skip-allow-1545-bi.txt; no check is pending or failing. Merging next with --squash --match-head-commit 7781964c8641ff5eda7df7a2906cdc788a1bb0e2.