feat(plugin): named page types as the unit for per-template config + metrics (v0.34.0)

packages/plugin/README.md

@@ -35,6 +35,9 @@ rest: true # required for the @export-ed table REST endpoints
   # --- options (all optional; defaults shown) ---
   domains: [] # indexable-host allowlist; empty = allow all hosts
 
+  # named templates; routes point at one with `pageType` — see "Page types (templates)"
+  pageTypes: [] # [{ name, renderInterval }] — several routes may share one name
+
   ingress: # how incoming bot requests are parsed (see "Ingress modes" below)
     mode: prefix # 'prefix' (native /p/<absolute-url>) or 'forwarded' (reverse proxy/CDN)
     botPathPrefix: /p/ # prefix mode: requests under this prefix are treated as bot requests
@@ -44,7 +47,8 @@ rest: true # required for the @export-ed table REST endpoints
     forwardedProtoHeader: x-forwarded-proto
     defaultProtocol: https
     # ordered, first match wins — see "Route classes"
-    routes: [] # [{ match: exact|prefix|contains, path, mode: prerender|passthrough, queryParams: [...] }]
+    routes: [] # [{ match: exact|prefix|contains, path, mode: prerender|passthrough,
+    #                queryParams: [...], pageType, renderInterval }]
     # compiled into `routes` as prepended passthrough entries; matched against the PATH
     excludePathPatterns: ['/search/'] # paths containing these are never auto-scheduled
     report: # periodic tally of paths served without prerendering
@@ -131,11 +135,13 @@ rest: true # required for the @export-ed table REST endpoints
     enabled: true # record bot analytics at all: bot_request (ingress volume by host/bot/device),
     # bot_serve (outcome by source/cache-status/bot — origin offload + cache hit rate),
     # page_age (ms since the served page rendered — freshness at serve, cache-served only),
-    # route_serve (outcome by route/cache-status/device — per-route delivery, for tuning each
-    # route's renderInterval), and route_page_age (served age by route/cache-status/device).
+    # pagetype_serve (outcome by page-type/cache-status/device — per-template delivery, for
+    # tuning each template's renderInterval), and pagetype_age (served age by the same three).
     # cache-status distinguishes 'hit' (within the page's renderInterval) from 'swr' (served
     # from the stale-while-revalidate window because the re-render is late/in flight) — both
     # are cache serves; 'hit' alone is the "is the TTL being met" signal.
+    # render_time carries the page type as its third dimension, so render COST and delivered
+    # freshness join on one key — see "Page types (templates)".
 
   crawlStats: # crawl breadth: distinct URLs crawled per bot per UTC day (HyperLogLog, ~0.8% error)
     enabled: true # also gated by analytics.enabled above; read via GET /prerender_admin/crawl-breadth?days=7
@@ -237,6 +243,72 @@ forwarding `/blog/*`"); passthrough is the coverage backlog ("we proxy this much
 on purpose"). The tally is in-process, so **every worker** flushes its own line — each carries
 `node=` and `worker=`, and a reader sums across them.
 
+### Page types (templates)
+
+A route says _whether_ a path is prerendered. A **page type** says _what kind of page it is_ —
+the site's own vocabulary for its templates: `home`, `category`, `pdp`. It is the unit that
+per-template settings and per-template metrics hang off.
+
+```yaml
+pageTypes:
+  - { name: home, renderInterval: 7200000 } # 2h
+  - { name: category, renderInterval: 43200000 } # 12h
+  - { name: pdp, renderInterval: 172800000 } # 48h
+
+ingress:
+  routes:
+    - { match: exact, path: '/', pageType: home }
+    - { match: prefix, path: '/catalog/', pageType: category }
+    - { match: contains, path: '/category/', pageType: category } # same template, second URL shape
+    - { match: prefix, path: '/product/prd-', pageType: pdp }
+```
+
+**Several routes may share one name, and that is the point.** A template reachable by two URL
+shapes is one template. Before page types, metrics were labelled with the matched route's _path_,
+so those two category routes produced two unrelated series that only a reader holding the route
+list could add back together — and a cadence set on both could silently drift, with only the
+first-matching copy ever taking effect.
+
+Declaring a type is **optional**. A route may name a type that appears in no `pageTypes` entry;
+it still labels metrics correctly. Declaring it is only needed to give the type settings, or to
+share one setting across the routes that carry it. A type declared but referenced by no route is
+reported as an `info` config finding — that state is almost always a spelling mismatch, and its
+symptom otherwise is silence.
+
+**Cadence precedence** (resolved at schedule time, every cycle — no data migration):
+
+```
+route renderInterval  >  pageType renderInterval  >  target's stored interval  >  render.defaultInterval
+```
+
+The route level still wins so a single URL can carve itself out of its template's cadence (an
+`exact` route ordered above the template's prefix) without inventing a one-member type.
+
+**What the name is used for:**
+
+| consumer                          | uses                                                                   |
+| --------------------------------- | ---------------------------------------------------------------------- |
+| `pagetype_serve` / `pagetype_age` | delivery + freshness per template — the "should this TTL move" numbers |
+| `render_time`                     | render **cost** per template — the fleet capacity input                |
+| `resolveRenderInterval`           | one cadence for every route sharing the template                       |
+| the queue job (`pageType`)        | lets browser-side render rules scope by template name                  |
+| `/prerender_admin` explain        | the template, whether it is declared, and the cadence that follows     |
+
+The job payload carries the **declared name only**, never the metrics fallback — a browser-side
+rule scoped to a template must not fire on a route that was never declared to be one. That field
+is what lets the render fleet stop re-describing the same URL shapes as regular expressions in its
+own config: two pattern lists for one routing fact, in two repositories, with nothing keeping them
+in step.
+
+**Adoption is incremental.** With no `pageType` on any route, the label falls back to the route's
+path and then to the route class — exactly the values emitted before page types existed. Name one
+route at a time; nothing resets.
+
+Metrics labels are bounded by construction: every label resolves to a configured name, a
+configured path, or one of three class constants. Nothing derives a label from the request itself
+— an unbounded label is how a monitoring backend gets taken down by a crawler walking a faceted
+URL space.
+
 ### Sitemaps are filtered to prerender routes
 
 A sitemap is written for search engines: it lists every indexable URL on the site, which is routinely

packages/plugin/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender",
-	"version": "0.33.0",
+	"version": "0.34.0",
 	"type": "module",
 	"description": "Configurable Harper plugin for prerendering pages for bots and crawlers",
 	"license": "Apache-2.0",

packages/plugin/src/admin/views/explain.js

@@ -119,6 +119,34 @@ function explanation(ctx, data) {
 							muted(`  ${data.ingress.route.source}`),
 						]),
 					],
+					data.eligibility.prerendered && [
+						'Page type',
+						el('span', null, [
+							el('code', { text: data.pageType.metricLabel }),
+							muted(
+								data.pageType.name === null
+									? '  no pageType on this route — reported under its path'
+									: data.pageType.declared
+										? '  declared in pageTypes'
+										: '  named but not declared — metrics only, no settings'
+							),
+						]),
+					],
+					data.eligibility.prerendered && [
+						'Render cadence',
+						el('span', null, [
+							el('code', { text: duration(data.pageType.cadence.effective) }),
+							muted(
+								`  ${
+									data.pageType.cadence.route !== null
+										? 'from the matched route'
+										: data.pageType.cadence.pageType !== null
+											? `from pageType ${data.pageType.name}`
+											: 'render.defaultInterval'
+								}${data.pageType.cadence.effectiveAssumesNoStoredInterval ? ', unless this target stores its own' : ''}`
+							),
+						]),
+					],
 				]),
 				verdictPills(data, page),
 			],

packages/plugin/src/config.js

@@ -35,7 +35,7 @@ import {
 // evaluation time. The count has to come from the compiler rather than from raw config,
 // because the finding's whole job is to catch entries the compiler REJECTED (a typo'd
 // `match`), which the raw array still contains.
-import { prerenderRouteCount } from './util/routeClass.js';
+import { declaredPageTypes, prerenderRouteCount, routePageTypes } from './util/routeClass.js';
 
 // Returns the Harper logger when running inside Harper, otherwise the console.
 // Unit tests run outside Harper where `logger` is undefined.
@@ -334,6 +334,19 @@ export const collectConfigWarnings = () => {
 				'check ingress.routes for entries dropped as invalid'
 		);
 	}
+	// A `pageTypes` entry nothing points at is almost always a typo on one side of the join, and
+	// its symptom is silence: the type's settings simply never apply and its name never appears
+	// in metrics, which reads exactly like "this template gets no traffic". Info, not warn — the
+	// same list is legitimately shared across deployments whose route sets differ.
+	const unusedPageTypes = declaredPageTypes().filter((name) => !routePageTypes().has(name));
+	if (unusedPageTypes.length > 0) {
+		add(
+			'info',
+			'pageTypes',
+			`pageTypes declared but not referenced by any route: ${unusedPageTypes.join(', ')} — their settings ` +
+				'will never apply; check for a spelling mismatch with ingress.routes[].pageType'
+		);
+	}
 	const { staging } = config.origin;
 	if (staging.ip) {
 		// Mirror stagingTargetIp's gate (ip AND header AND valid ip) so the finding never

packages/plugin/src/configSchema.js

@@ -62,6 +62,30 @@ export const configSchema = group('Prerender plugin configuration.', {
 		{ itemType: 'string' }
 	),
 
+	pageTypes: option(
+		[],
+		'Named page types (templates) — the site’s own vocabulary for the kinds of page it serves: ' +
+			'`home`, `category`, `pdp`. Each entry is { name: string, renderInterval?: number }.\n\n' +
+			'A page type is the unit that per-template settings and per-template METRICS hang off. ' +
+			'`ingress.routes` entries point at one with `pageType: <name>`, and SEVERAL routes may share ' +
+			'the same name — which is the point: a site whose category pages are reachable by two ' +
+			'different URL shapes gets ONE set of numbers for “category” instead of two unrelated rows ' +
+			'that a reader has to know to add together.\n\n' +
+			'Declaring a type here is optional. It is only required to (a) set a value once for several ' +
+			'routes, or (b) give the type a name in metrics; a route may name a type that is not declared ' +
+			'and it still labels metrics correctly — an undeclared name is not an error, just a type with ' +
+			'no settings of its own.\n\n' +
+			'`renderInterval` (ms) is the render cadence for every URL of this type. Full precedence, ' +
+			'resolved at schedule time on each cycle: route `renderInterval` > this > the target’s stored ' +
+			'interval (sitemap `changefreq` / explicit API write) > `render.defaultInterval`. The route ' +
+			'level still wins so a single URL (an `exact` route) can carve itself out of its type’s ' +
+			'cadence without inventing a one-member type.\n\n' +
+			'Names travel: the matched type is sent to the renderer on each queue job, so browser-side ' +
+			'render rules can be scoped by template name instead of re-describing the same URL shapes as ' +
+			'regular expressions in a second repository.',
+		{ itemType: 'object' }
+	),
+
 	ingress: group(
 		'Request-ingestion model: how incoming bot requests are recognized, which paths are ' +
 			'prerendered, and how the target URL and device type are derived.\n\n' +
@@ -97,7 +121,7 @@ export const configSchema = group('Prerender plugin configuration.', {
 				[],
 				'Ordered route list (forwarded mode). Each entry is ' +
 					"{ match: 'exact' | 'prefix' | 'contains', path: string, mode?: 'prerender' | 'passthrough', " +
-					'queryParams?: string[], renderInterval?: number }.\n\n' +
+					'queryParams?: string[], pageType?: string, renderInterval?: number }.\n\n' +
 					'FIRST MATCH WINS, so order most-specific first. That ordering is what lets a passthrough ' +
 					'carve-out sit inside a prerendered prefix (`/products/clearance/` above `/products/`) ' +
 					'without a second list and a precedence rule.\n\n' +
@@ -111,12 +135,17 @@ export const configSchema = group('Prerender plugin configuration.', {
 					'off the proxied origin fetch and hand the visitor the wrong page.\n\n' +
 					"A path matching NOTHING is 'unclassified': still proxied (never blocked), never cached, and " +
 					'counted for reporting so the gap can be fixed at the CDN or here.\n\n' +
+					'`pageType` (prerender routes only) names the template this route serves — see the top-level ' +
+					'`pageTypes`. It is what per-template settings and metrics key on, and SEVERAL routes may ' +
+					'carry the same name so that one template reachable by two URL shapes reports as one thing. ' +
+					'Omitted, the route reports under its own `path`, which is the pre-`pageTypes` behavior.\n\n' +
 					'`renderInterval` (ms, prerender routes only) sets the render cadence for every URL the route ' +
-					"matches. Precedence: route > the target's stored interval (sitemap `<changefreq>` or an " +
-					'explicit API write) > `render.defaultInterval` — resolved at schedule time on every cycle, so ' +
-					"changing it here takes effect on each URL's next render with no data migration. A per-URL " +
-					'exception is an `exact` route ordered above its class (e.g. the homepage `exact /` at 2h above ' +
-					'a 6h section prefix); a route that should defer to sitemap changefreq simply doesn’t set one.\n\n' +
+					"matches. Precedence: route > the route's `pageType` > the target's stored interval (sitemap " +
+					'`<changefreq>` or an explicit API write) > `render.defaultInterval` — resolved at schedule time ' +
+					"on every cycle, so changing it here takes effect on each URL's next render with no data " +
+					'migration. Prefer setting a cadence on the `pageType` when several routes share it; keep it ' +
+					'here for a per-URL exception (e.g. the homepage `exact /` at 2h above a 6h section prefix). A ' +
+					'route that should defer to sitemap changefreq simply doesn’t set one.\n\n' +
 					"OPERATIONAL NOTE: if the CDN edge-caches a route's responses with a fixed TTL from its own " +
 					"property settings (not from our response headers), that TTL and the route's renderInterval " +
 					'must be kept aligned BY HAND — rendering much faster than the edge TTL burns renders the edge ' +
@@ -391,8 +420,9 @@ export const configSchema = group('Prerender plugin configuration.', {
 			'How often a target is re-rendered when nothing more specific applies. Cadence is relative to ' +
 				'each render’s completion (not a fixed time-of-day), and a target’s first render is jittered ' +
 				'across its interval — so the fleet renders as a smooth stream rather than a daily herd. Full ' +
-				'precedence, resolved at schedule time: matched route `renderInterval` (ingress.routes) > the ' +
-				'target’s stored interval (sitemap `changefreq` / explicit API write) > this default.',
+				'precedence, resolved at schedule time: matched route `renderInterval` (ingress.routes) > that ' +
+				'route’s `pageType` `renderInterval` (top-level `pageTypes`) > the target’s stored interval ' +
+				'(sitemap `changefreq` / explicit API write) > this default.',
 			{ unit: 'ms', min: 1 }
 		),
 		suppression: group(