diff --git a/skills/capture-site/SKILL.md b/skills/capture-site/SKILL.md index 9dac815..2190d2e 100644 --- a/skills/capture-site/SKILL.md +++ b/skills/capture-site/SKILL.md @@ -124,6 +124,22 @@ source. Run each step as its own command, in this order: only)`: repeat that line to the person. Use `--sizes all` when the owner says old emails or other sites may link to those sizes. + Hosts that resize by query string (WordPress VIP, Jetpack Photon: `photo.jpg?w=1024`, + `?resize=300,200`) serve every size at the same path. The original is the bare path, so that + is what is requested and saved; each query string seen is listed under `variants` (not + requested; `--sizes all` saves them as `%3F` under a sibling folder, + `.conversion/media-cache-variants/`, so the media cache that is uploaded or copied into the + repo holds bare paths only). When the bare path answers an error (4xx or 5xx; a redirect is + recorded with its location, and a too-large file is asked again once the cap is raised), a + variant is saved in its place and the entry's `downloadedFrom` is the URL it came from. A cache-busting query (no + `w`, `h`, `resize`, `crop`, `fit`, `quality`, `format` or `s` parameter, e.g. `?ver=3`) is the + same file, so it is preferred and nothing else is marked; otherwise the widest variant is saved + and the entry also says `resized: true`. The run prints a line per file and a count (`2 + originals unavailable (resized: true) ...`), baseline/media.json keeps both fields, and the + media gate repeats the count as `N resized (original unavailable on live)` without failing, + since it compares against the bytes actually saved. Repeat that count to the person and list + those files in the plan and runbook as originals the live site no longer serves. + 4. **Templates.** Clusters the pages that answered 200 into `capture/templates.json`. On WordPress, pages group by template body class first (`home`, `page`, `single-post`, `archive`, template classes) and a page whose markup is clearly different (Jaccard below 0.5) @@ -242,7 +258,9 @@ redirects to a captcha path. Every page then looks the same, so a capture of it excluded prefix (kept in the HTML, never in internal, never requested) scripts.json third-party vendors with their ids and pages media.json every media file with sources, pages, status, type and bytes; a size - only the REST list names: downloaded: false, "unreferenced size" + only the REST list names: downloaded: false, "unreferenced size"; + query-string sizes under variants; resized: true when only a + resized copy could be saved, downloadedFrom when a variant stood in css/ every stylesheet, inline `; +}; --- @@ -20,8 +32,14 @@ const pageCss = pageSheet ? pageSheets[`../styles/pages/${pageSheet}`] : null; {canonical && } {robots && } {!INDEXABLE && } - - {pageCss && } + {css ? css.map((entry) => (entry.kind === 'inline' + ? + : )) : ( + <> + + {pageCss && } + + )} diff --git a/skills/parity-gates/SKILL.md b/skills/parity-gates/SKILL.md index 8366ed2..7821813 100644 --- a/skills/parity-gates/SKILL.md +++ b/skills/parity-gates/SKILL.md @@ -99,7 +99,7 @@ rebuild, rerun. One gate alone: `node /scripts/verify/.mjs`. | `links` | Every internal href and src resolves to a built page or file, a redirect source or baseline media, or lands on a built page in one hop through the compiled rules (`vercel.json` or `dist/_redirects`: the slash rules and every `redirects.csv` row), as live redirected it ("N links resolved through redirects"); two hops or a missing target fail; with no host config, `/about` where the build serves `/about/` fails | | | `media` | Every baseline media file is served by the build with the same bytes (`--paths` picks a file by its path or by a page that references it): in `dist/` in plan mode `repo`; under the plan's `prefix`, in the store manifest (`store`) or still on live (`live-temporary`); outside the prefix, in `dist/` in every mode. With `PUBLIC_INDEXABLE=true`, `live-temporary` fails every file under the prefix (a release build may not serve media from the live site; a plan without a prefix covers every file) and still checks the files outside it in `dist/` | Plan mode `live-temporary` on a staging build: files under the prefix are served from live, not checked | | `redirects` | Every baseline redirect and csv row answers in one hop with the same status, and the target answers 200 in `dist/` (or is external, or under a prefix in baseline `meta.json` `excluded`: then only the first hop is checked); no source is also a built file; no built page or file is redirected away from its own URL (`built page redirected by rule `; no exception covers it, fix the rule); a path with a "no redirect" exception (`kind: "redirect"`, `action: "none"`) is not redirected at all | A no-op query row, any query row on Cloudflare, or a source with no valid form that `redirects-skipped.json` lists with a reason (`not compiled`) | -| `routes` | Every page, post, archive and feed at 200 is built, noindex routes still say noindex, nothing extra is built (the capture's own scripts in `baseline/js.json` at their live paths, the live sitemap names in `baseline/sitemaps.json`, and the launch icon set are not extras); a route captured at any status other than 200, 3xx, 404 or 410 fails as `not captured: ` unless a `routes` exception skips it or names its built replacement | A `routes` exception with `to: "skip"`: that route was never captured | +| `routes` | Every page, post, archive and feed at 200 is built, noindex routes still say noindex, nothing extra is built (the capture's own scripts in `baseline/js.json` and stylesheets in `baseline/css.json` at their live paths, the live sitemap names in `baseline/sitemaps.json`, and the launch icon set are not extras); a route captured at any status other than 200, 3xx, 404 or 410 fails as `not captured: ` unless a `routes` exception skips it or names its built replacement | A `routes` exception with `to: "skip"`: that route was never captured | | `scripts` | Each kept vendor is on exactly its live pages with the same ids; a dropped one on none; no new vendor | | | `seo` | Title, description, canonical, robots, lang, h1, h2, og and JSON-LD @types equal per page; site URLs on the site's origin. Unless `PUBLIC_INDEXABLE=true`, robots must carry the staging `noindex, nofollow` and the rest of it equals live's past those two; a release build compares robots exactly | | | `sitemaps` | Every top sitemap in `dist/` (Astro's `sitemap-index.xml`, or a live name in `baseline/sitemaps.json` such as `wp-sitemap.xml` or `sitemap_index.xml`) lists exactly the indexable 200 pages on the origin (percent escapes compare case-blind: `%c3` is `%C3`); every robots.txt `Sitemap:` line is on the origin and names a file in `dist/`, or is a line the live robots.txt (`capture/robots.txt`) carried verbatim; every feed is built with as many items | | diff --git a/skills/parity-gates/references/contracts.md b/skills/parity-gates/references/contracts.md index 6e2136a..19cf825 100644 --- a/skills/parity-gates/references/contracts.md +++ b/skills/parity-gates/references/contracts.md @@ -78,6 +78,16 @@ The site's own script files the capture saved (`capture/js/manifest.json`; `file the capture), `[]` when there are none. The routes gate lets the build serve each at its live path; built or not, none is an extra. +## baseline/css.json + + [{ "href": "https://www.example.com/wp-content/themes/x/style.css?ver=1.2", "file": "css/1.css", + "pages": ["/", "/about/"] }] + +The site's own stylesheets the capture saved (`capture/css/manifest.json`, on the site's host +with or without `www.`, no inline ``; + +const document = (title, body) => ` + + + + +${escapeHtml(title)} +${STYLE} + + +${body} + + +`; + +// The route pages the last run wrote, so the next run removes only its own stale pages. +const SHEET_LIST = '.sheet.json'; + +// A file name for each route, unique within the sheet. +function pageNames(routes) { + const used = new Set(); + return new Map(routes.map((route) => { + const stem = String(route).replace(/[^A-Za-z0-9._-]+/g, '_').replace(/^[._]+/, '') || 'route'; + let name = stem; + for (let n = 2; used.has(name.toLowerCase()); n += 1) name = `${stem}-${n}`; + used.add(name.toLowerCase()); + return [route, `${name}.html`]; + })); +} + +// Writes the index at out and one page per route beside it; returns the counts. +export function writeSheet({ liveDir, newDir, diffFile, stylesFile, out }) { const live = indexManifest(liveDir); const next = indexManifest(newDir); const diff = readJson(diffFile, 'diff'); if (!Array.isArray(diff)) throw new UsageError(`${diffFile} is not a JSON array`); const styles = stylesFile ? readJson(stylesFile, 'styles') : []; + if (!Array.isArray(styles)) throw new UsageError(`${stylesFile} is not a JSON array`); const diffDir = dirname(diffFile); const diffByKey = new Map(diff.map((d) => [key(d.route, d.width), d])); - const order = []; - const seen = new Set(); + const widthsOf = new Map(); for (const { route, width } of [...diff, ...live.values(), ...next.values(), ...styles]) { - const k = key(route, width); - if (!seen.has(k)) { - seen.add(k); - order.push({ route, width }); - } + if (!widthsOf.has(route)) widthsOf.set(route, []); + if (!widthsOf.get(route).includes(width)) widthsOf.get(route).push(width); + } + const badStyles = new Map(); + for (const s of styles.filter((row) => !row.pass)) { + const k = key(s.route, s.width); + if (!badStyles.has(k)) badStyles.set(k, []); + badStyles.get(k).push(s); } - const routes = [...new Set(order.map((o) => o.route))]; + const routes = [...widthsOf.keys()]; + const names = pageNames(routes); + const stem = basename(out, extname(out)); + const pagesDir = join(dirname(out), `${stem}-pages`); + mkdirSync(pagesDir, { recursive: true }); let failing = 0; - const sections = routes.map((route) => { - const rows = order.filter((o) => o.route === route).map(({ width }) => { + let pairs = 0; + const indexRows = []; + for (const route of routes) { + const cells = []; + let routeFailing = 0; + const rows = widthsOf.get(route).map((width) => { + pairs += 1; const k = key(route, width); const a = live.get(k); const b = next.get(k); const d = diffByKey.get(k); - const relativeFile = a ? relative(liveDir, a.path) : b ? relative(newDir, b.path) : null; + const relativeFile = a?.path ? relative(liveDir, a.path) : b?.path ? relative(newDir, b.path) : null; const diffImage = d && relativeFile ? join(diffDir, relativeFile) : null; - const badStyles = styles.filter((s) => s.route === route && s.width === width && !s.pass); - if (d && !d.pass) failing += 1; - failing += badStyles.length; - const percent = Number.isFinite(d?.percent) ? `${d.percent.toFixed(2)} %` : 'n/a'; + const bad = badStyles.get(k) ?? []; + const rowFailing = (d && !d.pass ? 1 : 0) + bad.length; + routeFailing += rowFailing; + cells.push(`${escapeHtml(width)} px ${d ? `${d.pass ? 'pass' : 'fail'} ${escapeHtml(percentOf(d))}` : 'no diff'}${bad.length ? ` ${bad.length} style` : ''}`); const summary = d - ? `${verdict(d.pass)} ${escapeHtml(percent)} differing pixels heights ${escapeHtml(d.heightA ?? '?')} / ${escapeHtml(d.heightB ?? '?')} px` + ? `${verdict(d.pass)} ${escapeHtml(percentOf(d))} differing pixels heights ${escapeHtml(d.heightA ?? '?')} / ${escapeHtml(d.heightB ?? '?')} px` : `${a || b ? 'no diff' : 'style only'}`; const shots = a || b || d ? `
-${figure('Live', a?.path, a?.bannerClosed ? 'banner closed' : '')} -${figure('New', b?.path, b?.bannerClosed ? 'banner closed' : '')} -${d ? figure('Diff', diffImage) : ''} +${figure('Live', a?.path, pagesDir, a?.bannerClosed ? 'banner closed' : '')} +${figure('New', b?.path, pagesDir, b?.bannerClosed ? 'banner closed' : '')} +${d ? figure('Diff', diffImage, pagesDir) : ''}
` : ''; return `

${escapeHtml(width)} px ${summary}

${shots} -${styleTable(badStyles)} +${styleTable(bad)}
`; }); - return `

${escapeHtml(route)}

\n${rows.join('\n')}\n
`; - }); + failing += routeFailing; + const name = names.get(route); + writeFileSync(join(pagesDir, name), document(`Visual review: ${route}`, `

All routes

+

${escapeHtml(route)}

+

${rows.length} widths, ${routeFailing} failing rows.

+${rows.join('\n')} +
`)); + indexRows.push(`${escapeHtml(route)}${routeFailing ? `${routeFailing}` : '0'}${cells.join('
')}`); + } - const html = ` - - - - -Visual review - - - -

Visual review

-

${routes.length} routes, ${order.length} route-width rows, ${failing} failing rows.

-${sections.join('\n')} - - -`; - return { html, failing }; + writeFileSync(out, document('Visual review', `

Visual review

+

${routes.length} routes, ${pairs} route-width rows, ${failing} failing rows.

+ +${indexRows.join('\n')} +
RouteFailingWidths
`)); + // Route pages an earlier run wrote (listed in its .sheet.json) that this set no longer has. + const listFile = join(pagesDir, SHEET_LIST); + const written = new Set(names.values()); + let earlier = []; + try { + earlier = existsSync(listFile) ? JSON.parse(readFileSync(listFile, 'utf8')) : []; + } catch { + earlier = []; + } + const removed = (Array.isArray(earlier) ? earlier : []) + .filter((name) => typeof name === 'string' && /^[A-Za-z0-9][A-Za-z0-9._-]*\.html$/.test(name) && !written.has(name)); + for (const name of removed) rmSync(join(pagesDir, name), { force: true }); + writeFileSync(listFile, `${JSON.stringify([...written].sort(), null, 2)}\n`); + return { failing, routes: routes.length, pagesDir, removed: removed.length }; } function options(argv) { @@ -186,10 +251,9 @@ export function main(argv = process.argv.slice(2)) { console.log(HELP); return 0; } - const { html, failing } = buildSheet(opts); mkdirSync(dirname(opts.out), { recursive: true }); - writeFileSync(opts.out, html); - console.log(`Wrote ${opts.out} (${failing} failing rows)`); + const { failing, routes, pagesDir, removed } = writeSheet(opts); + console.log(`Wrote ${opts.out} and ${routes} route pages in ${pagesDir} (${failing} failing rows)${removed ? `; removed ${removed} route page${removed === 1 ? '' : 's'} no longer in the set` : ''}`); return 0; } catch (err) { if (!(err instanceof UsageError)) throw err; diff --git a/skills/visual-review/scripts/shoot.mjs b/skills/visual-review/scripts/shoot.mjs index 69ee374..d143ed4 100644 --- a/skills/visual-review/scripts/shoot.mjs +++ b/skills/visual-review/scripts/shoot.mjs @@ -9,10 +9,11 @@ import { fileURLToPath } from 'node:url'; import { assertNodeVersion } from './lib/node-version.mjs'; import { capture, launchBrowser, NavigationError, newShotContext, pngHeight } from './lib/browser.mjs'; import { readRoutes, RoutesError } from './lib/routes.mjs'; +import { parseBlockHosts } from './lib/blocked-hosts.mjs'; const HELP = `Usage: node shoot.mjs --base --routes --out [--widths 390,768,1440] [--delay 500] [--wait 1500] [--full] [--scale 1] - [--allow-analytics] + [--allow-analytics] [--block-hosts ] Takes a full-page Chromium screenshot of every route at every width and writes //.png and /manifest.json @@ -34,6 +35,9 @@ Takes a full-page Chromium screenshot of every route at every width and writes --allow-analytics let analytics, tag, pixel and chat hosts load (blocked by default: they do not draw the page, and a screenshot run must not count as visits in the site's reports) + --block-hosts also block these hosts (and their subdomains) for this run, with + or without --allow-analytics: a consent manager such as cmp.osano.com that + turns the theme's scripts off when the page is not on the live host Motion is frozen (prefers-reduced-motion plus injected CSS), scroll-reveal content is forced visible, lazy images are loaded by scrolling to the bottom and @@ -86,6 +90,7 @@ function parse(argv) { full: { type: 'boolean', default: false }, scale: { type: 'string', default: '1' }, 'allow-analytics': { type: 'boolean', default: false }, + 'block-hosts': { type: 'string' }, help: { type: 'boolean', default: false }, }, })); @@ -112,16 +117,26 @@ function parse(argv) { wait: number(values.wait, 'wait', { min: 0, max: 600000 }), scale: number(values.scale, 'scale', { min: 0.1, max: 1 }), allowAnalytics: values['allow-analytics'], + blockHosts: blockHostsFrom(values['block-hosts']), }; } +function blockHostsFrom(value) { + if (value === undefined) return []; + try { + return parseBlockHosts(value); + } catch (error) { + throw new UsageError(error.message); + } +} + const sleep = (ms) => new Promise((done) => setTimeout(done, ms)); const failedEntry = (route, width, error) => ({ route, width, file: null, heightPx: null, bannerClosed: false, scrollHeight: null, heightMismatch: false, error, }); -async function shoot({ base, routes, widths, out, delay, wait, scale, allowAnalytics }) { +async function shoot({ base, routes, widths, out, delay, wait, scale, allowAnalytics, blockHosts }) { const manifest = []; let stopped = false; let pace = delay; @@ -130,7 +145,7 @@ async function shoot({ base, routes, widths, out, delay, wait, scale, allowAnaly const browser = await launchBrowser(); try { widths: for (const width of widths) { - const context = await newShotContext(browser, { width, scale, allowAnalytics }); + const context = await newShotContext(browser, { width, scale, allowAnalytics, blockHosts }); let page = await context.newPage(); for (const { route, path } of routes) { const url = new URL(path, base).href; diff --git a/skills/visual-review/scripts/styles.mjs b/skills/visual-review/scripts/styles.mjs index 7f10d9d..4cfce85 100644 --- a/skills/visual-review/scripts/styles.mjs +++ b/skills/visual-review/scripts/styles.mjs @@ -6,13 +6,13 @@ import { dirname, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { parseArgs } from 'node:util'; import { assertNodeVersion } from './lib/node-version.mjs'; -import { blockAnalytics } from './lib/blocked-hosts.mjs'; +import { blockAnalytics, parseBlockHosts } from './lib/blocked-hosts.mjs'; import { readRoutes, RoutesError } from './lib/routes.mjs'; import { LENGTH_TOLERANCE_PX, PROPERTIES, SELECTORS } from './lib/selectors.mjs'; const HELP = `Usage: node styles.mjs --live --new --routes [--widths 390,1440] [--out styles.json] [--delay 500] - [--allow-analytics] + [--allow-analytics] [--block-hosts ] Opens every route at every width on both origins and compares, on the first match of each sampled selector, font-family (first family), font-size, @@ -30,6 +30,10 @@ Lengths pass within ${LENGTH_TOLERANCE_PX} px; colours and families must match e --delay milliseconds between page loads (default 500) --allow-analytics let analytics, tag, pixel and chat hosts load on both origins (blocked by default) + --block-hosts also block these hosts (and their subdomains) on both + origins for this run, with or without --allow-analytics: a consent + manager such as cmp.osano.com that turns the theme's scripts off when + the page is not on the live host Exit codes: 0 every row passes, 1 a row fails, 2 usage or environment error.`; @@ -153,6 +157,15 @@ export function compareSamples({ route, path }, width, liveSample, nextSample) { return rows; } +function blockHostsFrom(value) { + if (value === undefined) return []; + try { + return parseBlockHosts(value); + } catch (error) { + throw new UsageError(error.message); + } +} + function options(argv) { let values; try { @@ -166,6 +179,7 @@ function options(argv) { out: { type: 'string', default: 'styles.json' }, delay: { type: 'string', default: '500' }, 'allow-analytics': { type: 'boolean', default: false }, + 'block-hosts': { type: 'string' }, help: { type: 'boolean', short: 'h' }, }, })); @@ -189,7 +203,10 @@ function options(argv) { if (!Number.isFinite(delay) || delay < 0) throw new UsageError('--delay must be a number of milliseconds'); const routes = parseRoutes(values.routes); if (routes.length === 0) throw new UsageError('--routes names no routes'); - return { live: values.live, next: values.new, routes, widths, out: values.out, delay, allowAnalytics: values['allow-analytics'] }; + return { + live: values.live, next: values.new, routes, widths, out: values.out, delay, + allowAnalytics: values['allow-analytics'], blockHosts: blockHostsFrom(values['block-hosts']), + }; } export async function main(argv = process.argv.slice(2)) { @@ -222,7 +239,7 @@ export async function main(argv = process.argv.slice(2)) { for (const target of opts.routes) { for (const width of opts.widths) { const context = await browser.newContext({ viewport: { width, height: VIEWPORT_HEIGHT }, userAgent: USER_AGENT }); - if (!opts.allowAnalytics) await blockAnalytics(context); + await blockAnalytics(context, { extra: opts.blockHosts, analytics: !opts.allowAnalytics }); try { const samples = []; for (const origin of [opts.live, opts.next]) { diff --git a/test/capture/media.test.mjs b/test/capture/media.test.mjs index 9450fb2..d99260f 100644 --- a/test/capture/media.test.mjs +++ b/test/capture/media.test.mjs @@ -547,12 +547,182 @@ test('--max-file-bytes records a larger file "too-large" and saves the rest, str } }); +// An image host that serves every size at the same path with a query (WordPress VIP, Jetpack +// Photon): the bare path is the original, each ?w= / ?resize= is a smaller copy. +const ORIGINAL = Buffer.from('the full-size original bytes'); +const resizedBody = (query) => Buffer.from(`resized ${query}`); +function queryHost({ bareStatus = 200 } = {}) { + return serve((req, res) => { + const [, query = ''] = req.url.split('?'); + if (!query && bareStatus !== 200) { + res.writeHead(bareStatus); + res.end(); + return; + } + const body = query ? resizedBody(query) : ORIGINAL; + res.writeHead(200, { 'Content-Type': 'image/jpeg', 'Content-Length': body.length }); + res.end(req.method === 'HEAD' ? undefined : body); + }); +} + +test('a path seen only with query-string sizes downloads the bare original and lists each variant', async () => { + const server = await queryHost(); + try { + const dir = tinyCapture(server.origin, ''); + const files = join(dir, 'files'); + const result = await run(MEDIA, ['--capture', dir, '--download', files, '--delay', '0']); + assert.equal(result.code, 0, result.stderr); + const [entry, ...rest] = readJson(join(dir, 'media.json')); + assert.deepEqual(rest, []); + assert.deepEqual(entry, { + url: `${server.origin}/up/photo.jpg`, path: '/up/photo.jpg', source: ['html', 'srcset'], referencedBy: ['/'], + variants: [{ query: 'quality=99&w=1024', referencedBy: ['/'] }, { query: 'w=300', referencedBy: ['/'] }], + status: 200, contentType: 'image/jpeg', bytes: ORIGINAL.length, + }); + assert.deepEqual(server.requests, ['GET /up/photo.jpg'], 'only the original is requested'); + assert.deepEqual(readFileSync(join(files, 'up/photo.jpg')), ORIGINAL); + assert.match(result.stdout, /^2 query-string variants of 1 file \(listed only\)$/m); + + server.requests.length = 0; + const all = await run(MEDIA, ['--capture', dir, '--download', files, '--delay', '0', '--sizes', 'all']); + assert.equal(all.code, 0, all.stderr); + assert.deepEqual(server.requests.sort(), ['GET /up/photo.jpg?quality=99&w=1024', 'GET /up/photo.jpg?w=300'], 'the original already saved is kept'); + const [again] = readJson(join(dir, 'media.json')); + assert.equal(again.bytes, ORIGINAL.length); + for (const query of ['quality=99&w=1024', 'w=300']) { + const variant = again.variants.find((v) => v.query === query); + assert.deepEqual(variant, { query, referencedBy: ['/'], status: 200, contentType: 'image/jpeg', bytes: resizedBody(query).length, downloaded: true }); + assert.deepEqual(readFileSync(join(`${files}-variants`, `up/photo.jpg%3F${query}`)), resizedBody(query)); + } + assert.deepEqual(readFileSync(join(files, 'up/photo.jpg')), ORIGINAL, 'the original is not overwritten by a variant'); + assert.deepEqual(readdirSync(files, { recursive: true }).sort(), ['up', join('up', 'photo.jpg')], 'the upload cache holds bare paths only'); + assert.match(all.stdout, /^2 query-string variants of 1 file \(downloaded\)$/m); + assert.match(all.stdout, /^Variants saved under .*files-variants \(not part of the media cache\)\.$/m); + } finally { + await server.close(); + } +}); + +test('a query-string image whose bare path does not answer 200 falls back to the largest variant, marked resized', async () => { + const server = await queryHost({ bareStatus: 404 }); + try { + const dir = tinyCapture(server.origin, ''); + const files = join(dir, 'files'); + const result = await run(MEDIA, ['--capture', dir, '--download', files, '--delay', '0']); + assert.equal(result.code, 0, result.stderr); + const media = readJson(join(dir, 'media.json')); + const a = media.find((m) => m.path === '/up/a.jpg'); + assert.equal(a.url, `${server.origin}/up/a.jpg`); + assert.equal(a.status, 200); + assert.equal(a.resized, true); + assert.equal(a.downloadedFrom, `${server.origin}/up/a.jpg?resize=1024,683`); + assert.equal(a.bytes, resizedBody('resize=1024,683').length); + assert.deepEqual(readFileSync(join(files, 'up/a.jpg')), resizedBody('resize=1024,683')); + // No width to compare: the first variant seen. + const b = media.find((m) => m.path === '/up/b.jpg'); + assert.equal(b.resized, true); + assert.equal(b.downloadedFrom, `${server.origin}/up/b.jpg?crop=1`); + assert.match(result.stderr, /\/up\/a\.jpg: the original answered 404; saved the resized copy \?resize=1024,683 instead/); + assert.match(result.stdout, /^2 originals unavailable \(resized: true\): a resized copy was saved in their place$/m); + } finally { + await server.close(); + } +}); + +test('a bare path that fails with only a cache-busting variant saves that one, not marked resized', async () => { + const server = await queryHost({ bareStatus: 404 }); + try { + const dir = tinyCapture(server.origin, ''); + const files = join(dir, 'files'); + const result = await run(MEDIA, ['--capture', dir, '--download', files, '--delay', '0']); + assert.equal(result.code, 0, result.stderr); + const [d] = readJson(join(dir, 'media.json')); + assert.equal(d.status, 200); + assert.equal(d.resized, undefined); + assert.equal(d.downloadedFrom, `${server.origin}/up/d.jpg?ver=3`); + assert.deepEqual(readFileSync(join(files, 'up/d.jpg')), resizedBody('ver=3')); + assert.match(result.stderr, /\/up\/d\.jpg: the original answered 404; saved \?ver=3 instead \(a cache-busting query, not a resize\)/); + assert.doesNotMatch(result.stdout, /originals? unavailable/); + } finally { + await server.close(); + } +}); + +test('an original over --max-file-bytes is too-large, never replaced by a variant, and re-asked with a bigger cap', async () => { + const server = await queryHost(); + try { + const dir = tinyCapture(server.origin, ''); + const files = join(dir, 'files'); + const capped = await run(MEDIA, ['--capture', dir, '--download', files, '--delay', '0', '--max-file-bytes', '10']); + assert.equal(capped.code, 0, capped.stderr); + const [small] = readJson(join(dir, 'media.json')); + assert.equal(small.status, 'too-large'); + assert.equal(small.resized, undefined); + assert.equal(small.downloadedFrom, undefined); + assert.deepEqual(server.requests, ['GET /up/photo.jpg'], 'no variant is asked for'); + + server.requests.length = 0; + const raised = await run(MEDIA, ['--capture', dir, '--download', files, '--delay', '0']); + assert.equal(raised.code, 0, raised.stderr); + assert.deepEqual(server.requests, ['GET /up/photo.jpg']); + const [full] = readJson(join(dir, 'media.json')); + assert.equal(full.status, 200); + assert.equal(full.bytes, ORIGINAL.length); + assert.deepEqual(readFileSync(join(files, 'up/photo.jpg')), ORIGINAL); + } finally { + await server.close(); + } +}); + +test('a bare path that redirects is recorded with its location, not replaced by a variant', async () => { + const server = await serve((req, res) => { + if (req.url === '/up/moved.jpg') { + res.writeHead(301, { Location: '/up/real.jpg' }); + res.end(); + return; + } + res.writeHead(200, { 'Content-Type': 'image/jpeg', 'Content-Length': ORIGINAL.length }); + res.end(req.method === 'HEAD' ? undefined : ORIGINAL); + }); + try { + const dir = tinyCapture(server.origin, ''); + const result = await run(MEDIA, ['--capture', dir, '--download', join(dir, 'files'), '--delay', '0']); + assert.equal(result.code, 0, result.stderr); + const [moved] = readJson(join(dir, 'media.json')); + assert.equal(moved.status, 301); + assert.equal(moved.location, `${server.origin}/up/real.jpg`); + assert.equal(moved.resized, undefined); + assert.equal(moved.downloadedFrom, undefined); + assert.ok(!server.requests.some((r) => r.includes('?')), server.requests.join(', ')); + } finally { + await server.close(); + } +}); + +test('a query-string image with no working bare path or variant keeps the original answer', async () => { + const server = await serve((req, res) => { + res.writeHead(404); + res.end(); + }); + try { + const dir = tinyCapture(server.origin, ''); + const result = await run(MEDIA, ['--capture', dir, '--delay', '0']); + assert.equal(result.code, 0, result.stderr); + const [c] = readJson(join(dir, 'media.json')); + assert.equal(c.status, 404); + assert.equal(c.resized, undefined); + } finally { + await server.close(); + } +}); + test('media.mjs usage errors exit 2 and --help prints usage', async () => { const help = await run(MEDIA, ['--help']); assert.equal(help.code, 0); assert.match(help.stdout, /Usage: node media\.mjs --capture /); assert.match(help.stdout, /keyed by path/); - assert.match(help.stdout, /first URL/); + assert.match(help.stdout, /bare path/); + assert.match(help.stdout, /resized: true/); assert.equal((await run(MEDIA, [])).code, 2); assert.equal((await run(MEDIA, ['--capture', join(work, 'missing')])).code, 2); }); diff --git a/test/convert/copy-assets.test.mjs b/test/convert/copy-assets.test.mjs new file mode 100644 index 0000000..56dbe2d --- /dev/null +++ b/test/convert/copy-assets.test.mjs @@ -0,0 +1,175 @@ +import { test, before, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, relative, resolve } from 'node:path'; +import { startFixture } from '../fixture/serve.mjs'; + +const SKILLS = resolve(import.meta.dirname, '../../skills'); +const COPY = join(SKILLS, 'convert-site', 'scripts', 'copy-assets.mjs'); + +function run(script, args) { + return new Promise((done) => { + execFile(process.execPath, [script, ...args], (err, stdout, stderr) => done({ code: err ? err.code : 0, stdout, stderr })); + }); +} + +const files = (dir) => (existsSync(dir) + ? readdirSync(dir, { recursive: true, withFileTypes: true }).filter((d) => d.isFile()).map((d) => relative(dir, join(d.parentPath, d.name))).sort() + : []); + +const CONCAT_SRC = '/_static/??/wp-content/themes/fixture/js/util.js,/wp-content/themes/fixture/js/reveal.js?m=1'; +const OFF_SITE_FONT = 'https://fonts.example.com/s/display/v1/display-regular.woff2'; +let root; +let origin; + +before(async () => { + root = mkdtempSync(join(tmpdir(), 'copy-assets-test-')); + const capture = join(root, 'capture'); + const fixture = await startFixture(); + origin = fixture.origin; + try { + for (const [name, args] of [ + ['capture.mjs', [fixture.origin, '--out', capture, '--delay', '0', '--no-wayback']], + ['assets.mjs', ['--capture', capture, '--delay', '0']], + ['media.mjs', ['--capture', capture, '--delay', '0', '--download', join(root, 'cache')]], + ]) { + const result = await run(join(SKILLS, 'capture-site', 'scripts', name), args); + assert.equal(result.code, 0, `${name}: ${result.stderr}`); + } + } finally { + await fixture.close(); + } + // A WordPress VIP concatenation and a font from a font service, as a larger site has them. + const manifestFile = join(capture, 'js', 'manifest.json'); + const manifest = JSON.parse(readFileSync(manifestFile, 'utf8')); + writeFileSync(join(capture, 'js', 'concat-test.js'), 'console.log("concat");\n'); + manifest.push({ file: 'concat-test.js', src: `${origin}${CONCAT_SRC}`, kind: 'script', pages: ['/'] }); + writeFileSync(manifestFile, JSON.stringify(manifest)); + const indexFile = join(capture, 'assets', 'index.json'); + const index = JSON.parse(readFileSync(indexFile, 'utf8')); + writeFileSync(join(capture, 'assets', 'fonts', 'display-regular.woff2'), 'font'); + index.push({ kind: 'font', url: OFF_SITE_FONT, file: 'assets/fonts/display-regular.woff2', status: 200, note: null }); + writeFileSync(indexFile, JSON.stringify(index)); +}); +after(() => rmSync(root, { recursive: true, force: true })); + +// A fresh site folder holding the capture, the media download and a media plan. +function site(name, plan) { + const dir = join(root, name); + cpSync(join(root, 'capture'), join(dir, 'capture'), { recursive: true }); + cpSync(join(root, 'cache'), join(dir, '.conversion', 'media-cache'), { recursive: true }); + mkdirSync(join(dir, 'docs'), { recursive: true }); + writeFileSync(join(dir, 'docs', 'media-plan.json'), JSON.stringify(plan)); + return dir; +} +const copy = (dir, extra = []) => run(COPY, ['--capture', join(dir, 'capture'), '--site', dir, ...extra]); + +test('--help prints usage and exits 0', async () => { + const { code, stdout } = await run(COPY, ['--help']); + assert.equal(code, 0); + assert.match(stdout, /Usage: node copy-assets\.mjs/); +}); + +test('copies scripts, fonts, root icons and the media outside the prefix to public/ at their live paths', async () => { + const dir = site('store', { mode: 'store', prefix: '/wp-content/uploads/' }); + const result = await copy(dir); + assert.equal(result.code, 0, result.stderr); + const pub = join(dir, 'public'); + const capture = join(dir, 'capture'); + for (const entry of JSON.parse(readFileSync(join(capture, 'js', 'manifest.json'), 'utf8')).filter((e) => e.file && !e.src.includes('??'))) { + const target = join(pub, decodeURIComponent(new URL(entry.src).pathname)); + assert.ok(readFileSync(target).equals(readFileSync(join(capture, 'js', entry.file))), entry.src); + } + assert.ok(existsSync(join(pub, 'wp-content/themes/fixture/js/view.js')), 'the module script'); + assert.ok(existsSync(join(pub, 'wp-content/themes/fixture/js/reveal.js')), 'the classic script'); + for (const font of ['fixture-sans.woff2', 'fixture-serif.woff2']) { + assert.ok(existsSync(join(pub, 'wp-content/themes/fixture/fonts', font)), font); + } + assert.equal(readFileSync(join(pub, 'fonts', 'display-regular.woff2'), 'utf8'), 'font', 'a font service font goes under /fonts/'); + assert.ok(readFileSync(join(pub, 'favicon.ico')).equals(readFileSync(join(capture, 'assets', 'root-favicon.ico'))), 'the root favicon'); + // Nothing under the media prefix: the store or the rewrite serves it. + assert.deepEqual(files(join(pub, 'wp-content', 'uploads')), []); + // The concatenation is one file at /_static/concat/.js, listed in the manifest. + const hash = createHash('sha1').update(CONCAT_SRC).digest('hex'); + assert.equal(readFileSync(join(pub, '_static', 'concat', `${hash}.js`), 'utf8'), 'console.log("concat");\n'); + assert.deepEqual(JSON.parse(readFileSync(join(pub, '_static', 'concat', 'manifest.json'), 'utf8')), { [CONCAT_SRC]: `/_static/concat/${hash}.js` }); + assert.match(result.stdout, /scripts: \d+ \(1 concatenated\)/); + assert.match(result.stdout, /fonts: 3/); + assert.match(result.stdout, /root icons: 1/); + assert.match(result.stdout, /media outside \/wp-content\/uploads\/: \d+/); + + const before = files(pub).map((f) => [f, readFileSync(join(pub, f)).toString('base64')]); + const again = await copy(dir); + assert.equal(again.code, 0, again.stderr); + assert.equal(again.stdout, result.stdout, 'the same counts on a second run'); + assert.deepEqual(files(pub).map((f) => [f, readFileSync(join(pub, f)).toString('base64')]), before, 'idempotent'); +}); + +test('mode repo copies every downloaded media file, the prefix included', async () => { + const dir = site('repo', { mode: 'repo', prefix: '/wp-content/uploads/' }); + const result = await copy(dir); + assert.equal(result.code, 0, result.stderr); + assert.deepEqual(files(join(dir, 'public', 'wp-content', 'uploads')), files(join(dir, '.conversion', 'media-cache', 'wp-content', 'uploads'))); + assert.ok(files(join(dir, 'public', 'wp-content', 'uploads')).length > 10); + assert.match(result.stdout, /media \(repo mode, every file\): \d+/); +}); + +test('a listed file missing from the capture exits 1 and names it; the rest is still copied', async () => { + const dir = site('missing', { mode: 'store', prefix: '/wp-content/uploads/' }); + const manifest = JSON.parse(readFileSync(join(dir, 'capture', 'js', 'manifest.json'), 'utf8')); + const reveal = manifest.find((e) => e.src.includes('reveal.js') && !e.src.includes('??')); + rmSync(join(dir, 'capture', 'js', reveal.file)); + const result = await copy(dir); + assert.equal(result.code, 1); + assert.match(result.stderr, new RegExp(`missing: .*js/${reveal.file.replace('.', '\\.')}`)); + assert.ok(existsSync(join(dir, 'public', 'wp-content/themes/fixture/js/view.js'))); +}); + +test('media on disk without a media plan is a usage error', async () => { + const dir = site('noplan', { mode: 'store', prefix: '/wp-content/uploads/' }); + rmSync(join(dir, 'docs', 'media-plan.json')); + const result = await copy(dir); + assert.equal(result.code, 2); + assert.match(result.stderr, /media plan/); +}); + +test('a file the capture names outside the capture folder is refused with exit 1', async () => { + const dir = site('traversal', { mode: 'store', prefix: '/wp-content/uploads/' }); + writeFileSync(join(dir, '.env'), 'SECRET=1\n'); + const file = join(dir, 'capture', 'js', 'manifest.json'); + const manifest = JSON.parse(readFileSync(file, 'utf8')); + manifest.push({ file: '../../.env', src: `${origin}/wp-content/themes/fixture/js/env.js`, kind: 'script', pages: ['/'] }); + writeFileSync(file, JSON.stringify(manifest)); + const result = await copy(dir); + assert.equal(result.code, 1); + assert.match(result.stderr, /refused: script .*env\.js: .*\.env: outside /); + assert.ok(!existsSync(join(dir, 'public', 'wp-content/themes/fixture/js/env.js')), 'nothing written for it'); + assert.ok(existsSync(join(dir, 'public', 'wp-content/themes/fixture/js/view.js')), 'the rest is copied'); +}); + +test('nothing under an excluded prefix is copied, and the media prefix counts without its trailing slash', async () => { + const dir = site('excluded', { mode: 'store', prefix: '/wp-content/uploads' }); + const metaFile = join(dir, 'capture', 'meta.json'); + const meta = JSON.parse(readFileSync(metaFile, 'utf8')); + writeFileSync(metaFile, JSON.stringify({ ...meta, excluded: ['/wp-content/themes/fixture/fonts/', '/old-site'] })); + const cache = join(dir, '.conversion', 'media-cache'); + mkdirSync(join(cache, 'old-site', 'files'), { recursive: true }); + writeFileSync(join(cache, 'old-site', 'files', 'a.pdf'), 'pdf'); + mkdirSync(join(cache, 'wp-content', 'uploads-old'), { recursive: true }); + writeFileSync(join(cache, 'wp-content', 'uploads-old', 'b.jpg'), 'jpg'); + const result = await copy(dir); + assert.equal(result.code, 0, result.stderr); + const pub = join(dir, 'public'); + assert.deepEqual(files(join(pub, 'wp-content', 'themes', 'fixture', 'fonts')), [], 'excluded fonts'); + assert.ok(!existsSync(join(pub, 'old-site')), 'an excluded media folder'); + assert.match(result.stdout, /excluded: \d+ \(under a prefix the capture left out: \/wp-content\/themes\/fixture\/fonts\/, \/old-site\)/); + assert.deepEqual(files(join(pub, 'wp-content', 'uploads')), [], 'the prefix written without its slash still covers uploads/'); + assert.equal(readFileSync(join(pub, 'wp-content', 'uploads-old', 'b.jpg'), 'utf8'), 'jpg', 'but not a sibling folder that starts the same'); +}); + +test('convert-site carries a byte-identical copy of the gates\' excluded.mjs', () => { + assert.ok(readFileSync(join(SKILLS, 'convert-site', 'scripts', 'lib', 'excluded.mjs')).equals(readFileSync(join(SKILLS, 'parity-gates', 'scripts', 'lib', 'excluded.mjs')))); +}); diff --git a/test/convert/css.test.mjs b/test/convert/css.test.mjs index 867c26c..b6e3a69 100644 --- a/test/convert/css.test.mjs +++ b/test/convert/css.test.mjs @@ -7,7 +7,7 @@ import { join, resolve } from 'node:path'; import * as csstree from 'css-tree'; import { startFixture } from '../fixture/serve.mjs'; import { mergeRanges } from '../../skills/convert-site/scripts/css-coverage.mjs'; -import { mediaTested, pruneCss, rewriteRef } from '../../skills/convert-site/scripts/css-prune.mjs'; +import { mediaTested, pruneCss, rewriteRef, variantPath } from '../../skills/convert-site/scripts/css-prune.mjs'; import { stateBases } from '../../skills/convert-site/scripts/lib/state-selectors.mjs'; import { scopeInlineRules } from '../../skills/convert-site/scripts/lib/inline-scope.mjs'; @@ -163,7 +163,7 @@ test('coverage hovers the first link a person could point at, not a clipped skip test('pruning removes the unused rules, keeps pseudo rules and @font-face, and copies kept source as written', async () => { const out = join(TMP, 'site.css'); const report = join(TMP, 'prune-report.json'); - const result = await pruneCli(['--css', captureCss('capture-css'), '--coverage', COVERAGE, '--out', out, '--report', report]); + const result = await pruneCli(['--single-sheet', '--css', captureCss('capture-css'), '--coverage', COVERAGE, '--out', out, '--report', report]); assert.equal(result.code, 0, result.stderr); const css = readFileSync(out, 'utf8'); for (const selector of LEFTOVER) assert.ok(!css.includes(selector), `${selector} removed`); @@ -184,7 +184,7 @@ test('pruning removes the unused rules, keeps pseudo rules and @font-face, and c assert.deepEqual(errors, []); const summary = readJson(report); - assert.deepEqual(Object.keys(summary).sort(), ['after', 'before', 'keptBecause', 'pageSheets', 'removedRules', 'uncoveredInline', 'unmatched', 'unresolvedImports']); + assert.deepEqual(Object.keys(summary).sort(), ['after', 'before', 'keptBecause', 'mode', 'pageSheets', 'removedRules', 'uncoveredInline', 'unmatched', 'unparsed', 'unresolvedImports']); assert.deepEqual(summary.pageSheets, []); assert.deepEqual(Object.keys(summary.keptBecause).sort(), ['atRule', 'pseudo', 'state', 'untestedMedia']); assert.deepEqual(summary.unmatched, []); @@ -198,7 +198,7 @@ test('without a manifest, coverage URLs match files by basename and still resolv const out = join(TMP, 'basename.css'); const report = join(TMP, 'basename-report.json'); const dir = captureCss('capture-basename', { file: 'style.css', manifest: false }); - const result = await pruneCli(['--css', dir, '--coverage', COVERAGE, '--out', out, '--report', report]); + const result = await pruneCli(['--single-sheet', '--css', dir, '--coverage', COVERAGE, '--out', out, '--report', report]); assert.equal(result.code, 0, result.stderr); assert.match(result.stderr, /no manifest\.json/); const css = readFileSync(out, 'utf8'); @@ -213,7 +213,7 @@ test('coverage keys match the manifest on path and query whatever their origin', const rekeyed = Object.fromEntries(Object.entries(coverage).map(([key, value]) => [key.replace(fixture.origin, 'https://staging.example.com'), value])); writeFileSync(moved, JSON.stringify(rekeyed)); const out = join(TMP, 'other-origin.css'); - const result = await pruneCli(['--css', captureCss('capture-other-origin'), '--coverage', moved, '--out', out, '--report', join(TMP, 'other-origin.json')]); + const result = await pruneCli(['--single-sheet', '--css', captureCss('capture-other-origin'), '--coverage', moved, '--out', out, '--report', join(TMP, 'other-origin.json')]); assert.equal(result.code, 0, result.stderr); const css = readFileSync(out, 'utf8'); assert.ok(css.includes('.site-header {')); @@ -227,7 +227,7 @@ test('a file with no matching coverage is kept whole and the exit code is 1', as }); const out = join(TMP, 'unmatched.css'); const report = join(TMP, 'unmatched-report.json'); - const result = await pruneCli(['--css', dir, '--coverage', COVERAGE, '--out', out, '--report', report]); + const result = await pruneCli(['--single-sheet', '--css', dir, '--coverage', COVERAGE, '--out', out, '--report', report]); assert.equal(result.code, 1); assert.match(result.stderr, /2\.css/); const css = readFileSync(out, 'utf8'); @@ -250,7 +250,7 @@ test('inline `, + '', + ``, + ].join('\n') }, + { path: '/about/', slug: 'about', head: [ + ``, + '', + '', + '', + ``, + ].join('\n') }, + ], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ + // A rule only /about/ uses is used all the same: coverage merges every page's ranges. + [`${SITE_ORIGIN}/wp-content/themes/t/a.css?ver=1.2`]: { text: THEME_A, usedRanges: usedRules(THEME_A, ['.a-home', '.a-about']) }, + [`${SITE_ORIGIN}/wp-includes/css/b.css`]: { text: BLOCKS_B, usedRanges: usedRules(BLOCKS_B, ['.b-home']) }, + // Chromium applies no print rule on screen: coverage sees nothing used in the print sheet. + [`${SITE_ORIGIN}/wp-content/themes/t/print.css`]: { text: PRINT, usedRanges: [] }, + 'inline:1': { text: HOME_STYLE, usedRanges: usedRules(HOME_STYLE, ['.s-home']), pages: ['/'] }, + 'inline:2': { text: SHARED_STYLE, usedRanges: usedRules(SHARED_STYLE, ['.layout']), pages: ['/', '/about/'] }, + 'inline:3': { text: ABOUT_STYLE, usedRanges: [[0, ABOUT_STYLE.length]], pages: ['/about/'] }, + })); + return { dir, coverage }; +} + +const perSheetCli = (dir, coverage, extra = []) => pruneCli([ + '--css', join(dir, 'css'), '--coverage', coverage, '--report', join(dir, 'report.json'), + '--public', join(dir, 'public'), '--inline-out', join(dir, 'inline'), ...extra, +]); + +test('per-sheet: each linked sheet is pruned on its own at its live path, inline blocks per page, and each page keeps its head order', async () => { + const { dir, coverage } = twoSheetCapture('per-sheet'); + const pub = join(dir, 'public'); + const inlineOut = join(dir, 'inline'); + mkdirSync(inlineOut, { recursive: true }); + writeFileSync(join(inlineOut, 'inline-stale-1.css'), '.old { color: red; }\n'); + const result = await perSheetCli(dir, coverage); + assert.equal(result.code, 0, result.stderr); + + // Live paths, query string dropped from the file name. + const a = readFileSync(join(pub, 'wp-content', 'themes', 't', 'a.css'), 'utf8'); + assert.ok(a.includes('.a-home { color: red; }') && a.includes('.a-about { color: green; }'), a); + assert.ok(!a.includes('.a-unused'), 'an unused rule is removed'); + const b = readFileSync(join(pub, 'wp-includes', 'css', 'b.css'), 'utf8'); + assert.equal(b, '.b-home { margin: 0; }\n', 'one sheet\'s rules never land in another'); + // A print sheet: coverage is screen-only, so it is kept whole and its link keeps media="print". + assert.equal(readFileSync(join(pub, 'wp-content', 'themes', 't', 'print.css'), 'utf8'), PRINT); + + const summary = readJson(join(dir, 'report.json')); + assert.equal(summary.mode, 'per-sheet'); + assert.equal(summary.siteCss, null, 'nothing is left for a global site.css'); + assert.deepEqual(summary.sheets.map(({ href, file }) => ({ href, file })), [ + { href: `${SITE_ORIGIN}/wp-content/themes/t/a.css?ver=1.2`, file: join(pub, 'wp-content', 'themes', 't', 'a.css') }, + { href: `${SITE_ORIGIN}/wp-includes/css/b.css`, file: join(pub, 'wp-includes', 'css', 'b.css') }, + { href: `${SITE_ORIGIN}/wp-content/themes/t/print.css`, file: join(pub, 'wp-content', 'themes', 't', 'print.css') }, + ]); + for (const sheet of summary.sheets) { + assert.deepEqual(Object.keys(sheet).sort(), ['bytesAfter', 'bytesBefore', 'file', 'href', 'rulesRemoved']); + assert.equal(sheet.bytesAfter, statSync(sheet.file).size); + } + assert.deepEqual(summary.sheets.map((s) => s.rulesRemoved), [1, 1, 0]); + assert.deepEqual(summary.external, ['https://fonts.googleapis.com/css2?family=Inter']); + + // Each page's own order, links root-relative with their query, inline blocks as files. + assert.deepEqual(summary.heads['/'], [ + { kind: 'link', href: '/wp-content/themes/t/a.css?ver=1.2', media: 'all' }, + { kind: 'inline', file: 'inline-home-1.css' }, + { kind: 'link', href: '/wp-includes/css/b.css' }, + { kind: 'inline', file: 'inline-home-2.css' }, + ]); + assert.deepEqual(summary.heads['/about/'], [ + { kind: 'inline', file: 'inline-home-2.css' }, + { kind: 'link', href: 'https://fonts.googleapis.com/css2?family=Inter' }, + { kind: 'link', href: '/wp-content/themes/t/print.css', media: 'print' }, + { kind: 'link', href: '/wp-content/themes/t/a.css?ver=1.2', media: 'all' }, + { kind: 'inline', file: 'inline-about-2.css' }, + ]); + assert.deepEqual(readdirSync(inlineOut).sort(), ['inline-about-2.css', 'inline-home-1.css', 'inline-home-2.css'], 'stale inline files removed'); + assert.equal(readFileSync(join(inlineOut, 'inline-home-1.css'), 'utf8'), '.s-home { padding: 0; }\n'); + assert.equal(readFileSync(join(inlineOut, 'inline-home-2.css'), 'utf8'), '.layout { max-width: 640px; }\n'); + assert.equal(readFileSync(join(inlineOut, 'inline-about-2.css'), 'utf8'), `${ABOUT_STYLE}\n`); + assert.deepEqual(summary.inline.map(({ file, pages }) => ({ file, pages })), [ + { file: 'inline-home-1.css', pages: ['/'] }, + { file: 'inline-home-2.css', pages: ['/', '/about/'] }, + { file: 'inline-about-2.css', pages: ['/about/'] }, + ]); + assert.deepEqual(summary.pageSheets, [], 'each page keeps its own blocks, so nothing collides'); + assert.deepEqual(summary.unmatched, []); + assert.deepEqual(summary.unparsed, []); + assert.match(result.stdout, /wrote 3 sheet\(s\) at their live paths under .*public and 3 inline block file\(s\) to .*inline; no site\.css/); +}); + +test('per-sheet: identical sheets under two paths (a multisite) keep their own coverage, never each other\'s', async () => { + const text = '.c-block { margin-bottom: 2rem; }\n.b-gallery { margin-bottom: 0; }\n'; + const main = `${SITE_ORIGIN}/wp-content/themes/x/style.css`; + const sub = `${SITE_ORIGIN}/president/wp-content/themes/x/style.css`; + const dir = pageCapture('multisite', { + sheets: [{ file: '1.css', href: main, text }, { file: '2.css', href: sub, text }], + pages: [ + { path: '/', slug: 'home', head: '' }, + { path: '/president/', slug: 'president', head: '' }, + ], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ + [main]: { text, usedRanges: usedRules(text, ['.c-block']) }, + [sub]: { text, usedRanges: usedRules(text, ['.b-gallery']) }, + })); + const result = await perSheetCli(dir, coverage); + assert.equal(result.code, 0, result.stderr); + const pub = join(dir, 'public'); + assert.equal(readFileSync(join(pub, 'wp-content', 'themes', 'x', 'style.css'), 'utf8'), '.c-block { margin-bottom: 2rem; }\n'); + assert.equal(readFileSync(join(pub, 'president', 'wp-content', 'themes', 'x', 'style.css'), 'utf8'), '.b-gallery { margin-bottom: 0; }\n'); +}); + +test('a sheet css-tree cannot parse is kept byte for byte and listed under unparsed, without changing the exit code', async () => { + const broken = `${'@media screen{'.repeat(20000)}.deep { color: red; }`; + const href = `${SITE_ORIGIN}/wp-content/plugins/mc/embed.css`; + const dir = pageCapture('unparsed', { + sheets: [{ file: '1.css', href, text: broken }], + pages: [{ path: '/', slug: 'home', head: '' }], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ [href]: { text: broken, usedRanges: [] } })); + const result = await perSheetCli(dir, coverage); + assert.equal(result.code, 0, result.stderr); + assert.equal(readFileSync(join(dir, 'public', 'wp-content', 'plugins', 'mc', 'embed.css'), 'utf8'), broken); + const { unparsed } = readJson(join(dir, 'report.json')); + assert.equal(unparsed.length, 1); + assert.equal(unparsed[0].href, href); + assert.match(unparsed[0].error, /call stack/); + assert.equal(result.stderr.trim().split('\n').filter((line) => line.includes('could not parse')).length, 1, result.stderr); +}); + +test('--single-sheet wraps a sheet linked with a media attribute in that @media and keeps a print sheet whole', async () => { + const { dir, coverage } = twoSheetCapture('single-media'); + const out = join(dir, 'site.css'); + const result = await pruneCli(['--single-sheet', '--css', join(dir, 'css'), '--coverage', coverage, '--out', out, '--report', join(dir, 'r.json')]); + assert.equal(result.code, 0, result.stderr); + const css = readFileSync(out, 'utf8'); + assert.ok(css.includes(`@media print {\n${PRINT.trimEnd()}\n}`), css); + assert.ok(!/@media all/.test(css), 'media="all" adds no wrapper'); + assert.equal(readJson(join(dir, 'r.json')).mode, 'single-sheet'); +}); + +test('--single-sheet keeps a media-linked sheet\'s hoisted @import and @layer lines at the top, in order, the @import under the link\'s media', async () => { + const href = `${SITE_ORIGIN}/wp-content/themes/t/print.css`; + const text = '@layer base;\n@import url("https://cdn.example.org/p.css");\n@import url("https://cdn.example.org/q.css") screen;\n.print-only { display: block; }\n'; + const theme = '.t { color: red; }\n'; + const dir = pageCapture('single-hoist', { + sheets: [{ file: '1.css', href: `${SITE_ORIGIN}/wp-content/themes/t/a.css`, text: theme }, { file: '2.css', href, text }], + pages: [{ path: '/', slug: 'home', head: '\n' }], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ [`${SITE_ORIGIN}/wp-content/themes/t/a.css`]: { text: theme, usedRanges: [[0, theme.length]] }, [href]: { text, usedRanges: [] } })); + const out = join(dir, 'site.css'); + const result = await pruneCli(['--single-sheet', '--css', join(dir, 'css'), '--coverage', coverage, '--out', out, '--report', join(dir, 'r.json')]); + assert.equal(result.code, 0, result.stderr); + assert.equal(readFileSync(out, 'utf8'), [ + '@layer base;', + '@import url("https://cdn.example.org/p.css") print;', + '@import url("https://cdn.example.org/q.css") screen;', + '.t { color: red; }', + '@media print {\n.print-only { display: block; }\n}', + '', + ].join('\n')); + assert.match(result.stderr, /q\.css.*has its own media list.*print/); +}); + +test('a malformed url() is left as written, never rewritten', () => { + const result = pruneCss('.a { b: url(}', [[0, 13]], { base: 'https://www.example.com/x/a.css', siteOrigin: 'https://www.example.com' }); + assert.equal(result.css, '.a { b: url(}'); +}); + +// --- one path, several queries; sheets only an @import reaches; host variants ------------------ + +const T = `${SITE_ORIGIN}/wp-content/themes/t`; +const deep = (rules) => `${'@media screen{'.repeat(20000)}${rules}`; +const usedOf = (text, selectors) => ({ text, usedRanges: usedRules(text, selectors) }); + +test('per-sheet: one pathname under two queries with the same bytes is one file pruned with both coverages', async () => { + const text = '.a { color: red; }\n.b { color: green; }\n.c { color: blue; }\n'; + const dir = pageCapture('same-path-same-bytes', { + sheets: [{ file: '1.css', href: `${T}/style.css?ver=1`, text }, { file: '2.css', href: `${T}/style.css?ver=2`, text }], + pages: [ + { path: '/', slug: 'home', head: '' }, + { path: '/about/', slug: 'about', head: '' }, + ], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ [`${T}/style.css?ver=1`]: usedOf(text, ['.a']), [`${T}/style.css?ver=2`]: usedOf(text, ['.b']) })); + const result = await perSheetCli(dir, coverage); + assert.equal(result.code, 0, result.stderr); + const file = join(dir, 'public', 'wp-content', 'themes', 't', 'style.css'); + assert.equal(readFileSync(file, 'utf8'), '.a { color: red; }\n.b { color: green; }\n'); + const summary = readJson(join(dir, 'report.json')); + assert.deepEqual(summary.sheets.map(({ href, file: f }) => ({ href, file: f })), [{ href: `${T}/style.css?ver=1`, file }, { href: `${T}/style.css?ver=2`, file }]); + assert.deepEqual(summary.heads['/about/'], [{ kind: 'link', href: '/wp-content/themes/t/style.css?ver=2' }]); +}); + +test('per-sheet: one pathname under two queries with other bytes keeps both, the second at a query-named path its pages link', async () => { + const one = '.a { color: red; }\n.x { color: blue; }\n'; + const two = '.b { color: green; }\n.y { color: blue; }\n'; + const dir = pageCapture('same-path-other-bytes', { + sheets: [{ file: '1.css', href: `${T}/style.css?ver=1`, text: one }, { file: '2.css', href: `${T}/style.css?ver=2`, text: two }], + pages: [ + { path: '/', slug: 'home', head: '' }, + { path: '/about/', slug: 'about', head: '' }, + ], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ [`${T}/style.css?ver=1`]: usedOf(one, ['.a']), [`${T}/style.css?ver=2`]: usedOf(two, ['.b']) })); + const result = await perSheetCli(dir, coverage); + assert.equal(result.code, 0, result.stderr); + const folder = join(dir, 'public', 'wp-content', 'themes', 't'); + assert.equal(readFileSync(join(folder, 'style.css'), 'utf8'), '.a { color: red; }\n'); + assert.equal(readFileSync(join(folder, 'style.ver-2.css'), 'utf8'), '.b { color: green; }\n'); + assert.equal(variantPath('/wp-content/themes/t/style.css', '?ver=2'), '/wp-content/themes/t/style.ver-2.css'); + const summary = readJson(join(dir, 'report.json')); + assert.deepEqual(summary.sheets.map((s) => s.file), [join(folder, 'style.css'), join(folder, 'style.ver-2.css')]); + assert.deepEqual(summary.heads['/'], [{ kind: 'link', href: '/wp-content/themes/t/style.css?ver=1' }]); + assert.deepEqual(summary.heads['/about/'], [{ kind: 'link', href: '/wp-content/themes/t/style.ver-2.css?ver=2' }]); +}); + +test('per-sheet: a sheet only an @import reaches is written at its live path when its parent keeps the @import (an unparsable parent or child)', async () => { + const child = '.c1 { color: red; }\n.c2 { color: blue; }\n'; + const parentBroken = `@import url("child.css");\n${deep('.p { color: red; }')}`; + const parent = '@import url("child2.css");\n.p { color: red; }\n'; + const child2Broken = deep('.q { background: url(img/q.png); }'); + const dir = pageCapture('import-only', { + sheets: [ + { file: '1.css', href: `${T}/broken-parent.css`, text: parentBroken }, + { file: '2.css', href: `${T}/child.css`, text: child }, + { file: '3.css', href: `${T}/parent.css`, text: parent }, + { file: '4.css', href: `${T}/child2.css`, text: child2Broken }, + ], + pages: [{ path: '/', slug: 'home', head: '\n' }], + }); + const manifest = readJson(join(dir, 'css', 'manifest.json')); + manifest[1].importedBy = [`${T}/broken-parent.css`]; + manifest[3].importedBy = [`${T}/parent.css`]; + writeFileSync(join(dir, 'css', 'manifest.json'), JSON.stringify(manifest)); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ + [`${T}/broken-parent.css`]: { text: parentBroken, usedRanges: [] }, + [`${T}/child.css`]: usedOf(child, ['.c1']), + [`${T}/parent.css`]: usedOf(parent, ['.p']), + [`${T}/child2.css`]: { text: child2Broken, usedRanges: [] }, + })); + const result = await perSheetCli(dir, coverage); + assert.equal(result.code, 0, result.stderr); + const folder = join(dir, 'public', 'wp-content', 'themes', 't'); + assert.equal(readFileSync(join(folder, 'broken-parent.css'), 'utf8'), parentBroken, 'the unparsable parent is kept byte for byte'); + assert.equal(readFileSync(join(folder, 'child.css'), 'utf8'), '.c1 { color: red; }\n', 'its child is served at its live path, pruned with its own coverage'); + assert.equal(readFileSync(join(folder, 'parent.css'), 'utf8'), '@import url("/wp-content/themes/t/child2.css");\n.p { color: red; }\n', 'an unparsable child stays an @import'); + assert.equal(readFileSync(join(folder, 'child2.css'), 'utf8'), child2Broken, 'and is served byte for byte at its live path, its relative url()s intact'); + const { sheets, unparsed } = readJson(join(dir, 'report.json')); + assert.deepEqual(sheets.map((s) => s.href), [`${T}/broken-parent.css`, `${T}/parent.css`, `${T}/child.css`, `${T}/child2.css`]); + assert.deepEqual(unparsed.map((u) => u.href), [`${T}/broken-parent.css`, `${T}/child2.css`]); +}); + +test('--single-sheet: an unparsable imported sheet inlined byte for byte has its relative url()s made root-relative first', async () => { + const parent = '@import url("sub/child.css");\n.p { color: red; }\n'; + const child = deep('.q { background: url(img/q.png); } .r { background: url("../r.png"); }'); + const dir = pageCapture('single-unparsed-child', { + sheets: [{ file: '1.css', href: `${T}/parent.css`, text: parent }, { file: '2.css', href: `${T}/sub/child.css`, text: child }], + pages: [{ path: '/', slug: 'home', head: '' }], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ [`${T}/parent.css`]: usedOf(parent, ['.p']), [`${T}/sub/child.css`]: { text: child, usedRanges: [] } })); + const out = join(dir, 'site.css'); + const result = await pruneCli(['--single-sheet', '--css', join(dir, 'css'), '--coverage', coverage, '--out', out, '--report', join(dir, 'r.json')]); + assert.equal(result.code, 0, result.stderr); + const css = readFileSync(out, 'utf8'); + assert.ok(css.includes('url(/wp-content/themes/t/sub/img/q.png)'), css.slice(-200)); + assert.ok(css.includes('url("/wp-content/themes/t/r.png")'), css.slice(-200)); + assert.ok(!css.includes('url(img/q.png)')); +}); + +test('per-sheet: a link to the site under another scheme or the www/apex variant is the site\'s own sheet, served from public/', async () => { + const text = '.a { color: red; }\n.b { color: blue; }\n'; + const dir = pageCapture('host-variants', { + sheets: [{ file: '1.css', href: `${T}/a.css`, text }], + pages: [{ path: '/', slug: 'home', head: '' }], + }); + const coverage = join(dir, 'coverage.json'); + writeFileSync(coverage, JSON.stringify({ [`${T}/a.css`]: usedOf(text, ['.a']) })); + const result = await perSheetCli(dir, coverage); + assert.equal(result.code, 0, result.stderr); + const summary = readJson(join(dir, 'report.json')); + assert.deepEqual(summary.heads['/'], [{ kind: 'link', href: '/wp-content/themes/t/a.css' }]); + assert.deepEqual(summary.external, []); + assert.equal(readFileSync(join(dir, 'public', 'wp-content', 'themes', 't', 'a.css'), 'utf8'), '.a { color: red; }\n'); +}); + +test('css-prune usage: --out needs --single-sheet, and per-sheet mode needs the capture\'s inventory', async () => { + const { dir, coverage } = twoSheetCapture('usage'); + const out = await perSheetCli(dir, coverage, ['--out', join(dir, 'site.css')]); + assert.equal(out.code, 2); + assert.match(out.stderr, /--out is for --single-sheet/); + rmSync(join(dir, 'inventory.json')); + const bare = await perSheetCli(dir, coverage); + assert.equal(bare.code, 2); + assert.match(bare.stderr, /inventory\.json/); +}); diff --git a/test/convert/init-site.test.mjs b/test/convert/init-site.test.mjs index c6c836d..a842e44 100644 --- a/test/convert/init-site.test.mjs +++ b/test/convert/init-site.test.mjs @@ -167,6 +167,12 @@ test('init scaffolds templates, copies scripts, commits on main and switches to assert.doesNotMatch(base, /^import '\.\.\/styles\/site\.css';$/m); const order = ['', '{pageCss && }', ''].map((s) => base.indexOf(s)); assert.ok(order.every((at, i) => at > 0 && (i === 0 || at > order[i - 1])), `site.css, the page sheet, then the head slot: ${order}`); + // The default build: the page's own head CSS list (css-prune's heads) in live order, links to + // the sheets at their live paths and the pruned