Feedback from a 1,285-shot parity run (0.5.0): computed origin noise, progress, base-vs-base, sharing reports #59

Status (2026-11-02)

• Done: computed origin normalization (#62); heartbeat, dead-origin abort, CSP-safe CSS, selector-aware ceiling, geojs blocked, results.json docs (#96); hook timeout (Deadline per shot so a hung renderer can't hang the run #54), probe, video, sitemap (--sitemap: take routes from the reference's sitemap.xml #46), serve docs (Docs: serve static builds with serve.json (clean URLs, trailing slash, rewrites) #52); CLI subsets keep config hooks via --only (#118); a route hook can run the top-level one with ctx.topLevel while replacing it stays the default (#117).
• Open: where unstable pixels are, folding tiny unstable rows into clean, a name field on route objects, --rerun-failed with a changed config.

Feedback from a real run on renderdiff 0.5.0: a CSS-only cleanup on metabase.com, base build vs PR build, both served locally.

The runs:

• Docs: 49 routes × 5 widths = 245 shots.
• Marketing: 208 routes × 5 widths = 1040 shots.

Both runs used --computed, and agents drove them through the skill. The result was parity, with only per-load noise on marketing. Items are ranked by time lost. The ones already tracked link to their issue.

Cost the most time

• --computed doesn't normalize the origin.
  • Absolute hrefs (a[href="http://localhost:4611/..."]) and url() values differ only by port.
  • Docs reported "dom identical on 2 of 49" and 53 findings, every one of them the port.
  • Marketing reported "7 of 208" identical, with about 200 routes "unmatched".
  • Both agents wrote a filter script. After it, there were 0 findings.
  • This also breaks the skill's rule "CSS-only change ⇒ expect dom identical".
  • Fix: rewrite the left origin to the right one in keys, attributes and values before comparing.
• No progress in long background runs.
  • Marketing runs took 15–31 min.
  • The agent judged progress by counting PNGs in the out dir, and its first baseline outlived a 600 s Bash timeout.
  • --progress exists, but neither agent found it: the skill doesn't say to use it for background runs.
  • Fix: mention --progress in the skill's loop, and print a heartbeat like 412/1040 shots even when output is piped.
• An origin that dies mid-run doesn't stop the run.
  • Both static servers were killed from outside partway through.
  • renderdiff kept going and logged 258 ERR_CONNECTION_REFUSED errors; the agent then had to redo the whole 15-minute run.
  • Fix: after a few connection-refused errors in a row on one origin, abort with exit 2.
• A before hook that hangs stalls the whole run, with no output.
  • A hook awaited an iframe's fonts.ready, but the page swapped the iframe's src, so the promise never resolved.
  • The run sat at 6 of 15 shots for over 10 minutes until it was killed.
  • Fix: a per-hook timeout that fails the shot with before hook timed out on left. Related: Deadline per shot so a hung renderer can't hang the run #54.
• No base-vs-base mode. Both agents could use only two ports. To get a baseline, they served the base build on both ports, killed the right-hand server, and restarted it on the change build. On marketing they did this three times. Fixes:
  • --baseline, or a self-check that loads the right origin twice. Related: Recheck differing shots for load-to-load variation #47.
  • Something that subtracts a baseline results.json from a change run. Today that's done by eye.
• Page-height ceiling.
  • /docs/latest/configuring-metabase/environment-variables is 119k–158k px tall and errored at all widths, which cost a 12-minute rerun.
  • Adding a selector didn't help, because the full-page height check runs before the selector is applied.
  • Fix: skip that check when a selector is set, or clip to the first N px and report the rest as not compared. Related: Re-read page height in the lazy-load scroll pass #53.

Noise handling

• Masked iframes still change height.
  • maskMedia paints iframes over, but auto-sizing embeds still move the layout. Examples:
    • a newsletter iframe that only grows, by up to 400 px;
    • the Lever jobs list, which sometimes doesn't load on one side, scoring 51%;
    • a Fillout fallback driven by a timer.
  • Each needed a hand-written hook. Fixes:
    • a per-route and global waitFor: { selector, timeout };
    • an option to pin iframe heights at shot time.
• Video still shifts the layout. video.js and <video> heights still move 0.5–2 px while the video loads. That produces bands, which can't be accepted. Fix: wait for loadedmetadata before the shot.
• CSP violations show up as renderdiff errors.
  • Pages with embedded iframes failed with page.addStyleTag: ... violates ... CSP at random widths, in src/capture.ts (lines 183 and 220).
  • It reads like a renderdiff bug. The workaround was blocking the embed hosts.
  • Fix: retry the injection. If it still fails, report it as a failed asset (csp from iframe <host>) instead of an unmeasured error.
• Default blocklist misses geo lookups. The cookie banner depended on a 3 s get.geojs.io timeout, which flipped one route to 70%. Blocking the host made the banner deterministic.
• Tiny unstable counts with no location.
  • Seventeen docs routes showed rows like 55 px unstable, with no location given.
  • Finding the source took a PIL magenta search on left.png. It turned out to be anti-aliased rounded corners.
  • Fixes:
    • give a bounding box or crop for unstable pixels;
    • fold rows that only have under ~100 px unstable into the clean: line.
• No suggested masks.
  • On docs, the same element (p.gdpr-cookie-notice-description) differed on about 30 routes, and the right selector had to be dug out of minified JS.
  • Fix: a baseline summary like "same element differs on N routes: candidate mask".

Config and CLI

• CLI routes lose their config hooks.
  • renderdiff ... /love?state=nav ran without the config route's before, so it shot the closed nav, with no warning.
  • Fix: when a CLI route matches a config route's label, reuse that route's hook, or warn.
• A route's before replaces the global before instead of chaining.
  • This forced a withSettle(hook) wrapper on every interactive route.
  • Fix: run the global hook, then the route's.
• No name for hooked variants.
  • The agents added ?state=x to keep each hooked variant of a path unique.
  • Fix: a name (or state) field on route objects.
• --rerun-failed was unusable while iterating.
  • It exits 2 on any config change, and the agents were editing hooks to chase noise, so they used separate --out dirs with route subsets instead.
  • Fix: allow it with a warning.
• goto timeouts.
  • page.goto: Timeout 45000ms errors hit different routes on each run. Dropping to --concurrency 4 fixed them, but that was a guess.
  • Fix: retry a goto timeout once, and add the hint "try a lower --concurrency".
• The results.json docs mislead a line-by-line parser.
  • The reference says "one result per line", and that's true. But the lines sit inside a JSON document, with trailing commas.
  • An agent's line-by-line JSON.parse crashed on them.
  • Fix: say that in the reference, or write an NDJSON sibling.
• Missing flags. --routes-file, and --sitemap (--sitemap: take routes from the reference's sitemap.xml #46). Both agents parsed routes.txt inside the config themselves.
• No static server. npx serve doesn't resolve slashless URLs to .html, so a custom server was written twice. Fix: renderdiff serve <dir> with clean-URL support. Related: Docs: serve static builds with serve.json (clean URLs, trailing slash, rewrites) #52.

Report and sharing

0.6.0 fixed most of the navigation: sidebar, search and summary.

• file:// previews drop the images.

  • The report's images are relative paths, so a preview that renders file:// HTML as a static snapshot (the Claude desktop file pane) shows it without images. It had to be served over HTTP.
  • Fix: have --open serve the out dir instead of opening the file.

• The output is too big to share. Docs was 454 MB (735 PNGs). The marketing out dir reached 2.5 GB, including files left over from earlier runs. Neither fits a PR attachment or an Artifact. To publish on GitHub Pages:

  • Convert to WebP. WebP is capped at 16 383 px, so taller shots were cut into 16 000 px tiles and stacked in the report.
  • Keep only the left image for zero-diff shots, and replace their diff with "No differing pixels".

  That brought docs to 74 MB (275 files), and marketing to 514 MB (1425 files, mostly long release and license pages). Fixes:

  • renderdiff report --compact <dir> (or a default output format) that does this;
  • optionally, a publish helper for gh-pages.

• Stale files pile up. Earlier runs' files stay in the out dir. Fix: clear the files renderdiff owns at the start of a run, or write a manifest.

• Noise shows as red.

  • Without a threshold, any differing shot is red in the sidebar, so 18 noise-only routes look like failures.
  • Accepting them needs exact bands and regions, which don't hold for animation.
  • Fix: accept "unstable on base-vs-base" by element or selector, not by pixel region. Related: Recheck differing shots for load-to-load variation #47.

Worked well

• The before hooks (hover, click, Cmd+K) opened every state, which the screenshots confirmed.
• --dry-run was handy for checking route and selector parsing.
• The .mjs config, where routes are read from a file, was flexible.
• renderdiff report --verdict --title is instant.
• The skill's mode table (CSS-only → pixels + --computed), the rationalizations table and the "verdict only after a fresh run" rule kept the agents honest.