Skip to content

Visual testing 4/8: shoot renders, with the variant-to-story join and pin rules #714

Description

@nathanacurtis

Parent: #588 — step 4 of 8.

Problem

The render side of each pair. Screenshot the Storybook render of every shootable variant, join each Figma variant to the story that actually renders that configuration, and do it deterministically enough that a diff of zero means something.

Two things make this harder than "screenshot every story". Stories carry config-correct JSX slot children baked in, and children cannot be passed through a URL — so the variant-to-story join is a real decision per variant. And Figma constrains a FILL root by the frame it sits in while Storybook has no equivalent, so an unpinned render measures the preview viewport rather than the component.

Potential solution(s)

  • specs testing visual shoot [--components <keys...>] [--workers N] [--port <port>] [--target react|webcomponents]. Writes one render per variant plus a record of exactly how each pair was produced — story id, URL, join kind. Parallel page pool, default 8 workers. Infers the dev-server port the way the Storybook server already does, with an override.
  • Shot mechanics, already settled: Chromium at device scale 2, transitions and animations and caret frozen by injected CSS, fonts awaited, short settle, tight crop of the render root's child, background omitted.
  • Component pin: pin the render to the variant's authored width unless the spec opts out. Do not narrow the pin to FILL roots only — it reads as the more principled rule and was measured to be worse, costing roughly 30 passing pairs with every hug component regressing.
  • Composition pin: pin width unconditionally, since compositions are always fixed-width, and pin height only when the frame's vertical sizing is FIXED; a hugging frame keeps its natural height. Vertical overflow then shows as visible pixels rather than a dimension mismatch. This rule is decided but unmeasured — settle it against one real composition before capturing others, and record the measurement beside the existing fill-versus-hug notes.
  • The join, in preference order: a story whose scalar args complete via contract defaults to exactly the variant's configuration, so its baked children are correct by construction; else the highest-overlap story with the remainder forced through the iframe args parameter, weighting children-selecting props double, a known approximation; else reported as a coverage gap and counted as a failure, because an axis the emitter dropped is itself signal. A composition always takes the first path — one node, one story.
  • Staleness guard: a scoped shoot against a Storybook that has not picked up the latest regeneration measures the old code and reads as a pass. Compare the live story index against the manifest and say so.
  • Playwright is the customer's, not ours. It is declared in the scaffold init writes and installed by the customer in their own testing/visual/ directory — the same contract as the Storybook host, where we never ship or vendor Storybook (ADR A). shoot resolves it from there at runtime and, when it is absent, fails with the init command and the install line rather than a module-not-found trace. This step adds no dependency to the published CLI.

Acceptance criteria

  • Three consecutive full shoots, diffed render-against-render, produce exactly 0 differing pixels per pair. This is the gate — the original 142-story precursor already cleared it across three runs, and a regression here invalidates everything downstream.
  • A scoped shoot of one component writes only that component's renders.
  • A composition's shot dimensions equal its Figma export's dimensions under the pin rule above.
  • A manifest naming a story the running index does not have produces a stated mismatch, not a silent pass.
  • Both emitted targets shoot.
  • With Playwright not installed, shoot fails naming init and the install command, and no other subcommand is affected.
  • The published CLI's dependency list is unchanged by this step.

Case data

  • Territory: cli
  • Size: l

Notes

Depends on step 2. Can be validated render-against-render before step 3 exists, since baselines are only consulted at diff time.

Activity

  1. self-assigned this
    on Oct 8, 2026
  2. nathanacurtis commented on Oct 8, 2026

    @nathanacurtis
    MemberAuthor

    Built and verified on feature/testing-visual in specs (commit 0198a8e), on top of feature/compositions-cli. Validation evidence is in the commit message; #718 carries the remaining fixture sweep and docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

clispecs-cli commandstestingspecs-testing parity validation

Type

Fields

Priority

None yet

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions