Repository navigation
Visual testing 8/8: broaden to the remaining library fixtures, and document #718
Description
Activity
Full-catalog sweep complete for the three live fixtures, run through the shipped commands (PR #721). The fourth fixture (legacy flat layout) was skipped — it is being retired; the flat-layout guard rests on the unit test from step 1.
Fixture Specs Shootable Captured Shot Pass Fail vs September A (muse) 77 components + 12 compositions 1,109 1,106 (3 nodes Figma returns no render for) 1,103, 0 failed 550 71 Sept 513/33 on fewer components and fewer baselines; see failure themes below B 32 components 636 636/636 636, 0 failed, zero mapping problems 394 48 Sept 334/12 on 377 pairs — the delta is wider coverage plus one theme below D (adversarial) 150 components + 1 composition 465 465/465 465, 0 failed 238 208 (53%) Sept 55% — the fixtures doing their job September deltas are first-class again: fixture B's recovered report carried into
previous, so every row shows its Δ against the last pre-storybook-generation run.Failure themes, observed (causes not verified):
- Near-miss threshold overshoots (fixture A: Badge 10.6% vs a 10% bar, Pill family 3.x% vs 3%): dimensions exact, diffs marginally over bars tuned against September's renderer. This run installed a newer Chromium, which is the class of shift the pass-bar floor documentation describes. Retuning the ignore files is a judgment call left open.
- Image-content mismatches (fixture B: Avatar 0→16, Image 0→6): dimensions exact, 85–93% pixel difference — right-sized boxes showing almost entirely different content.
- Composition height pin did not take on the one measured composition (dh −184 with the pin recorded in the manifest). The crop-rule measurement belongs to this fixture round once fixture B has composition specs.
Still open on this issue: docs-site pages (CLI reference for
specs testing visual+ the measurement guide), the composition crop-rule measurement, and reconciling the recovered ignore files (three entries already warn for missingnote:).- added a commit that references this issue
on Oct 8, 2026 Docs are in on the PR branch: the
testing visualcommand reference (stage cost table, never-implicit capture, ignore keys by stage, both report modes) and the measurement guide (pin-everything and ink-bounds rules with the measurements that settled them, composition pin rule, threshold floor, failure-signature table). Site builds clean.Remaining on this issue, both deliberately left:
- Recovered ignore-file reconciliation — three entries still warn for a missing
note:; whether each bar survives the newer Chromium is a judgment call, and every failing row has its composite on the report page. - Composition crop-rule measurement — blocked until the intended fixture workspace has composition specs; the first measurement (height pin not taking, dh −184) is recorded on PR specs testing visual: visual testing over the emitted Storybook #721.
- Recovered ignore-file reconciliation — three entries still warn for a missing
The composition crop rule is now measured, on the intended fixture and on the muse's 12 compositions. It holds.
The truncation defect it uncovered first: the height pin was applying correctly, but a screenshot clip beyond the viewport is silently truncated to it — a pinned 876px composition came back 784px (viewport minus body margin) and read as a dimension defect. Fixed on the PR branch: the shoot grows the viewport to the measured box before capturing.
The rule, measured:
Case Height sizing Result Fixture B sign-in pattern FIXED → pinned Dimensions exactly match the Figma export; remaining 13.2% is content fidelity Fixture B filter page pattern HUG → natural Renders at its content height; the design frame holds 1295px more vertical space — reported as real signal, not punished as a pin failure 10 of 12 muse compositions FIXED → pinned dh exactly 0; one passes outright at 2.9% The fixture pair exercised both branches — fixed-width always held (as predicted: compositions are page-node frames), and height splits by the frame's own sizing.
Open edge, recorded not chased: very wide frames (≥1440 CSS px) show dw −8 CSS px on the render — the signature of a classic scrollbar taking width in headless Chromium once content exceeds the viewport. Narrow frames are exact. Likely fix is forcing overlay scrollbars or hiding them in the freeze CSS; needs its own measurement before trusting.
Remaining on this issue: the four noteless ignore entries (presented for decisions separately) and the scrollbar edge above.
Everything this issue named is done and verified:
- Full-catalog sweep on all three live fixtures (results above); the retired flat-layout fixture skipped by decision.
- Docs shipped as a Testing section (Overview + Getting Started) plus the command reference, after a rework to house style.
- Composition crop rule measured on the intended fixture and the muse — both branches hold; the viewport-truncation and re-centering defects it surfaced are fixed, and every fixed-height composition now diffs at exactly 0/0 dimensions.
- Ignore files reconciled: every suppressing entry carries its note (reasons promoted from comments, one measured before restoring), the warning no longer fires on coverage-adding keys, and one inert entry is deleted.
- The scrollbar/re-centering width edge is closed (dw −16 → 0 on every wide page).
All on PR #721.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Parent: #588 — step 8 of 8.
Problem
Steps 1 through 7 are validated against one library. It is a large one — 80 components and 12 compositions — but it is one library's conventions, and a rule proven on one library's data is not proven. Three more libraries exist as fixtures and each exercises something the muse does not.
There is also no documentation. A customer cannot discover that capture is deliberately never automatic, or that two of the measurement rules are the opposite of the principled-looking choice, by reading the source.
Potential solution(s)
Run the finished suite end to end against each remaining fixture, then document it.
Each needs its unmapped-prop check driven to zero, baselines captured, a full shoot and diff, and its recovered ignore file reconciled against current behaviour.
Each of these libraries has a recovered September run on disk — the hand-tuned ignore file, and the prior report so the first new run shows a delta rather than starting blank. Two things to expect: the ignore files are components-only and predate kind-aware keying, so they need the kind dimension and every entry re-checked against a
note:that is now months old, some of them temporary allowances that may no longer apply. And the recovered manifests are stale and must be rebuilt rather than reused, since they encode the variant-to-story join against specs and contracts regenerated repeatedly since.Documentation: a CLI reference page carrying the subcommand table with its cost columns, the ignore keys with which stage reads each, and the two baseline modes. It must state plainly that capture is never automatic and that there is no
buildorrunintegration, so nobody goes looking for one. Plus a guide page carrying the fill-versus-hug measurement and the composition pin measurement — the two places where the principled-looking rule is the wrong one, both settled by measurement, both of which a reader who does not know that will re-litigate.Acceptance criteria
Case data
Notes
Depends on steps 1 through 7.