Visual testing 5/8: diff, scoring, ignore resolution, and the report #715

Parent: #588 — step 5 of 8.

Problem

Deciding and reporting pass or fail. This is also the stage a customer re-runs most often — tuning a tolerance and re-scoring must cost seconds with no browser, or nobody tunes anything and every library ships with the default that happens to be wrong for it.

Potential solution(s)

• specs testing visual diff [--components <keys...>] [--against figma|accepted], specs testing visual report [--format json|md], and the bare specs testing visual as shoot → diff → report. Capture is not in the chain.
• Diff mechanics, already settled: both images flatten onto an opaque-white union canvas, top-left anchored, before comparison — Figma exports and transparent shots disagree about alpha wherever a surface is white rather than transparent, and raw comparison would flag every such pixel. Then a per-pixel threshold compare with anti-aliasing excluded. Pass means dimensions match within tolerance and the diff percentage is at or under the pass threshold; dimension deltas are reported separately as first-class defect signal, so a render 2px wider than its baseline fails even at a low pixel diff.
• Every diffed pair gets a baseline-render-diff triptych. That composite is the unit of diagnosis for a human or an agent, and it is why writing a diff tree is worth the disk.
• Ignore resolution runs defaults → per-spec → variant matcher, now keyed by kind. Scoring keys (passPct, dimTolerancePx, threshold, skip, skipVariants, note) are read here and re-score in seconds; manifest-time keys (sampleVariants, pinWidth) need a rebuild and a re-shoot. Which stage reads which must stay true.
• Two rules enforced rather than documented: every ignore entry needs a note:, because these suppress real signal and the note is what separates a permanent measurement fact from a temporary allowance — an entry without one is indistinguishable from a hidden bug. And the default pass threshold must be set deliberately per library, since falling through to 1% is below the floor Chrome and Figma rasterize to, making the same spec read green in one library and red in another.
• Compositions are advisory: captured, diffed, reported with pixel counts, but their pass/fail does not decide exit status. A composition's diff is dominated by its children, so one leaf defect lights up every composition containing it. Component and composition totals report separately. Composition output is Pro, so a free run filters them before the loop and reports the count once.
• Ranking comes from the dependency analysis — leaves at depth 0, else max transitive depth; sort depth ascending, then fail count and worst diff descending. Fixing leaves first shrinks downstream diffs for free, which is also why compositions sort last. Ranking is catalogue-wide, so a scoped diff reuses the stored graph rather than recomputing it.
• The report JSON is the machine-readable truth; markdown and the status page are regenerated views. A scoped run replaces only its own keys and preserves each key's prior summary, so improvement deltas survive across rounds.

Acceptance criteria

• Re-diffing an unchanged library reproduces the stored report exactly.
• Changing one spec's pass threshold and re-diffing that spec alone completes in seconds, opens no browser, and changes only that spec's row.
• A scoped diff leaves every other spec's numbers and prior values untouched.
• A failing composition does not change the exit status; a failing component does.
• An ignore entry with no note: warns.
• --against accepted with no accepted tree fails with a message pointing at the regression-mode work, not a stack trace.
• A free-tier run reports the filtered composition count once and emits no composition results.

Case data

• Territory: cli
• Size: l

Notes

Depends on steps 3 and 4.