Skip to content

Visual testing: make baseline capture self-scoping via node-subtree hashing #719

Description

@nathanacurtis

Parent: #588 — follow-on. Not required for the suite to ship.

Problem

Baseline capture is the slow, rate-limited stage, and in v1 it is all-or-name-them-yourself: capture everything missing, or type out the components you believe changed. On an 80-component library, "I think something moved in Figma" costs a full recapture to discover that three things moved.

The obvious signal is the wrong one. The library file's lastModified moves on any edit anywhere in the file, so keying off it means recapturing the whole catalogue to find that nothing relevant changed — exactly the waste this would exist to remove.

The spec version is tempting, since the versioning machinery already computes per-component change class. But the spec is lossy by design, and visual testing earns its keep precisely at the fidelity the spec drops — so a signal derived from the spec goes blind in the one region that matters.

Potential solution(s)

  • Hash the component's Figma node subtree out of the already-fetched payload. The payload holds every component's full subtree with nested instances expanded, so a change inside a leaf propagates to every parent containing it — which is the correct behaviour for a visual baseline.
  • Record the hash at capture time alongside the existing provenance.
  • status recomputes it from the payload on disk: same hash means current, different means changed, no record means missing. Offline, free, exact per component.
  • baseline with no component scope then captures missing and changed, rather than everything missing.
  • Report honestly on what it cannot know: the payload is only as fresh as the last fetch, so status states the payload's fetch time and never claims knowledge of unfetched edits.
  • Keep the spec version in the report as human context. Do not make it the signal.

This makes the overt command self-scoping. It does not make capture automatic — that stays ruled out.

Acceptance criteria

  • Capture baselines, edit an unrelated page in the Figma file, re-fetch: zero components reported changed.
  • Change one component in Figma, re-fetch: exactly that component reported changed, and capture takes exactly it.
  • Change a leaf component used by several others, re-fetch: every dependent reported changed too.
  • A payload older than the capture record is reported as such rather than as current.
  • Hashes are stable across two fetches of an unchanged file.
  • Capture still cannot be invoked by anything but the command itself.

Case data

  • Territory: cli
  • Size: m

Notes

Depends on steps 2 and 3 of #588.

Activity

  1. self-assigned this
    on Oct 8, 2026
  2. nathanacurtis commented on Oct 8, 2026

    @nathanacurtis
    MemberAuthor

    Implemented on PR #721 (commit fdb1f33), per this issue's approach: the manifest hashes each spec's node subtree, capture records the hash it photographed, status compares and reports current / changed / missing, and a bare baseline captures missing + changed.

    Acceptance notes:

    • Mismatch detection verified mechanically (a tampered record flags exactly that spec's variants, nothing else; restoring clears it). The real-Figma-edit path — edit a leaf, refetch, see every dependent flagged — awaits the next genuine design change; the propagation rests on REST payloads inlining instance subtrees.
    • Hash stability across rebuilds holds (manifests remain byte-identical). Stability across two fetches of an unchanged file is untested until a no-op refetch happens.
    • Pre-fingerprint captures report as not comparable and are treated as current — no bulk recapture was forced on the three live fixtures.
    • Capture remains reachable only by the command itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

clispecs-cli commandstestingspecs-testing parity validation

Type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions