Static Full-Coverage Mode for the Capability Drift Check

[QuintinWillison]

Background: how the check works today

The public API drift check is a two-checkout diff, not a static audit of the branch under test.

The composite action .github/actions/sdk-compliance-check-setup/action.yml checks out two trees:

• _sdk-pr - actions/checkout with no ref on a pull_request event, which resolves to GitHub's ephemeral test-merge commit (refs/pull/N/merge), that is the PR head merged into the current target tip.
• _sdk-base - github.event.pull_request.base.sha, the tip of the branch the PR targets, captured in the event payload when the triggering event fired.

The language extractor emits the full public symbol surface of each tree and checkNewSymbols in scripts/capability-matrix/src/api-check.ts diffs them:

const newSymbolObjs = prSymbols.filter((s) => !baseNames.has(s.name));
const uncoveredSymbols = newSymbolObjs.filter((s) => !symbolIndex.has(s.name));

Only symbols new relative to the base extraction must be covered by sdk-compliance.yaml. Everything already present on the target branch is grandfathered, however uncatalogued it may be. A third guard (removedRegisteredSymbols) fails the PR when a symbol registered in the manifest disappears from the surface.

The rationale is the familiar lint-retrofit ratchet, in the same family as golangci-lint --new-from-rev or Rubocop's TODO baseline: a mature SDK can adopt the gate immediately with a sparse manifest, every PR accounts for what it adds and coverage only ever tightens. Using git itself as the baseline also avoids a committed allowlist snapshot per repository, an artefact that would bloat, go stale and merge-conflict.

What surprised us in review

While landing an auto-retry capability in supabase/supabase-go, the check flagged exactly two symbols as "New public API detected that is not in the capability matrix": configuration.Configuration.Retry and postgrest.Option. Meanwhile configuration.Option, a neighbouring exported type in the same file family with identical manifest status (absent), passed silently, as did dozens of other uncatalogued exported identifiers such as configuration.New, postgrest.Client and postgrest.Response.

Until you know the two-checkout mechanism, that verdict looks arbitrary. The failure message never states that "new" means "relative to the base branch", and nothing in the consuming repository makes the differential nature of the check visible. The word "New" is technically the clue, but it reads equally well as "newly detected".

The core concern: what the design imposes on consuming repositories

The check's verdict is not a property of the tree it inspects. It is a property of the tree, the branch the PR targets and the moment the triggering event fired. That shapes how every consuming SDK repository must work, whether or not the shape suits it:

• No repository can answer "is my manifest complete?" from its own checkout. The check is not reproducible locally from a single ref, and the same tree can pass in one PR and fail in another depending on target and timing.
• Manifest coverage becomes a function of history rather than of state. Whether a symbol needs cataloguing depends on when and how it first crossed a PR boundary, not on whether it is public API now.
• Enforcement exists only at the PR boundary. The Go reusable workflow gates the check job with if: github.event_name == 'pull_request', and any language's check needs github.event.pull_request.base.sha, so pushes to the default branch and workflow_dispatch runs are never checked.
• Repositories inherit branch and re-run discipline they did not choose: stacked work must target parent branches to avoid being billed for the parent's symbols, and verdicts silently age as the target branch moves.

Failure and confusion modes of the current design

1. Grandfathered debt is permanent and invisible. A symbol that crosses unclaimed once, whether by direct push, by a reviewer waving it through or by merging before the gate existed, is never flagged again. There is no audit mode to surface the backlog later.
2. Boundary gaps. Direct pushes to the default branch bypass the check entirely, which is also the mechanism by which permanent grandfathering happens.
3. Stale base snapshots. github.event.pull_request.base.sha is frozen at event creation. Movement of the target branch does not re-trigger the check, re-running the same event reuses the frozen payload and a retarget alone fires edited, which is not among the default pull_request trigger types (opened, synchronize, reopened). A retargeted PR therefore shows verdicts computed against its old target until its next push.
4. Stack sensitivity. Properly stacked PRs (child targets parent branch) behave correctly by construction, with each layer accountable for its own delta. But a branch cut from an unmerged parent whose PR targets the default branch is billed for the parent's symbols too, and the difference between the two setups is invisible in the failure output.
5. Merge-commit dependency. The PR-side extraction reads the test-merge commit, so the check cannot run at all for a PR with conflicts against its target.
6. Diagnostic opacity. The failure message omits the reference point. Two flagged symbols beside dozens of silently ignored equals cost real review time to decode, as we can attest.

Where this bites hardest: incubating SDKs

supabase/supabase-go is pre-public and early in incubation, with a deliberately small, carefully curated surface. Its goal is the opposite of a legacy retrofit: every public symbol accounted for in the manifest from day one. The ratchet cannot express that goal, let alone enforce it. It only ever asserts "no new unaccounted symbols crossed this PR", and only on PRs. A repository aiming for a fully-accounted surface gets no way to prove it holds that state and no protection against quietly regressing from it.

Ways out

Option 1 - require full coverage everywhere

Every SDK catalogues its entire public surface and the check becomes a static audit for all repositories.

Strongest guarantee and a single mental model, but it puts an adoption cliff in front of the mature SDKs: a large, blocking, one-off cataloguing effort per repository, which is exactly the cost the ratchet was designed to avoid. In practice it parks the gate for the repositories that most need something in place now. Right end-state, wrong forcing function.

Option 2 - two declared paths through the same checker (recommended)

Let each repository declare its coverage mode in its own manifest:

sdk: go
api_coverage: full   # default: additions (today's ratchet)

Semantics of full:

• One checkout of the branch under test, one extraction, no base ref anywhere.
• Every extracted public symbol must appear in the manifest, under symbols or supporting_symbols (per-feature or top-level).
• Every manifest symbol must exist in the extraction, which subsumes the removed-symbol guard as a plain staleness check.
• Runs identically on pull_request, push and workflow_dispatch, and is exactly reproducible locally by running the extractor and checker against a working tree.

Under full, every entry in the inventory above disappears for the opted-in repository: no grandfathering, no boundary gap, no base snapshot to go stale, no stack sensitivity and nothing differential left to explain in the failure message.

The ratchet then becomes what it always really was, adoption scaffolding rather than a terminal state. A mature SDK stays on additions until it chooses to do its one-off backlog-cataloguing PR, then flips to full. New and incubating repositories start on full from their first commit, and it should probably be the recommended default for them.

Independent of the choice

The additions-mode failure message should say what it means: "New public API detected relative to the base branch (<ref>) that is not in the capability matrix". One line in formatErrorMessage, and it would have saved this entire investigation.

Suggested implementation shape

• Manifest schema: optional api_coverage: additions | full, defaulting to additions, validated with the rest of the manifest.
• Checker: a checkFullCoverage(prSymbols, compliance) beside checkNewSymbols, returning uncovered symbols and stale manifest entries. The CLI reads the mode from the manifest itself so the mode travels with the repository's own claim rather than with workflow wiring.
• Reusable workflows: in full mode skip the base checkout and base extraction and drop the pull_request gate so the check job also runs on push and dispatch. No extractor changes in any language, since extractors already emit the full surface per checkout - the mode branches purely in shared TypeScript and workflow YAML.
• Docs: capability-matrix guidance describing both modes, when to choose which and the expected migration direction from additions to full.

supabase-go's position

supabase/supabase-go would opt into full immediately. Its uncatalogued backlog is small - construction and plumbing types such as client constructors, option types, response and error types, mostly destined for the top-level supporting_symbols list - and it is happy to be the pilot repository for the mode.

---

Generated by Pi v0.84.3 using Claude Fable 5 (claude-fable-5).
Session working directory: /Users/quintin/code/supabase/supabase-go
Resume this session from that directory with: pi --session 01a038f0-9d52-7c5a-8d61-2e7b89928967
Reviewed and Refined by @QuintinWillison.