The observation
While fixing the brace-expansion advisory (#331), we hit something worth generalising.
main's HEAD commit still carries a green Dependency Audit result — the one from when #325 merged. But a re-run on that same commit fails today. No code changed. The advisory simply landed in the database after the check ran.
So the green badge on main is not evidence the tree is clean. It is evidence the tree was clean against the advisory data available at that moment.
Why this matters more than the individual advisory
This is the mechanism that let #325's non-fix survive. That PR pinned brace-expansion to ^2.1.2 — inside the affected range <=5.0.7. It merged green, and stayed apparently green, for weeks. There was no moment at which anything announced the regression, because our audit only ever runs in reaction to a PR.
Vulnerability disclosure is asynchronous to our commits. Every other gate we run is a pure function of the code — tests, types, lint — so "green at merge" stays true. Audit is the one gate whose correct answer changes while the code sits still, and we run it as though it were the same kind of check.
The result is a gate that reports on the past and is read as reporting on the present.
Proposed fix
Run the dependency audit on a schedule against main, not only on pull requests.
on:
schedule:
- cron: "0 7 * * *"
pull_request:
workflow_dispatch:A daily run turns "silently vulnerable for weeks" into "loud within a day", and it is the only way a newly disclosed advisory against unchanged code can ever reach us.
Two details that make it useful rather than noisy:
- Failure must reach a person. A scheduled run that goes red on a branch nobody watches reproduces the original problem in a new place. It needs to notify.
- Do not let it be advisory-only. A
|| true / continue-on-error audit is strictly worse than none — it looks like coverage and can never fail. We have already shipped that shape once.
Applies fleet-wide
The same reasoning holds for every repo running an audit gate; this is filed here because this is where the evidence is. Related: the sibling issue about an audit left red on main, which is the same gate failing in the opposite direction — red-and-ignored rather than green-and-stale. Both end with the check meaning nothing.
The observation
While fixing the
brace-expansionadvisory (#331), we hit something worth generalising.main's HEAD commit still carries a greenDependency Auditresult — the one from when #325 merged. But a re-run on that same commit fails today. No code changed. The advisory simply landed in the database after the check ran.So the green badge on
mainis not evidence the tree is clean. It is evidence the tree was clean against the advisory data available at that moment.Why this matters more than the individual advisory
This is the mechanism that let #325's non-fix survive. That PR pinned
brace-expansionto^2.1.2— inside the affected range<=5.0.7. It merged green, and stayed apparently green, for weeks. There was no moment at which anything announced the regression, because our audit only ever runs in reaction to a PR.Vulnerability disclosure is asynchronous to our commits. Every other gate we run is a pure function of the code — tests, types, lint — so "green at merge" stays true. Audit is the one gate whose correct answer changes while the code sits still, and we run it as though it were the same kind of check.
The result is a gate that reports on the past and is read as reporting on the present.
Proposed fix
Run the dependency audit on a schedule against
main, not only on pull requests.A daily run turns "silently vulnerable for weeks" into "loud within a day", and it is the only way a newly disclosed advisory against unchanged code can ever reach us.
Two details that make it useful rather than noisy:
|| true/continue-on-erroraudit is strictly worse than none — it looks like coverage and can never fail. We have already shipped that shape once.Applies fleet-wide
The same reasoning holds for every repo running an audit gate; this is filed here because this is where the evidence is. Related: the sibling issue about an audit left red on
main, which is the same gate failing in the opposite direction — red-and-ignored rather than green-and-stale. Both end with the check meaning nothing.