test(doctor): assert the hanging-backend property instead of a stopwatch - #82
Merged
Conversation
"a hanging backend cannot stall the fast report" injects clients that never
resolve, sets timeoutMs to 50, and then asserted:
assert.ok(Date.now() - started < 500);
That measures the machine, not the behaviour. The 50ms timeout can fire exactly
as designed and the assertion still fails because the runner was busy. Observed
failing on windows-latest at 689ms, and locally at 1818ms under full-suite load
while passing in isolation at 82ms.
Replaced with the property the test exists to defend: a probe fed by a hanging
client must never come back claiming it verified anything. agent.transport,
auth.credential, agent.catalog and mcp.broker are named explicitly rather than
filtered on an axis, because local checks like workspace.git legitimately do
verify in this fixture — nothing about them touches the backend that is hanging.
The wall clock is still bounded, but as a hang detector rather than a stopwatch:
30s distinguishes "returned" from "awaited forever", which is the failure the
test was written to catch. A slow runner no longer registers as a bug.
Stronger than what it replaces, not weaker. Mutation-checked: making notChecked
return a verified axis fails this test along with two others (5 pass / 3 fail);
restoring gives 8 / 8. The old assertion passed that mutation untouched — it
never looked at a single axis, so a false green would have gone through while a
busy runner did not.
Gates at this commit:
npm run typecheck exit 0
npm test 922 pass / 0 fail
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
test/diagnostics.test.ts— "a hanging backend cannot stall the fast report" fails intermittently on loaded runners.It injects clients that never resolve, sets
timeoutMs: 50, and then asserts:That measures the machine, not the behaviour. The 50ms timeout can fire exactly as designed and the assertion still fails because the runner was busy.
Observed:
windows-latestCIThis is on
main, so it can redden any branch that runs at the wrong moment. It has already done so once.Contract
Assert the property the test exists to defend, not the clock: a probe fed by a hanging client must never come back claiming it verified anything.
agent.transport,auth.credential,agent.catalogandmcp.brokerare named explicitly rather than filtered on an axis. A first attempt filtered onreachable !== "na"and failed onworkspace.git— a local check that legitimately verifies in this fixture, because nothing about it touches the backend that is hanging.The wall clock stays, but as a hang detector rather than a stopwatch. The failure this test guards against is an unbounded
awaitthat never returns at all; 30s separates that from a slow runner, and any finite margin would do. A busy machine is no longer a bug.This is stronger than what it replaces, not weaker
Relaxing a timing bound looks like weakening a test, so here is the evidence it is not.
Mutation: make
notChecked()return a verified axis — the exact false-green this suite exists to prevent.The old assertion passed that mutation untouched. It never looked at a single axis. So the previous test would let a false green through while failing on a busy runner — precisely backwards.
Tests
npm run typechecknpm test922 is the full suite on
mainat41a7e261; this PR changes no production code and adds no test, so the count is unchanged by design.Scope
One file, one test. No production code.
git diff --stat:test/diagnostics.test.ts | 27 +++++++++-.Why this is separate
It surfaced while working an unrelated lane, where it turned
windows-latestred on a PR that had not touched anything near it. Fixing it there would have buried an independentmain-line defect inside an unrelated change.Recommend landing this first, ahead of the supercluster lanes (#72, #73, #74, #75, #77, #78) — every one of them carries this flake until it does.
The same fix is currently also committed on
supercluster/a5-usage-continuitybecause that is the branch it was blocking; that copy should be dropped once this lands.