verify(trace): promote 105 verification artifacts from EXECUTED gate evidence (TRACE-P02) - #365
Conversation
…evidence (TRACE-P02) The right side of the V was almost entirely unpromoted: 22 of 247 requirements `verified`, with the evidence already existing and simply unbound. This binds the part that a gate run can actually justify. METHOD — executed evidence, not a `verifies` link. A full sweep (`--filter '(has-tag "falcon")'`) ran 323 steps with 0 FAIL. An artifact qualifies only if it had >=1 step that genuinely EXECUTED and none that failed. Skipped steps do not count: bench-only, `enforced-by-kani-gate` and `skip-no-steps` are all exclusions, because "the gate did not run it" is not evidence that it passes. artifacts in the sweep 162 >=1 EXECUTED step, zero failures 130 <- promoted only skipped steps 32 <- left unpromoted, deliberately any FAIL 0 Net: 105 promoted here (the other 25 were already `verified`). Verification artifacts: 130 verified / 123 approved / 34 implemented. --- A THIRD CATEGORY TRACE-P02 DID NOT ANTICIPATE, FOUND BY DOING IT --- The mechanical sweep initially promoted SIX artifacts whose own text says they must NOT be verified: FV-FALCON-OCI-002 "stays `implemented` until the fuse is run" FV-FALCON-OCI-003 wasm.directory criteria (b) and (c) NOT MET FV-FALCON-OCI-004 kill-criterion owned by gale/meld, unrun FV-FALCON-OCI-005 ditto FV-FALCON-PERF-002 criterion (d), the PX4 head-to-head, not done (#362) FV-FALCON-REL-001 failure injection not run All six have PASSING executed steps. The gate result is real. It is also NOT SUFFICIENT, because each names a criterion the gate cannot check — an external kill-criterion, a live-page fetch, a comparison nobody has run. TRACE-P02 anticipated two populations: gate-runnable (promote) and bench-only (waive). This is a third: **gate-passing AND self-blocked**. Promoting them would have asserted precisely the outcomes those criteria exist to test — the error v1.133 avoided and that jess's meld run then vindicated by REJECTING. All six reverted to `implemented`. The rule this establishes: an artifact that names an undischarged criterion is excluded from mechanical promotion no matter how green its steps are. Gate evidence is necessary, never sufficient. WHAT IS DELIBERATELY NOT PROMOTED: - 32 artifacts whose steps only skipped (bench-only / kani-enforced) - the 6 self-blocked above - 45 artifacts with NO steps at all — separately tracked; they cannot be evidence as they stand and need steps written or an honest demotion - requirements themselves — this promotes VERIFIERS; requirement promotion follows once their verifiers are all `verified` Code-free, per the two-commit rule. rivet validate: PASS, exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
Automated review for PR #365pulseengine/relay: Verdict: 💬 Comment Summary: The code snippet provided appears to be a list of verification artifacts for a software project, specifically related to the FALCON (Flight Automation and Control Library) HAL (Hardware Abstraction Layer). Each artifact has an ID, type, title, status, and description. The status of each artifact is updated from 'approved' to 'verified'. This suggests that the verification process has been complete Findings: 0 mechanical (rivet) · 1 from local AI model. Findings (1):
Generated by a local AI model and post-validated against a strict JSON contract. Each finding includes the verbatim line being criticised — verify by reading the file at the cited location. Reviewed at |
…eads as a regression (#383) PR #365's verification gate failed on one step out of 163: [ FAIL (rc=1)] FV-FALCON-WORLD-001: scripts/falcon-hello-demo.sh [falcon-hello-demo] vehicle sent 0 heartbeat(s) [falcon-hello-demo] FAIL: expected at least 8 heartbeats, got 0 error: bind 127.0.0.1:14701: Address already in use (os error 98) Nothing was wrong with the code. The demo hardcoded the 14700/14701 pair, the gate runners are SHARED, and two jobs on one runner fight for the same socket. That is the worst shape a flake can take on a REQUIRED check: it fails loudly, names a plausible-sounding functional symptom ("got 0 heartbeats"), and the actual cause is one line further down. #365 is a 105-artifact promotion PR, so the obvious reading was that a promotion had broken something. FIX: probe for a free port pair instead of assuming one. An explicit FALCON_HELLO_PORT_BASE still wins, so a bench operator can pin it, and if 64 probes somehow all fail the script falls back to the historical default rather than erroring — a demo that cannot find a port should still try, not add a second failure mode. MEASURED, both directions, three concurrent runs: forced to the old fixed base 2 of 3 FAIL "Address already in use" x3 "expected at least 8 heartbeats, got 0" x2 with the probe 3 of 3 PASS, 14 heartbeats each The negative case is the real bug reproduced on purpose, not a synthetic stand-in — #365's exact error text, from the exact condition. Refs #365, #345 Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The right side of the V was almost entirely unpromoted — 22 of 247 requirements
verified, with the evidence already existing and simply unbound. This binds the part a gate run can actually justify.Method: executed evidence, not a
verifieslinkA full sweep ran 323 steps, 0 FAIL. An artifact qualifies only if it had ≥1 step that genuinely executed and none that failed. Skipped steps don't count — bench-only,
enforced-by-kani-gateandskip-no-stepsare all exclusions, because "the gate did not run it" is not evidence that it passes.Net 105 promoted here (25 were already
verified). Verification artifacts now 130 verified / 123 approved / 34 implemented.A third category TRACE-P02 didn't anticipate — found by doing it
The mechanical sweep initially promoted six artifacts whose own text says they must not be verified:
implementedFV-FALCON-OCI-002implementeduntil the fuse is run"FV-FALCON-OCI-003FV-FALCON-OCI-004FV-FALCON-OCI-005FV-FALCON-PERF-002FV-FALCON-REL-001All six have passing executed steps. The gate result is real. It is also not sufficient, because each names a criterion the gate cannot check — an external kill-criterion, a live-page fetch, a comparison nobody has run.
TRACE-P02 anticipated two populations: gate-runnable (promote) and bench-only (waive). This is a third: gate-passing and self-blocked. Promoting them would have asserted exactly the outcomes those criteria exist to test — the error v1.133 avoided, and which jess's meld run then vindicated by rejecting.
All six reverted. The rule this establishes: an artifact naming an undischarged criterion is excluded from mechanical promotion no matter how green its steps are. Gate evidence is necessary, never sufficient.
Deliberately not promoted
verifiedCode-free, per the two-commit rule.
rivet validatePASS, exit 0.🤖 Generated with Claude Code