Repository navigation
feat: a push may defer a broken CodeQL to the required sdlc ci (#33) - #36
Conversation
Constitution 1.35.0 amended XII to add exit `3`, could not run, deferred to CI (constitution#99), so the implementation reverted out of #34 comes back, with the narrowings that decision called for. This repo's pin moves to v1.35.0 in the same change, because the rule it implements lives there. `3` is granted by this driver, per check, and today only CodeQL has it: `codeql_broke()` raises `DeferredError` when `ci_judges()` holds, a tool's own `3` maps to `2`, and `run_step` turns a `3` from any tier but `push` back into a `2`. A configuration error stays `2`. `ci_judges()` asks whether CI will really judge this commit, which is two tests, not the presence of the text `sdlc ci` anywhere in a workflow file: - a LINE THAT RUNS the gate -- a path ending in `bin/sdlc` with `ci` as its argument, outside comments and echoed strings. The Gatehouse finding on #34, and not theoretical: trentina carries `VERIFIER_JOB: sdlc ci` as a job name and `sdlc ci` inside an error message. - a TRIGGER THAT COVERS THE COMMIT: push, pull_request, pull_request_target or merge_group. Gatehouse raised this one on the restore -- a workflow_dispatch-only job judges nothing anybody pushed, so it must not unlock a deferral. A workflow that does not parse grants nothing. An unreadable workflow directory or file now says so on its way to refusing the deferral, instead of looking like a repo with no CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx
| return False | ||
| if not isinstance(doc, dict): | ||
| return False | ||
| on = doc.get("on", doc.get(True)) |
There was a problem hiding this comment.
HIGH (Bug Hunter): judges_a_commit treats the presence of a judging event as proof that this commit will trigger the workflow. It ignores event filters: for example, a workflow with on: push restricted to branches: [release] still passes this check when the commit is pushed to another branch. CodeQL can then defer even though CI will not judge that commit.
Suggestion: Do not grant deferral based only on the event name. Account for branch/path filters and other conditions that can exclude the commit, or conservatively require an unfiltered trigger before allowing deferral.
Evidence:
if isinstance(on, dict):
return bool(JUDGING_EVENTS & set(on))
There was a problem hiding this comment.
fixed in cb251cd — branch filters are now evaluated: a filter must reach the default branch under GitHub's glob rules (* stops at a /, ** does not), and branches-ignore or any branches list carrying a negation grants nothing, since GitHub applies those in order and ['**', '!main'] excludes main while matching everything. on: push filtered to [release] no longer unlocks a deferral.
| """ | ||
| if not judges_a_commit(text): | ||
| return False | ||
| for raw in text.splitlines(): |
There was a problem hiding this comment.
HIGH (Bug Hunter): runs_sdlc_ci searches every non-comment workflow line rather than only executable run steps. A workflow with a judging trigger and a YAML name such as name: bin/sdlc ci matches the regex and grants a deferral even though it never runs the gate. This can let a push through with no CI verdict.
Suggestion: Parse the workflow and inspect executable step commands for the gate invocation instead of scanning arbitrary YAML text.
Evidence:
for raw in text.splitlines():
line = raw.split("#", 1)[0]
if "echo" in line or "printf" in line:
continue
if _RUNS_SDLC_CI.search(line):
return True
There was a problem hiding this comment.
fixed in cb251cd — the file is parsed and only unconditional jobs.*.steps[].run scripts are searched. A YAML name:, a step name:, an env: value, a comment and an echoed string are mentions, not runs, and a job or step behind an if: grants nothing.
|
|
||
| # A path ending in `bin/sdlc`, then `ci` as its argument: how a workflow | ||
| # actually invokes the gate (`$RUNNER_TEMP/sdlc/bin/sdlc ci`). | ||
| _RUNS_SDLC_CI = re.compile(r"""(?:^|[\s;&|("'=])\S*bin/sdlc\s+ci\b""") |
There was a problem hiding this comment.
HIGH (Security Scan): The deferral check scans raw workflow text for a command-shaped substring instead of verifying that an enabled workflow step actually runs it. For example, a workflow with a push trigger and name: ./bin/sdlc ci satisfies this check even if no step runs the gate. A CodeQL failure can then be deferred and the push allowed through without CI judging the commit.
Suggestion: Parse the workflow YAML and inspect executable steps[].run content rather than arbitrary lines. Also verify that the relevant job and step are enabled and that workflow filters cover the commit before granting a deferral.
Evidence:
_RUNS_SDLC_CI = re.compile(r"""(?:^|[\s;&|("'=])\S*bin/sdlc\s+ci\b""")
...
for raw in text.splitlines():
line = raw.split("#", 1)[0]
if "echo" in line or "printf" in line:
continue
if _RUNS_SDLC_CI.search(line):
return True
There was a problem hiding this comment.
fixed in cb251cd — same fix: parsed steps[].run only, if: on either job or step disqualifies it, and the trigger must reach the default branch. Verified against the fleet: trentina, acquacotta, rotv, petit and shiva still grant a deferral; ashigaru does not, because its sdlc ci job sits behind ASHIGARU_ENABLED, which is the correct answer.
|
|
||
|
|
||
| def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None: | ||
| """Build the database and analyse it, or fail as 'could not run'.""" |
There was a problem hiding this comment.
HIGH (Documentation): The existing run_codeql docstring is stale: CodeQL failures can now defer to CI when CI will judge the same commit, rather than always failing as “could not run.”
Suggestion: Update the docstring to describe the conditional deferral and clarify that the failure remains “could not run” when CI will not judge the commit.
Evidence:
"""Build the database and analyse it, or fail as 'could not run'."""
There was a problem hiding this comment.
fixed in cb251cd — the run_codeql docstring and the analyze comment now say a failure is a 2 unless the push is eligible to defer, in which case codeql_broke makes it a 3.
|
|
||
|
|
||
| def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None: | ||
| """Build the database and analyse it, or fail as 'could not run'.""" |
There was a problem hiding this comment.
LOW (General Review): The run_codeql docstring and analysis-failure comment still describe every failure as a CANNOT_RUN/exit-2 outcome. The new codeql_broke path can instead raise DeferredError and return exit 3, so these comments no longer accurately describe the function.
Suggestion: Update the docstring and comment to say that CodeQL failures become exit 2 unless the push is eligible for deferral, in which case they become exit 3.
Evidence:
```python
"""Build the database and analyse it, or fail as 'could not run'."""
...
# Analysis failing is not the tool's verdict ON the code -- it means no
# verdict was reached, which the exit-code contract makes a 2.
codeql_broke(...)
<!-- gatehouse agent=general confidence=99 votes=1/1 -->
There was a problem hiding this comment.
fixed in cb251cd — the run_codeql docstring and the analyze comment now say a failure is a 2 unless the push is eligible to defer, in which case codeql_broke makes it a 3.
Gatehouse's findings on the restore, in two rounds, and every one was right: no amount of text matching answers "will CI judge this commit", and each wrong answer lets a push through with no verdict. `runs_sdlc_ci` now parses the file and asks two structural questions: - is there an UNCONDITIONAL `steps[].run` invoking a path ending in `bin/sdlc` with `ci` as its argument? A job name, a step name, an `env:` value, a comment and an echoed string are mentions, not runs. A job or step behind an `if:` may be skipped on this very commit, so it grants nothing -- ashigaru's `sdlc ci` sits behind `ASHIGARU_ENABLED`, and a broken CodeQL there is correctly still a 2. - does a trigger REACH the default branch, which is the branch XII's required check guards? `push`, `pull_request`, `pull_request_target` or `merge_group`, matched with GitHub's glob rules, where `*` stops at a `/` and `**` does not. `on: push` filtered to `[release]` was the hole: the right event name over a workflow that judges nothing anybody pushed. `branches-ignore` and any `branches` list carrying a negation grant nothing, because GitHub applies those in order and `['**', '!main']` excludes `main` while matching everything -- which a set of names cannot express. The default branch is read once in `ci_judges` from `SDLC_DEFAULT_BRANCH`, which the launcher now passes from its existing `default_branch()`, and handed down as an argument, so the classifier is a function of its inputs. Where it cannot be determined, a filtered trigger grants nothing. The `run_codeql` docstring and the analyze comment said every failure is a 2, which stopped being true with exit 3. Verified against the fleet: trentina, acquacotta, rotv, petit and shiva still grant a deferral; ashigaru does not, which is the right answer for a gate behind a kill switch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx
There was a problem hiding this comment.
Gatehouse found no issues.
Reviewed 1 commit since a90860e (incremental).
8 finding(s) dropped: their evidence is not in the diff.
Constitution 1.35.0 amended XII to add exit
3, could not run, deferred to CI (crunchtools/constitution#99). This restores the implementation that was reverted out of #34, with the narrowings that decision called for, and moves this repo's pin to v1.35.0 since the rule it implements lives there.3is the driver's to grant, per check.codeql_broke()raisesDeferredErroronly whenci_judges()holds; a tool's own3maps to2;run_stepturns a3from any tier butpushback into a2. A configuration error stays2. Today only CodeQL can defer, and a second check wanting it is a new conversation, not a config toggle.ci_judges()asks whether CI will really judge this commit, which is two tests rather than the textsdlc ciappearing somewhere in a workflow file:bin/sdlcwithcias its argument, outside comments and echoed strings. This was Gatehouse's finding on fix: CodeQL gets its memory budget and rebuilds a half-built database (#33) #34 and it is not theoretical — trentina carriesVERIFIER_JOB: sdlc cias a job name andsdlc ciinside an error message, and either would have unlocked a deferral on its own.push,pull_request,pull_request_targetormerge_group. Gatehouse raised this one against the restore itself: aworkflow_dispatch-only job judges nothing anybody pushed. A workflow that cannot be parsed grants nothing.An unreadable workflow directory or file now says so on the way to refusing the deferral, instead of looking like a repo with no CI.
Tests:
tests/test_codeql_resilience.pycovers the tier and CI conditions, both mention classes, both trigger classes and the unparsable file;tests/test_host_contract.shcovers the launcher's handling of3.Closes #33
🤖 Generated with Claude Code
https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx