Skip to content

feat: a push may defer a broken CodeQL to the required sdlc ci (#33) - #36

Merged
fatherlinux merged 2 commits into
mainfrom
fix/33-defer-to-ci
Nov 8, 2026
Merged

fatherlinux merged 2 commits into
mainfrom
fix/33-defer-to-ci

Conversation

@fatherlinux

Copy link
Copy Markdown
Member

Constitution 1.35.0 amended XII to add exit 3, could not run, deferred to CI (crunchtools/constitution#99). This restores the implementation that was reverted out of #34, with the narrowings that decision called for, and moves this repo's pin to v1.35.0 since the rule it implements lives there.

3 is the driver's to grant, per check. codeql_broke() raises DeferredError only when ci_judges() holds; a tool's own 3 maps to 2; run_step turns a 3 from any tier but push back into a 2. A configuration error stays 2. Today only CodeQL can defer, and a second check wanting it is a new conversation, not a config toggle.

ci_judges() asks whether CI will really judge this commit, which is two tests rather than the text sdlc ci appearing somewhere in a workflow file:

  • A line that runs the gate: a path ending in bin/sdlc with ci as its argument, outside comments and echoed strings. This was Gatehouse's finding on fix: CodeQL gets its memory budget and rebuilds a half-built database (#33) #34 and it is not theoretical — trentina carries VERIFIER_JOB: sdlc ci as a job name and sdlc ci inside an error message, and either would have unlocked a deferral on its own.
  • A trigger that covers the commit: push, pull_request, pull_request_target or merge_group. Gatehouse raised this one against the restore itself: a workflow_dispatch-only job judges nothing anybody pushed. A workflow that cannot be parsed grants nothing.

An unreadable workflow directory or file now says so on the way to refusing the deferral, instead of looking like a repo with no CI.

Tests: tests/test_codeql_resilience.py covers the tier and CI conditions, both mention classes, both trigger classes and the unparsable file; tests/test_host_contract.sh covers the launcher's handling of 3.

Closes #33

🤖 Generated with Claude Code

https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx

Constitution 1.35.0 amended XII to add exit `3`, could not run, deferred to
CI (constitution#99), so the implementation reverted out of #34 comes back,
with the narrowings that decision called for. This repo's pin moves to
v1.35.0 in the same change, because the rule it implements lives there.

`3` is granted by this driver, per check, and today only CodeQL has it:
`codeql_broke()` raises `DeferredError` when `ci_judges()` holds, a tool's
own `3` maps to `2`, and `run_step` turns a `3` from any tier but `push`
back into a `2`. A configuration error stays `2`.

`ci_judges()` asks whether CI will really judge this commit, which is two
tests, not the presence of the text `sdlc ci` anywhere in a workflow file:

- a LINE THAT RUNS the gate -- a path ending in `bin/sdlc` with `ci` as its
  argument, outside comments and echoed strings. The Gatehouse finding on
  #34, and not theoretical: trentina carries `VERIFIER_JOB: sdlc ci` as a
  job name and `sdlc ci` inside an error message.
- a TRIGGER THAT COVERS THE COMMIT: push, pull_request, pull_request_target
  or merge_group. Gatehouse raised this one on the restore -- a
  workflow_dispatch-only job judges nothing anybody pushed, so it must not
  unlock a deferral. A workflow that does not parse grants nothing.

An unreadable workflow directory or file now says so on its way to refusing
the deferral, instead of looking like a repo with no CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 5 issues (4 high, 1 low)

Full review.

2 finding(s) dropped: their evidence is not in the diff.

Comment thread sdlc
return False
if not isinstance(doc, dict):
return False
on = doc.get("on", doc.get(True))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Bug Hunter): judges_a_commit treats the presence of a judging event as proof that this commit will trigger the workflow. It ignores event filters: for example, a workflow with on: push restricted to branches: [release] still passes this check when the commit is pushed to another branch. CodeQL can then defer even though CI will not judge that commit.

Suggestion: Do not grant deferral based only on the event name. Account for branch/path filters and other conditions that can exclude the commit, or conservatively require an unfiltered trigger before allowing deferral.

Evidence:

if isinstance(on, dict):
    return bool(JUDGING_EVENTS & set(on))

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in cb251cd — branch filters are now evaluated: a filter must reach the default branch under GitHub's glob rules (* stops at a /, ** does not), and branches-ignore or any branches list carrying a negation grants nothing, since GitHub applies those in order and ['**', '!main'] excludes main while matching everything. on: push filtered to [release] no longer unlocks a deferral.

Comment thread sdlc Outdated
"""
if not judges_a_commit(text):
return False
for raw in text.splitlines():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Bug Hunter): runs_sdlc_ci searches every non-comment workflow line rather than only executable run steps. A workflow with a judging trigger and a YAML name such as name: bin/sdlc ci matches the regex and grants a deferral even though it never runs the gate. This can let a push through with no CI verdict.

Suggestion: Parse the workflow and inspect executable step commands for the gate invocation instead of scanning arbitrary YAML text.

Evidence:

for raw in text.splitlines():
    line = raw.split("#", 1)[0]
    if "echo" in line or "printf" in line:
        continue
    if _RUNS_SDLC_CI.search(line):
        return True

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in cb251cd — the file is parsed and only unconditional jobs.*.steps[].run scripts are searched. A YAML name:, a step name:, an env: value, a comment and an echoed string are mentions, not runs, and a job or step behind an if: grants nothing.

Comment thread sdlc Outdated

# A path ending in `bin/sdlc`, then `ci` as its argument: how a workflow
# actually invokes the gate (`$RUNNER_TEMP/sdlc/bin/sdlc ci`).
_RUNS_SDLC_CI = re.compile(r"""(?:^|[\s;&|("'=])\S*bin/sdlc\s+ci\b""")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Security Scan): The deferral check scans raw workflow text for a command-shaped substring instead of verifying that an enabled workflow step actually runs it. For example, a workflow with a push trigger and name: ./bin/sdlc ci satisfies this check even if no step runs the gate. A CodeQL failure can then be deferred and the push allowed through without CI judging the commit.

Suggestion: Parse the workflow YAML and inspect executable steps[].run content rather than arbitrary lines. Also verify that the relevant job and step are enabled and that workflow filters cover the commit before granting a deferral.

Evidence:

_RUNS_SDLC_CI = re.compile(r"""(?:^|[\s;&|("'=])\S*bin/sdlc\s+ci\b""")
...
for raw in text.splitlines():
    line = raw.split("#", 1)[0]
    if "echo" in line or "printf" in line:
        continue
    if _RUNS_SDLC_CI.search(line):
        return True

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in cb251cd — same fix: parsed steps[].run only, if: on either job or step disqualifies it, and the trigger must reach the default branch. Verified against the fleet: trentina, acquacotta, rotv, petit and shiva still grant a deferral; ashigaru does not, because its sdlc ci job sits behind ASHIGARU_ENABLED, which is the correct answer.

Comment thread sdlc Outdated


def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None:
"""Build the database and analyse it, or fail as 'could not run'."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Documentation): The existing run_codeql docstring is stale: CodeQL failures can now defer to CI when CI will judge the same commit, rather than always failing as “could not run.”

Suggestion: Update the docstring to describe the conditional deferral and clarify that the failure remains “could not run” when CI will not judge the commit.

Evidence:

"""Build the database and analyse it, or fail as 'could not run'."""

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in cb251cd — the run_codeql docstring and the analyze comment now say a failure is a 2 unless the push is eligible to defer, in which case codeql_broke makes it a 3.

Comment thread sdlc Outdated


def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None:
"""Build the database and analyse it, or fail as 'could not run'."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW (General Review): The run_codeql docstring and analysis-failure comment still describe every failure as a CANNOT_RUN/exit-2 outcome. The new codeql_broke path can instead raise DeferredError and return exit 3, so these comments no longer accurately describe the function.

Suggestion: Update the docstring and comment to say that CodeQL failures become exit 2 unless the push is eligible for deferral, in which case they become exit 3.

Evidence:

```python
"""Build the database and analyse it, or fail as 'could not run'."""
...
# Analysis failing is not the tool's verdict ON the code -- it means no
# verdict was reached, which the exit-code contract makes a 2.
codeql_broke(...)

<!-- gatehouse agent=general confidence=99 votes=1/1 -->

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in cb251cd — the run_codeql docstring and the analyze comment now say a failure is a 2 unless the push is eligible to defer, in which case codeql_broke makes it a 3.

Gatehouse's findings on the restore, in two rounds, and every one was right:
no amount of text matching answers "will CI judge this commit", and each
wrong answer lets a push through with no verdict.

`runs_sdlc_ci` now parses the file and asks two structural questions:

- is there an UNCONDITIONAL `steps[].run` invoking a path ending in
  `bin/sdlc` with `ci` as its argument? A job name, a step name, an `env:`
  value, a comment and an echoed string are mentions, not runs. A job or
  step behind an `if:` may be skipped on this very commit, so it grants
  nothing -- ashigaru's `sdlc ci` sits behind `ASHIGARU_ENABLED`, and a
  broken CodeQL there is correctly still a 2.
- does a trigger REACH the default branch, which is the branch XII's
  required check guards? `push`, `pull_request`, `pull_request_target` or
  `merge_group`, matched with GitHub's glob rules, where `*` stops at a `/`
  and `**` does not. `on: push` filtered to `[release]` was the hole: the
  right event name over a workflow that judges nothing anybody pushed.
  `branches-ignore` and any `branches` list carrying a negation grant
  nothing, because GitHub applies those in order and `['**', '!main']`
  excludes `main` while matching everything -- which a set of names cannot
  express.

The default branch is read once in `ci_judges` from `SDLC_DEFAULT_BRANCH`,
which the launcher now passes from its existing `default_branch()`, and
handed down as an argument, so the classifier is a function of its inputs.
Where it cannot be determined, a filtered trigger grants nothing. The
`run_codeql` docstring and the analyze comment said every failure is a 2,
which stopped being true with exit 3.

Verified against the fleet: trentina, acquacotta, rotv, petit and shiva
still grant a deferral; ashigaru does not, which is the right answer for a
gate behind a kill switch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found no issues.

Reviewed 1 commit since a90860e (incremental).

8 finding(s) dropped: their evidence is not in the diff.

@fatherlinux
fatherlinux merged commit bae0de6 into main Nov 8, 2026
31 checks passed
@fatherlinux
fatherlinux deleted the fix/33-defer-to-ci branch November 8, 2026 15:19
This was referenced Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A codeql gate that cannot run blocks the push instead of reporting it

1 participant