Skip to content

A codeql gate that cannot run blocks the push instead of reporting it #33

Closed

Description

@fatherlinux

For Scott

What: The pre-push security scan in josui-skills crashed instead of finishing, and because it is designed to fail closed, it refused every push to that repo until I went around it by hand.

Why it matters: The scan failing closed is correct — a gate that silently passes when it breaks is worse than no gate. But a gate that cannot run is indistinguishable, from the user's seat, from a gate that found a problem, and the only way past it is --no-verify, which skips all the local gates rather than the one that is broken. Teach people that reflex and the next real finding gets skipped the same way. If we never fix it, the fallback becomes habit.

Recommendation: Give the scan enough memory to finish, and when it still cannot run, let it say so and defer to CI rather than block. Size S.

Needs from you: nothing.

Where it fits

Feature or program: standalone, python-sdlc tooling. Related: crunchtools/josui-skills#91, which had to be pushed with --no-verify and merged on CI's verdict instead.
Why now: it blocks pushes to josui-skills today, and that repo holds the skills every session loads. It can wait only as long as nobody needs to change a skill.

For the agent

Problem: sdlc-push runs codeql database create then codeql database analyze over python-security-extended.qls. In josui-skills the database builds cleanly, then analyze dies:

[2026-11-08 11:27:28] Calling plumbing command: codeql resolve ram --dataset=/cache/codeql/josui-skills/db-python
[2026-11-08 11:27:28] Plumbing command codeql resolve ram completed:
                      [ "-J-Xmx1253M" ]
[2026-11-08 11:27:28] Plumbing command codeql database run-queries completed with status 70.
[2026-11-08 11:27:28] Exiting with code 70

The gate then reports GATE CANNOT RUN: codeql database analyze failed (exit 2); nothing was judged and the push is refused. Failed on 3 of 3 push attempts this session.

1253 MB of evaluator heap is the number to look at. The repo has 4583 *.py files. Partial results were still written (results/ appeared 12s after exit 70), so this looks like the evaluator being killed mid-run rather than a malformed query set.

Two separate faults, and the first is already fixed by hand:

  1. A previous run had been killed mid-build and left /cache/codeql/josui-skills holding trap/ and working/ with no db-python. CodeQL then refused to overwrite it: Ran with database overwriting enabled, but the directory does not appear to be a CodeQL database or database cluster. Please check you do indeed wish to delete it, and do so manually. I deleted the directory; creation works again. A half-built cache should not require a human to clear it — the gate should detect an unfinalized database and rebuild it.
  2. Analyze then failed as above, which is the open part.

Measured, so the scope is known rather than guessed: this is not a general failure and not a simple function of repo size. Every other repo in crunchtools-hook-cache/codeql/ has a complete db-python + results + src.zip, including trentina at 8245 *.py files — more than josui-skills. So something about this repo's closure, not its file count, is pushing the evaluator over. Worth finding out what before raising the limit blindly.

C=~/.local/share/containers/storage/volumes/crunchtools-hook-cache/_data/codeql
for d in "$C"/*/; do [ -d "$d/db-python" ] || echo "unfinalized: $(basename $d)"; done

Files: the sdlc launcher's codeql gate and wherever --ram / container memory for it is set; [tool.sdlc.push] in a consuming repo's pyproject.toml for how the gate is selected.

Proposed fix:

  1. Raise the evaluator's memory, or pass --ram explicitly instead of letting codeql resolve ram derive 1253 MB from a constrained container.
  2. Treat an unfinalized database in the cache as a cache miss: detect missing db-python and rebuild rather than failing.
  3. Separate "the gate ran and found nothing" from "the gate could not run". A gate that cannot run should fail the push only when no CI will judge the same commit; otherwise it should warn loudly, name CI as the judge, and let the push through. Blocking with no selective override is what makes --no-verify the only door.

Done when:

  • git push on josui-skills passes sdlc-push with no --no-verify
  • Deleting db-python from a cached database and pushing rebuilds it instead of erroring
  • A gate forced to fail to start produces a distinct, non-blocking verdict naming CI, verified on a scratch branch

Out of scope: changing which queries the suite runs, .trivyignore, and any narrowing of scanned paths — reducing coverage to fit the memory limit is the wrong fix and would quietly shrink what gets scanned.

Activity

  1. fatherlinux commented on Nov 8, 2026

    @fatherlinux
    MemberAuthor

    Item 3 is unblocked. I approved the XII amendment as CTO on crunchtools/constitution#99; it merged as constitution#100 and constitution.md now carries exit 3, "could not run, deferred to CI", at version 1.35.0.

    The amendment grants 3 on tighter terms than this issue proposed, so the implementation has to match them: 3 is granted per check by python-sdlc and is never a status a tool returns, today only CodeQL; it requires push tier, outside CI, that check also in [tool.sdlc.ci], a workflow line that actually runs bin/sdlc ci (not a comment or a job name — the Gatehouse finding on #34), and the tool itself broke. A config error stays 2. Anywhere else 3 is 2.

    So the work here is to restore 6a11c8a plus the 33c9b9e shellcheck fix on top of main, with ci_judges() tightened to that bin/sdlc ci match. Items 1 and 2 ship in #34 as they stand.

    One sequencing note: the constitution refs were bumped to @v1.35.0 on main but the annotated tag is not cut yet, so pin python-sdlc's constitution reference only once v1.35.0 exists.

  2. fatherlinux commented on Nov 8, 2026

    @fatherlinux
    MemberAuthor

    Correcting the sequencing note above. v1.35.0 is now tagged and released (annotated tag on d7261ac), so the literal condition I gave is met — but do not move python-sdlc's constitution pin on the strength of it.

    Creating the missing release objects for 1.33.0 and 1.34.0 exposed that the whole fleet, this repo included, is still inheriting v1.31.0; scripts/fleet-drift.py had been measuring against v1.32.0 because it reads releases/latest and those two versions had tags with no release. The fleet bump is one decision for all 70 repos, it retags latest/main and touches systemd units on gloval, and it is Scott's call. It is in front of him now.

    So: restoring 6a11c8a + 33c9b9e with the tightened ci_judges() is unblocked and can land against the current pin. The pin bump rides with the fleet, separately.

  3. fatherlinux commented on Nov 8, 2026

    @fatherlinux
    MemberAuthor

    Reopening: this closed on a commit message, not on the work.

    #34's branch kept 6a11c8a's original message, which ended "Closes #33", after the deferral code had been stripped out of the diff. So merging 16c4c2f closed this issue while items 1 and 2 shipped and item 3 did not. Verified against origin/main: no DeferredError, no codeql_broke, no ci_judges, no SDLC_TIER, no DEFERRED.

    What shipped in #34: the --ram budget and the per-project flock with a proven-empty database directory. Those close items 1 and 2, and they fix the josui-skills blocker.

    What remains, and it is now unblocked rather than waiting:

    Constitution XII was amended and ratified — exit 3, "could not run, deferred to CI", in v1.35.0 (crunchtools/constitution#99, merged as #100). The implementation has to match the amendment's terms, which are tighter than this issue originally proposed: 3 is granted per check by python-sdlc and is never a status a tool returns, today only CodeQL; it requires push tier, outside CI, that check also present in [tool.sdlc.ci], a workflow line that actually runs bin/sdlc ci (not a comment or a job name), and the tool itself to have broken. A configuration error stays 2. Anywhere else 3 is 2.

    The reference implementation is 6a11c8a plus the 33c9b9e shellcheck fix, in #34's history. It needs restoring as the deferral subset only, since its memory and locking halves are already on main: DEFERRED/DeferredError, ci_judges(), codeql_broke(), the SDLC_TIER export and pass-through, run_step's demotion of a non-push 3 to 2, the deferred() warning, the 3 branches in run_serial/run_lane/_close_check, run_plan's own _plan_rc, the usage text, docs/codeql.md, and the deferral() case in tests/test_codeql_resilience.py plus the SDLC_TIER host-contract case. ci_judges() gets the bin/sdlc ci tightening, which the reference version does not have.

    Note for whoever picks this up: this repo carries the ashigaru-observe topic, so it is watch-only and Ashigaru will not take this itself.

    Done when: a CodeQL forced to fail to start at the push tier, in a repo whose CI runs bin/sdlc ci, prints DEFERRED TO CI, records it in last-push.json, and lets the push through; the same failure under sdlc ci, under sdlc codeql, with CI set, or in a repo without that workflow line still exits 2 and blocks; and a config error still exits 2.

  4. fatherlinux commented on Nov 8, 2026

    @fatherlinux
    MemberAuthor

    Items 1 and 2 shipped in #34 and are released in 0.5.4 (#35).

    • CodeQL's evaluator now gets --ram = ¾ of the memory budget: 6144 MB at the default 8g, a 4.6 GB heap where it used to get 1253 MB.
    • Runs on the same project take a lock and wait for each other instead of deleting each other's database mid-run.
    • A half-built database is cleared, checked to be gone, and rebuilt. Verified on josui-skills with db-python deleted, and again with a read-only trap/ left behind.

    On the cause: a clean josui-skills checkout analyses in 9s (13 tracked .py files), and CodeQL already leaves out .venv and gitignored worktrees. So the 4583 files were not the cause. Two runs sharing /cache/codeql/josui-skills fits the evidence best, including results/ appearing after the exit 70.

    Item 3, the non-blocking "deferred to CI" status, contradicts constitution XII ("Every check exits 0, 1, or 2"). It is proposed as an amendment in crunchtools/constitution#99 rather than shipped here. Leaving this open until that is decided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions