Skip to content

fix: CodeQL gets its memory budget and rebuilds a half-built database (#33) - #34

Merged
fatherlinux merged 4 commits into
mainfrom
fix/33-codeql-cannot-run
Nov 8, 2026
Merged

fatherlinux merged 4 commits into
mainfrom
fix/33-codeql-cannot-run

Conversation

@fatherlinux

@fatherlinux fatherlinux commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Part of #33. The third item, a non-blocking "deferred to CI" status, needs an amendment to constitution XII (which allows only exit codes 0/1/2), so it is proposed in crunchtools/constitution#99 instead of shipped here.

What was wrong

  • Memory. CodeQL picks its own evaluator size and doesn't look at the container's limit. codeql resolve ram gave a 1509 MB heap under both --memory 8g and --memory 2g, and 1253 MB on josui-skills' dataset. The 8g the launcher budgets was never used.
  • Shared database directory. The database lives at /cache/codeql/<worktree basename>. Two checkouts with the same basename, or a push retried while the last one was still running, could delete each other's database mid-run. That leaves half-built caches, and analyses that die with nothing to read. This is the likely cause of josui-skills' exit 70: a clean checkout of josui-skills (13 tracked .py files) analyses in 9s, and CodeQL already leaves out .venv and gitignored worktrees.

Fix

  • --ram = ¾ of the smallest of SDLC_MEMORY (now passed in by the launcher), cgroup memory.max and MemTotal. That is 6144 MB at the default 8g; the measured heap went from 1253M to 4693M.
  • A per-project flock beside the database. The previous database is cleared, read-only parts included, and checked to be gone before create runs.
  • run_plan no longer reads run_serial's global _rc as its own result.

Verified on josui-skills, with sdlc layered onto the 0.5.3 image

  • sdlc codeql passes, and --ram=6144 reaches CodeQL.
  • After deleting db-python, and separately after leaving a read-only trap/ with no database, the next run rebuilds and passes.
  • New tests/test_codeql_resilience.py (added to the CI unit-test step) and a host-contract case for SDLC_MEMORY.

🤖 Generated with Claude Code

Three faults behind josui-skills' refused pushes:

- CodeQL sized its own evaluator (~1.2-1.5 GB heap) and ignored the
  container's 8g. `--ram` is now three quarters of the smallest of
  SDLC_MEMORY, the cgroup limit and MemTotal: 6144 MB at the default.
- Runs on one project shared one database directory and could delete
  each other's mid-run. They now take a lock, and the previous database
  is cleared and checked gone, so a half-built one rebuilds itself.
- When CodeQL itself breaks at the push tier, outside CI, in a repo whose
  workflow runs `sdlc ci`, the check exits 3: DEFERRED TO CI, printed and
  recorded, and the push goes through. `sdlc ci`, `sdlc codeql` and
  config errors stay a 2.

Also fixes run_plan reading run_serial's global `_rc` as its own result.

Closes #33

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 7 issues (2 critical, 4 high, 1 medium)

Full review.

4 finding(s) dropped: their evidence is not in the diff.

Comment thread sdlc
for line in Path("/proc/meminfo").read_text(encoding="utf-8").splitlines():
if line.startswith("MemTotal:"):
found.append(int(line.split()[1]) // 1024)
except OSError, ValueError, IndexError:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL (Bug Hunter): This except clause is invalid Python 3 syntax. The module fails to parse, so the sdlc image cannot run any gate.

Suggestion: Catch the exceptions as a tuple: except (OSError, ValueError, IndexError):.

Evidence:

except OSError, ValueError, IndexError:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive. except OSError, ValueError, IndexError: is valid on Python 3.14 (PEP 758), the interpreter this image runs. ruff format writes it this way for target 3.14, the file already uses the form at self_reported(), and every unit test ran this module in the image.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: except A, B, C: is valid Python 3.14 syntax (PEP 758), the image's interpreter; ruff writes it for target 3.14 and the tests ran this module in the image.

Comment thread sdlc Outdated
workflows = REPO / ".github" / "workflows"
try:
files = [*workflows.glob("*.yml"), *workflows.glob("*.yaml")]
return any(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Bug Hunter): ci_judges() treats any occurrence of sdlc ci in any workflow file as proof that CI will judge the commit. A comment, example, or inactive workflow step can match without actually running the check, causing CodeQL failures to be deferred when no CI verdict will follow.

Suggestion: Determine whether an applicable workflow actually invokes sdlc ci for the current push, rather than searching raw workflow text for the phrase.

Evidence:

return any(
    re.search(r"\bsdlc\s+ci\b", f.read_text(encoding="utf-8", errors="replace"))
    for f in files
)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moot as of 80c1f70: ci_judges() was removed along with the deferral. The tightening (match a workflow line that runs bin/sdlc ci, not any mention of the text) is recorded in crunchtools/constitution#99 for whenever the deferral comes back.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 80c1f70
ci_judges() was removed with the deferral.

Comment thread sdlc Outdated
workflows = REPO / ".github" / "workflows"
try:
files = [*workflows.glob("*.yml"), *workflows.glob("*.yaml")]
return any(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Security Scan): ci_judges() treats any workflow file containing the text sdlc ci as proof that CI will judge the commit. A comment, disabled workflow, or workflow that runs only for unrelated events can satisfy this check. If CodeQL then fails at push time, the gate defers and lets the push through without a CI verdict, bypassing the intended security check.

Suggestion: Validate the workflow structure and relevant triggers rather than searching for a text match; only defer when an enabled workflow is configured to run sdlc ci for the commit being pushed. If that cannot be established reliably, do not defer.

Evidence:

return any(
    re.search(r"\bsdlc\s+ci\b", f.read_text(encoding="utf-8", errors="replace"))
    for f in files
)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate of the thread above. ci_judges() was removed in 80c1f70.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 80c1f70
ci_judges() was removed with the deferral.

Comment thread sdlc
the dataset, and /tmp, which is a tmpfs and so is charged to this same
cgroup. 2048 is CodeQL's own floor; it rounds anything smaller up.
"""
limit = memory_limit_mb()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Performance Check): When the detected memory limit is below 2048 MB, this forces CodeQL's RAM setting above that limit. For example, a 1 GB budget produces --ram=2048, despite the code having detected the lower ceiling. CodeQL can then exceed the container's budget and be killed for OOM, making analysis less reliable and potentially triggering memory pressure on the host.

Suggestion: Avoid setting --ram above the detected limit. For limits below CodeQL's supported minimum, handle that case explicitly rather than raising the configured budget beyond the available memory.

Evidence:

limit = memory_limit_mb()
return max(2048, limit * 3 // 4) if limit else 2048

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted as is. 2048 MB is CodeQL's own floor: codeql resolve ram --help says smaller values "will be transparently rounded up". Passing less would change nothing, and passing 2048 makes the effective value visible in the log. A container under 2 GB cannot run CodeQL at all, and that failure is a 2.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: 2048 MB is CodeQL's own floor, and resolve ram --help says smaller values are transparently rounded up, so passing less changes nothing. Under 2 GB CodeQL cannot run, which is a 2.

Comment thread sdlc


def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None:
"""Build the database and analyse it, or fail as 'could not run'."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Documentation): run_codeql's docstring says failures are reported as “could not run,” but the function now may raise DeferredError when CodeQL fails at an eligible push tier. The docstring omits this changed outcome.

Suggestion: Update the docstring to describe both outcomes: CodeQL failures raise DeferredError when CI will judge the commit, and otherwise raise the normal cannot-run error.

Evidence:

"""Build the database and analyse it, or fail as 'could not run'."""

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by 80c1f70: run_codeql calls cannot() again and never raises anything else, so the docstring is accurate.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 80c1f70
run_codeql only calls cannot() again, so the docstring is accurate.

Comment thread sdlc Outdated
Only for the TOOL breaking. A config the repo got wrong stays a 2 wherever
it is found, because CI would fail on it in exactly the same way.
"""
if ci_judges():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL (Constitution): A CodeQL execution failure is not the tool's verdict, so the constitution requires exit 2 and says it must never be worked around. This code instead turns that failure into a deferral when CI may judge later; the launcher allows that deferral through on push.

Suggestion: Keep CodeQL execution failures as CannotRunError/exit 2. Do not let a later CI run convert a check that could not run into a passing push.

Evidence:

if ci_judges():
    raise DeferredError(msg)
cannot(msg)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and fixed in 80c1f70. XII allows only 0/1/2, so the exit-3 deferral has been removed from this PR and proposed as an amendment in crunchtools/constitution#99. A broken CodeQL is a 2 again in every tier.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 80c1f70
The exit-3 deferral was removed; it is proposed as a constitution amendment in crunchtools/constitution#99.

Comment thread sdlc
for line in Path("/proc/meminfo").read_text(encoding="utf-8").splitlines():
if line.startswith("MemTotal:"):
found.append(int(line.split()[1]) // 1024)
except OSError, ValueError, IndexError:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH (Constitution): This except clause is invalid Python 3 syntax, so sdlc cannot execute. The constitution’s contract test requires every tool to execute, not merely be present.

Suggestion: Use a tuple of exception types: except (OSError, ValueError, IndexError):.

Evidence:

except OSError, ValueError, IndexError:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

False positive, same as the thread above: PEP 758 syntax, valid on the image's Python 3.14 and tested there.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: except A, B, C: is valid Python 3.14 syntax (PEP 758), the image's interpreter; tested in the image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found no issues.

Reviewed 1 commit since 6a11c8a (incremental).

Constitution XII says every check exits 0, 1 or 2. A fourth status that
lets a push through is an amendment to that rule for every Python repo,
so it is proposed to the constitution instead of shipped here. What stays
is the cause of josui-skills' refused pushes: CodeQL's memory and the
shared, half-built database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found 1 issues (1 medium)

Reviewed 1 commit since 33c9b9e (incremental).

3 finding(s) dropped: their evidence is not in the diff.

Comment thread sdlc
def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None:
"""Build the database and analyse it, or fail as 'could not run'."""
step("codeql database create")
rc, _ = run(create, env=project_env())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Test Coverage): The changed CodeQL failure paths are not covered by the modified resilience test, which now exercises database handling and memory only. Neither a nonzero database-create result nor a nonzero analyze result is tested for the new CannotRunError behavior.

Suggestion: Add tests that mock run to return a nonzero status for each command in turn, then assert run_codeql raises CannotRunError and the command-specific failure is reported.

Evidence:

if rc != 0:
    cannot(f"codeql could not build a database (exit {rc}); nothing was analysed")
...
if rc != 0:
    cannot(f"codeql database analyze failed (exit {rc}); nothing was judged")

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the next commit. failures() in tests/test_codeql_resilience.py stubs run to fail create, then analyze, and asserts each raises CannotRunError naming the step and the exit status.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 95a01eb
failures() in tests/test_codeql_resilience.py covers a failed create and a failed analyze.

@fatherlinux fatherlinux changed the title fix: a CodeQL that cannot run no longer blocks the push (#33) fix: CodeQL gets its memory budget and rebuilds a half-built database (#33) Nov 8, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gatehouse found no issues.

Reviewed 1 commit since 80c1f70 (incremental).

@fatherlinux
fatherlinux merged commit 16c4c2f into main Nov 8, 2026
44 of 52 checks passed
@fatherlinux
fatherlinux deleted the fix/33-codeql-cannot-run branch November 8, 2026 15:01
fatherlinux added a commit that referenced this pull request Nov 8, 2026
Constitution 1.35.0 amended XII to add exit `3`, could not run, deferred to
CI (constitution#99), so the implementation reverted out of #34 comes back,
with the narrowings that decision called for. This repo's pin moves to
v1.35.0 in the same change, because the rule it implements lives there.

`3` is granted by this driver, per check, and today only CodeQL has it:
`codeql_broke()` raises `DeferredError` when `ci_judges()` holds, a tool's
own `3` maps to `2`, and `run_step` turns a `3` from any tier but `push`
back into a `2`. A configuration error stays `2`.

`ci_judges()` asks whether CI will really judge this commit, which is two
tests, not the presence of the text `sdlc ci` anywhere in a workflow file:

- a LINE THAT RUNS the gate -- a path ending in `bin/sdlc` with `ci` as its
  argument, outside comments and echoed strings. The Gatehouse finding on
  #34, and not theoretical: trentina carries `VERIFIER_JOB: sdlc ci` as a
  job name and `sdlc ci` inside an error message.
- a TRIGGER THAT COVERS THE COMMIT: push, pull_request, pull_request_target
  or merge_group. Gatehouse raised this one on the restore -- a
  workflow_dispatch-only job judges nothing anybody pushed, so it must not
  unlock a deferral. A workflow that does not parse grants nothing.

An unreadable workflow directory or file now says so on its way to refusing
the deferral, instead of looking like a repo with no CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx
fatherlinux added a commit that referenced this pull request Nov 8, 2026
#36)

* feat: a push may defer a broken CodeQL to the required `sdlc ci` (#33)

Constitution 1.35.0 amended XII to add exit `3`, could not run, deferred to
CI (constitution#99), so the implementation reverted out of #34 comes back,
with the narrowings that decision called for. This repo's pin moves to
v1.35.0 in the same change, because the rule it implements lives there.

`3` is granted by this driver, per check, and today only CodeQL has it:
`codeql_broke()` raises `DeferredError` when `ci_judges()` holds, a tool's
own `3` maps to `2`, and `run_step` turns a `3` from any tier but `push`
back into a `2`. A configuration error stays `2`.

`ci_judges()` asks whether CI will really judge this commit, which is two
tests, not the presence of the text `sdlc ci` anywhere in a workflow file:

- a LINE THAT RUNS the gate -- a path ending in `bin/sdlc` with `ci` as its
  argument, outside comments and echoed strings. The Gatehouse finding on
  #34, and not theoretical: trentina carries `VERIFIER_JOB: sdlc ci` as a
  job name and `sdlc ci` inside an error message.
- a TRIGGER THAT COVERS THE COMMIT: push, pull_request, pull_request_target
  or merge_group. Gatehouse raised this one on the restore -- a
  workflow_dispatch-only job judges nothing anybody pushed, so it must not
  unlock a deferral. A workflow that does not parse grants nothing.

An unreadable workflow directory or file now says so on its way to refusing
the deferral, instead of looking like a repo with no CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx

* fix: read the parsed workflow, not its text, before deferring (#33)

Gatehouse's findings on the restore, in two rounds, and every one was right:
no amount of text matching answers "will CI judge this commit", and each
wrong answer lets a push through with no verdict.

`runs_sdlc_ci` now parses the file and asks two structural questions:

- is there an UNCONDITIONAL `steps[].run` invoking a path ending in
  `bin/sdlc` with `ci` as its argument? A job name, a step name, an `env:`
  value, a comment and an echoed string are mentions, not runs. A job or
  step behind an `if:` may be skipped on this very commit, so it grants
  nothing -- ashigaru's `sdlc ci` sits behind `ASHIGARU_ENABLED`, and a
  broken CodeQL there is correctly still a 2.
- does a trigger REACH the default branch, which is the branch XII's
  required check guards? `push`, `pull_request`, `pull_request_target` or
  `merge_group`, matched with GitHub's glob rules, where `*` stops at a `/`
  and `**` does not. `on: push` filtered to `[release]` was the hole: the
  right event name over a workflow that judges nothing anybody pushed.
  `branches-ignore` and any `branches` list carrying a negation grant
  nothing, because GitHub applies those in order and `['**', '!main']`
  excludes `main` while matching everything -- which a set of names cannot
  express.

The default branch is read once in `ci_judges` from `SDLC_DEFAULT_BRANCH`,
which the launcher now passes from its existing `default_branch()`, and
handed down as an argument, so the classifier is a function of its inputs.
Where it cannot be determined, a filtered trigger grants nothing. The
`run_codeql` docstring and the analyze comment said every failure is a 2,
which stopped being true with exit 3.

Verified against the fleet: trentina, acquacotta, rotv, petit and shiva
still grant a deferral; ashigaru does not, which is the right answer for a
gate behind a kill switch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant