Repository navigation
fix: CodeQL gets its memory budget and rebuilds a half-built database (#33) - #34
Conversation
Three faults behind josui-skills' refused pushes: - CodeQL sized its own evaluator (~1.2-1.5 GB heap) and ignored the container's 8g. `--ram` is now three quarters of the smallest of SDLC_MEMORY, the cgroup limit and MemTotal: 6144 MB at the default. - Runs on one project shared one database directory and could delete each other's mid-run. They now take a lock, and the previous database is cleared and checked gone, so a half-built one rebuilds itself. - When CodeQL itself breaks at the push tier, outside CI, in a repo whose workflow runs `sdlc ci`, the check exits 3: DEFERRED TO CI, printed and recorded, and the push goes through. `sdlc ci`, `sdlc codeql` and config errors stay a 2. Also fixes run_plan reading run_serial's global `_rc` as its own result. Closes #33 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| for line in Path("/proc/meminfo").read_text(encoding="utf-8").splitlines(): | ||
| if line.startswith("MemTotal:"): | ||
| found.append(int(line.split()[1]) // 1024) | ||
| except OSError, ValueError, IndexError: |
There was a problem hiding this comment.
CRITICAL (Bug Hunter): This except clause is invalid Python 3 syntax. The module fails to parse, so the sdlc image cannot run any gate.
Suggestion: Catch the exceptions as a tuple: except (OSError, ValueError, IndexError):.
Evidence:
except OSError, ValueError, IndexError:
There was a problem hiding this comment.
False positive. except OSError, ValueError, IndexError: is valid on Python 3.14 (PEP 758), the interpreter this image runs. ruff format writes it this way for target 3.14, the file already uses the form at self_reported(), and every unit test ran this module in the image.
There was a problem hiding this comment.
not a bug: except A, B, C: is valid Python 3.14 syntax (PEP 758), the image's interpreter; ruff writes it for target 3.14 and the tests ran this module in the image.
| workflows = REPO / ".github" / "workflows" | ||
| try: | ||
| files = [*workflows.glob("*.yml"), *workflows.glob("*.yaml")] | ||
| return any( |
There was a problem hiding this comment.
HIGH (Bug Hunter): ci_judges() treats any occurrence of sdlc ci in any workflow file as proof that CI will judge the commit. A comment, example, or inactive workflow step can match without actually running the check, causing CodeQL failures to be deferred when no CI verdict will follow.
Suggestion: Determine whether an applicable workflow actually invokes sdlc ci for the current push, rather than searching raw workflow text for the phrase.
Evidence:
return any(
re.search(r"\bsdlc\s+ci\b", f.read_text(encoding="utf-8", errors="replace"))
for f in files
)
There was a problem hiding this comment.
Moot as of 80c1f70: ci_judges() was removed along with the deferral. The tightening (match a workflow line that runs bin/sdlc ci, not any mention of the text) is recorded in crunchtools/constitution#99 for whenever the deferral comes back.
There was a problem hiding this comment.
fixed in 80c1f70
ci_judges() was removed with the deferral.
| workflows = REPO / ".github" / "workflows" | ||
| try: | ||
| files = [*workflows.glob("*.yml"), *workflows.glob("*.yaml")] | ||
| return any( |
There was a problem hiding this comment.
HIGH (Security Scan): ci_judges() treats any workflow file containing the text sdlc ci as proof that CI will judge the commit. A comment, disabled workflow, or workflow that runs only for unrelated events can satisfy this check. If CodeQL then fails at push time, the gate defers and lets the push through without a CI verdict, bypassing the intended security check.
Suggestion: Validate the workflow structure and relevant triggers rather than searching for a text match; only defer when an enabled workflow is configured to run sdlc ci for the commit being pushed. If that cannot be established reliably, do not defer.
Evidence:
return any(
re.search(r"\bsdlc\s+ci\b", f.read_text(encoding="utf-8", errors="replace"))
for f in files
)
There was a problem hiding this comment.
Duplicate of the thread above. ci_judges() was removed in 80c1f70.
There was a problem hiding this comment.
fixed in 80c1f70
ci_judges() was removed with the deferral.
| the dataset, and /tmp, which is a tmpfs and so is charged to this same | ||
| cgroup. 2048 is CodeQL's own floor; it rounds anything smaller up. | ||
| """ | ||
| limit = memory_limit_mb() |
There was a problem hiding this comment.
MEDIUM (Performance Check): When the detected memory limit is below 2048 MB, this forces CodeQL's RAM setting above that limit. For example, a 1 GB budget produces --ram=2048, despite the code having detected the lower ceiling. CodeQL can then exceed the container's budget and be killed for OOM, making analysis less reliable and potentially triggering memory pressure on the host.
Suggestion: Avoid setting --ram above the detected limit. For limits below CodeQL's supported minimum, handle that case explicitly rather than raising the configured budget beyond the available memory.
Evidence:
limit = memory_limit_mb()
return max(2048, limit * 3 // 4) if limit else 2048
There was a problem hiding this comment.
Accepted as is. 2048 MB is CodeQL's own floor: codeql resolve ram --help says smaller values "will be transparently rounded up". Passing less would change nothing, and passing 2048 makes the effective value visible in the log. A container under 2 GB cannot run CodeQL at all, and that failure is a 2.
There was a problem hiding this comment.
not a bug: 2048 MB is CodeQL's own floor, and resolve ram --help says smaller values are transparently rounded up, so passing less changes nothing. Under 2 GB CodeQL cannot run, which is a 2.
|
|
||
|
|
||
| def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None: | ||
| """Build the database and analyse it, or fail as 'could not run'.""" |
There was a problem hiding this comment.
HIGH (Documentation): run_codeql's docstring says failures are reported as “could not run,” but the function now may raise DeferredError when CodeQL fails at an eligible push tier. The docstring omits this changed outcome.
Suggestion: Update the docstring to describe both outcomes: CodeQL failures raise DeferredError when CI will judge the commit, and otherwise raise the normal cannot-run error.
Evidence:
"""Build the database and analyse it, or fail as 'could not run'."""
There was a problem hiding this comment.
Resolved by 80c1f70: run_codeql calls cannot() again and never raises anything else, so the docstring is accurate.
There was a problem hiding this comment.
fixed in 80c1f70
run_codeql only calls cannot() again, so the docstring is accurate.
| Only for the TOOL breaking. A config the repo got wrong stays a 2 wherever | ||
| it is found, because CI would fail on it in exactly the same way. | ||
| """ | ||
| if ci_judges(): |
There was a problem hiding this comment.
CRITICAL (Constitution): A CodeQL execution failure is not the tool's verdict, so the constitution requires exit 2 and says it must never be worked around. This code instead turns that failure into a deferral when CI may judge later; the launcher allows that deferral through on push.
Suggestion: Keep CodeQL execution failures as CannotRunError/exit 2. Do not let a later CI run convert a check that could not run into a passing push.
Evidence:
if ci_judges():
raise DeferredError(msg)
cannot(msg)
There was a problem hiding this comment.
Agreed, and fixed in 80c1f70. XII allows only 0/1/2, so the exit-3 deferral has been removed from this PR and proposed as an amendment in crunchtools/constitution#99. A broken CodeQL is a 2 again in every tier.
There was a problem hiding this comment.
fixed in 80c1f70
The exit-3 deferral was removed; it is proposed as a constitution amendment in crunchtools/constitution#99.
| for line in Path("/proc/meminfo").read_text(encoding="utf-8").splitlines(): | ||
| if line.startswith("MemTotal:"): | ||
| found.append(int(line.split()[1]) // 1024) | ||
| except OSError, ValueError, IndexError: |
There was a problem hiding this comment.
HIGH (Constitution): This except clause is invalid Python 3 syntax, so sdlc cannot execute. The constitution’s contract test requires every tool to execute, not merely be present.
Suggestion: Use a tuple of exception types: except (OSError, ValueError, IndexError):.
Evidence:
except OSError, ValueError, IndexError:
There was a problem hiding this comment.
False positive, same as the thread above: PEP 758 syntax, valid on the image's Python 3.14 and tested there.
There was a problem hiding this comment.
not a bug: except A, B, C: is valid Python 3.14 syntax (PEP 758), the image's interpreter; tested in the image.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Gatehouse found no issues.
Reviewed 1 commit since 6a11c8a (incremental).
Constitution XII says every check exits 0, 1 or 2. A fourth status that lets a push through is an amendment to that rule for every Python repo, so it is proposed to the constitution instead of shipped here. What stays is the cause of josui-skills' refused pushes: CodeQL's memory and the shared, half-built database. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Gatehouse found 1 issues (1 medium)
Reviewed 1 commit since 33c9b9e (incremental).
3 finding(s) dropped: their evidence is not in the diff.
| def run_codeql(create: list[str], analyze: list[str], sarif: Path) -> None: | ||
| """Build the database and analyse it, or fail as 'could not run'.""" | ||
| step("codeql database create") | ||
| rc, _ = run(create, env=project_env()) |
There was a problem hiding this comment.
MEDIUM (Test Coverage): The changed CodeQL failure paths are not covered by the modified resilience test, which now exercises database handling and memory only. Neither a nonzero database-create result nor a nonzero analyze result is tested for the new CannotRunError behavior.
Suggestion: Add tests that mock run to return a nonzero status for each command in turn, then assert run_codeql raises CannotRunError and the command-specific failure is reported.
Evidence:
if rc != 0:
cannot(f"codeql could not build a database (exit {rc}); nothing was analysed")
...
if rc != 0:
cannot(f"codeql database analyze failed (exit {rc}); nothing was judged")
There was a problem hiding this comment.
Fixed in the next commit. failures() in tests/test_codeql_resilience.py stubs run to fail create, then analyze, and asserts each raises CannotRunError naming the step and the exit status.
There was a problem hiding this comment.
fixed in 95a01eb
failures() in tests/test_codeql_resilience.py covers a failed create and a failed analyze.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Gatehouse found no issues.
Reviewed 1 commit since 80c1f70 (incremental).
Constitution 1.35.0 amended XII to add exit `3`, could not run, deferred to CI (constitution#99), so the implementation reverted out of #34 comes back, with the narrowings that decision called for. This repo's pin moves to v1.35.0 in the same change, because the rule it implements lives there. `3` is granted by this driver, per check, and today only CodeQL has it: `codeql_broke()` raises `DeferredError` when `ci_judges()` holds, a tool's own `3` maps to `2`, and `run_step` turns a `3` from any tier but `push` back into a `2`. A configuration error stays `2`. `ci_judges()` asks whether CI will really judge this commit, which is two tests, not the presence of the text `sdlc ci` anywhere in a workflow file: - a LINE THAT RUNS the gate -- a path ending in `bin/sdlc` with `ci` as its argument, outside comments and echoed strings. The Gatehouse finding on #34, and not theoretical: trentina carries `VERIFIER_JOB: sdlc ci` as a job name and `sdlc ci` inside an error message. - a TRIGGER THAT COVERS THE COMMIT: push, pull_request, pull_request_target or merge_group. Gatehouse raised this one on the restore -- a workflow_dispatch-only job judges nothing anybody pushed, so it must not unlock a deferral. A workflow that does not parse grants nothing. An unreadable workflow directory or file now says so on its way to refusing the deferral, instead of looking like a repo with no CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx
#36) * feat: a push may defer a broken CodeQL to the required `sdlc ci` (#33) Constitution 1.35.0 amended XII to add exit `3`, could not run, deferred to CI (constitution#99), so the implementation reverted out of #34 comes back, with the narrowings that decision called for. This repo's pin moves to v1.35.0 in the same change, because the rule it implements lives there. `3` is granted by this driver, per check, and today only CodeQL has it: `codeql_broke()` raises `DeferredError` when `ci_judges()` holds, a tool's own `3` maps to `2`, and `run_step` turns a `3` from any tier but `push` back into a `2`. A configuration error stays `2`. `ci_judges()` asks whether CI will really judge this commit, which is two tests, not the presence of the text `sdlc ci` anywhere in a workflow file: - a LINE THAT RUNS the gate -- a path ending in `bin/sdlc` with `ci` as its argument, outside comments and echoed strings. The Gatehouse finding on #34, and not theoretical: trentina carries `VERIFIER_JOB: sdlc ci` as a job name and `sdlc ci` inside an error message. - a TRIGGER THAT COVERS THE COMMIT: push, pull_request, pull_request_target or merge_group. Gatehouse raised this one on the restore -- a workflow_dispatch-only job judges nothing anybody pushed, so it must not unlock a deferral. A workflow that does not parse grants nothing. An unreadable workflow directory or file now says so on its way to refusing the deferral, instead of looking like a repo with no CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx * fix: read the parsed workflow, not its text, before deferring (#33) Gatehouse's findings on the restore, in two rounds, and every one was right: no amount of text matching answers "will CI judge this commit", and each wrong answer lets a push through with no verdict. `runs_sdlc_ci` now parses the file and asks two structural questions: - is there an UNCONDITIONAL `steps[].run` invoking a path ending in `bin/sdlc` with `ci` as its argument? A job name, a step name, an `env:` value, a comment and an echoed string are mentions, not runs. A job or step behind an `if:` may be skipped on this very commit, so it grants nothing -- ashigaru's `sdlc ci` sits behind `ASHIGARU_ENABLED`, and a broken CodeQL there is correctly still a 2. - does a trigger REACH the default branch, which is the branch XII's required check guards? `push`, `pull_request`, `pull_request_target` or `merge_group`, matched with GitHub's glob rules, where `*` stops at a `/` and `**` does not. `on: push` filtered to `[release]` was the hole: the right event name over a workflow that judges nothing anybody pushed. `branches-ignore` and any `branches` list carrying a negation grant nothing, because GitHub applies those in order and `['**', '!main']` excludes `main` while matching everything -- which a set of names cannot express. The default branch is read once in `ci_judges` from `SDLC_DEFAULT_BRANCH`, which the launcher now passes from its existing `default_branch()`, and handed down as an argument, so the classifier is a function of its inputs. Where it cannot be determined, a filtered trigger grants nothing. The `run_codeql` docstring and the analyze comment said every failure is a 2, which stopped being true with exit 3. Verified against the fleet: trentina, acquacotta, rotv, petit and shiva still grant a deferral; ashigaru does not, which is the right answer for a gate behind a kill switch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Part of #33. The third item, a non-blocking "deferred to CI" status, needs an amendment to constitution XII (which allows only exit codes 0/1/2), so it is proposed in crunchtools/constitution#99 instead of shipped here.
What was wrong
codeql resolve ramgave a 1509 MB heap under both--memory 8gand--memory 2g, and 1253 MB on josui-skills' dataset. The 8g the launcher budgets was never used./cache/codeql/<worktree basename>. Two checkouts with the same basename, or a push retried while the last one was still running, could delete each other's database mid-run. That leaves half-built caches, and analyses that die with nothing to read. This is the likely cause of josui-skills' exit 70: a clean checkout of josui-skills (13 tracked.pyfiles) analyses in 9s, and CodeQL already leaves out.venvand gitignored worktrees.Fix
--ram= ¾ of the smallest ofSDLC_MEMORY(now passed in by the launcher), cgroupmemory.maxand MemTotal. That is 6144 MB at the default 8g; the measured heap went from 1253M to 4693M.flockbeside the database. The previous database is cleared, read-only parts included, and checked to be gone beforecreateruns.run_planno longer readsrun_serial's global_rcas its own result.Verified on josui-skills, with
sdlclayered onto the 0.5.3 imagesdlc codeqlpasses, and--ram=6144reaches CodeQL.db-python, and separately after leaving a read-onlytrap/with no database, the next run rebuilds and passes.tests/test_codeql_resilience.py(added to the CI unit-test step) and a host-contract case forSDLC_MEMORY.🤖 Generated with Claude Code