From c6b5720400635509272b323f19088c1300ef335b Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 19:56:16 -0500 Subject: [PATCH 01/10] fix: a release builds the commit once, and promotes only a verified digest `container.yml` triggers on a push to main AND on a `v*` tag, and `release.yml` tags the commit that just merged -- so both events fired for the same sha and both ran `build`, which pushes `:`. The second push replaced the first's manifest. It cost v0.3.5 its image: a git tag and a GitHub release with no `0.3.5` behind them, which for a fleet that pins by exact release is an exit 2 and nothing checked. A tag event now builds nothing. It resolves the digest the main-branch run of that same commit already pushed, and `promote` copies that. Two things worth naming, because both are easy to get wrong: The skip is per STEP, not per job. A job-level `if:` on `build` skips every job that `needs:` it unless each one grows `always()` logic -- so the fix would have looked right and quietly promoted nothing on its first tag. The steps carry the condition and the job's outputs fall back from `push` to `resolve`, so the plumbing downstream is untouched. `verify-digest` still runs on the tag path. `:` is pushed BEFORE it is verified, so a main run whose contract test failed leaves a pushed but untested digest on the registry; promoting that on the strength of the push alone would publish an image no job ever tested. Resolving the digest and then verifying it again is what keeps the invariant this workflow's own header states. The concurrency group stays, with a new reason: it is what guarantees the main-branch run has finished before the tag run looks for `:`. Closes #14 --- .github/workflows/container.yml | 74 ++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 14 deletions(-) diff --git a/.github/workflows/container.yml b/.github/workflows/container.yml index 5dc36e1..7baae5d 100644 --- a/.github/workflows/container.yml +++ b/.github/workflows/container.yml @@ -9,6 +9,11 @@ # Why that matters more here than anywhere else: 35 repos will pin this image. # A published tag that was never the thing the contract test ran against is a # tool version nobody can actually verify. +# +# A TAG event therefore builds NOTHING. `release.yml` tags the commit that just +# merged, so the main-branch run of that same sha has already built, tested and +# pushed `:`; the tag run resolves that digest, verifies it again and +# promotes it. Building it twice is what cost v0.3.5 its image (#14). name: Container @@ -26,17 +31,17 @@ on: workflow_dispatch: # Keyed on the COMMIT, not the run, so the push-to-main run and the tag run of -# the same commit SERIALIZE instead of racing. `github.run_id` is unique per -# run, so the old group put every run in a group of its own and controlled -# nothing: both builds pushed the same `:` tag, the second replaced the -# first's manifest, and the loser's "Verify the pushed digest" step then failed -# with `manifest unknown` on a digest that no longer existed. +# the same commit SERIALIZE. `github.run_id` is unique per run, so the old group +# put every run in a group of its own and controlled nothing: both builds pushed +# the same `:` tag and the second replaced the first's manifest. That cost +# v0.3.5 its image -- a GitHub release with no image behind it, which is the one +# state a pinned fleet cannot survive. # -# That is not theoretical. It cost v0.3.5 its image: the main run won, `latest` -# moved, the tag run lost, and the `0.3.5` tag was never created -- a GitHub -# release with no image behind it, which is the one state a pinned fleet cannot -# survive. v0.3.4 hit the same race and happened to win it. Serialized, the -# second run pushes and then verifies its own digest, which is the current one. +# The tag run no longer builds, so there is no race left to lose. The group is +# still load-bearing, for a different reason: it is what guarantees the +# main-branch run has FINISHED before the tag run looks for the `:` it +# pushed. Without it the tag run could reach `skopeo inspect` first and fail on +# an image that was still being built. # # cancel-in-progress stays false off pull requests: cancelling the main run # when its tag arrives would abandon a push half-done. @@ -51,19 +56,27 @@ env: jobs: build: - name: Build and test + # A tag event builds nothing; it only resolves what main already pushed. + name: ${{ startsWith(github.ref, 'refs/tags/v') && 'Resolve the tested digest' || 'Build and test' }} runs-on: ubuntu-latest permissions: contents: read + # Read by the step conditions below. The skip is per STEP, never per job: + # a job-level `if:` would skip every job that `needs:` this one unless each + # of them grew `always()` logic, which is how this fix fails silently on its + # first tag. + env: + IS_TAG: ${{ startsWith(github.ref, 'refs/tags/v') }} outputs: - digest: ${{ steps.push.outputs.digest }} - ref: ${{ steps.push.outputs.ref }} + digest: ${{ steps.push.outputs.digest || steps.resolve.outputs.digest }} + ref: ${{ steps.push.outputs.ref || steps.resolve.outputs.ref }} steps: - uses: actions/checkout@v7 with: persist-credentials: false - name: Shell and host-contract checks + if: env.IS_TAG != 'true' run: | set -eu podman run --rm -v "$PWD:/mnt:ro" -w /mnt \ @@ -72,6 +85,7 @@ jobs: tests/test_host_contract.sh - name: Build + if: env.IS_TAG != 'true' env: # A commit-driven build uses the cache; a rebuild that no commit # triggered must not, or it ships nothing new. @@ -91,11 +105,13 @@ jobs: # The central test for THIS image: its whole purpose is "these # executables exist at these versions". Run before anything is published. - name: Contract test + if: env.IS_TAG != 'true' run: tests/contract.sh "localhost/python-sdlc:$GITHUB_SHA" # The planner and the CodeQL path matching, run in the image because # they need its interpreter and PyYAML. - name: Unit tests + if: env.IS_TAG != 'true' run: | set -eu for t in tests/test_plan.py tests/test_codeql_globs.py; do @@ -113,7 +129,7 @@ jobs: - name: Push the SHA tag only id: push - if: github.event_name != 'pull_request' + if: github.event_name != 'pull_request' && env.IS_TAG != 'true' run: | set -eu podman push "localhost/python-sdlc:$GITHUB_SHA" "$QUAY:$GITHUB_SHA" @@ -123,6 +139,36 @@ jobs: echo "ref=$QUAY@$digest" >> "$GITHUB_OUTPUT" echo "pushed $QUAY@$digest" + # The tag path, in place of everything above. Nothing is built and nothing + # is pushed: this reads the digest the main-branch run of this same commit + # already published, and `verify-digest` below tests it again before + # `promote` moves any tag onto it. + # + # Re-verifying is not ceremony. `:` is pushed BEFORE it is verified, + # so a main run whose contract test failed leaves a pushed-but-untested + # digest sitting on the registry. Promoting that on the strength of the + # push alone would publish an image no job ever tested. + - name: Resolve the digest the main-branch run pushed + id: resolve + if: env.IS_TAG == 'true' + run: | + set -eu + # stderr is NOT suppressed: "not found" and "authentication + # required" are different problems and the log has to say which. + digest=$(skopeo inspect --format '{{.Digest}}' \ + "docker://$QUAY:$GITHUB_SHA") || digest= + if [ -z "$digest" ]; then + echo "$QUAY:$GITHUB_SHA does not exist." >&2 + echo "The main-branch run for this commit never pushed it, so there" >&2 + echo "is no tested image to promote. Promoting anything else would" >&2 + echo "publish an image this release was never built from. Fix that" >&2 + echo "run and re-push the tag." >&2 + exit 1 + fi + echo "digest=$digest" >> "$GITHUB_OUTPUT" + echo "ref=$QUAY@$digest" >> "$GITHUB_OUTPUT" + echo "promoting the already-built $QUAY@$digest" + # Re-runs the contract test against what is ACTUALLY on the registry, not # against the local build. The profile requires this on every trigger: a # rebuild changes packages without a commit, and a digest nobody tested is a From 2725c5ac583201a318a39ff7e3cae856d98f8bc2 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:03:18 -0500 Subject: [PATCH 02/10] fix: a test run on the laptop no longer decides the gate's verdict The gate mounts the worktree at /repo, so the container and the laptop share the tree's `__pycache__`. Bytecode carries the path it was compiled from, and pytest's assertion rewriter reads its cached code back as marshalled -- `_read_pyc` in `_pytest/assertion/rewrite.py` returns `marshal.load(fp)` as it stands, because the point of that cache is the REWRITTEN code. So a module compiled on the host keeps `/home//.../x.py` in `co_filename` while `__file__` is `/repo/.../x.py`, and any test comparing the two fails in the gate and nowhere else. That is four of the five pushes on crunchtools/trentina#706, about 30 minutes, spent on a test with no defect behind it. `-e PYTHONPYCACHEPREFIX=/tmp/pycache`, in BOTH containers that mount the tree: `run_in_image` here and the `pytest-in-image` argv the image builds. #16 named only the first. /tmp is already the 4g tmpfs, pytest's `get_cache_dir` honours `sys.pycache_prefix`, and with it set the container neither reads the tree's bytecode nor writes any back into it -- the second direction matters because the commit tier mounts the tree read-write. A plain `import` never showed this: the stdlib's loader calls `_imp._fix_co_filename` after unmarshalling. tests/test_pycache_prefix.py pins both halves of that -- the rewriter's path survives, the stdlib's does not -- and reproduces the mismatch before proving the prefix removes it, so it cannot pass by testing nothing. The env var reaching each container is asserted where each argv is built: tests/test_host_contract.sh for the gate's own, tests/test_plan.py for `pytest-in-image`. Closes #16 --- .github/workflows/container.yml | 3 +- bin/sdlc | 12 ++ pyproject.toml | 7 +- sdlc | 5 + tests/test_host_contract.sh | 15 +++ tests/test_plan.py | 11 ++ tests/test_pycache_prefix.py | 201 ++++++++++++++++++++++++++++++++ 7 files changed, 252 insertions(+), 2 deletions(-) create mode 100755 tests/test_pycache_prefix.py diff --git a/.github/workflows/container.yml b/.github/workflows/container.yml index 7baae5d..ab6ee7a 100644 --- a/.github/workflows/container.yml +++ b/.github/workflows/container.yml @@ -114,7 +114,8 @@ jobs: if: env.IS_TAG != 'true' run: | set -eu - for t in tests/test_plan.py tests/test_codeql_globs.py; do + for t in tests/test_plan.py tests/test_codeql_globs.py \ + tests/test_pycache_prefix.py; do podman run --rm -v "$PWD:/src:ro,z" -w /src \ --entrypoint /opt/constitution-venv/bin/python3 \ "localhost/python-sdlc:$GITHUB_SHA" "$t" diff --git a/bin/sdlc b/bin/sdlc index 3d4b1d7..48274c0 100755 --- a/bin/sdlc +++ b/bin/sdlc @@ -191,6 +191,17 @@ run_in_image() { # is writable, with no --user and no --userns=keep-id. Two bugs came from # trying to be cleverer than this. # + # PYTHONPYCACHEPREFIX is what keeps the shared tree from deciding a verdict. + # /repo is the developer's worktree, so its `__pycache__` is the laptop's, and + # bytecode carries the path it was compiled from. pytest's assertion rewriter + # reads its cached code back as marshalled and cannot fix that path -- the + # whole point of its cache is the REWRITTEN code -- so a module compiled on + # the host keeps `/home//.../x.py` while `__file__` is `/repo/.../x.py`, + # and any test comparing the two fails here and nowhere else. That was four of + # the five pushes on trentina#706. /tmp is the tmpfs below, so with the prefix + # set the container neither reads the tree's bytecode nor writes any back into + # it -- which matters at commit, where the mount is read-write. + # # ONE deliberate difference: `z`, not `Z`. A tier runs several of these and # `Z` relabels the tree EXCLUSIVELY, taking it out from under whichever # container started first (gates.sh:157-161). Those two run one at a time and @@ -202,6 +213,7 @@ run_in_image() { -v "$CACHE_VOLUME:/cache" \ --tmpfs /tmp:rw,exec,size=4g,mode=1777 \ -e HOME=/tmp \ + -e PYTHONPYCACHEPREFIX=/tmp/pycache \ -e SEMGREP_SETTINGS_FILE=/tmp/semgrep.yml \ -e "SDLC_PROJECT=$PROJECT" \ -e "SDLC_JOBS=$GATE_CPUS" \ diff --git a/pyproject.toml b/pyproject.toml index 08252b5..8231656 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -49,4 +49,9 @@ max-complexity = 10 "sdlc" = ["S404", "S108", "ARG001"] "validate" = ["S404"] # A test asserts. That is what it is for. -"tests/*" = ["S101", "PT018"] +# S302: test_pycache_prefix.py unmarshals bytecode to show what pytest's +# assertion rewriter sees. The input is a .pyc that test compiled itself, +# seconds earlier, into its own temporary directory -- there is no untrusted +# bytecode anywhere near it, and reading it any other way would not be a copy +# of the behaviour under test. +"tests/*" = ["S101", "PT018", "S302"] diff --git a/sdlc b/sdlc index 7198e42..523d5cb 100755 --- a/sdlc +++ b/sdlc @@ -540,6 +540,11 @@ def kind_pytest_in_image(name: str, settings: dict) -> list[tuple[str, str, str] f"PYTHONPATH=/sdlc-cache/pytest/{project_name()}", "-e", "HOME=/tmp", + # The same reason as bin/sdlc's run_in_image: this mounts the repo's + # test files, so without a prefix pytest reads bytecode the host + # compiled and judges paths that do not exist in here. + "-e", + "PYTHONPYCACHEPREFIX=/tmp/pycache", "-w", "/repo", "--entrypoint", diff --git a/tests/test_host_contract.sh b/tests/test_host_contract.sh index 2a27edb..cb29649 100755 --- a/tests/test_host_contract.sh +++ b/tests/test_host_contract.sh @@ -441,6 +441,21 @@ else bad "podman was called but never asked to run the image" fi +printf '\n== what the containers are given\n\n' + +# The worktree is the developer's, so its __pycache__ is the laptop's. Every +# container that mounts it must keep its bytecode somewhere else, or a module +# compiled on the host decides a verdict in here -- four of the five pushes on +# trentina#706. tests/test_pycache_prefix.py tests the property; this asserts +# the env var reaches every such container. +mounts=$(grep -c ':/repo:' "$LOG" 2>/dev/null || true) +prefixed=$(grep ':/repo:' "$LOG" 2>/dev/null | grep -c 'PYTHONPYCACHEPREFIX=/tmp/pycache' || true) +if [ "${mounts:-0}" -gt 0 ] && [ "$prefixed" = "$mounts" ]; then + ok "all $mounts container(s) mounting the worktree got their own bytecode cache" +else + bad "$prefixed of ${mounts:-0} containers mounting the worktree set PYTHONPYCACHEPREFIX" +fi + printf '\n' if [ "$FAILED" -eq 0 ]; then printf 'host contract PASSED\n'; exit 0 diff --git a/tests/test_plan.py b/tests/test_plan.py index 0a755d2..8cd8fca 100644 --- a/tests/test_plan.py +++ b/tests/test_plan.py @@ -409,6 +409,17 @@ def plan_of(target: str, pyproject: str, **env) -> list[list[str]]: runs = [c for i, c in enumerate(seq) if i == 0 or seq[i - 1] != c] check(f"lane {lane}: each check's steps are contiguous", len(runs) == len(set(runs)), str(seq)) +# The in-image pytest run mounts the repo's test files, so it has the same +# exposure to the host's bytecode that bin/sdlc's own containers have: pytest's +# rewriter reads its cached code as marshalled and cannot correct the path it +# was compiled from. See tests/test_pycache_prefix.py. +argv = next(r[3].split(S.UNIT_SEPARATOR) for r in rows if r[0] == "image-tests" and r[2] == "host") +check( + "the in-image pytest run keeps its bytecode out of the tree", + "PYTHONPYCACHEPREFIX=/tmp/pycache" in argv, + str(argv), +) + single = plan_of("push", '[project]\nname = "fx"\nversion = "0.0.0"\n') check( "push is one lane, so it runs exactly as it did before", diff --git a/tests/test_pycache_prefix.py b/tests/test_pycache_prefix.py new file mode 100755 index 0000000..36d09db --- /dev/null +++ b/tests/test_pycache_prefix.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +"""A `.pyc` compiled on the host must not decide a verdict inside the gate. + +The gate mounts the developer's worktree at `/repo`, so the container and the +laptop share the tree's `__pycache__`. Bytecode records the path it was +compiled from in `co_filename`, so a module compiled on the laptop carries +`/home//.../m.py` while `__file__` inside the container is +`/repo/.../m.py`. Any test comparing the two then fails in the gate and nowhere +else: trentina's `test_library_lines_are_fingerprinted` keys a dict on +`__file__` and matches it against `record.pathname`, which comes from +`co_filename`. That cost four of the five pushes on crunchtools/trentina#706 +before anyone suspected the cache. + +It is specifically PYTEST's cache, not Python's. The stdlib's source loader +calls `_imp._fix_co_filename` after unmarshalling (`_bootstrap_external. +_compile_bytecode`), so a plain `import` rewrites the stale path and never +shows this. pytest's assertion rewriter keeps its own +`*.cpython-NNN-pytest-X.Y.Z.pyc` beside it and its `_read_pyc` +(`_pytest/assertion/rewrite.py`) returns `marshal.load(fp)` as it stands -- it +has to, because the point of that cache is the REWRITTEN code. So the foreign +path survives, and only there. + +`PYTHONPYCACHEPREFIX` is the fix at both ends: the rewriter's `get_cache_dir` +honours `sys.pycache_prefix`, so with one set it neither reads the tree's +bytecode nor writes any back into the mount. + +Both halves of pytest's behaviour are reimplemented below rather than imported, +because this image has no pytest -- a repo's pytest comes from its own +`uv.lock`. They are small, and each says which pytest function it +copies so a drift in pytest can be checked against it. + +The env var reaching the containers is asserted separately: +`tests/test_host_contract.sh` for the gate's own, `tests/test_plan.py` for +`pytest-in-image`. + +Run: tests/test_pycache_prefix.py (needs nothing but python3) +""" + +from __future__ import annotations + +import importlib.util +import marshal +import os +import py_compile +import sys +import tempfile +import types +from pathlib import Path + +# What a host-compiled module carries: a path that does not exist in the +# container. That it differs from the real one is the whole point. +FOREIGN = "/home/somebody/elsewhere/subject.py" + +SOURCE = '''\ +"""A module whose only job is to report the path its bytecode came from.""" + + +def where() -> str: + return where.__code__.co_filename +''' + +# The fixed part of a .pyc: magic, flags, source mtime, source size. +PYC_HEADER = 16 + +FAILED: list[str] = [] + + +def ok(msg: str) -> None: + print(f"ok {msg}") + + +def bad(msg: str) -> None: + FAILED.append(msg) + print(f"FAIL {msg}") + + +def check(name: str, got: object, want: object) -> None: + if got == want: + ok(f"{name}: {got}") + else: + bad(f"{name}: {got!r}, want {want!r}") + + +# --------------------------------------------- pytest's two behaviours, copied + + +def rewriter_cache_dir(source: Path) -> Path: + """`_pytest.assertion.rewrite.get_cache_dir`, copied. + + With a prefix, the tree's own directory is not involved at all: the cache + goes under the prefix at the source's absolute path. That is the line this + whole change rests on. + """ + if sys.pycache_prefix: + return Path(sys.pycache_prefix) / Path(*source.parts[1:-1]) + return source.parent / "__pycache__" + + +def rewriter_read_pyc(source: Path, pyc: Path) -> types.CodeType | None: + """`_pytest.assertion.rewrite._read_pyc`, copied: the magic/mtime/size + header checks, then the code object exactly as marshalled. + + No `_imp._fix_co_filename`. That absence IS the bug, and it is not a pytest + defect -- the cached code is rewritten code, and pytest reads it back the + way it wrote it. + """ + try: + with open(pyc, "rb") as fp: + stat = os.stat(source) + header = fp.read(PYC_HEADER) + if len(header) != PYC_HEADER or header[:4] != importlib.util.MAGIC_NUMBER: + return None + if header[4:8] != b"\x00\x00\x00\x00": + return None + if int.from_bytes(header[8:12], "little") != int(stat.st_mtime) & 0xFFFFFFFF: + return None + if int.from_bytes(header[12:16], "little") != stat.st_size & 0xFFFFFFFF: + return None + code = marshal.load(fp) + except OSError: + return None + return code if isinstance(code, types.CodeType) else None + + +# ------------------------------------------------------------------- the cases + + +def host_compiled(source: Path, into: Path) -> Path: + """Bytecode for `source`, in `into`, stamped with a path from another machine.""" + into.mkdir(parents=True, exist_ok=True) + pyc = into / f"{source.stem}.{sys.implementation.cache_tag}-pytest-9.1.1.pyc" + py_compile.compile(str(source), cfile=str(pyc), dfile=FOREIGN, doraise=True) + return pyc + + +def main() -> int: + with tempfile.TemporaryDirectory() as tmp: + tree = (Path(tmp) / "tree").resolve() + tree.mkdir() + source = tree / "subject.py" + source.write_text(SOURCE, encoding="utf-8") + + # ---- the bug, as the gate meets it. The laptop left bytecode in the + # tree; the container shares the tree, so the rewriter reads it. + sys.pycache_prefix = None + shared = rewriter_cache_dir(source) + check("without a prefix the cache is in the tree", shared, tree / "__pycache__") + pyc = host_compiled(source, shared) + code = rewriter_read_pyc(source, pyc) + if code is None: + bad("the copied header checks rejected bytecode they should accept") + return 1 + check("co_filename the rewriter would use", code.co_filename, FOREIGN) + if code.co_filename == str(source): + bad("no mismatch, so nothing below proves anything") + + # ---- and why a plain import never showed it: the stdlib fixes the path. + sys.modules.pop("subject_stdlib", None) + spec = importlib.util.spec_from_file_location("subject_stdlib", source) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + check("co_filename through the stdlib loader", module.where(), str(source)) + + # ---- the fix. With a prefix the rewriter looks outside the tree, so + # the laptop's bytecode is not on its path at all. + prefix = (Path(tmp) / "pycache").resolve() + sys.pycache_prefix = str(prefix) + try: + prefixed = rewriter_cache_dir(source) + if prefixed.is_relative_to(tree): + bad(f"with a prefix the cache is still in the tree: {prefixed}") + else: + ok(f"with a prefix the cache is outside the tree: {prefixed}") + check( + "the stale bytecode is not where the rewriter now looks", + (prefixed / pyc.name).exists(), + False, + ) + # The other direction, which matters because the commit tier mounts + # the tree READ-WRITE: nothing the container compiles lands in it. + before = sorted(p.name for p in shared.iterdir()) + host_compiled(source, prefixed) + check( + "the tree's __pycache__ is untouched", + sorted(p.name for p in shared.iterdir()), + before, + ) + finally: + sys.pycache_prefix = None + + print() + if FAILED: + print(f"PYTHONPYCACHEPREFIX FAILED: {len(FAILED)} check(s)") + return 1 + print("PYTHONPYCACHEPREFIX PASSED") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From cb7690cd695d557648cf9cb85ab271d6762eceaa Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:06:18 -0500 Subject: [PATCH 03/10] feat: the gate's own containers run with no capabilities A gate runs the repo's own code -- conftest, mypy plugins, semgrep rules, a Containerfile's RUN lines -- so the container it runs in is a boundary, not a convenience. `kind_pytest_in_image` has known that since it was written: `--read-only`, `--cap-drop=all`, `--security-opt no-new-privileges`, `--network none` unless asked. The gate's own containers, which run far more repo-supplied code than that one does, had none of it. `--cap-drop=all --security-opt=no-new-privileges` on both, in `run_in_image` and on `run_host`'s `run` prefix. Not `--network none`: `uv sync`, `pip-audit` and Gatehouse need it. Not `--read-only`: ruff rewrites the tree and that is the point of the commit tier. Nothing in the image wants a capability. The tools read the tree, write the cache volume and talk to the network, and rootless podman maps the container's root to the invoking user, so file access is the developer's own either way. Checked rather than assumed: `sdlc versions` probes all seven binaries, `sdlc ruff-check` rewrites the tree, and trentina's `sdlc mypy` runs a full `uv sync` into the shared cache volume and type-checks 235 files -- all of it with no capabilities. tests/test_host_contract.sh asserts it over every container the launcher builds, so a future argv cannot quietly drop the flags. --- bin/sdlc | 11 +++++++++++ tests/test_host_contract.sh | 12 ++++++++++++ 2 files changed, 23 insertions(+) diff --git a/bin/sdlc b/bin/sdlc index 48274c0..d6bd8c0 100755 --- a/bin/sdlc +++ b/bin/sdlc @@ -191,6 +191,15 @@ run_in_image() { # is writable, with no --user and no --userns=keep-id. Two bugs came from # trying to be cleverer than this. # + # No capabilities, and no way to acquire one. A gate runs a repo's own code -- + # conftest, mypy plugins, semgrep rules, a Containerfile's RUN lines -- so the + # container it runs in is a boundary, not a convenience. `kind_pytest_in_image` + # has locked its container down this way since it was written; the gate's own + # containers never did. Nothing here needs a capability: the tools read the + # tree, write the cache volume and talk to the network, and rootless podman + # maps the container's root to the invoking user, so file access is the + # developer's own either way. + # # PYTHONPYCACHEPREFIX is what keeps the shared tree from deciding a verdict. # /repo is the developer's worktree, so its `__pycache__` is the laptop's, and # bytecode carries the path it was compiled from. pytest's assertion rewriter @@ -208,6 +217,7 @@ run_in_image() { # never hit it. set -- \ --events-backend=none run --rm --log-driver=none \ + --cap-drop=all --security-opt=no-new-privileges \ --cpus "$GATE_CPUS" --memory "$GATE_MEMORY" --cpu-shares 256 \ -v "$REPO:/repo:$_mode,z" -w /repo \ -v "$CACHE_VOLUME:/cache" \ @@ -325,6 +335,7 @@ run_host() { build) set -- build --cpu-period 100000 --cpu-quota "$(( GATE_CPUS * 100000 ))" \ --memory "$GATE_MEMORY" "$@" ;; run) set -- --events-backend=none run --log-driver=none \ + --cap-drop=all --security-opt=no-new-privileges \ --cpus "$GATE_CPUS" --memory "$GATE_MEMORY" --cpu-shares 256 "$@" ;; *) die "a host step may only build or run; refused: podman $_sub" ;; esac diff --git a/tests/test_host_contract.sh b/tests/test_host_contract.sh index cb29649..2b831df 100755 --- a/tests/test_host_contract.sh +++ b/tests/test_host_contract.sh @@ -456,6 +456,18 @@ else bad "$prefixed of ${mounts:-0} containers mounting the worktree set PYTHONPYCACHEPREFIX" fi +# A gate runs the repo's own code, so its containers get no capabilities and no +# way to acquire one. Asserted over every `run` the launcher itself builds -- +# `podman volume` lines and a repo's own host steps are not it. +runs=$(grep -c 'run --rm' "$LOG" 2>/dev/null || true) +locked=$(grep 'run --rm' "$LOG" 2>/dev/null \ + | grep 'cap-drop=all' | grep -c 'no-new-privileges' || true) +if [ "${runs:-0}" -gt 0 ] && [ "$locked" = "$runs" ]; then + ok "all $runs gate container(s) ran with no capabilities" +else + bad "$locked of ${runs:-0} gate containers dropped capabilities" +fi + printf '\n' if [ "$FAILED" -eq 0 ]; then printf 'host contract PASSED\n'; exit 0 From d8f172cef899d5a48e6134b2117cbd73645247eb Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:07:35 -0500 Subject: [PATCH 04/10] feat: Gourmand runs at commit, not only at push Gourmand's findings are deterministic: the same tree gives the same answer, with no model and no network. Paying for them at push, where each rejection costs a six-to-eight minute cycle and a re-review of the whole diff, was paying the wrong price for an answer a three-second check at commit would have given. One of the five pushes on crunchtools/trentina#706 was exactly this. Measured in full mode, warm: 1.7s on this repo, 3.2s on trentina. Full rather than `--diff-source`, because full mode also sees duplication across files, which is most of what it is for. It runs after both ruff steps, so it judges the formatted tree rather than the one you typed, and tests/test_plan.py pins that order -- a commit tier that reviewed the unformatted source would report findings the same commit had already fixed. It stays in push as well: a commit can be made with --no-verify, and push is the tier that renders the verdicts. Semgrep deliberately does not join it. It takes 27s, and `t_semgrep` exits 2 on a repo with no `.semgrep`, so as a default it would break the commit hook across the fleet. A repo that wants it there writes `[tool.sdlc.commit] checks = ["semgrep"]`. Constitution XII's commit sentence gains Gourmand in a separate prose change; the validator inspects only that the two hooks exist, so nothing waits on it. Refs #18 --- README.md | 9 ++++++--- sdlc | 16 +++++++++++++--- tests/test_plan.py | 12 +++++++++++- 3 files changed, 30 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 0f44d9c..c669f16 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,11 @@ the image contains. | `sdlc push` | the pre-push hook | `[tool.sdlc.push]` | | `sdlc ci` | the CI job named `sdlc ci` | `[tool.sdlc.ci]` | -- **commit** takes seconds, works offline and costs nothing: formatting and - lint fixes. +- **commit** takes seconds, works offline and costs nothing: formatting, lint + fixes, and Gourmand's structural review. Everything here is deterministic — + the same tree gives the same answer, with no model and no network — so there + is no reason to find it at push, where each rejection costs a six-to-eight + minute cycle. - **push** renders every verdict, once, over `origin/...HEAD`. It never builds a container — that is minutes, and a push you wait minutes for is a push people stop making. @@ -80,7 +83,7 @@ nested `pyproject.toml` still applies to the files under it. | Command | Runs | |---|---| -| `commit` | ruff-check, ruff-format, hooks-installed | +| `commit` | ruff-check, ruff-format, gourmand, hooks-installed | | `push` | mypy, pytest, gourmand, gatehouse, semgrep, pip-audit, constitution, codeql | | `ci` | mypy, pytest, semgrep, pip-audit, constitution, codeql | diff --git a/sdlc b/sdlc index 523d5cb..6a26e28 100755 --- a/sdlc +++ b/sdlc @@ -146,9 +146,19 @@ def pyproject() -> dict: COMMANDS = ("commit", "push", "ci") DEFAULTS = { - # Seconds, offline, free: what rewrites your code, plus the one check that - # notices the push hook was never installed. - "commit": ["ruff-check", "ruff-format", "hooks-installed"], + # Seconds, offline, free: what rewrites your code, the structural review + # that is cheap enough to sit beside it, plus the one check that notices the + # push hook was never installed. + # + # Gourmand runs AFTER the two ruff steps, so it judges the formatted tree + # rather than the one you typed. It is here because its findings are + # deterministic -- the same tree gives the same answer, with no model and no + # network -- so paying for them at push, where each rejection costs a + # six-to-eight minute cycle, was paying the wrong price. Measured at 1.7s on + # this repo and 3.2s on trentina, warm, in full mode. It stays in push too: + # a commit can be made with --no-verify, and push is the tier that renders + # the verdicts. + "commit": ["ruff-check", "ruff-format", "gourmand", "hooks-installed"], # Everything that renders a verdict, once, over origin/...HEAD. # Never a container build: that is minutes, and a push you wait minutes for # is a push people stop making. diff --git a/tests/test_plan.py b/tests/test_plan.py index 8cd8fca..c73fae7 100644 --- a/tests/test_plan.py +++ b/tests/test_plan.py @@ -64,7 +64,7 @@ def cannot_run(fn, *args) -> str | None: empty: dict = {} check( "commit defaults", - S.resolve(empty, "commit") == ["ruff-check", "ruff-format", "hooks-installed"], + S.resolve(empty, "commit") == ["ruff-check", "ruff-format", "gourmand", "hooks-installed"], ) check("push defaults", S.resolve(empty, "push") == S.DEFAULTS["push"]) ci = S.resolve(empty, "ci") @@ -331,6 +331,16 @@ def sarif(results: list[tuple[str, str, str]]) -> dict: set(lanes.values()) == {S.MAIN_LANE}, str(lanes), ) +# Gourmand judges what ruff leaves behind, so it must come after both ruff +# steps. A commit tier that reviewed the unformatted tree would report findings +# that the same commit had already fixed. +order = S.resolve(empty, "commit") +check( + "gourmand runs after the two ruff steps", + order.index("gourmand") > max(order.index("ruff-check"), order.index("ruff-format")), + str(order), +) + commit = S.lane_of(empty, S.resolve(empty, "commit")) check("commit is always one lane", set(commit.values()) == {S.MAIN_LANE}, str(commit)) From 34d21646283990f6ee1df5afdabc16ea728cb549 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:19:02 -0500 Subject: [PATCH 05/10] feat: the review runs beside the rest, and every command says what it cost Two things, and the second is why the first could be judged at all. PER-CHECK DURATIONS AND A RECORD. The serial path -- which is the path every `sdlc push` took -- printed no durations whatsoever, so nothing about the slowest tier in this tool could be tuned by anyone who had not sat watching it with a stopwatch. It now prints the same per-check footer the lane path always had, and both paths write: /sdlc/last-.json check, exit, seconds, log /sdlc/logs-/*.log what a buffered check printed That is also what an agent needs after a rejected push: which check, how long, and where its output went, without running the tier again to find out. In the git directory, not a committed `.sdlc/`, because it is one machine's last run. Asked of `git rev-parse --git-path`, since in a linked worktree `.git` is a file. Written, never read -- a verdict comes from running the check, and a file that could stand in for one would be a way past a gate. Failing to write it never fails a gate. A LANE FOR THE REVIEW. Gatehouse is the one check that does not read the worktree: its input is the `origin/...HEAD` diff, committed objects that bin/sdlc computes on the host and pipes in on stdin. So the rule that sends a tree-rewriting command down one lane does not apply to it, and it is also the only check whose time is not CPU -- one API call and then waiting, measured at 35s on a 365-line diff, spent with the CPUs idle. #18 rejected push lanes for want of durations. That was the wrong reason to wait: the argument is that Gatehouse reads git rather than the tree, and it needs no measurement. What makes it free is the budget rule. A lane whose every step is `local` takes one CPU and is LEFT OUT of the division, so the main lane keeps the whole budget it had. Dividing by the lane count would have handed half the machine to a lane that wants no CPU, which is precisely how lanes were measured as a regression on a 4-vCPU runner (15m27s to 24m38s). And the working lane keeps the terminal. run_lanes buffers every lane and prints each block when it finishes, which is right when both lanes are computing and wrong here: the buffered lane would be the whole of a push, and three minutes of silence reads like a hang. So when exactly one lane runs containers, that lane runs in the foreground through the ordinary sequential path -- same output, live -- and the others are printed whole after it. The multi-container-lane path is untouched. tests/test_host_contract.sh covers the new executor end to end: both lanes run, the buffered block lands after the streamed one, the banner names what is running beside (from the plan, since no check is named in bin/sdlc), the working lane keeps all 4 of a budget of 4 while the other takes 1, and a finding in the lane off the CPU still fails the command. Refs #18 --- README.md | 57 +++++++-- bin/sdlc | 243 ++++++++++++++++++++++++++++++++++-- sdlc | 26 +++- tests/test_host_contract.sh | 77 +++++++++++- tests/test_plan.py | 24 +++- 5 files changed, 402 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index c669f16..d8b2a11 100644 --- a/README.md +++ b/README.md @@ -96,24 +96,32 @@ A command's checks run in lanes. Steps of one lane run in sequence; lanes run at the same time, so a command takes about as long as its longest chain rather than the sum of everything in it. -The image decides the lanes, by two rules: +The image decides the lanes, by three rules: - An `image-build` check and every `pytest-in-image` check that runs the image it tags are **one lane** — the build has to finish before the tests start. +- Gatehouse is the `review` lane, always. It is the one check that does not + read the worktree: its input is the `origin/...HEAD` diff, + which `bin/sdlc` computes on the host and pipes in. So it can run beside + anything, including a command that rewrites the tree. - Everything else is the `main` lane, in the order `tiers` prints. That is where `uv sync` happens, and one lane keeps two syncs out of one venv. A command that rewrites the tree — anything running `ruff-check`, -`ruff-format` or `gourmand` — gets no lanes at all, because a build context -reading those files while they change is judging a tree that no longer exists. -That makes `sdlc commit` sequential by definition. +`ruff-format` or `gourmand` — gets no lanes beside it except that reviewer, +because a build context reading those files while they change is judging a tree +that no longer exists. That makes `sdlc commit` sequential by definition. -A repo with no `image-build` check has one lane and runs exactly as it did -before lanes existed. So does `sdlc push`, which never builds a container. +So `sdlc push` has two lanes: everything that computes, and the review. `sdlc +ci` has one unless the repo builds an image, because CI gets Gatehouse from its +own workflow. The CPU budget is for the whole gate, not for each container: it is divided -between the lanes, floor of one CPU each. Two lanes are no busier on your -laptop than one was. +between the lanes that run containers, floor of one CPU each. A lane whose +every step is a question `bin/sdlc` answers itself — the reviewer, which is one +API call and then waiting — takes a single CPU and is **left out of that +division**, so putting it beside the others costs them nothing. Two lanes are +no busier on your laptop than one was. The budget itself is half the machine, so a gate cannot take the desktop away from the person running it — except where there is no such person. In CI, or @@ -125,8 +133,37 @@ changelog entry. Each lane's output is buffered and printed whole, in the order `tiers` lists the checks, so two lanes never cut into each other's log. Each block ends with -how long its check took, because buffering is what the `== ` timestamps -used to tell you. +how long its check took. + +With one exception, which is the shape `sdlc push` has: when exactly one lane +runs containers and the rest do not, that lane runs in the **foreground** and +streams to your terminal as it always did, and the others are printed after it. +Buffering the lane that is doing the work would mean watching a silent terminal +for the length of the test suite, which reads like a hang. + +## What the last run found + +Every command leaves a record, inside the git directory rather than in the +worktree: + +| Path | What is in it | +|---|---| +| `/sdlc/last-.json` | each check, its exit, its seconds, its log | +| `/sdlc/logs-/` | what each buffered check printed | + +``` +$ jq -r '.checks[] | "\(.seconds)s \(.check) exit \(.exit)"' .git/sdlc/last-push.json +``` + +The checks appear in the order they **finished**, which with lanes is not the +order they were listed in. A check that streamed to your terminal has no log +file: the terminal had it. + +The record is written, never read. Nothing in this tool consults it to decide +anything, because a verdict comes from running the check — if a file could +stand in for one, a stale or hand-edited file would be a way past a gate. +Failing to write it never fails a gate either; a repo with nowhere to write +simply has no record. If no check finishes for `SDLC_LANE_STALL` seconds (default 3600) the printer stops waiting on the order, says so, waits for every lane to stop and then diff --git a/bin/sdlc b/bin/sdlc index d6bd8c0..d146208 100755 --- a/bin/sdlc +++ b/bin/sdlc @@ -301,6 +301,89 @@ t_gatehouse() { --env-file "$_env" -i "$IMAGE" gatehouse } +# ------------------------------------------------------------------- the record +# +# What the last run of each command found, and how long each check took: +# +# /sdlc/last-.json check, exit, seconds, log +# /sdlc/logs-/*.log what a buffered check printed +# +# The serial path printed no durations at all, so nothing about this gate could +# be tuned by anyone who had not sat watching it. It is also what an agent needs +# after a rejected push: which check, how long, and where its output went, +# without running the whole tier again to find out. +# +# Inside the git directory rather than in a committed `.sdlc/`, because this is +# one machine's last run and belongs to nobody else. `git rev-parse --git-path` +# is asked rather than assembled: in a linked worktree `.git` is a file. +# +# The record is WRITE-ONLY. Nothing in either half of this tool reads it to +# decide anything, and nothing may start: a verdict comes from running the +# check. An input would make a stale or hand-edited file into a way past a gate, +# and the gate is the only thing here worth having. +# +# Failing to write it is never a failure of the gate. Every function below +# degrades to doing nothing. + +# Whole seconds. Every duration printed and recorded is measured with this, in +# both the serial and the lane path. +now_s() { date +%s 2>/dev/null || printf 0; } + +STATE= # /sdlc, or empty when there is nowhere to write +LOGS= # the log directory for the running command +PART= # the per-check lines, rendered to JSON when the run ends +RECORD_FOR= # the command being recorded + +state_init() { # command + STATE=$(git rev-parse --git-path sdlc 2>/dev/null) || STATE= + [ -n "$STATE" ] || return 0 + LOGS="$STATE/logs-$1" + mkdir -p "$LOGS" 2>/dev/null || { STATE=; LOGS=; return 0; } + # One run's worth. A history nobody prunes is a directory that grows until + # someone notices it, and the question is always about the last run. + rm -f "$LOGS"/*.log 2>/dev/null || true + PART="$STATE/last-$1.json.part" + : > "$PART" 2>/dev/null || { PART=; return 0; } + RECORD_FOR=$1 +} + +check_log() { # index check -> a path, or nothing when there is no state + [ -n "$LOGS" ] || return 0 + # A check name reaches a filename here, and `plain()` in the image refuses a + # tab, a newline and the separator but not a slash. The index keeps two + # checks that sanitize to the same name from sharing one log. + printf '%s/%s-%s.log' "$LOGS" "$1" "$(printf '%s' "$2" | tr -c 'A-Za-z0-9._-' '_')" +} + +record_check() { # check rc seconds log + [ -n "$PART" ] || return 0 + # Appended from the lane subshells as well as from here, so the order is the + # order checks FINISHED. One short line per check, one write each. + printf '%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "${4:-}" >> "$PART" 2>/dev/null || true +} + +record_finish() { # worst status + [ -n "$PART" ] && [ -f "$PART" ] || return 0 + # awk because the escaping has to be done once and correctly: a check name or + # a path is the only thing here that can carry a quote or a backslash. + awk -F'\t' -v cmd="$RECORD_FOR" -v worst="$1" \ + -v when="$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || printf unknown)" ' + function esc(s) { gsub(/\\/, "\\\\", s); gsub(/"/, "\\\"", s); return s } + BEGIN { + printf "{\n \"command\": \"%s\",\n \"finished\": \"%s\",\n", esc(cmd), esc(when) + printf " \"exit\": %s,\n \"checks\": [\n", worst + } + { + if (n++) printf ",\n" + printf " {\"check\": \"%s\", \"exit\": %s, \"seconds\": %s, \"log\": ", \ + esc($1), $2, $3 + if ($4 == "") printf "null}"; else printf "\"%s\"}", esc($4) + } + END { printf "\n ]\n}\n" } + ' "$PART" > "$STATE/last-$RECORD_FOR.json" 2>/dev/null || true + rm -f "$PART" 2>/dev/null || true +} + # --------------------------------------------------------------------- the plan # # Which checks run is decided by the repo's [tool.sdlc.commit], [tool.sdlc.push] @@ -367,9 +450,22 @@ run_step() { # mode payload -> the step's exit status esac } +_close_serial() { # the open check is finished: footer and record + [ -n "${_scur:-}" ] || return 0 + # Only a PASS prints here. A failure has already said so, at the moment it + # happened, rather than when the next check starts. + [ "$_scur_rc" -eq 0 ] && printf '\n== %s passed in %ss\n' "$_scur" "$_scur_took" + record_check "$_scur" "$_scur_rc" "$_scur_took" "" + _scur= +} + run_serial() { # the plan -> its worst step status _worst=0 _failed= + _scur= + _scur_rc=0 + _scur_took=0 + _sidx=0 _ifs=$IFS set -f IFS=' @@ -383,18 +479,32 @@ run_serial() { # the plan -> its worst step status _payload=${_rest#*"$TAB"} # One failed step ends that check; the next check still runs. case " $_failed " in *" $_check "*) continue ;; esac + # A check's steps are contiguous, so a new name closes the previous block. + # The timing is per CHECK, not per step: `pytest-in-image` is a deps step + # and then the run, and what anyone wants to know is what pytest cost. + if [ "$_check" != "$_scur" ]; then + _close_serial + _sidx=$((_sidx + 1)) + _scur=$_check + _scur_rc=0 + _scur_took=0 + fi + _t0=$(now_s) _rc=0 run_step "$_mode" "$_payload" || _rc=$? + _scur_took=$(( _scur_took + $(now_s) - _t0 )) if [ "$_rc" -ne 0 ]; then _failed="$_failed $_check" - printf '\n%s: %s %s\n' "$ME" "$_check" \ - "$([ "$_rc" -eq 2 ] && echo 'COULD NOT RUN' || echo 'FAILED')" >&2 + _scur_rc=$_rc + printf '\n%s: %s %s after %ss\n' "$ME" "$_check" \ + "$([ "$_rc" -eq 2 ] && echo 'COULD NOT RUN' || echo 'FAILED')" "$_scur_took" >&2 # A later 1 must not mask an earlier 2. if [ "$_rc" -eq 2 ] || [ "$_worst" -eq 2 ]; then _worst=2; else _worst=1; fi fi done IFS=$_ifs set +f + _close_serial return "$_worst" } @@ -432,8 +542,6 @@ OUT= # the lane scratch directory, while lanes are running STALL_MAX=${SDLC_LANE_STALL:-3600} case "$STALL_MAX" in ''|*[!0-9]*) die "SDLC_LANE_STALL must be a whole number" ;; esac -now_s() { date +%s 2>/dev/null || printf 0; } - _lanes_running() { # 0 while any lane has not yet written its done marker _j=0 while [ "$_j" -lt "$NLANES" ]; do @@ -456,6 +564,13 @@ _close_check() { # the open check's block is complete "$_cur_took" >> "$OUT/out.$_cur" fi printf '%s\n' "$_cur_rc" > "$OUT/rc.$_cur" + # Kept where the record can point at it. The scratch directory goes away + # when the command ends; this is what someone reads afterwards. + _log=$(check_log "$_cur" "$_cur_name") + if [ -n "$_log" ]; then + cat "$OUT/out.$_cur" > "$_log" 2>/dev/null || _log= + fi + record_check "$_cur_name" "$_cur_rc" "$_cur_took" "$_log" _cur= fi } @@ -514,9 +629,93 @@ run_lane() { # lane-index steps-file lane-cpus -> the lane's worst return "$_worst" } +lane_has_container() { # plan lane -> 0 when any of its steps runs a container + printf '%s\n' "$1" | awk -F"$TAB" -v l="$2" ' + $2 == l && $3 != "local" { found = 1 } + END { exit found ? 0 : 1 }' +} + +run_split() { # the plan -> the worst status, one lane streaming + # The case where exactly one lane runs containers and the others only ask this + # script questions -- today that is a push, with the reviewer beside the rest. + # + # run_lanes buffers EVERY lane to a file and prints each block when it is + # finished, because two lanes writing one terminal produce a log nobody can + # read. That is the right trade when both lanes are working. It is the wrong + # one here: the lane that is working is the whole of a push, and buffering it + # would mean watching nothing for the length of the test suite. Someone + # waiting on a silent terminal for three minutes concludes it has hung. + # + # So the lane with the containers runs in the FOREGROUND, through the ordinary + # sequential path -- same output, live, as a repo with one lane has always had + # -- and the others run beside it, buffered, and are printed whole after it. + OUT=$(mktemp -d) || die "cannot create a scratch directory for the lane logs" + trap 'rm -rf "$OUT"; record_finish 2' EXIT HUP INT TERM + + printf '%s\n' "$1" | cut -f1 | awk '!seen[$0]++' > "$OUT/order" + printf '%s\n' "$1" | cut -f2 | awk '!seen[$0]++' > "$OUT/lanes" + + # Once, here, rather than from whichever lane gets there first. + need_volume "$CACHE_VOLUME" + + _spfg= + _spk=0 + _sppids= + _spidx= + while IFS= read -r _splane; do + if lane_has_container "$1" "$_splane"; then + _spfg=$_splane + continue + fi + _spk=$((_spk + 1)) + # The same global numbering run_lanes uses, so an index names the same check + # in both paths. + printf '%s\n' "$1" | awk -F"$TAB" -v OFS="$TAB" -v l="$_splane" ' + !($1 in idx) { idx[$1] = ++n } + $2 == l { print idx[$1], $1, $3, $4 }' > "$OUT/steps.$_spk" + printf '%s: beside the rest: %s(output follows)\n' "$ME" \ + "$(cut -f2 "$OUT/steps.$_spk" | awk '!seen[$0]++' | tr '\n' ' ')" + _spidx="$_spidx $(cut -f1 "$OUT/steps.$_spk" | awk '!seen[$0]++' | tr '\n' ' ')" + run_lane "$_spk" "$OUT/steps.$_spk" 1 & + _sppids="$_sppids $!" + done < "$OUT/lanes" + + _spworst=0 + _spplan=$(printf '%s\n' "$1" | awk -F"$TAB" -v l="$_spfg" '$2 == l') + if [ -n "$_spplan" ]; then + run_serial "$_spplan" || _spworst=$? + fi + + # Reaped BEFORE the blocks below are printed: the status is the answer and is + # never guessed from what a file happened to contain, and once every lane has + # stopped nothing is still writing. + for _spp in $_sppids; do + _sprc=0 + wait "$_spp" || _sprc=$? + case "$_sprc" in + 0) ;; + 2) _spworst=2 ;; + *) [ "$_spworst" -eq 2 ] || _spworst=1 ;; + esac + done + + for _spi in $_spidx; do + if [ -f "$OUT/out.$_spi" ]; then + cat "$OUT/out.$_spi" + else + printf '\n%s: %s NEVER RAN: its lane stopped before reaching it\n' "$ME" \ + "$(awk -v n="$_spi" 'NR == n {print; exit}' "$OUT/order")" >&2 + fi + done + + rm -rf "$OUT" + trap - EXIT HUP INT TERM + return "$_spworst" +} + run_lanes() { # the plan -> the worst status of every lane OUT=$(mktemp -d) || die "cannot create a scratch directory for the lane logs" - trap 'rm -rf "$OUT"' EXIT HUP INT TERM + trap 'rm -rf "$OUT"; record_finish 2' EXIT HUP INT TERM # The checks in plan order, first appearance: the order blocks are printed in. printf '%s\n' "$1" | cut -f1 | awk '!seen[$0]++' > "$OUT/order" @@ -524,10 +723,23 @@ run_lanes() { # the plan -> the worst status of every lane NCHECKS=$(awk 'END {print NR}' "$OUT/order") NLANES=$(awk 'END {print NR}' "$OUT/lanes") + # Divided between the lanes that actually run something on a CPU. A lane whose + # every step is `local` -- a question this script answers itself, or a reviewer + # that is one API call and then waiting -- takes a single CPU and is left out + # of the division, so putting one beside the others costs the others nothing. + # Without that, adding a lane that does no computing would have halved the + # budget of the lane that does, which is exactly how lanes were measured as a + # regression (see the budget, above). + # # Floor of one, so a budget smaller than the lane count does overshoot it by # a CPU or two. The alternative is running the lanes one after another on a # single-CPU budget, which is the thing this exists to stop doing. - _lane_cpus=$(( GATE_CPUS / NLANES )) + _oncpu=0 + while IFS= read -r _lane; do + lane_has_container "$1" "$_lane" && _oncpu=$((_oncpu + 1)) + done < "$OUT/lanes" + [ "$_oncpu" -lt 1 ] && _oncpu=1 + _lane_cpus=$(( GATE_CPUS / _oncpu )) [ "$_lane_cpus" -lt 1 ] && _lane_cpus=1 # Once, here, rather than from whichever lane gets there first. @@ -545,7 +757,8 @@ run_lanes() { # the plan -> the worst status of every lane $2 == l { print idx[$1], $1, $3, $4 }' > "$OUT/steps.$_k" printf '%s: lane %s: %s\n' "$ME" "$_lane" \ "$(cut -f2 "$OUT/steps.$_k" | awk '!seen[$0]++' | tr '\n' ' ')" - run_lane "$_k" "$OUT/steps.$_k" "$_lane_cpus" & + if lane_has_container "$1" "$_lane"; then _cpus=$_lane_cpus; else _cpus=1; fi + run_lane "$_k" "$OUT/steps.$_k" "$_cpus" & _pids="$_pids $!" done < "$OUT/lanes" @@ -614,13 +827,29 @@ run_plan() { # commit | push | ci | || die "the image could not plan '$1'; nothing was checked" [ -n "$_plan" ] || die "the image planned nothing for '$1'" + state_init "$1" + # However this ends -- a step's `die`, a signal -- the record is rendered from + # whatever was finished. 2 because the only way out through here is a gate + # that could not run; every ordinary ending goes through the call below. + trap 'record_finish 2' EXIT HUP INT TERM + _n=$(printf '%s\n' "$_plan" | awk -F"$TAB" '!seen[$2]++ {n++} END {print n + 0}') + # How many of those lanes run a container, which decides which executor can + # keep the terminal live. Exactly one means everything else is waiting on + # something that is not a CPU, so that one can stream. + _oncpu=$(printf '%s\n' "$_plan" \ + | awk -F"$TAB" '$3 != "local" && !seen[$2]++ {n++} END {print n + 0}') _rc=0 if [ "${_n:-1}" -le 1 ]; then run_serial "$_plan" || _rc=$? + elif [ "${_oncpu:-0}" -eq 1 ]; then + run_split "$_plan" || _rc=$? else run_lanes "$_plan" || _rc=$? fi + + trap - EXIT HUP INT TERM + record_finish "$_rc" return "$_rc" } diff --git a/sdlc b/sdlc index 6a26e28..cca4965 100755 --- a/sdlc +++ b/sdlc @@ -368,6 +368,24 @@ def plan_lines(cfg: dict, check: str) -> list[tuple[str, str, str]]: # existed. `sdlc push` has no image chain either, so push is unchanged too. MAIN_LANE = "main" +REVIEW_LANE = "review" + +# Checks that run beside anything, including a command that rewrites the tree. +# +# The one-lane rule above protects readers of the TREE. Gatehouse does not read +# the tree: its input is `git diff origin/...HEAD`, committed objects +# that bin/sdlc computes on the host before the container starts and pipes in on +# stdin. So ruff can be rewriting your source while Gatehouse reviews the diff +# and neither notices the other. +# +# It is also the only check whose time is not CPU -- it is one API call and then +# waiting -- which is why it is worth moving. On push it was 35 seconds of a +# six-to-eight minute tier spent with the CPUs idle. The launcher gives a lane +# with no containers in it a single CPU and divides the budget between the rest, +# so the main lane keeps the whole budget it had and this costs nothing to +# overlap. That is the difference from the lanes that were measured as a +# regression: those were two CPU-bound lanes splitting one budget. +TREE_INDEPENDENT = ("gatehouse",) def lane_of(cfg: dict, checks: list[str]) -> dict[str, str]: @@ -377,9 +395,10 @@ def lane_of(cfg: dict, checks: list[str]) -> dict[str, str]: `pytest-in-image` may be listed before the `image-build` that makes what it runs, and two builds may legitimately produce two different tags. """ - # Nothing runs beside a check that rewrites the tree: one lane, in order. + # Nothing runs beside a check that rewrites the tree -- except the ones that + # do not read it. if any(check in WRITES_TREE for check in checks): - return dict.fromkeys(checks, MAIN_LANE) + return {c: REVIEW_LANE if c in TREE_INDEPENDENT else MAIN_LANE for c in checks} lanes: dict[str, str] = {} by_tag: dict[str, str] = {} for check in checks: @@ -394,6 +413,9 @@ def lane_of(cfg: dict, checks: list[str]) -> dict[str, str]: # No build of that image in this command: the repo is testing something # already built, so it owes nothing to another lane and gets its own. lanes[check] = by_tag.get(image, check) + for check in checks: + if check in TREE_INDEPENDENT: + lanes[check] = REVIEW_LANE return {check: lanes.get(check, MAIN_LANE) for check in checks} diff --git a/tests/test_host_contract.sh b/tests/test_host_contract.sh index 2b831df..751662f 100755 --- a/tests/test_host_contract.sh +++ b/tests/test_host_contract.sh @@ -48,7 +48,7 @@ done # POSIX utilities. The rule is about language toolchains and linters, not about # rebuilding coreutils -- every machine with a shell has these. POSIX_UTILS="sh awk sed tr grep basename dirname nproc cat mktemp rm ln head id - cut sleep date wc" + cut sleep date wc mkdir tail" for prog in $POSIX_UTILS; do if full=$(command -v "$prog" 2>/dev/null); then ln -sf "$full" "$SHIM/$prog"; fi done @@ -98,6 +98,9 @@ case " \$* " in *" plan lanes-stall "*) printf 'f${TAB}one${TAB}image-ro${TAB}quiet-ok\n' printf 's${TAB}two${TAB}image-ro${TAB}very-slow-rc-one\n' ;; + *" plan lanes-review "*) + printf 'x${TAB}main${TAB}image-ro${TAB}tick-x\n' + printf 'rev${TAB}review${TAB}local${TAB}gatehouse\n' ;; *" plan lanes-budget "*) printf 'p${TAB}one${TAB}host${TAB}podman${US}build${US}-t${US}localhost/x:sdlc${US}.\n' printf 'q${TAB}two${TAB}image-ro${TAB}quiet-ok\n' ;; @@ -119,6 +122,9 @@ case "\$_last" in step-one) printf 'RAN step-one\n'; sleep 1 ;; step-two) printf 'RAN step-two\n'; sleep 1 ;; slow-rc-two) printf 'RAN slow-rc-two\n'; sleep 2; exit 2 ;; + gatehouse) + if [ -f "$SHIM/review-fails" ]; then printf 'REVIEW FOUND SOMETHING\n'; exit 1; fi + printf 'REVIEW CLEAN\n' ;; must-not-run) printf 'RAN must-not-run\n'; touch "$SHIM/must-not-run" ;; after-the-failure|other-lane-first|other-lane-second) printf 'RAN %s\n' "\$_last" ;; @@ -417,13 +423,80 @@ chmod +x "$GITSHIM/git" ENVFILE="$GITSHIM/gatehouse.env" : > "$ENVFILE" PATH="$GITSHIM" GATEHOUSE_ENV="$ENVFILE" sh bin/sdlc gatehouse-direct >/dev/null 2>&1 && rc=0 || rc=$? -rm -rf "$GITSHIM" if [ "$rc" -eq 0 ]; then ok "a passing review after a non-empty diff reports success" else bad "a passing review after a non-empty diff exited $rc" fi +printf '\n== a lane that is not on a CPU\n\n' + +# A reviewer waiting on an API is not competing for the machine, so it runs +# BESIDE the lane that is -- and the lane that is keeps the whole budget and +# keeps the terminal. Buffering the working lane too, as run_lanes does, would +# mean a push shows nothing for the length of the test suite. +review() { # [fails] -> exit status, output in $RUN/out + rm -f "$SHIM/review-fails" + [ "${1:-}" = fails ] && touch "$SHIM/review-fails" + # Not truncated: the argv assertions at the end of this file read the whole + # log, so this run's lines are taken from where it was instead. + _was=$(wc -l < "$LOG" 2>/dev/null || printf 0) + _rc=0 + PATH="$GITSHIM" GATEHOUSE_ENV="$ENVFILE" SDLC_CPUS=4 \ + sh bin/sdlc lanes-review > "$RUN/out" 2>&1 || _rc=$? + tail -n "+$((_was + 1))" "$LOG" > "$RUN/podman.log" 2>/dev/null || : > "$RUN/podman.log" + rm -f "$SHIM/review-fails" + return "$_rc" +} + +review && rc=0 || rc=$? +if [ "$rc" -eq 0 ]; then + ok "two lanes, one of them off the CPU, pass together" +else + bad "a passing split run exited $rc" + sed 's/^/ /' "$RUN/out" +fi + +# The working lane streamed: its output is there, and the buffered lane's block +# comes after all of it rather than interleaved into it. +fg_end=$(grep -n 'END tick-x' "$RUN/out" | cut -d: -f1) +rev_at=$(grep -n 'REVIEW CLEAN' "$RUN/out" | cut -d: -f1) +if [ -n "$fg_end" ] && [ -n "$rev_at" ] && [ "$rev_at" -gt "$fg_end" ]; then + ok "the lane off the CPU is printed whole, after the one that streamed" +else + bad "the two lanes' output was not separated (fg=$fg_end review=$rev_at)" + sed 's/^/ /' "$RUN/out" +fi + +# Said up front, before the streaming starts, so nobody wonders what else is +# running. The check's name comes from the plan: no check is named in bin/sdlc. +if grep -q 'beside the rest: rev' "$RUN/out"; then + ok "the run says what is going on beside it, and names it from the plan" +else + bad "the split run did not say what was running beside the main lane" +fi + +# The regression guard. Dividing the budget by the LANE count would have given +# the working lane 2 of 4 to make room for a lane that wants no CPU at all -- +# which is how lanes were measured as a regression in the first place. +if grep -q -- '--cpus 4 ' "$RUN/podman.log" && grep -q -- '--cpus 1 ' "$RUN/podman.log"; then + ok "the working lane kept the whole budget; the other took one CPU" +else + bad "the budget was divided with a lane that runs no container" + grep -o -- '--cpus [0-9]*' "$RUN/podman.log" | sort -u | sed 's/^/ /' +fi + +# And the verdict is the worst of the two, whichever lane it came from. +review fails && rc=0 || rc=$? +if [ "$rc" -eq 1 ]; then + ok "a finding in the lane off the CPU fails the command" +else + bad "a failing review in a split run exited $rc, want 1" + sed 's/^/ /' "$RUN/out" +fi + +rm -rf "$GITSHIM" + # The targets that run a container must actually have tried to. Without this the # loop above would also pass if every target silently did nothing. if [ -s "$LOG" ]; then diff --git a/tests/test_plan.py b/tests/test_plan.py index c73fae7..51a8e7b 100644 --- a/tests/test_plan.py +++ b/tests/test_plan.py @@ -430,11 +430,27 @@ def plan_of(target: str, pyproject: str, **env) -> list[list[str]]: str(argv), ) -single = plan_of("push", '[project]\nname = "fx"\nversion = "0.0.0"\n') +# Push has two lanes: everything that computes, and the reviewer. Gatehouse +# reads the diff bin/sdlc computed on the host, never the worktree, so it is the +# one check that can run beside a command that rewrites the tree -- and the one +# whose time is an API call rather than a CPU, so overlapping it is free. +pushed = plan_of("push", '[project]\nname = "fx"\nversion = "0.0.0"\n') check( - "push is one lane, so it runs exactly as it did before", - {r[1] for r in single} == {S.MAIN_LANE}, - str({r[1] for r in single}), + "push puts the reviewer in its own lane", + {r[0] for r in pushed if r[1] == S.REVIEW_LANE} == {"gatehouse"}, + str([(r[0], r[1]) for r in pushed]), +) +check( + "everything that computes stays in one lane", + {r[1] for r in pushed} == {S.MAIN_LANE, S.REVIEW_LANE}, + str({r[1] for r in pushed}), +) +# CI gets Gatehouse from its own workflow, so there is nothing to split off. +ci_rows = plan_of("ci", '[project]\nname = "fx"\nversion = "0.0.0"\n') +check( + "ci is one lane, having no reviewer to run beside", + {r[1] for r in ci_rows} == {S.MAIN_LANE}, + str({r[1] for r in ci_rows}), ) From df8bdcd5cd22450fb926bee19d36169963a1a1f3 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:24:58 -0500 Subject: [PATCH 06/10] feat: `sdlc -- `, so one test can be run as one test `sdlc pytest tests/test_x.py` ran the whole suite. The dispatcher passed "$1" to run_plan and dropped every argument after it without a word, so the first thing anyone reaches for after a rejected push -- rerun just that test -- silently did the most expensive possible thing instead. Arguments after `--` are joined with the unit separator, exported as SDLC_ARGS, and appended by `t_pytest` after the config's `args` and after `-n $SDLC_JOBS`, so the caller's flag wins: `-- -n0` turns off the parallelism the gate just added, which is how you read a traceback. sdlc pytest -- tests/test_x.py::test_y -n0 Refusals rather than guesses, because a gate that quietly does something else is the thing this repo exists to delete: sdlc pytest tests/x.py exit 2, and says where the arguments go sdlc pytest -- exit 2 sdlc push -- -x exit 2; a tier is several checks an argument containing the separator exit 2 It is not a plan field. The plan is the image telling the host what to run and `plain()` guards it precisely so a repo's TOML cannot forge a line into this shell's argv; this travels the other way, by name through the environment, and lands in a Python argv list with no shell in between. The test asserts that: podman is handed `-e SDLC_ARGS` and the value appears nowhere in the argv. A tier CLEARS the variable rather than refusing it, so an `SDLC_ARGS` exported in someone's shell cannot reach a push's pytest by the back door. pytest first because that is where the cycles go. mypy, semgrep and ruff can read the same variable later. Refs #18 --- bin/sdlc | 44 ++++++++++++++++++++++- sdlc | 29 ++++++++++++++- tests/test_host_contract.sh | 72 ++++++++++++++++++++++++++++++++++--- tests/test_plan.py | 29 +++++++++++++++ 4 files changed, 167 insertions(+), 7 deletions(-) diff --git a/bin/sdlc b/bin/sdlc index d146208..b14405e 100755 --- a/bin/sdlc +++ b/bin/sdlc @@ -230,6 +230,7 @@ run_in_image() { -e "SDLC_HOST_REPO=$REPO" \ -e "SDLC_CACHE_VOLUME=$CACHE_VOLUME" \ ${SDLC_SKIP:+-e SDLC_SKIP} \ + ${SDLC_ARGS:+-e SDLC_ARGS} \ ${SDLC_GATEHOUSE_SAMPLES:+-e SDLC_GATEHOUSE_SAMPLES} \ "$@" @@ -865,6 +866,8 @@ usage: $ME commit | push | ci | | tiers | list | versions | codeql-basel ci what CI runs: push's checks except the two reviewers, plus the repo's own one check by name -- the inner loop + -- one check, with arguments for the tool itself, as: + $ME pytest -- tests/test_x.py::test_y -n0 tiers what each command runs in this repo list the built-in checks and the kinds versions the exact tool versions the image carries @@ -878,5 +881,44 @@ EOF ;; tiers|list|versions) run_in_image ro nostdin "$IMAGE" "$1" ;; codeql-baseline) run_in_image rw nostdin "$IMAGE" "$1" ;; - *) SDLC_DEPTH=0 run_plan "$1" ;; + *) + _target=$1 + shift + # Arguments for the check itself, after `--`. Without this, `sdlc pytest + # tests/x.py` ran the WHOLE suite: the dispatcher passed "$1" to run_plan + # and dropped everything after it, silently, so the one thing an agent + # reaches for after a rejected push -- rerun just that test -- quietly did + # the most expensive possible thing instead. + # + # Joined with the unit separator and handed over in the environment, like + # SDLC_SKIP. It is NOT a plan field: the plan is the image telling this + # script what to run, and `plain()` guards it so a repo's config can never + # forge a line into this shell's argv. This goes the other way and lands in + # a Python argv list, with no shell between here and the tool. + case "$_target" in + commit|push|ci) + # A tier is several checks; arguments for "the check" have no meaning. + # Cleared rather than refused when it merely arrives in the + # environment, so an exported SDLC_ARGS cannot reach a tier's pytest. + [ $# -eq 0 ] || die "$_target runs several checks; '--' arguments belong to a single check" + SDLC_ARGS= + ;; + *) + if [ $# -gt 0 ]; then + [ "$1" = -- ] || die "unexpected argument '$1'; arguments for a check go after '--', as: $ME $_target -- " + shift + [ $# -gt 0 ] || die "'--' with nothing after it; give the check's arguments, or drop it" + SDLC_ARGS= + for _a in "$@"; do + # The one character that could not survive the round trip, refused + # for the same reason the plan refuses it. + case $_a in *"$US"*) die "an argument contains a control separator, which is refused" ;; esac + SDLC_ARGS="${SDLC_ARGS:+$SDLC_ARGS$US}$_a" + done + export SDLC_ARGS + fi + ;; + esac + SDLC_DEPTH=0 run_plan "$_target" + ;; esac diff --git a/sdlc b/sdlc index cca4965..31e72e8 100755 --- a/sdlc +++ b/sdlc @@ -795,6 +795,28 @@ def sync_project(env: dict[str, str]) -> None: # `[tool.pytest] testpaths`), so there is one place to say what gets checked. +def passthrough() -> list[str]: + """What the caller put after `--` on a single-check run. + + `bin/sdlc` joins those words with the unit separator and exports + SDLC_ARGS. It is deliberately not a plan field: the plan is this script + telling the host what to run, and `plain()` guards it so that a repo's + config can never forge a line into the host's shell. This travels the other + way and ends up in an argv list, so there is no shell between the caller + and the tool. + + Only a check that is a tool invocation reads this, and only `pytest` does + today. A tier never sees it -- `bin/sdlc` clears the variable for + commit/push/ci rather than refusing it, so an exported SDLC_ARGS cannot + reach a tier's pytest -- and no `host` step argv is built from it. + """ + raw = os.environ.get("SDLC_ARGS") + if not raw: + return [] + # Not filtered: `-k ''` is a legitimate empty argument. + return raw.split(UNIT_SEPARATOR) + + def t_ruff_check(s: dict) -> int: need_project() step("ruff check --fix .") @@ -838,7 +860,8 @@ def t_pytest(s: dict) -> int: sync_project(env) jobs = os.environ.get("SDLC_JOBS", "0") args = names(s.get("args", ["-q", "-rfE"]), "pytest.args") - step("pytest") + extra = passthrough() + step("pytest" + ("".join(f" {a}" for a in extra) if extra else "")) cmd = [ "uv", "run", @@ -852,6 +875,10 @@ def t_pytest(s: dict) -> int: ] if jobs != "0" and "pytest-xdist" in (project_dir() / "uv.lock").read_text(encoding="utf-8"): cmd += ["-n", jobs] + # LAST, after the config's args and after `-n`, so what the caller asked + # for on the command line wins: `-- -n0` turns off the parallelism this + # just added, which is how you read a traceback. + cmd += extra return run(cmd, env=env, cwd=project_dir())[0] diff --git a/tests/test_host_contract.sh b/tests/test_host_contract.sh index 751662f..2175d92 100755 --- a/tests/test_host_contract.sh +++ b/tests/test_host_contract.sh @@ -27,6 +27,16 @@ FAILED=0 ok() { printf 'ok %s\n' "$*"; } bad() { printf 'FAIL %s\n' "$*"; FAILED=$((FAILED + 1)); } +# The fake podman appends every call to one cumulative log, which the argv +# assertions at the end of this file read whole. A block that wants only its own +# calls marks the position and reads from there, rather than truncating it. +LOG_AT=0 +log_mark() { LOG_AT=$(wc -l < "$LOG" 2>/dev/null || printf 0); } +log_since() { + tail -n "+$((LOG_AT + 1))" "$LOG" > "$RUN/podman.log" 2>/dev/null \ + || : > "$RUN/podman.log" +} + printf '== host contract: bin/sdlc\n\n' case "$(head -1 bin/sdlc)" in @@ -388,13 +398,16 @@ done # The CPU budget is for the whole gate, not for each container: two lanes out # of a budget of 4 get 2 each, so a laptop running `sdlc ci` is no worse off # than it was when the checks were sequential. -: > "$LOG" +log_mark lanes lanes-budget || true -if grep -q -- '--cpus 2 ' "$LOG" && grep -q -- '--cpu-quota 200000' "$LOG"; then +log_since +if grep -q -- '--cpus 2 ' "$RUN/podman.log" \ + && grep -q -- '--cpu-quota 200000' "$RUN/podman.log"; then ok "a budget of 4 is split 2 and 2 between two lanes" else bad "the lanes did not divide the CPU budget" - grep -o -- '--cpus [0-9]*\|--cpu-quota [0-9]*' "$LOG" | sort -u | sed 's/^/ /' + grep -o -- '--cpus [0-9]*\|--cpu-quota [0-9]*' "$RUN/podman.log" | sort -u \ + | sed 's/^/ /' fi # A container that succeeds must report success, whatever the caller's last @@ -440,11 +453,11 @@ review() { # [fails] -> exit status, output in $RUN/out [ "${1:-}" = fails ] && touch "$SHIM/review-fails" # Not truncated: the argv assertions at the end of this file read the whole # log, so this run's lines are taken from where it was instead. - _was=$(wc -l < "$LOG" 2>/dev/null || printf 0) + log_mark _rc=0 PATH="$GITSHIM" GATEHOUSE_ENV="$ENVFILE" SDLC_CPUS=4 \ sh bin/sdlc lanes-review > "$RUN/out" 2>&1 || _rc=$? - tail -n "+$((_was + 1))" "$LOG" > "$RUN/podman.log" 2>/dev/null || : > "$RUN/podman.log" + log_since rm -f "$SHIM/review-fails" return "$_rc" } @@ -514,6 +527,55 @@ else bad "podman was called but never asked to run the image" fi +printf '\n== arguments for one check\n\n' + +# `sdlc pytest tests/x.py` used to run the whole suite: the dispatcher read +# "$1" and dropped the rest without a word. Silently doing the most expensive +# possible thing is worse than refusing, so an argument that is not behind `--` +# is now an exit 2, and `--` on a tier is too. +args_case() { # expect-rc target [args...] + _want=$1; shift + _what="$*" + _got=0 + PATH="$SHIM" sh bin/sdlc "$@" >/dev/null 2>&1 || _got=$? + if [ "$_got" -eq "$_want" ]; then + ok "'$_what' exits $_got" + else + bad "'$_what' exits $_got, want $_want" + fi +} + +args_case 2 mypy tests/x.py +args_case 2 mypy -- +args_case 2 push -- -x +args_case 2 commit -- -x +args_case 0 mypy -- -q + +# Handed over in the ENVIRONMENT, not built into the argv: podman is told the +# variable's name and the kernel carries the value, so no word of it is ever a +# word of a command line here. +log_mark +PATH="$SHIM" sh bin/sdlc mypy -- --tb=short -n0 >/dev/null 2>&1 || true +log_since +if grep -q -- '-e SDLC_ARGS' "$RUN/podman.log" && ! grep -q -- '--tb=short' "$RUN/podman.log" +then + ok "the check's arguments reach the container by name, not as argv words" +else + bad "SDLC_ARGS was not passed, or its value leaked into the podman argv" + grep -o -- '-e SDLC_[A-Z]*\|--tb=short' "$RUN/podman.log" | sort -u | sed 's/^/ /' +fi + +# A tier clears it rather than refusing it, so an exported one cannot reach a +# tier's pytest by the back door. +log_mark +PATH="$SHIM" SDLC_ARGS="--lf" sh bin/sdlc push >/dev/null 2>&1 || true +log_since +if ! grep -q -- '-e SDLC_ARGS' "$RUN/podman.log"; then + ok "an inherited SDLC_ARGS does not reach a tier" +else + bad "a tier forwarded SDLC_ARGS from the environment" +fi + printf '\n== what the containers are given\n\n' # The worktree is the developer's, so its __pycache__ is the laptop's. Every diff --git a/tests/test_plan.py b/tests/test_plan.py index 51a8e7b..85976db 100644 --- a/tests/test_plan.py +++ b/tests/test_plan.py @@ -454,6 +454,35 @@ def plan_of(target: str, pyproject: str, **env) -> list[list[str]]: ) +# ------------------------------------------------ arguments for a single check +# +# bin/sdlc joins the words after `--` with the unit separator and exports +# SDLC_ARGS. It is not a plan field: the plan is this script telling the host +# what to run, and `plain()` guards it. This goes the other way and ends up in +# an argv list, so there is no shell between the caller and the tool. + +PASSTHROUGH = [ + (None, []), + ("", []), + ("-q", ["-q"]), + (f"-k{S.UNIT_SEPARATOR}", ["-k", ""]), # `-k ''` is a legitimate argument + (f"tests/x.py::test_y{S.UNIT_SEPARATOR}-n0", ["tests/x.py::test_y", "-n0"]), + # A word with spaces survives as ONE word, which is the whole reason the + # separator is used instead of splitting on whitespace. + (f"-k{S.UNIT_SEPARATOR}not slow", ["-k", "not slow"]), +] +for raw, want in PASSTHROUGH: + saved = os.environ.pop("SDLC_ARGS", None) + if raw is not None: + os.environ["SDLC_ARGS"] = raw + try: + check(f"passthrough {raw!r}", S.passthrough() == want, str(S.passthrough())) + finally: + os.environ.pop("SDLC_ARGS", None) + if saved is not None: + os.environ["SDLC_ARGS"] = saved + + def main() -> int: print() if FAILED: From 19848ad7dec713bf41f6550fd893f4f7e29d4f44 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:28:02 -0500 Subject: [PATCH 07/10] docs: say how to work a change, and stop relabelling the tree exclusively Three things, all found by using this repo's own gates on this repo. `--help` and the README now say what order to do things in: `sdlc commit && sdlc push` before the first push, `sdlc -- ` to reproduce one failure, and `/sdlc/last-push.json` for what the last run cost. Plus the two things that reliably mislead people -- that the Python running the tests is the image's and not the machine's, so a host `pytest` result says nothing about this gate; and that CI triage, not the local record, is the authority on a review finding. The path in `--help` is asked of `git rev-parse`, because in a linked worktree the git directory is elsewhere. `z`, not `Z`, in this repo's own three local hooks. bin/sdlc has carried the reason since it was written: `Z` relabels the worktree with a category pair private to one container, which takes the tree out from under every other container that mounts it. Two of those hooks did it anyway, and the result was that `podman run -v "$PWD:/mnt:ro" ...` -- the shellcheck step, as container.yml still spells it -- could not read the repo at all: "openBinaryFile: permission denied". Half an hour, on the tool whose job is to delete exactly this. The CI step gets `,z` as well, since a fresh checkout is the only reason it has not bitten there. And the local Gatehouse hook pinned 0.22.0 while the image bakes 0.23.0 -- precisely the drift this repo exists to delete, in its own config. The state directory is also made absolute at the point it is computed, so the log paths in the record can be opened from anywhere rather than only from the top of the worktree. --- .github/workflows/container.yml | 2 +- .pre-commit-config.yaml | 13 ++++++++++--- README.md | 17 +++++++++++++++++ bin/sdlc | 18 ++++++++++++++++++ 4 files changed, 46 insertions(+), 4 deletions(-) diff --git a/.github/workflows/container.yml b/.github/workflows/container.yml index ab6ee7a..70afd4c 100644 --- a/.github/workflows/container.yml +++ b/.github/workflows/container.yml @@ -79,7 +79,7 @@ jobs: if: env.IS_TAG != 'true' run: | set -eu - podman run --rm -v "$PWD:/mnt:ro" -w /mnt \ + podman run --rm -v "$PWD:/mnt:ro,z" -w /mnt \ docker.io/koalaman/shellcheck:v0.11.0 bin/sdlc tests/contract.sh \ tests/test_host_contract.sh tests/test_host_contract.sh diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index acd671f..7ba99dd 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -35,7 +35,7 @@ repos: hooks: - id: shellcheck name: shellcheck (bin/sdlc and the tests) - entry: podman --events-backend=none run --rm --log-driver=none -v .:/mnt:ro,Z -w /mnt docker.io/koalaman/shellcheck:v0.11.0 bin/sdlc tests/contract.sh tests/test_host_contract.sh + entry: podman --events-backend=none run --rm --log-driver=none -v .:/mnt:ro,z -w /mnt docker.io/koalaman/shellcheck:v0.11.0 bin/sdlc tests/contract.sh tests/test_host_contract.sh language: system pass_filenames: false always_run: true @@ -61,13 +61,20 @@ repos: # it is right. The validator matches the image NAME, so exact tags still pass # `Constitution / Validate constitution`. # + # A SECOND deviation, for the reason bin/sdlc's run_in_image states: `z`, not + # `Z`. `Z` relabels the worktree with a category pair private to one + # container, taking it out from under every other container that mounts the + # same tree -- including a plain `-v .:/mnt:ro` in another hook or workflow + # step, which then cannot read the repo at all. It cost half an hour here. + # These three hooks mount one tree, so they share the label. + # # The fleet example should change the same way; raised separately. # `--events-backend=none --log-driver=none` keeps a gate out of the journal. - repo: local hooks: - id: gourmand name: Gourmand (AI slop detection) - entry: podman --events-backend=none run --rm --log-driver=none -v .:/src:Z -w /src quay.io/crunchtools/gourmand:1.0.0 check --full + entry: podman --events-backend=none run --rm --log-driver=none -v .:/src:z -w /src quay.io/crunchtools/gourmand:1.0.0 check --full language: system pass_filenames: false always_run: true @@ -75,7 +82,7 @@ repos: - id: gatehouse name: Gatehouse (AI code review, staged diff) - entry: bash -c 'git diff --cached | podman --events-backend=none run --rm --log-driver=none -i --env-file "$HOME/.config/mcp-env/gatehouse.env" -v .:/src:ro,Z -w /src quay.io/crunchtools/gatehouse:0.22.0 --stdin' + entry: bash -c 'git diff --cached | podman --events-backend=none run --rm --log-driver=none -i --env-file "$HOME/.config/mcp-env/gatehouse.env" -v .:/src:ro,z -w /src quay.io/crunchtools/gatehouse:0.23.0 --stdin' language: system pass_filenames: false always_run: true diff --git a/README.md b/README.md index d8b2a11..70536e2 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,23 @@ the image contains. Because the laptop and CI run the same image and the same commands, a green push predicts a green CI. +## Working a change + +In the order that costs least: + +``` +sdlc commit && sdlc push every verdict, before a CI cycle +sdlc pytest -- tests/x.py::test_y -n0 reproduce one failure, just that one +jq . .git/sdlc/last-push.json what the last run cost, per check +``` + +Two things mislead people here. The Python running your tests is the image's, +not your machine's, so a result from a host `pytest` says nothing about this +gate — and bytecode your machine compiled is kept out of the container +deliberately, so it cannot say anything either. And on a review finding, CI +triage is the authority, not the local record: push predicts CI, it does not +replace it. + ## Wiring a repo `.pre-commit-config.yaml`, the whole file: diff --git a/bin/sdlc b/bin/sdlc index b14405e..9ab75ba 100755 --- a/bin/sdlc +++ b/bin/sdlc @@ -858,6 +858,10 @@ run_plan() { # commit | push | ci | case "${1:---help}" in -h|--help|help) + # Asked of git rather than written as `.git/sdlc`: in a linked worktree the + # git directory is somewhere else entirely. + STATE_HINT=$(git rev-parse --git-path sdlc 2>/dev/null) || STATE_HINT= + [ -n "$STATE_HINT" ] || STATE_HINT='/sdlc' cat < | tiers | list | versions | codeql-baseline @@ -876,6 +880,20 @@ usage: $ME commit | push | ci | | tiers | list | versions | codeql-basel Configure each command in pyproject.toml: [tool.sdlc.commit], [tool.sdlc.push], [tool.sdlc.ci]. Skip one check once: SDLC_SKIP=name. Exit 2 means a check COULD NOT RUN and is never to be worked around. + +Working a change, in the order that costs least: + + $ME commit && $ME push every verdict, before spending a CI cycle on + a finding you could have read in two minutes + $ME -- reproduce one failure without rerunning the tier + $STATE_HINT/last-push.json + what the last run cost, per check, and where + each buffered check's output went + +Two things mislead people here. The Python running your tests is the image's, +not this machine's, so a result from a host 'pytest' says nothing about this +gate. And on a review finding CI triage is the authority, not this record, +which is only what happened on this machine last time. Image: $IMAGE EOF ;; From f3b09f4979f28eb7926c972eece2563f9e0ec2b9 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:35:49 -0500 Subject: [PATCH 08/10] chore: python-sdlc 0.4.0 Fewer push cycles, and a gate you can see into. Measured on the way through, with the 0.4.0 image built locally: - the whole commit tier, Gourmand included, is 2.6s on this repo - `sdlc pytest -- -n0` runs one test in 15s, where the same command used to run the entire suite - #16 reproduced and fixed end to end on trentina's own test: with the tree carrying host-compiled bytecode, the gate FAILS `test_library_lines_are_fingerprinted` and prints the laptop's path in the traceback; with PYTHONPYCACHEPREFIX set and nothing else changed, the same poisoned tree passes - the contract test, the three unit suites and 54 host-contract checks pass against the locally built image The version moves in pyproject.toml, IMAGE_TAG and the Containerfile LABEL together, which is what `Constitution / Validate constitution` and tests/contract.sh check between them. v0.3.5 shipped with those out of step and had no image at all; the workflow change in this release is why that cannot happen again. --- CHANGELOG.md | 162 +++++++++++++++++++++++++++++++++++++++++++++++++ Containerfile | 2 +- IMAGE_TAG | 2 +- pyproject.toml | 2 +- 4 files changed, 165 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8319ed7..3cc1d41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,168 @@ All notable changes to this project are documented here, following ## [Unreleased] +## [0.4.0] - 2026-11-04 + +Fewer push cycles, and a gate you can see into. Every change here comes from +one piece of evidence: crunchtools/trentina#706 took five pushes, four of them +rejected, about 28 minutes of gate time, and not one of those four rejections +was something the developer could have seen before pushing. + +### Added + +- **Gourmand runs at commit, not only at push.** Its findings are + deterministic -- the same tree gives the same answer, with no model and no + network -- so paying for them at push, where each rejection costs a + six-to-eight minute cycle and a re-review of the whole diff, was paying the + wrong price. Measured in full mode, warm: 1.7s on this repo, 3.2s on + trentina. Full rather than `--diff-source`, because full mode also sees + duplication across files, which is most of what it is for. It runs after both + ruff steps so it judges the formatted tree, and it stays in push as well: a + commit can be made with `--no-verify`, and push is the tier that renders the + verdicts. + + Semgrep deliberately does not join it. It takes 27s, and `semgrep` exits 2 on + a repo with no `.semgrep`, so as a default it would break the commit hook + across the fleet. A repo that wants it there writes + `[tool.sdlc.commit] checks = ["semgrep"]`. + +- **`sdlc -- `.** `sdlc pytest tests/test_x.py` used to run the + whole suite: the dispatcher read `"$1"` and dropped everything after it + without a word, so the first thing anyone reaches for after a rejected push + -- rerun just that test -- silently did the most expensive possible thing + instead. + + ``` + sdlc pytest -- tests/test_x.py::test_y -n0 + ``` + + The arguments are appended after the config's `args` and after + `-n $SDLC_JOBS`, so the caller's flag wins: `-- -n0` turns off the + parallelism the gate just added, which is how you read a traceback. An + argument that is not behind `--` is an exit 2 rather than a guess, and so is + `--` on a tier, which is several checks. They travel by name through the + environment and land in an argv list, never through the plan and never + through a shell. + +- **A record of what the last run cost.** + + | Path | What is in it | + |---|---| + | `/sdlc/last-.json` | each check, its exit, its seconds, its log | + | `/sdlc/logs-/` | what each buffered check printed | + + The serial path -- which is the path every `sdlc push` took -- printed no + durations whatsoever, so nothing about the slowest tier in this tool could be + tuned by anyone who had not sat watching it with a stopwatch. It now prints + the same per-check footer the lane path always had. The record is also what + an agent needs after a rejected push: which check, how long, where its output + went. + + In the git directory rather than a committed `.sdlc/`, because it is one + machine's last run, and asked of `git rev-parse --git-path` because in a + linked worktree `.git` is a file. It is **written, never read**: a verdict + comes from running the check, and a file that could stand in for one would be + a way past a gate. Failing to write it never fails a gate. + +- **A recipe in `--help` and the README** for the order that costs least, plus + the two things that reliably mislead people here: that the Python running the + tests is the image's and not the machine's, and that CI triage rather than + the local record is the authority on a review finding. + +### Changed + +- **The review runs beside the rest.** Gatehouse is the one check that does not + read the worktree -- its input is the `origin/...HEAD` diff, + committed objects that `bin/sdlc` computes on the host and pipes in on stdin + -- so the rule that sends a tree-rewriting command down a single lane does + not apply to it. It is also the only check whose time is not CPU: one API + call and then waiting, measured at 35s on a 365-line diff, previously spent + with the CPUs idle. `sdlc push` therefore has two lanes now; `sdlc ci` still + has one, because CI gets Gatehouse from its own workflow. + + What makes it free is a change to the budget: a lane whose every step is a + question `bin/sdlc` answers itself takes one CPU and is **left out of the + division**, so the main lane keeps the whole budget it had. Dividing by the + lane count would have handed half the machine to a lane that wants no CPU at + all, which is precisely how lanes were measured as a regression on a 4-vCPU + runner in 0.3.2. + + And the working lane keeps the terminal. Buffering every lane is right when + both are computing and wrong here -- the buffered lane would be the whole of + a push, and three minutes of silence reads like a hang -- so when exactly one + lane runs containers, that lane streams as it always did and the others are + printed whole after it. + +- **A release builds the commit once.** `container.yml` fired on a push to main + *and* on a `v*` tag, and `release.yml` tags the commit that just merged, so + both events built the same sha and both pushed `:`; the second push + replaced the first's manifest. That cost v0.3.5 its image -- a git tag and a + GitHub release with no `0.3.5` behind them, which for a fleet pinning by + exact release is an exit 2 and nothing checked. A tag event now builds + nothing: it resolves the digest the main-branch run already pushed, **runs + the contract test against it again**, and promotes that. + + The re-verification is not ceremony. `:` is pushed before it is + verified, so a main run whose contract test failed leaves a pushed but + untested digest on the registry; promoting that on the strength of the push + alone would publish an image no job ever tested. The skip is per step rather + than per job, because a job-level `if:` would skip every job that `needs:` it + and the fix would have looked right while promoting nothing. + +- **`z`, not `Z`, in this repo's own hooks**, for the reason `bin/sdlc` has + carried since it was written: `Z` relabels the worktree with a category pair + private to one container, taking the tree out from under every other + container that mounts it. Two local hooks did it anyway, and the result was + that a plain `-v "$PWD:/mnt:ro"` -- the shellcheck step -- could not read the + repo at all. The local Gatehouse hook also pinned 0.22.0 while the image + bakes 0.23.0: the drift this repo exists to delete, in its own config. + +### Fixed + +- **A test run on the laptop no longer decides the gate's verdict.** The gate + mounts the worktree at `/repo`, so the container and the laptop share the + tree's `__pycache__`. Bytecode carries the path it was compiled from, and + pytest's assertion rewriter reads its cached code back as marshalled -- + `_read_pyc` returns `marshal.load(fp)` as it stands, because the point of + that cache is the rewritten code. So a module compiled on the host kept + `/home//.../x.py` in `co_filename` while `__file__` was + `/repo/.../x.py`, and any test comparing the two failed in the gate and + nowhere else. That was four of the five pushes on trentina#706. + + `PYTHONPYCACHEPREFIX=/tmp/pycache`, in **both** containers that mount the + tree: `run_in_image` and the `pytest-in-image` argv. With it set, the + container neither reads the tree's bytecode nor writes any back into it -- + the second direction matters because the commit tier mounts the tree + read-write. A plain `import` never showed this, because the stdlib's loader + calls `_imp._fix_co_filename`; the regression test pins both halves of that. + +### Security + +- **The gate's own containers run with no capabilities.** A gate runs the + repo's own code -- conftest, mypy plugins, semgrep rules, a Containerfile's + `RUN` lines -- so the container it runs in is a boundary, not a convenience. + `pytest-in-image` has been locked down since it was written; the gate's own + containers, which run far more repo-supplied code, had none of it. + `--cap-drop=all --security-opt=no-new-privileges` on both. Not + `--network none`, which `uv sync`, `pip-audit` and Gatehouse need, and not + `--read-only`, because rewriting the tree is the point of the commit tier. + Checked rather than assumed: all seven tool binaries probe, `ruff-check` + rewrites the tree, and trentina's `mypy` runs a full `uv sync` into the + shared cache volume and type-checks 235 files. + +### Not done, and why + +- **Incremental local review** (#18's 0.5.0) is out. It trades the second look + at unchanged code -- which on trentina#706 found two real crashes -- for + speed; it depends on two unshipped changes in `crunchtools/gatehouse`; and + with the review now running beside the rest, it only saves wall time if the + review is the longest lane, which nobody has measured. The durations added + here are how that gets decided. +- **`--consensus-json` from the local Gatehouse run** is out. Gatehouse has the + flag, but the container mounts the worktree read-only, so collecting the file + would mean a new writable bind mount inside the git directory -- a real cost + against a convenience the terminal output already provides. + ## [0.3.7] - 2026-11-04 ### Added diff --git a/Containerfile b/Containerfile index 6ac20af..526a76a 100644 --- a/Containerfile +++ b/Containerfile @@ -44,7 +44,7 @@ LABEL description="Every CrunchTools Python SDLC check in one image: ruff, mypy, LABEL org.opencontainers.image.source="https://github.com/crunchtools/python-sdlc" LABEL org.opencontainers.image.description="Every CrunchTools Python SDLC check in one image." LABEL org.opencontainers.image.licenses="AGPL-3.0-or-later" -LABEL version="0.3.7" +LABEL version="0.4.0" # What the image carries, readable without starting anything. `sdlc versions` # prints these and a consuming repo's CI pins them. diff --git a/IMAGE_TAG b/IMAGE_TAG index 0f82685..1d0ba9e 100644 --- a/IMAGE_TAG +++ b/IMAGE_TAG @@ -1 +1 @@ -0.3.7 +0.4.0 diff --git a/pyproject.toml b/pyproject.toml index 8231656..88c365e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -10,7 +10,7 @@ [project] name = "python-sdlc" -version = "0.3.7" +version = "0.4.0" description = "One container image and one script for every CrunchTools Python SDLC check" requires-python = ">=3.14" license = "AGPL-3.0-or-later" From 4400c0da633c9675f22e42e48a4966e83b62bd07 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:51:03 -0500 Subject: [PATCH 09/10] fix: CodeQL could not read its own queries once the gate dropped capabilities The capability drop two commits ago broke `codeql`, and the commit that made it said "checked rather than assumed" while having checked three of the eight checks. It exercised `sdlc versions`, `ruff-check` and `mypy`. Not CodeQL. Running trentina's whole `sdlc push` against the image is what found it: 355 seconds and an exit 2 where there should have been a verdict. WARNING: Not using precompiled UseofInput.qlx: ... java.nio.file.AccessDeniedException: /opt/codeql/qlpacks/codeql/python-queries/1.9.0/.codeql/precompiled/*.qlx ... all 55 queries recompiled ... Oops! A fatal internal error occurred. (StringPool.noSuchStringError) codeql database analyze failed (exit 100) The CodeQL bundle extracts 25 precompiled `.qlx` files at mode 0600 owned by the uid that built it, 1001. The container runs as root, so root was reading them purely on CAP_DAC_OVERRIDE -- and the gate now drops every capability. Without them CodeQL recompiles the lot and then dies. Fixed in the image, not by handing the capability back: `chown -R root:root` and `chmod -R a+rX /opt/codeql` after extraction. `a+rX`, so execute is added to directories and to files that already had it and to nothing else. This also restores the precompiled queries the gate had been using all along, which is most of that 355 seconds. The real lesson is that a bare `podman run` still holds the capability, so a tool broken this way works in every hand-test and fails only in the gate. tests/contract.sh now asserts that no file under /opt needs CAP_DAC_OVERRIDE, so the next bundle -- or the next tool -- cannot reintroduce it quietly. --- CHANGELOG.md | 13 ++++++++++--- Containerfile | 19 +++++++++++++++++++ tests/contract.sh | 22 ++++++++++++++++++++++ 3 files changed, 51 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3cc1d41..b6d6bff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -151,9 +151,16 @@ was something the developer could have seen before pushing. `--cap-drop=all --security-opt=no-new-privileges` on both. Not `--network none`, which `uv sync`, `pip-audit` and Gatehouse need, and not `--read-only`, because rewriting the tree is the point of the commit tier. - Checked rather than assumed: all seven tool binaries probe, `ruff-check` - rewrites the tree, and trentina's `mypy` runs a full `uv sync` into the - shared cache volume and type-checks 235 files. + Normalizing `/opt/codeql`'s ownership and modes is part of this, not + housekeeping. The bundle extracts 25 precompiled `.qlx` files at mode 0600 + owned by uid 1001, and the container's root had been reading them purely on + `CAP_DAC_OVERRIDE`; without it `codeql database analyze` recompiled all 55 + queries and then died with an internal error -- 355 seconds and an exit 2 + instead of a verdict. Found by running trentina's whole `sdlc push` against + the image, which is the only place it shows: a bare `podman run` still holds + the capability, so a tool broken this way passes every hand-test. + `tests/contract.sh` now asserts that no file under `/opt` needs that + capability. ### Not done, and why diff --git a/Containerfile b/Containerfile index 526a76a..7fb85a3 100644 --- a/Containerfile +++ b/Containerfile @@ -150,6 +150,23 @@ RUN ln -s /app/venv/bin/gatehouse /usr/local/bin/gatehouse # 110 MB, which is the whole reason CodeQL can be in a pre-push gate at all. # Fetched by exact release and checksum-verified: a tool that decides whether # code ships is not downloaded from a moving target. +# +# The ownership and modes are NORMALIZED after extraction, and that is load +# bearing rather than tidiness. The bundle carries 25 precompiled `.qlx` files +# under `.codeql/precompiled/` at mode 0600, owned by the uid that built it +# (1001). The container runs as root, so root read them only by holding +# CAP_DAC_OVERRIDE -- and the gate now drops every capability. Without the +# chmod, `codeql database analyze` cannot read its own precompiled queries, +# recompiles all 55 of them, and then dies with an internal StringPool error: +# +# WARNING: Not using precompiled UseofInput.qlx: ... +# java.nio.file.AccessDeniedException: /opt/codeql/.../*.bytes.qlx +# Oops! A fatal internal error occurred. +# codeql database analyze failed (exit 100) +# +# Found by running trentina's `sdlc push` against this image: 355 seconds and +# an exit 2 instead of a verdict. `a+rX` rather than `a+rx`, so execute is +# added to directories and to files that already had it, and nothing else. RUN set -eu; \ base="https://github.com/github/codeql-action/releases/download/${CODEQL_BUNDLE}"; \ curl -fsSL --retry 5 --retry-delay 5 --retry-all-errors --connect-timeout 20 -o /tmp/codeql.tar.zst "${base}/codeql-bundle-python-linux64.tar.zst"; \ @@ -157,6 +174,8 @@ RUN set -eu; \ mkdir -p /opt; \ tar --use-compress-program=unzstd -xf /tmp/codeql.tar.zst -C /opt; \ rm -f /tmp/codeql.tar.zst /tmp/codeql.sha; \ + chown -R root:root /opt/codeql; \ + chmod -R a+rX /opt/codeql; \ ln -s /opt/codeql/codeql /usr/local/bin/codeql; \ codeql --version | head -1 diff --git a/tests/contract.sh b/tests/contract.sh index 570cad0..cefa427 100755 --- a/tests/contract.sh +++ b/tests/contract.sh @@ -31,6 +31,28 @@ inside() { podman run --rm --entrypoint="" "$IMAGE" "$@"; } printf '== contract test: %s\n\n' "$IMAGE" +# ------------------------------------------------- every tool's files are its own +# +# The gate runs its containers with `--cap-drop=all`, so the container's root +# has no CAP_DAC_OVERRIDE and cannot read a file it does not own whose mode says +# no. An image that ships such a file under /opt has a tool that works in +# testing -- where a bare `podman run` still has the capability -- and fails in +# the gate. +# +# This is not hypothetical. The CodeQL bundle extracts 25 precompiled `.qlx` +# files at mode 0600 owned by uid 1001, and with them unreadable +# `codeql database analyze` recompiled all 55 queries and then died with an +# internal error: 355 seconds and an exit 2 instead of a verdict, on trentina. +# The Containerfile normalizes them; this is what keeps the next bundle, or the +# next tool, from reintroducing it silently. +unreadable=$(inside sh -c 'find /opt -xdev -type f ! -perm -o=r 2>/dev/null | head -20') +if [ -z "$unreadable" ]; then + ok "every file under /opt is readable without CAP_DAC_OVERRIDE" +else + bad "files under /opt need a capability the gate drops:" + printf '%s\n' "$unreadable" | sed 's/^/ /' +fi + # ---------------------------------------------------------- every tool executes # # `command -v` is not enough: a COPY --from can land a binary that is present From 2f47094be6a2d23e0863f54986fb71b7c03bc574 Mon Sep 17 00:00:00 2001 From: Scott McCarty Date: Wed, 4 Nov 2026 20:57:18 -0500 Subject: [PATCH 10/10] fix: the recorded log paths resolve from anywhere, as the last commit claimed 19848ad's message said the state directory was made absolute where it is computed. It was not: that edit was in a patch that aborted partway, and only the other half of it landed. The claim was in the commit and the code was not. `git rev-parse --git-path sdlc` answers relative to the current directory -- `.git/sdlc` from the top, `../.git/sdlc` from a subdirectory -- and bin/sdlc chdirs to the toplevel before asking, so every path in `last-.json` came out relative to the repo root. Which is one directory out of however many an agent might be sitting in when it reads the record. "log": ".git/sdlc/logs-push/4-gatehouse.log" before "log": "/home/.../trentina/.git/sdlc/logs-push/4-gatehouse.log" after The hint in `--help` stays relative, because that one is being read by a person looking at their own repo. --- bin/sdlc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/bin/sdlc b/bin/sdlc index 9ab75ba..9879d60 100755 --- a/bin/sdlc +++ b/bin/sdlc @@ -338,6 +338,10 @@ RECORD_FOR= # the command being recorded state_init() { # command STATE=$(git rev-parse --git-path sdlc 2>/dev/null) || STATE= [ -n "$STATE" ] || return 0 + # `--git-path` answers relative to the current directory, so the paths this + # records would only resolve from the top of the worktree. An agent reading + # the record is as likely to be somewhere else. + case $STATE in /*) ;; *) STATE="$REPO/$STATE" ;; esac LOGS="$STATE/logs-$1" mkdir -p "$LOGS" 2>/dev/null || { STATE=; LOGS=; return 0; } # One run's worth. A history nobody prunes is a directory that grows until