Repository navigation
Fewer push cycles: 0.4.0 - #22
Merged
Merged
Conversation
…igest `container.yml` triggers on a push to main AND on a `v*` tag, and `release.yml` tags the commit that just merged -- so both events fired for the same sha and both ran `build`, which pushes `:<sha>`. The second push replaced the first's manifest. It cost v0.3.5 its image: a git tag and a GitHub release with no `0.3.5` behind them, which for a fleet that pins by exact release is an exit 2 and nothing checked. A tag event now builds nothing. It resolves the digest the main-branch run of that same commit already pushed, and `promote` copies that. Two things worth naming, because both are easy to get wrong: The skip is per STEP, not per job. A job-level `if:` on `build` skips every job that `needs:` it unless each one grows `always()` logic -- so the fix would have looked right and quietly promoted nothing on its first tag. The steps carry the condition and the job's outputs fall back from `push` to `resolve`, so the plumbing downstream is untouched. `verify-digest` still runs on the tag path. `:<sha>` is pushed BEFORE it is verified, so a main run whose contract test failed leaves a pushed but untested digest on the registry; promoting that on the strength of the push alone would publish an image no job ever tested. Resolving the digest and then verifying it again is what keeps the invariant this workflow's own header states. The concurrency group stays, with a new reason: it is what guarantees the main-branch run has finished before the tag run looks for `:<sha>`. Closes #14
The gate mounts the worktree at /repo, so the container and the laptop share the tree's `__pycache__`. Bytecode carries the path it was compiled from, and pytest's assertion rewriter reads its cached code back as marshalled -- `_read_pyc` in `_pytest/assertion/rewrite.py` returns `marshal.load(fp)` as it stands, because the point of that cache is the REWRITTEN code. So a module compiled on the host keeps `/home/<user>/.../x.py` in `co_filename` while `__file__` is `/repo/.../x.py`, and any test comparing the two fails in the gate and nowhere else. That is four of the five pushes on crunchtools/trentina#706, about 30 minutes, spent on a test with no defect behind it. `-e PYTHONPYCACHEPREFIX=/tmp/pycache`, in BOTH containers that mount the tree: `run_in_image` here and the `pytest-in-image` argv the image builds. #16 named only the first. /tmp is already the 4g tmpfs, pytest's `get_cache_dir` honours `sys.pycache_prefix`, and with it set the container neither reads the tree's bytecode nor writes any back into it -- the second direction matters because the commit tier mounts the tree read-write. A plain `import` never showed this: the stdlib's loader calls `_imp._fix_co_filename` after unmarshalling. tests/test_pycache_prefix.py pins both halves of that -- the rewriter's path survives, the stdlib's does not -- and reproduces the mismatch before proving the prefix removes it, so it cannot pass by testing nothing. The env var reaching each container is asserted where each argv is built: tests/test_host_contract.sh for the gate's own, tests/test_plan.py for `pytest-in-image`. Closes #16
A gate runs the repo's own code -- conftest, mypy plugins, semgrep rules, a Containerfile's RUN lines -- so the container it runs in is a boundary, not a convenience. `kind_pytest_in_image` has known that since it was written: `--read-only`, `--cap-drop=all`, `--security-opt no-new-privileges`, `--network none` unless asked. The gate's own containers, which run far more repo-supplied code than that one does, had none of it. `--cap-drop=all --security-opt=no-new-privileges` on both, in `run_in_image` and on `run_host`'s `run` prefix. Not `--network none`: `uv sync`, `pip-audit` and Gatehouse need it. Not `--read-only`: ruff rewrites the tree and that is the point of the commit tier. Nothing in the image wants a capability. The tools read the tree, write the cache volume and talk to the network, and rootless podman maps the container's root to the invoking user, so file access is the developer's own either way. Checked rather than assumed: `sdlc versions` probes all seven binaries, `sdlc ruff-check` rewrites the tree, and trentina's `sdlc mypy` runs a full `uv sync` into the shared cache volume and type-checks 235 files -- all of it with no capabilities. tests/test_host_contract.sh asserts it over every container the launcher builds, so a future argv cannot quietly drop the flags.
Gourmand's findings are deterministic: the same tree gives the same answer, with no model and no network. Paying for them at push, where each rejection costs a six-to-eight minute cycle and a re-review of the whole diff, was paying the wrong price for an answer a three-second check at commit would have given. One of the five pushes on crunchtools/trentina#706 was exactly this. Measured in full mode, warm: 1.7s on this repo, 3.2s on trentina. Full rather than `--diff-source`, because full mode also sees duplication across files, which is most of what it is for. It runs after both ruff steps, so it judges the formatted tree rather than the one you typed, and tests/test_plan.py pins that order -- a commit tier that reviewed the unformatted source would report findings the same commit had already fixed. It stays in push as well: a commit can be made with --no-verify, and push is the tier that renders the verdicts. Semgrep deliberately does not join it. It takes 27s, and `t_semgrep` exits 2 on a repo with no `.semgrep`, so as a default it would break the commit hook across the fleet. A repo that wants it there writes `[tool.sdlc.commit] checks = ["semgrep"]`. Constitution XII's commit sentence gains Gourmand in a separate prose change; the validator inspects only that the two hooks exist, so nothing waits on it. Refs #18
… cost Two things, and the second is why the first could be judged at all. PER-CHECK DURATIONS AND A RECORD. The serial path -- which is the path every `sdlc push` took -- printed no durations whatsoever, so nothing about the slowest tier in this tool could be tuned by anyone who had not sat watching it with a stopwatch. It now prints the same per-check footer the lane path always had, and both paths write: <git-dir>/sdlc/last-<command>.json check, exit, seconds, log <git-dir>/sdlc/logs-<command>/*.log what a buffered check printed That is also what an agent needs after a rejected push: which check, how long, and where its output went, without running the tier again to find out. In the git directory, not a committed `.sdlc/`, because it is one machine's last run. Asked of `git rev-parse --git-path`, since in a linked worktree `.git` is a file. Written, never read -- a verdict comes from running the check, and a file that could stand in for one would be a way past a gate. Failing to write it never fails a gate. A LANE FOR THE REVIEW. Gatehouse is the one check that does not read the worktree: its input is the `origin/<default>...HEAD` diff, committed objects that bin/sdlc computes on the host and pipes in on stdin. So the rule that sends a tree-rewriting command down one lane does not apply to it, and it is also the only check whose time is not CPU -- one API call and then waiting, measured at 35s on a 365-line diff, spent with the CPUs idle. #18 rejected push lanes for want of durations. That was the wrong reason to wait: the argument is that Gatehouse reads git rather than the tree, and it needs no measurement. What makes it free is the budget rule. A lane whose every step is `local` takes one CPU and is LEFT OUT of the division, so the main lane keeps the whole budget it had. Dividing by the lane count would have handed half the machine to a lane that wants no CPU, which is precisely how lanes were measured as a regression on a 4-vCPU runner (15m27s to 24m38s). And the working lane keeps the terminal. run_lanes buffers every lane and prints each block when it finishes, which is right when both lanes are computing and wrong here: the buffered lane would be the whole of a push, and three minutes of silence reads like a hang. So when exactly one lane runs containers, that lane runs in the foreground through the ordinary sequential path -- same output, live -- and the others are printed whole after it. The multi-container-lane path is untouched. tests/test_host_contract.sh covers the new executor end to end: both lanes run, the buffered block lands after the streamed one, the banner names what is running beside (from the plan, since no check is named in bin/sdlc), the working lane keeps all 4 of a budget of 4 while the other takes 1, and a finding in the lane off the CPU still fails the command. Refs #18
`sdlc pytest tests/test_x.py` ran the whole suite. The dispatcher passed "$1" to run_plan and dropped every argument after it without a word, so the first thing anyone reaches for after a rejected push -- rerun just that test -- silently did the most expensive possible thing instead. Arguments after `--` are joined with the unit separator, exported as SDLC_ARGS, and appended by `t_pytest` after the config's `args` and after `-n $SDLC_JOBS`, so the caller's flag wins: `-- -n0` turns off the parallelism the gate just added, which is how you read a traceback. sdlc pytest -- tests/test_x.py::test_y -n0 Refusals rather than guesses, because a gate that quietly does something else is the thing this repo exists to delete: sdlc pytest tests/x.py exit 2, and says where the arguments go sdlc pytest -- exit 2 sdlc push -- -x exit 2; a tier is several checks an argument containing the separator exit 2 It is not a plan field. The plan is the image telling the host what to run and `plain()` guards it precisely so a repo's TOML cannot forge a line into this shell's argv; this travels the other way, by name through the environment, and lands in a Python argv list with no shell in between. The test asserts that: podman is handed `-e SDLC_ARGS` and the value appears nowhere in the argv. A tier CLEARS the variable rather than refusing it, so an `SDLC_ARGS` exported in someone's shell cannot reach a push's pytest by the back door. pytest first because that is where the cycles go. mypy, semgrep and ruff can read the same variable later. Refs #18
…vely Three things, all found by using this repo's own gates on this repo. `--help` and the README now say what order to do things in: `sdlc commit && sdlc push` before the first push, `sdlc <check> -- <args>` to reproduce one failure, and `<git-dir>/sdlc/last-push.json` for what the last run cost. Plus the two things that reliably mislead people -- that the Python running the tests is the image's and not the machine's, so a host `pytest` result says nothing about this gate; and that CI triage, not the local record, is the authority on a review finding. The path in `--help` is asked of `git rev-parse`, because in a linked worktree the git directory is elsewhere. `z`, not `Z`, in this repo's own three local hooks. bin/sdlc has carried the reason since it was written: `Z` relabels the worktree with a category pair private to one container, which takes the tree out from under every other container that mounts it. Two of those hooks did it anyway, and the result was that `podman run -v "$PWD:/mnt:ro" ...` -- the shellcheck step, as container.yml still spells it -- could not read the repo at all: "openBinaryFile: permission denied". Half an hour, on the tool whose job is to delete exactly this. The CI step gets `,z` as well, since a fresh checkout is the only reason it has not bitten there. And the local Gatehouse hook pinned 0.22.0 while the image bakes 0.23.0 -- precisely the drift this repo exists to delete, in its own config. The state directory is also made absolute at the point it is computed, so the log paths in the record can be opened from anywhere rather than only from the top of the worktree.
Fewer push cycles, and a gate you can see into.
Measured on the way through, with the 0.4.0 image built locally:
- the whole commit tier, Gourmand included, is 2.6s on this repo
- `sdlc pytest -- <nodeid> -n0` runs one test in 15s, where the same
command used to run the entire suite
- #16 reproduced and fixed end to end on trentina's own test: with the
tree carrying host-compiled bytecode, the gate FAILS
`test_library_lines_are_fingerprinted` and prints the laptop's path in
the traceback; with PYTHONPYCACHEPREFIX set and nothing else changed,
the same poisoned tree passes
- the contract test, the three unit suites and 54 host-contract checks
pass against the locally built image
The version moves in pyproject.toml, IMAGE_TAG and the Containerfile LABEL
together, which is what `Constitution / Validate constitution` and
tests/contract.sh check between them. v0.3.5 shipped with those out of step
and had no image at all; the workflow change in this release is why that
cannot happen again.
…bilities
The capability drop two commits ago broke `codeql`, and the commit that made
it said "checked rather than assumed" while having checked three of the eight
checks. It exercised `sdlc versions`, `ruff-check` and `mypy`. Not CodeQL.
Running trentina's whole `sdlc push` against the image is what found it: 355
seconds and an exit 2 where there should have been a verdict.
WARNING: Not using precompiled UseofInput.qlx: ...
java.nio.file.AccessDeniedException:
/opt/codeql/qlpacks/codeql/python-queries/1.9.0/.codeql/precompiled/*.qlx
... all 55 queries recompiled ...
Oops! A fatal internal error occurred. (StringPool.noSuchStringError)
codeql database analyze failed (exit 100)
The CodeQL bundle extracts 25 precompiled `.qlx` files at mode 0600 owned by
the uid that built it, 1001. The container runs as root, so root was reading
them purely on CAP_DAC_OVERRIDE -- and the gate now drops every capability.
Without them CodeQL recompiles the lot and then dies.
Fixed in the image, not by handing the capability back: `chown -R root:root`
and `chmod -R a+rX /opt/codeql` after extraction. `a+rX`, so execute is added
to directories and to files that already had it and to nothing else. This also
restores the precompiled queries the gate had been using all along, which is
most of that 355 seconds.
The real lesson is that a bare `podman run` still holds the capability, so a
tool broken this way works in every hand-test and fails only in the gate.
tests/contract.sh now asserts that no file under /opt needs
CAP_DAC_OVERRIDE, so the next bundle -- or the next tool -- cannot
reintroduce it quietly.
… claimed 19848ad's message said the state directory was made absolute where it is computed. It was not: that edit was in a patch that aborted partway, and only the other half of it landed. The claim was in the commit and the code was not. `git rev-parse --git-path sdlc` answers relative to the current directory -- `.git/sdlc` from the top, `../.git/sdlc` from a subdirectory -- and bin/sdlc chdirs to the toplevel before asking, so every path in `last-<command>.json` came out relative to the repo root. Which is one directory out of however many an agent might be sitting in when it reads the record. "log": ".git/sdlc/logs-push/4-gatehouse.log" before "log": "/home/.../trentina/.git/sdlc/logs-push/4-gatehouse.log" after The hint in `--help` stays relative, because that one is being read by a person looking at their own repo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14. Closes #16. Works #18.
One release for all three open issues. Every change here comes from one piece of evidence: crunchtools/trentina#706 took five pushes, four rejected, about 28 minutes of gate time, and not one of those four rejections was something the developer could have seen before pushing.
Reviewed against the source with the
ctoagent before anything was written; where it overrode an issue, the reason is in the commit.Measured, against this image built locally
sdlc pushon trentinacodeqlsdlc pushexits 0 on trentina; the contract test, four unit suites and 54 host-contract checks pass.Fixed
#16 — a laptop test run no longer decides the gate's verdict. The container shares the worktree's
__pycache__, and pytest's assertion rewriter reads its cached code back as marshalled, so a module compiled on the host kept the laptop's path inco_filenamewhile__file__was/repo/….PYTHONPYCACHEPREFIX=/tmp/pycachein both containers that mount the tree — this issue named only one.Reproduced and fixed on trentina's own
test_library_lines_are_fingerprinted: with host-compiled bytecode planted in the tree the gate fails it and prints/home/smccarty/…in the traceback; with the prefix set and nothing else changed, the same poisoned tree passes.#14 — a release builds the commit once. Both the main-branch push and the tag fired for the same sha and both pushed
:<sha>; the second replaced the first's manifest, which cost v0.3.5 its image. A tag event now builds nothing and promotes the digest main already pushed.Two things that were easy to get wrong. The skip is per step, not per job: a job-level
if:skips every job thatneeds:it unless each growsalways()logic, so the fix would have looked right and promoted nothing on its first tag. Andverify-digeststill runs on the tag path —:<sha>is pushed before it is verified, so a main run whose contract test failed leaves a pushed-but-untested digest that the tag run would otherwise have promoted. That hole was in #14 as filed.Faster
Gourmand at commit. Deterministic findings — same tree, same answer, no model, no network — so paying for them at push was the wrong price. 1.7s here, 3.2s on trentina. Semgrep stays out: 27s, and it exits 2 on a repo with no
.semgrep, which as a default would break the commit hook fleet-wide.A lane for the review. #18 rejected push lanes "for lack of durations". Wrong reason to wait: Gatehouse reads the
origin/<default>...HEADdiff the host computes, never the worktree, so the single-lane-when-rewriting rule does not apply to it — and that needs no measurement. A lane whose every step islocaltakes one CPU and is left out of the division, so the main lane keeps the whole budget; dividing by lane count is precisely how lanes were measured as a regression in 0.3.2. And the lane that runs containers keeps the terminal, because buffering it would mean watching nothing for the length of the test suite.Easier
sdlc <check> -- <args>, sosdlc pytest -- tests/x.py::test_y -n0runs one test. An argument not behind--is now an exit 2 rather than a silent full-suite run. A verdict record at<git-dir>/sdlc/last-<command>.jsonwith per-check seconds and logs — the serial path printed no durations at all, so nothing about the slowest tier here could be tuned by anyone without a stopwatch. Written, never read. Plus a recipe in--helpand the README.More secure
--cap-drop=all --security-opt=no-new-privilegeson the gate's own containers.pytest-in-imagehas been locked down since it was written; the containers that run far more repo-supplied code had none of it.That broke CodeQL, and the commit which made the change claimed verification it did not have — it exercised three of eight checks. Running trentina's whole push is what found it: the CodeQL bundle extracts 25 precompiled
.qlxfiles at mode 0600 owned by uid 1001, and root had been reading them purely onCAP_DAC_OVERRIDE. Fixed in the image rather than by handing the capability back, which also restored the precompiled queries and took CodeQL from 355s to 95s.tests/contract.shnow asserts no file under/optneeds that capability, because a barepodman runstill holds it — so a tool broken this way passes every hand-test and fails only in the gate.Also
z, notZ, in this repo's own three hooks:Zhad relabelled the worktree exclusively and taken it out from under the shellcheck step, which is the bugbin/sdlchas documented since it was written. And the local Gatehouse hook pinned 0.22.0 while the image bakes 0.23.0.Not in this release
Incremental local review (#18's 0.5.0) is out: it trades the second look at unchanged code — which on trentina#706 found two real crashes — against accuracy; it depends on two unshipped
crunchtools/gatehousechanges; and with the review now beside the rest it only saves wall time if the review is the longest lane, which nobody has measured. The durations here are how that gets decided. It would be 0.6.0; 0.5.0 is #19.--consensus-jsonis out: the reviewer's container mounts the worktree read-only, so collecting the file needs a new writable bind mount inside the git directory — a real cost against a convenience the terminal already gives.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01R7Y1jfBgAyvK9xarny6EHx