Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .github/workflows/container.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,23 @@
- cron: "0 6 * * 1"
workflow_dispatch:

# Keyed on the COMMIT, not the run, so the push-to-main run and the tag run of
# the same commit SERIALIZE instead of racing. `github.run_id` is unique per
# run, so the old group put every run in a group of its own and controlled
# nothing: both builds pushed the same `:<sha>` tag, the second replaced the
# first's manifest, and the loser's "Verify the pushed digest" step then failed
# with `manifest unknown` on a digest that no longer existed.
#
# That is not theoretical. It cost v0.3.5 its image: the main run won, `latest`
# moved, the tag run lost, and the `0.3.5` tag was never created -- a GitHub
# release with no image behind it, which is the one state a pinned fleet cannot
# survive. v0.3.4 hit the same race and happened to win it. Serialized, the
# second run pushes and then verifies its own digest, which is the current one.
#
# cancel-in-progress stays false off pull requests: cancelling the main run
# when its tag arrives would abandon a push half-done.
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.sha }}

Check failure on line 44 in .github/workflows/container.yml

View workflow job for this annotation

GitHub Actions / Gatehouse triage / Gatehouse triage

Check failure on line 44 in .github/workflows/container.yml

View workflow job for this annotation

GitHub Actions / Gatehouse triage / Gatehouse triage

Check failure on line 44 in .github/workflows/container.yml

View workflow job for this annotation

GitHub Actions / Gatehouse triage / Gatehouse triage

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Bug Hunter): GitHub Actions keeps at most one pending run per concurrency group; when another run enters the same group, it cancels and replaces the existing pending run, even with cancel-in-progress: false. If the main build is running and the tag build is pending, a workflow_dispatch or scheduled run for the same SHA can replace the tag build, leaving the release tag without its image.

Suggestion: Ensure required tag builds cannot be displaced by other runs in the same group—for example, consolidate the main and tag handling into one run or use a queuing design that preserves every required run.

Evidence:

group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.sha }}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug for the path this fixes, and you are right about the limit. Worth being precise about both.

You are correct that a concurrency group holds at most one PENDING run and a third entrant cancels the one waiting. The release path has exactly two: the push to main, and the tag release.yml creates from that same commit. Two runs, so one runs and one pends, and nothing is cancelled. A third run on the SAME sha would have to be a manual re-run or a schedule firing on a head that has not moved — and today that third run does not queue, it races, which is how v0.3.5 lost its image. Serialized-and-possibly-cancelled is a better failure than concurrent-and-silently-overwritten.

On the end state, which matters more: promote already has its own fixed group (promote-python-sdlc, never cancelled) and a backwards-move guard, so promotions cannot interleave. Serialized, the main run builds and promotes, then the tag run builds, verifies ITS digest, and promotes that one — so latest, :0.3.6 and :<sha> all end at the second build. Consistent, with one orphaned digest from the first build as the cost.

What this does NOT fix, and what you have really put your finger on: the same commit is built twice at all. The workflow's own header says the design is "the build pushes ONLY the commit SHA tag... latest plus the semver tags move in a separate promote job that copies that tested digest without rebuilding" — and a tag event should therefore promote the :<sha> image that already exists rather than rebuild it. That removes the second build, the orphaned digest and the pending-run question in one go. It is a restructure of build/promote wiring rather than a one-line group change, so I am not folding it into a pull request whose subject is jq; filing it as the follow-up.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: the two-run release path cannot hit the pending-run cancellation, and the end state is consistent.

Restating in the form the triage check accepts — my earlier reply on this thread opened "not a bug for the path this fixes", without the colon, so it did not count as an answer. Same content.

You are right that a concurrency group holds at most one PENDING run and a third entrant cancels the one waiting. The release path has exactly two runs: the push to main, and the tag release.yml creates from that same commit. One runs, one pends, nothing is cancelled. A third run on the same sha would need a manual re-run or a schedule firing on an unmoved head — and today that third run does not queue at all, it races, which is how v0.3.5 lost its image.

On the end state, which matters more than the queue: promote already has its own fixed group (promote-python-sdlc, never cancelled) and a backwards-move guard, so promotions cannot interleave. Serialized, the main run builds and promotes, then the tag run builds, verifies its own digest and promotes that — so latest, 0.3.6 and the sha tag all land on the second build. One orphaned digest from the first build is the cost.

What this does not fix is the real thing you found: the same commit is built twice at all. The workflow header already says a tag should promote the digest the main run built rather than rebuild it, and doing that removes the second build, the orphan and this question together. It is a restructure of the build/promote wiring, not a one-line group change, so it is #14 rather than this pull request, whose subject is jq.

cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions: {}
Expand Down
47 changes: 47 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,53 @@ All notable changes to this project are documented here, following

## [Unreleased]

## [0.3.6] - 2026-11-04

### Fixed

- **Two builds of the same commit raced and one of them lost its image.**
`container.yml`'s concurrency group was keyed on `github.run_id`, which is
unique per run -- so every run sat in a group of its own and the group
controlled nothing. A merge to main and the tag that `release.yml` then
creates are two pushes of the SAME commit, so both built it and both pushed
the same `:<sha>` tag; the second replaced the first's manifest, and the
loser's "Verify the pushed digest" step failed with `manifest unknown` on a
digest that no longer existed.

v0.3.4 won that race. **v0.3.5 lost it**: the main run moved `latest`, the tag
run failed, and the `0.3.5` tag was never created -- a GitHub release and a
git tag with no image behind them, which is the one state a fleet pinning this
image cannot survive. Nothing had pinned it yet only because the promote job
that opens the bump pull requests is in the run that failed.

The group is keyed on `github.sha` now, so the two serialize and the second
verifies the digest it just pushed. v0.3.5 is superseded by this release
rather than backfilled.

### Added

- **jq, and a stated boundary for the shell test toolbox.** Claude Code hands a
hook its event as JSON on stdin, hooks are shell scripts, and jq is the
smallest correct way to read JSON from bash. So a repo that ships such a hook
has unit tests that run the script as a subprocess -- and those failed here
with `jq: command not found`, which is not a verdict about the code. Two
repos hit it within a day of the fleet conversion (crunchtools/josui's
entrypoint tests; 4 of crunchtools/claude-agent's 103 shell-hook tests), and
both declare `Requires: jq` in their own package manifests.

The alternative was to run those tests inside each repo's own image. That is
right for a per-repo runtime dependency and wrong here: claude-agent's
reference Containerfile installs the PUBLISHED package, so building it in the
gate would test the last release rather than the pull request.

The boundary now lives as a comment above the `dnf install` where it is read:
beyond the checks, this image carries bash, coreutils, git, curl, python3 and
jq, and nothing else. A test that reaches for socat, nc, node or a repo's own
runtime is an integration test and belongs in `pytest-in-image` against that
repo's image, or in a new kind. `tests/contract.sh` now asserts jq, curl and
bash execute, so the toolbox is a tested property rather than an accident of
the base image.

## [0.3.5] - 2026-11-04

### Changed
Expand Down
20 changes: 18 additions & 2 deletions Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ LABEL description="Every CrunchTools Python SDLC check in one image: ruff, mypy,
LABEL org.opencontainers.image.source="https://github.com/crunchtools/python-sdlc"
LABEL org.opencontainers.image.description="Every CrunchTools Python SDLC check in one image."
LABEL org.opencontainers.image.licenses="AGPL-3.0-or-later"
LABEL version="0.3.5"
LABEL version="0.3.6"

# What the image carries, readable without starting anything. `sdlc versions`
# prints these and a consuming repo's CI pins them.
Expand Down Expand Up @@ -87,7 +87,23 @@ ENV SDLC_GOURMAND_VERSION=${GOURMAND_VERSION} \
# Hummingbird does not ship it. Without it `codeql database create` fails with
# "Could not find a valid Python distribution" on an image that obviously has
# one.
RUN dnf install -y git-core tar zstd which \
#
# THE SHELL TEST TOOLBOX, and its boundary. Beyond the checks themselves this
# image carries exactly: bash, coreutils, git, curl, python3 and jq. Nothing
# else. jq is here because Claude Code hands hooks their event as JSON on
# stdin, hooks are shell scripts, and jq is the smallest correct way to read
# JSON from bash -- so a repo that ships such a hook has unit tests that run
# the script as a subprocess, and two of them (crunchtools/josui,
# crunchtools/claude-agent) declare `Requires: jq` in their own packages. That
# is a shared protocol, not one repo's quirk.
#
# A test that reaches for socat, nc, node, or a repo's own runtime is an
# integration test: it runs through the `pytest-in-image` kind against the
# repo's OWN built image, or it gets a new kind. It does not get a package
# here. Without that line this list becomes a host-tools grab bag, one repo at
# a time. tests/contract.sh asserts the toolbox, so it is a tested property
# rather than an accident of the base image.
RUN dnf install -y git-core tar zstd which jq \
&& dnf clean all \
&& rm -rf /var/cache/dnf

Expand Down
2 changes: 1 addition & 1 deletion IMAGE_TAG
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.3.5
0.3.6
22 changes: 22 additions & 0 deletions gatehouse-findings.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,25 @@
this image produces, and all of those are pinned exactly -- the contract test \
asserts each reported version equals the version pinned at build time.
"""

[[findings]]
agent = "constitution"
category = "constitution"
file = "Containerfile"
evidence = "RUN dnf install -y git-core tar zstd which jq \\"

Check failure on line 39 in gatehouse-findings.toml

View workflow job for this annotation

GitHub Actions / Gatehouse triage / Gatehouse triage

Check failure on line 39 in gatehouse-findings.toml

View workflow job for this annotation

GitHub Actions / Gatehouse triage / Gatehouse triage

Check failure on line 39 in gatehouse-findings.toml

View workflow job for this annotation

GitHub Actions / Gatehouse triage / Gatehouse triage

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Ledger): This branch silences a constitution finding in Containerfile as not-a-bug: Same reasoning as the base-image finding above, and the same four unpinned neighbours: git-core, tar, zstd and which have never carried exact RPM versions on this line. The Container Image profile requires a weekly rebuild (0 6 * * 1) whose entire purpose is to pick up OS security updates; pinning an OS package to an exact version is how you build an image that stops receiving them, which is the opposite of what the cron is for. What the constitution pins to exact versions is the CHECK tools -- ruff, mypy, semgrep, pip-audit, CodeQL, Gourmand, Gatehouse -- because a verdict that changes under you is not a verdict; those are pinned by ARG and reported by sdlc versions. jq is not a check, it is part of the shell test toolbox, and its filter language is stable across every version the fleet ships. Pinning it would freeze a CVE-bearing RPM into the gate image for no gain in reproducibility that the digest-addressed release does not already provide.

Suggestion: Confirm the judgement is right, or drop the entry and fix the finding. The entry stops applying on its own once the code it quotes is gone.

Evidence:

RUN dnf install -y git-core tar zstd which jq \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: the ledger is right to stop on it, and here is the reasoning for the record.

The silenced finding asks for jq to be pinned to an exact RPM version. Three things say otherwise:

  1. Its four neighbours are unpinned too. git-core, tar, zstd and which have never carried exact versions on that line. jq is consistent with them, not an exception to a rule being followed.
  2. The Container Image profile mandates a weekly rebuild (0 6 * * 1) whose entire purpose is to pick up base-image errata. Pinning an OS package to an exact version is how an image stops receiving them — the opposite of what that cron exists for. The same reasoning is already recorded in this file for the base image itself, a few entries up.
  3. What the constitution pins to exact versions is the CHECK tools — ruff, mypy, semgrep, pip-audit, CodeQL, Gourmand, Gatehouse — because a verdict that changes underneath you is not a verdict. Those are pinned by ARG and reported by sdlc versions. jq is not a check; it is part of the shell test toolbox, and its filter language is stable across every version the fleet ships.

Reproducibility is not lost either: consumers pin this image by exact release (rev: v0.3.6), and promote copies a digest rather than rebuilding, so what a repo gets is a fixed set of bytes regardless of which jq build went into it.

The one thing that would change my mind is jq becoming load-bearing for a verdict rather than for a test fixture. It is not, and the comment above the dnf install now says where that line is.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Ledger): This branch silences a constitution finding in Containerfile as not-a-bug: Same reasoning as the base-image finding above, and the same four unpinned neighbours: git-core, tar, zstd and which have never carried exact RPM versions on this line. The Container Image profile requires a weekly rebuild (0 6 * * 1) whose entire purpose is to pick up OS security updates; pinning an OS package to an exact version is how you build an image that stops receiving them, which is the opposite of what the cron is for. What the constitution pins to exact versions is the CHECK tools -- ruff, mypy, semgrep, pip-audit, CodeQL, Gourmand, Gatehouse -- because a verdict that changes under you is not a verdict; those are pinned by ARG and reported by sdlc versions. jq is not a check, it is part of the shell test toolbox, and its filter language is stable across every version the fleet ships. Pinning it would freeze a CVE-bearing RPM into the gate image for no gain in reproducibility that the digest-addressed release does not already provide.

Suggestion: Confirm the judgement is right, or drop the entry and fix the finding. The entry stops applying on its own once the code it quotes is gone.

Evidence:

RUN dnf install -y git-core tar zstd which jq \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: the ledger is right to stop on it, and here is the reasoning for the record.

The silenced finding asks for jq to be pinned to an exact RPM version. Three things say otherwise:

  1. Its four neighbours are unpinned too. git-core, tar, zstd and which have never carried exact versions on that line. jq is consistent with them, not an exception to a rule being followed.
  2. The Container Image profile mandates a weekly rebuild (0 6 * * 1) whose entire purpose is to pick up base-image errata. Pinning an OS package to an exact version is how an image stops receiving them — the opposite of what that cron exists for. The same reasoning is already recorded in this file for the base image itself, a few entries up.
  3. What the constitution pins to exact versions is the CHECK tools — ruff, mypy, semgrep, pip-audit, CodeQL, Gourmand, Gatehouse — because a verdict that changes underneath you is not a verdict. Those are pinned by ARG and reported by sdlc versions. jq is not a check; it is part of the shell test toolbox, and its filter language is stable across every version the fleet ships.

Reproducibility is not lost either: consumers pin this image by exact release (rev: v0.3.6), and promote copies a digest rather than rebuilding, so what a repo gets is a fixed set of bytes regardless of which jq build went into it.

The one thing that would change my mind is jq becoming load-bearing for a verdict rather than for a test fixture. It is not, and the comment above the dnf install now says where that line is.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Ledger): This branch silences a constitution finding in Containerfile as not-a-bug: Same reasoning as the base-image finding above, and the same four unpinned neighbours: git-core, tar, zstd and which have never carried exact RPM versions on this line. The Container Image profile requires a weekly rebuild (0 6 * * 1) whose entire purpose is to pick up OS security updates; pinning an OS package to an exact version is how you build an image that stops receiving them, which is the opposite of what the cron is for. What the constitution pins to exact versions is the CHECK tools -- ruff, mypy, semgrep, pip-audit, CodeQL, Gourmand, Gatehouse -- because a verdict that changes under you is not a verdict; those are pinned by ARG and reported by sdlc versions. jq is not a check, it is part of the shell test toolbox, and its filter language is stable across every version the fleet ships. Pinning it would freeze a CVE-bearing RPM into the gate image for no gain in reproducibility that the digest-addressed release does not already provide.

Suggestion: Confirm the judgement is right, or drop the entry and fix the finding. The entry stops applying on its own once the code it quotes is gone.

Evidence:

RUN dnf install -y git-core tar zstd which jq \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: the ledger is right to stop on it, and here is the reasoning for the record.

The silenced finding asks for jq to be pinned to an exact RPM version. Three things say otherwise:

  1. Its four neighbours are unpinned too. git-core, tar, zstd and which have never carried exact versions on that line. jq is consistent with them, not an exception to a rule being followed.
  2. The Container Image profile mandates a weekly rebuild (0 6 * * 1) whose entire purpose is to pick up base-image errata. Pinning an OS package to an exact version is how an image stops receiving them — the opposite of what that cron exists for. The same reasoning is already recorded in this file for the base image itself, a few entries up.
  3. What the constitution pins to exact versions is the CHECK tools — ruff, mypy, semgrep, pip-audit, CodeQL, Gourmand, Gatehouse — because a verdict that changes underneath you is not a verdict. Those are pinned by ARG and reported by sdlc versions. jq is not a check; it is part of the shell test toolbox, and its filter language is stable across every version the fleet ships.

Reproducibility is not lost either: consumers pin this image by exact release (rev: v0.3.6), and promote copies a digest rather than rebuilding, so what a repo gets is a fixed set of bytes regardless of which jq build went into it.

The one thing that would change my mind is jq becoming load-bearing for a verdict rather than for a test fixture. It is not, and the comment above the dnf install now says where that line is.

status = "not-a-bug"
reason = """
Same reasoning as the base-image finding above, and the same four unpinned \
neighbours: git-core, tar, zstd and which have never carried exact RPM \
versions on this line. The Container Image profile requires a weekly rebuild \
(`0 6 * * 1`) whose entire purpose is to pick up OS security updates; pinning \
an OS package to an exact version is how you build an image that stops \
receiving them, which is the opposite of what the cron is for. What the \
constitution pins to exact versions is the CHECK tools -- ruff, mypy, semgrep, \
pip-audit, CodeQL, Gourmand, Gatehouse -- because a verdict that changes under \
you is not a verdict; those are pinned by ARG and reported by `sdlc versions`. \
jq is not a check, it is part of the shell test toolbox, and its filter \
language is stable across every version the fleet ships. Pinning it would \
freeze a CVE-bearing RPM into the gate image for no gain in reproducibility \
that the digest-addressed release does not already provide.
"""
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

[project]
name = "python-sdlc"
version = "0.3.5"
version = "0.3.6"
description = "One container image and one script for every CrunchTools Python SDLC check"
requires-python = ">=3.14"
license = "AGPL-3.0-or-later"
Expand Down
5 changes: 4 additions & 1 deletion tests/contract.sh
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@ for probe in \
"gourmand:gourmand --version" \
"codeql:codeql version --format=terse" \
"uv:uv --version" \
"git:git --version"
"git:git --version" \
"jq:jq --version" \
"curl:curl --version" \
"bash:bash --version"
do
name=${probe%%:*}
cmd=${probe#*:}
Expand Down
Loading