Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 48 additions & 20 deletions .github/workflows/container.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Build, test and publish quay.io/crunchtools/python-sdlc.
# Build, test and publish quay.io/crunchtools/python-sdlc, and the same digest
# as quay.io/crunchtools/sdlc, the fleet-wide name it is becoming.
#
# The shape is the Container Image profile's, and it is the same shape trentina
# arrived at the hard way in #689: the build pushes ONLY the commit SHA tag, the
Expand Down Expand Up @@ -54,6 +55,11 @@ permissions: {}

env:
QUAY: quay.io/crunchtools/python-sdlc
# The fleet-wide name for the same image: one digest, two repositories, the
# same tags on both, until every consumer has moved and python-sdlc retires.
# It is never built or pushed separately -- it is a COPY of the digest pushed
# above, so the two names cannot disagree about what they hold.
SDLC: quay.io/crunchtools/sdlc

jobs:
build:
Expand All @@ -72,7 +78,7 @@ jobs:
digest: ${{ steps.push.outputs.digest || steps.resolve.outputs.digest }}
ref: ${{ steps.push.outputs.ref || steps.resolve.outputs.ref }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

Expand All @@ -81,9 +87,11 @@ jobs:
run: |
set -eu
podman run --rm -v "$PWD:/mnt:ro,z" -w /mnt \
docker.io/koalaman/shellcheck:v0.11.0 bin/sdlc tests/contract.sh \
tests/test_host_contract.sh scripts/release-version.sh
docker.io/koalaman/shellcheck:v0.11.0@sha256:61862eba1fcf09a484ebcc6feea46f1782532571a34ed51fedf90dd25f925a8d bin/sdlc tests/contract.sh \
tests/test_host_contract.sh tests/test_container_stack.sh scripts/release-version.sh \
scripts/revision-of.sh tests/test_revision_of.sh
tests/test_host_contract.sh
tests/test_revision_of.sh
tests/test_release_version.py

- name: Build
Expand Down Expand Up @@ -124,6 +132,12 @@ jobs:
"localhost/python-sdlc:$GITHUB_SHA" "$t"
done

# The container stack through the real launcher and a real podman: the
# image mount grype scans is something only podman can prove.
- name: Container stack end to end
if: env.IS_TAG != 'true'
run: tests/test_container_stack.sh "localhost/python-sdlc:$GITHUB_SHA"

- name: Log in to Quay
if: github.event_name != 'pull_request'
env:
Expand All @@ -139,9 +153,15 @@ jobs:
podman push "localhost/python-sdlc:$GITHUB_SHA" "$QUAY:$GITHUB_SHA"
digest=$(skopeo inspect --format '{{.Digest}}' "docker://$QUAY:$GITHUB_SHA")
[ -n "$digest" ] || { echo "the registry reported no digest" >&2; exit 1; }
# The fleet name gets the SAME manifest, copied rather than pushed a
# second time: a second podman push recompresses the layers and can
# produce a different digest, and then the two names hold two images.
skopeo copy --all --preserve-digests "docker://$QUAY@$digest" "docker://$SDLC:$GITHUB_SHA"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Test Coverage): The new registry-copy path publishes the built digest under a second repository and verifies the read-back, but the changed tests do not exercise this external registry interaction. The separate promotion path also now copies tags to both repositories, with no test of the two-repository behavior or its failure cases.

Suggestion: Add a workflow-level test using a stubbed skopeo (and git for the ancestry guard) to verify the copy and digest check for both repositories, including copy failure, digest mismatch, and a guard refusal when either repository is ahead.

Evidence:

skopeo copy --all --preserve-digests "docker://$QUAY@$digest" "docker://$SDLC:$GITHUB_SHA"
got=$(skopeo inspect --format '{{.Digest}}' "docker://$SDLC:$GITHUB_SHA")
[ "$got" = "$digest" ] || { echo "$SDLC:$GITHUB_SHA is $got, not $digest" >&2; exit 1; }

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a bug: the copy step checks itself. It reads the digest back from quay.io/crunchtools/sdlc and fails the job unless that digest equals the one it just tested. That is the same check the existing push and promote steps rely on, and none of these registry steps can run offline from a pull request, since PR runs have no Quay credentials by design.

got=$(skopeo inspect --format '{{.Digest}}' "docker://$SDLC:$GITHUB_SHA")
[ "$got" = "$digest" ] || { echo "$SDLC:$GITHUB_SHA is $got, not $digest" >&2; exit 1; }
echo "digest=$digest" >> "$GITHUB_OUTPUT"
echo "ref=$QUAY@$digest" >> "$GITHUB_OUTPUT"
echo "pushed $QUAY@$digest"
echo "pushed $QUAY@$digest and $SDLC@$digest"

# The tag path, in place of everything above. Nothing is built and nothing
# is pushed: this reads the digest the main-branch run of this same commit
Expand Down Expand Up @@ -185,7 +205,7 @@ jobs:
permissions:
contents: read
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Contract test against the pushed digest
Expand All @@ -211,7 +231,7 @@ jobs:
steps:
# Full history: the backwards-move guard below asks git whether the
# currently published commit already contains this one.
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
fetch-depth: 0
Expand Down Expand Up @@ -262,22 +282,30 @@ jobs:
# RT#1550): --preserve-digests refuses a copy that would change the
# manifest, and the read-back fails the job if the tag resolves to
# anything but the digest that was tested.
#
# Every tag moves on BOTH names, python-sdlc and sdlc, to the same
# digest, and the guard reads both: either one ahead of this commit
# refuses the whole promotion rather than leaving them split.
want=${REF#*@}
promote() {
skopeo copy --all --preserve-digests "docker://$REF" "docker://$QUAY:$1"
got=$(skopeo inspect --format '{{.Digest}}' "docker://$QUAY:$1")
[ "$got" = "$want" ] || { echo ":$1 resolves to $got, not the tested $want" >&2; exit 1; }
for repo in "$QUAY" "$SDLC"; do

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Bug Hunter): Promotion updates the two repositories sequentially. If copying or verifying the second repository fails after the first succeeds, the workflow exits with the tags pointing at different digests, contrary to the stated same-digest promotion behavior.

Suggestion: Handle partial promotion explicitly: on failure, retry or reconcile both repositories to the tested digest, and report the partial state so it cannot be mistaken for a completed promotion.

Evidence:

for repo in "$QUAY" "$SDLC"; do
  skopeo copy --all --preserve-digests "docker://$REF" "docker://$repo:$1"
  got=$(skopeo inspect --format '{{.Digest}}' "docker://$repo:$1")
  [ "$got" = "$want" ] || { echo "$repo:$1 resolves to $got, not the tested $want" >&2; exit 1; }
done

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a bug: no registry can move tags on two repositories atomically. What matters is that a split is loud and self-healing, and it is. Any copy or read-back that fails exits non-zero, so the job goes red. A rerun converges because skopeo copy --preserve-digests is idempotent and the backwards-move guard lets a tag that already holds this same commit through (current == GITHUB_SHA). Copying both names first and tagging second would not shrink the window, since tagging is itself the copy.

skopeo copy --all --preserve-digests "docker://$REF" "docker://$repo:$1"
got=$(skopeo inspect --format '{{.Digest}}' "docker://$repo:$1")
[ "$got" = "$want" ] || { echo "$repo:$1 resolves to $got, not the tested $want" >&2; exit 1; }
done
}
if [ -n "$floating" ]; then
current=$(skopeo inspect --format \
'{{ index .Labels "org.opencontainers.image.revision" }}' \
"docker://$QUAY:$floating" 2>/dev/null || true)
if [ -n "${current:-}" ] && [ "$current" != "$GITHUB_SHA" ]; then
if git merge-base --is-ancestor "$GITHUB_SHA" "$current" 2>/dev/null; then
echo "refusing: :$floating is $current, which already contains $GITHUB_SHA" >&2
exit 1
for repo in "$QUAY" "$SDLC"; do
# Fails closed: only an absent tag reads as empty
# (scripts/revision-of.sh, tests/test_revision_of.sh).
current=$(scripts/revision-of.sh "$repo:$floating")
if [ -n "${current:-}" ] && [ "$current" != "$GITHUB_SHA" ]; then
if git merge-base --is-ancestor "$GITHUB_SHA" "$current" 2>/dev/null; then
echo "refusing: $repo:$floating is $current, which already contains $GITHUB_SHA" >&2
exit 1
fi
fi
fi
done
promote "$floating"
fi
case "$GITHUB_REF" in
Expand Down Expand Up @@ -311,13 +339,13 @@ jobs:
group: bump-consumers-python-sdlc
cancel-in-progress: false
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Mint CI token
id: ci-token
uses: actions/create-github-app-token@v3
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
client-id: ${{ vars.CRUNCHTOOLS_CI_CLIENT_ID }}
private-key: ${{ secrets.CRUNCHTOOLS_CI_APP_KEY }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
permissions:
contents: read
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Is there an untagged version?
id: version
Expand All @@ -41,7 +41,7 @@ jobs:
- name: Mint CI token
if: steps.version.outputs.tag != ''
id: ci-token
uses: actions/create-github-app-token@v3
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
client-id: ${{ vars.CRUNCHTOOLS_CI_CLIENT_ID }}
private-key: ${{ secrets.CRUNCHTOOLS_CI_APP_KEY }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/update-tools.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,13 @@ jobs:
# merge itself.
- name: Mint CI token
id: ci-token
uses: actions/create-github-app-token@v3
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
client-id: ${{ vars.CRUNCHTOOLS_CI_CLIENT_ID }}
private-key: ${{ secrets.CRUNCHTOOLS_CI_APP_KEY }}
owner: crunchtools

- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
token: ${{ steps.ci-token.outputs.token }}

Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ repos:
hooks:
- id: shellcheck
name: shellcheck (bin/sdlc, the tests and scripts)
entry: podman --events-backend=none run --rm --log-driver=none -v .:/mnt:ro,z -w /mnt docker.io/koalaman/shellcheck:v0.11.0 bin/sdlc tests/contract.sh tests/test_host_contract.sh scripts/release-version.sh
entry: podman --events-backend=none run --rm --log-driver=none -v .:/mnt:ro,z -w /mnt docker.io/koalaman/shellcheck:v0.11.0@sha256:61862eba1fcf09a484ebcc6feea46f1782532571a34ed51fedf90dd25f925a8d bin/sdlc tests/contract.sh tests/test_host_contract.sh tests/test_container_stack.sh scripts/release-version.sh scripts/revision-of.sh tests/test_revision_of.sh
language: system
pass_filenames: false
always_run: true
Expand Down
57 changes: 57 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,63 @@ All notable changes to this project are documented here, following

## [Unreleased]

### Added

- **Stacks.** One image for every kind of repo: the planner detects which
stacks a repo has from its files -- `pyproject.toml` is python, a root
`Containerfile` or `Dockerfile` is container, `package.json` is node, a
`bootc` base image is bootc, nothing is docs -- and runs each stack's checks
plus the ones every repo gets (Gourmand, Gatehouse, the constitution
validator, the hook check). `[tool.sdlc] stacks` adds a stack and
`skip-stacks` drops one; neither can name a command. `sdlc stacks` shows what
was detected and what runs. node, bootc and docs are named but not yet
checked, so a repo that has one is refused (exit 2) until it skips it, rather
than going green on checks that never ran.
- **A Python repo that names no stacks runs exactly what it ran before**, with
or without a Containerfile. Proved by planning every public consumer's
`commit`, `push`, `ci` and `tiers` with v0.7.4 and with this image and
diffing the output.
- **The container stack**: `containerfile-lint` (hadolint 2.14.0, in push and
ci) and, in ci only, `container-build`, `container-smoke` (the built image
starts detached with no network and no capabilities, and podman's report of
its state decides: still running after `seconds`, default 10, or exited 0,
passes; an entrypoint that cannot start fails) and `container-scan` (grype 0.121.0 over the
built image's filesystem, failing on a fixable Critical by default; the
repo's `.grype.yaml` applies).
- `sdlc.toml`: the same `[tool.sdlc]` tables, for a repo with no
`pyproject.toml`. Refused beside one.
- **Main before a release, as `:main`.** Every push to main publishes its
tested image as `quay.io/crunchtools/sdlc:main`: the digest the contract
test ran against, read back after the copy, never moved onto an older
commit. No pull request run touches a registry. On a developer's machine,
`SDLC_IMAGE=main` (or any tag or image reference) or
`[tool.sdlc] image = "<tag>"` runs it; CI and `sdlc ci` refuse the repo key
(exit 2). Every run on anything but the pinned release prints which image
it is and what chose it, and a registry tag is pulled when newer. With
neither set, nothing changes.
- The image is also published as `quay.io/crunchtools/sdlc`: the same digest
copied, never rebuilt, with the same tags as `python-sdlc`.

### Security

- **A repo's config cannot inject podman options.** Every image and tag a
check uses -- `container-build`'s tag, smoke's and scan's image, and the
`image-build` and `pytest-in-image` kinds (the last two since they were
added) -- must be an image reference: nothing starting with `-`, no comma,
no whitespace. Otherwise the planner exits 2. The smoke and scan runs put
the image after `--`, and `bin/sdlc` checks that word again on the host. A
repo path that begins with `-` is refused the same way.
- The promote guard fails closed: it treats only "manifest unknown" as an
absent tag, and any other failure to read a tag's revision stops the job.

### Fixed

- `bin/sdlc`'s `run_host` kept its exit status in `_rc`, which `run_serial`
also uses, so a host step could leak its own status. It has its own name now.
- The hadolint, grype and shellcheck images are pinned by digest, and every
`actions/*` step in `container.yml`, `release.yml` and `update-tools.yml`
by commit SHA.

## [0.7.5] - 2026-11-17

### Changed
Expand Down
27 changes: 24 additions & 3 deletions Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,27 @@
ARG GOURMAND_VERSION=1.0.0
ARG GATEHOUSE_VERSION=0.29.1
ARG UV_VERSION=0.13.7
ARG HADOLINT_VERSION=2.14.0
ARG GRYPE_VERSION=0.121.0

FROM quay.io/crunchtools/gourmand:${GOURMAND_VERSION} AS gourmand
FROM quay.io/crunchtools/gatehouse:${GATEHOUSE_VERSION} AS gatehouse
# Distroless, binary only -- the documented way to get a pinned uv.
FROM ghcr.io/astral-sh/uv:${UV_VERSION} AS uv
# The container stack's two tools, each a static binary in a scratch image.
# Pinned by digest as well as tag: the digest is what is fetched, the tag says
# which release it is, and the contract test asserts the two agree.
FROM ghcr.io/hadolint/hadolint:v${HADOLINT_VERSION}@sha256:27086352fd5e1907ea2b934eb1023f217c5ae087992eb59fde121dce9c9ff21e AS hadolint
FROM docker.io/anchore/grype:v${GRYPE_VERSION}@sha256:621d56d1d68106b787022c81546f7e0a36202a25d1239f0c20195fab8c16f7ea AS grype

FROM quay.io/hummingbird/python:3.14-builder

# Repeated after FROM: an ARG before the first FROM is out of scope here.
ARG GOURMAND_VERSION=1.0.0
ARG GATEHOUSE_VERSION=0.29.1
ARG UV_VERSION=0.13.7
ARG HADOLINT_VERSION=2.14.0
ARG GRYPE_VERSION=0.121.0
ARG RUFF_VERSION=0.17.1
ARG SEMGREP_VERSION=1.177.0
ARG PIP_AUDIT_VERSION=2.9.0
Expand All @@ -40,9 +49,9 @@ ARG CODEQL_BUNDLE=codeql-bundle-v2.27.4
USER root

LABEL maintainer="fatherlinux <scott.mccarty@crunchtools.com>"
LABEL description="Every CrunchTools Python SDLC check in one image: ruff, mypy, pytest, Gourmand, Gatehouse, Semgrep, CodeQL, pip-audit and the constitution validator."
LABEL description="Every CrunchTools SDLC check in one image: ruff, mypy, pytest, Gourmand, Gatehouse, Semgrep, CodeQL, pip-audit, hadolint, grype and the constitution validator."
LABEL org.opencontainers.image.source="https://github.com/crunchtools/python-sdlc"
LABEL org.opencontainers.image.description="Every CrunchTools Python SDLC check in one image."
LABEL org.opencontainers.image.description="Every CrunchTools SDLC check in one image, for Python and container repos."
LABEL org.opencontainers.image.licenses="AGPL-3.0-or-later"
LABEL version="0.7.5"

Expand All @@ -55,6 +64,8 @@ LABEL com.crunchtools.sdlc.semgrep="${SEMGREP_VERSION}"
LABEL com.crunchtools.sdlc.pip-audit="${PIP_AUDIT_VERSION}"
LABEL com.crunchtools.sdlc.codeql="${CODEQL_BUNDLE}"
LABEL com.crunchtools.sdlc.uv="${UV_VERSION}"
LABEL com.crunchtools.sdlc.hadolint="${HADOLINT_VERSION}"
LABEL com.crunchtools.sdlc.grype="${GRYPE_VERSION}"

# The commit and build time. NOT optional, and the reason is a trap trentina hit
# in #689: its promote job refuses a release unless it can read
Expand All @@ -77,7 +88,9 @@ ENV SDLC_GOURMAND_VERSION=${GOURMAND_VERSION} \
SDLC_SEMGREP_VERSION=${SEMGREP_VERSION} \
SDLC_PIP_AUDIT_VERSION=${PIP_AUDIT_VERSION} \
SDLC_CODEQL_BUNDLE=${CODEQL_BUNDLE} \
SDLC_UV_VERSION=${UV_VERSION}
SDLC_UV_VERSION=${UV_VERSION} \
SDLC_HADOLINT_VERSION=${HADOLINT_VERSION} \
SDLC_GRYPE_VERSION=${GRYPE_VERSION}

# git-core, not git: no perl, no tk, and nothing here runs `git send-email`.
# tar and zstd are for the CodeQL bundle below.
Expand Down Expand Up @@ -133,6 +146,14 @@ RUN uv venv /opt/constitution-venv \
# so it is portable out of its image. Verified before relying on it.
COPY --from=gourmand /usr/local/sbin/gourmand /usr/local/bin/gourmand

# The container stack. Both are static Go/Haskell binaries with no runtime, so
# they copy out of their scratch images as Gourmand does. hadolint lints a
# Containerfile; grype scans a built image's filesystem, which the launcher
# mounts read-only with `podman run --mount type=image`, so neither needs a
# container engine in here.
COPY --from=hadolint /bin/hadolint /usr/local/bin/hadolint
COPY --from=grype /grype /usr/local/bin/grype

# Gatehouse is a Python app in a self-contained venv. Copying it works ONLY
# because that image and this one are both Hummingbird with python3.14 at the
# same path -- its pyvenv.cfg records `home = /usr/sbin`. A different base would
Expand Down
Loading
Loading