Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,22 @@ All notable changes to this project are documented here, following

## [Unreleased]

## [0.5.1] - 2026-11-06

### Fixed

- **The constitution gate no longer depends on a warm cache.** `validate`
clones crunchtools/constitution from inside the container, which on a gateway
seat goes through a CONNECT proxy that refuses github.com -- the route to
GitHub there is a git `insteadOf` in the host's git config, which the
container never reads. A seat whose cache volume already held the pinned tag
passed; an empty volume, or a repo moving to a tag nobody had fetched, was a
gate that could not run. `bin/sdlc` now makes that clone with the host's git,
once per tag under `~/.cache/python-sdlc/constitution/`, and mounts it
read-only where `validate` already looks. The container still gets no git
config or credentials. If the host clone fails, nothing changes: `validate`
tries its own and reports a 2 as before.

## [0.5.0] - 2026-11-05

### Changed
Expand Down
2 changes: 1 addition & 1 deletion Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ LABEL description="Every CrunchTools Python SDLC check in one image: ruff, mypy,
LABEL org.opencontainers.image.source="https://github.com/crunchtools/python-sdlc"
LABEL org.opencontainers.image.description="Every CrunchTools Python SDLC check in one image."
LABEL org.opencontainers.image.licenses="AGPL-3.0-or-later"
LABEL version="0.5.0"
LABEL version="0.5.1"

# What the image carries, readable without starting anything. `sdlc versions`
# prints these and a consuming repo's CI pins them.
Expand Down
2 changes: 1 addition & 1 deletion IMAGE_TAG
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.5.0
0.5.1
43 changes: 43 additions & 0 deletions bin/sdlc
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,48 @@ need_volume() {
podman volume exists "$1" 2>/dev/null || die "cannot create the podman volume $1"
}

# ------------------------------------------------- the constitution, by the host
#
# `validate` clones crunchtools/constitution at the tag the repo pins, into the
# cache volume. From inside the container that clone takes whatever route the
# network gives a container, and on a gateway seat that is a CONNECT proxy that
# refuses github.com: the route to GitHub there is a git `insteadOf` in the
# HOST's git config, which the container never reads. A warm cache volume hid
# it; an empty one, or a repo moving to a tag nobody has fetched, was a gate
# that could not run (claude-agent#9, developer3, 2026-11-06).
#
# So the host's git makes the clone, once per tag, and the checkout is mounted
# read-only at the path `validate` already looks in. The container still gets
# no git config and no credentials -- what crosses is a public repo's files.
# Done once here, before any lane forks. Two sdlc runs -- two repos on one tag
# -- can still meet, so each clones into its own directory and publishes it
# with `ln -sn`, which only one of them can win; the loser removes only its own
# copy and uses the winner's. Every failure is silent on purpose: without the
# mount `validate` tries its own clone and reports a 2 exactly as it did before.
CONSTITUTION_MOUNT=
host_constitution() {
_manifest=$REPO/.specify/memory/constitution.md
[ -f "$_manifest" ] || return 0
_tag=$(awk '/^>[ \t]*\*\*Inherits:\*\*/ {
if (match($0, /v[0-9]+\.[0-9]+\.[0-9]+/)) print substr($0, RSTART, RLENGTH)
exit
}' "$_manifest")
[ -n "$_tag" ] || return 0
_root=${XDG_CACHE_HOME:-$HOME/.cache}/python-sdlc/constitution
_dir=$_root/$_tag
if [ ! -f "$_dir/constitution.md" ]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Bug Hunter): Concurrent sdlc invocations can delete or interfere with each other's in-progress clone. Both can observe the missing file, then one invocation's rm -rf can remove the directory while the other is cloning; a failed clone can likewise remove the other invocation's checkout. On seats where the container cannot reach GitHub, this can make the constitution gate fail intermittently.

Suggestion: Serialize cache population with a per-tag lock, or clone into a unique temporary directory and atomically publish it without deleting a directory another process may be populating.

Evidence:

if [ ! -f "$_dir/constitution.md" ]; then
  rm -rf "$_dir"
  mkdir -p "$_dir" 2>/dev/null || return 0
  git clone --quiet --depth 1 --branch "$_tag" \
      https://github.com/crunchtools/constitution "$_dir" >/dev/null 2>&1 \
    || { rm -rf "$_dir"; return 0; }
fi

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 0e58fd4

Each run now clones into its own mktemp -d directory and publishes it with ln -sn, so only one run can win. The loser deletes only its own copy and uses the winner's. Nothing removes a shared path anymore. I tested three runs started at once on an empty cache: all three passed and left one checkout and one link.

mkdir -p "$_root" 2>/dev/null || return 0
_own=$(mktemp -d "$_root/.$_tag.XXXXXX" 2>/dev/null) || return 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Performance Check): Concurrent runs that need the same uncached tag each create a temporary directory and clone the full shallow repository before racing to publish it. The losing run then deletes its clone, so concurrent runs cause redundant network and disk work.

Suggestion: Serialize cache population per tag, then recheck whether the cache exists before cloning. Alternatively, use a mechanism that lets later runs wait for and reuse the in-progress clone.

Evidence:

_own=$(mktemp -d "$_root/.$_tag.XXXXXX" 2>/dev/null) || return 0
git clone --quiet --depth 1 --branch "$_tag" \
    https://github.com/crunchtools/constitution "$_own" >/dev/null 2>&1 \
  || { rm -rf "$_own"; return 0; }
ln -sn "${_own##*/}" "$_dir" 2>/dev/null || rm -rf "$_own"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a bug: the duplicate clone only happens when two runs start on a cold cache for a tag nobody has fetched, which is about once per seat per constitution release. It costs one --depth 1 clone of a small repo, about a second, and the losing copy is deleted at once. A lock to avoid that would mean a waiter, a timeout and stale-lock handling in a POSIX-only launcher. That's more failure modes than the second it saves, so I'm keeping the race lock-free and correct instead.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Test Coverage): The new concurrent cache-publication logic has no corresponding test coverage in the changes shown. Its behavior depends on two runs racing to publish the symlink, including the losing run cleaning up only its own clone and then using the winner's checkout.

Suggestion: Add a test that runs two cache population attempts for the same tag concurrently and verifies that one checkout is published, the loser does not remove the winner's data, and both runs can use the published constitution. Also cover clone failure cleanup.

Evidence:

_own=$(mktemp -d "$_root/.$_tag.XXXXXX" 2>/dev/null) || return 0
git clone --quiet --depth 1 --branch "$_tag" \
    https://github.com/crunchtools/constitution "$_own" >/dev/null 2>&1 \
  || { rm -rf "$_own"; return 0; }
ln -sn "${_own##*/}" "$_dir" 2>/dev/null || rm -rf "$_own"
[ -f "$_dir/constitution.md" ] || return 0

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in ccaf13b

tests/test_host_contract.sh now starts three bin/sdlc runs at once against an offline git whose clone is slow. It asserts one checkout, one link to it and no stray copies. It passes with ln -sn, and it fails against the earlier ln -s bug, which left 3 directories and 2 nested links.

git clone --quiet --depth 1 --branch "$_tag" \
https://github.com/crunchtools/constitution "$_own" >/dev/null 2>&1 \
|| { rm -rf "$_own"; return 0; }
ln -sn "${_own##*/}" "$_dir" 2>/dev/null || rm -rf "$_own"
[ -f "$_dir/constitution.md" ] || return 0
fi
CONSTITUTION_MOUNT="$_dir:/cache/constitution/$_tag:ro,z"
}
host_constitution

# ------------------------------------------------------------------ podman run
#
# `z`, not `Z`: more than one of these can mount the same tree at once, and `Z`
Expand Down Expand Up @@ -221,6 +263,7 @@ run_in_image() {
--cpus "$GATE_CPUS" --memory "$GATE_MEMORY" --cpu-shares 256 \
-v "$REPO:/repo:$_mode,z" -w /repo \
-v "$CACHE_VOLUME:/cache" \
${CONSTITUTION_MOUNT:+-v "$CONSTITUTION_MOUNT"} \
--tmpfs /tmp:rw,exec,size=4g,mode=1777 \
-e HOME=/tmp \
-e PYTHONPYCACHEPREFIX=/tmp/pycache \
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

[project]
name = "python-sdlc"
version = "0.5.0"
version = "0.5.1"
description = "One container image and one script for every CrunchTools Python SDLC check"
requires-python = ">=3.14"
license = "AGPL-3.0-or-later"
Expand Down
51 changes: 51 additions & 0 deletions tests/test_host_contract.sh
Original file line number Diff line number Diff line change
Expand Up @@ -607,6 +607,57 @@ else
bad "$locked of ${runs:-0} gate containers dropped capabilities"
fi

# The constitution is cloned by the HOST's git and published with `ln -sn`, so
# runs that meet on a cold cache never delete each other's clone. Three run at
# once against a git whose clone is slow and offline; what must be left is one
# checkout, one link to it, and no loser's copy.
CLONESHIM=$(mktemp -d)
CLONECACHE=$(mktemp -d)
for prog in $POSIX_UTILS podman pre-commit gh; do
[ -e "$SHIM/$prog" ] && ln -s "$(readlink -f "$SHIM/$prog")" "$CLONESHIM/$prog"
done
REALGIT=$(command -v git)
cat > "$CLONESHIM/git" <<EOF
#!/bin/sh
case "\$1" in
clone) for dest; do :; done; sleep 1; : > "\$dest/constitution.md" ;;
*) exec "$REALGIT" "\$@" ;;
esac
EOF
chmod +x "$CLONESHIM/git"
# The check's own verdict depends on the checkout (CI has no pre-push hook), so
# each concurrent run must match one that had a cache to itself.
_solo=$(mktemp -d)
PATH="$CLONESHIM" XDG_CACHE_HOME="$_solo" sh bin/sdlc hooks-installed >/dev/null 2>&1 \
&& _want=0 || _want=$?
rm -rf "$_solo"
_pids=
for _ in 1 2 3; do

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM (Test Coverage): The concurrency test verifies the final cache layout but never verifies that all three hooks-installed runs succeeded. A run that fails during a clone race can be silently ignored as long as another run leaves the expected checkout and link, so the test can pass despite broken behavior for concurrent callers.

Suggestion: Capture each background process ID and check the result of each corresponding wait; fail the test if any invocation exits nonzero.

Evidence:

for _ in 1 2 3; do
  PATH="$CLONESHIM" XDG_CACHE_HOME="$CLONECACHE" sh bin/sdlc hooks-installed >/dev/null 2>&1 &
done
wait

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in b6fc416

Each background run is now waited on by its PID, and its exit status is compared with a baseline run that had a cache to itself. Comparing with a baseline rather than 0 is deliberate: hooks-installed legitimately fails on a checkout with no pre-push hook, as in CI. A run whose exit differs from the baseline fails the test.

PATH="$CLONESHIM" XDG_CACHE_HOME="$CLONECACHE" sh bin/sdlc hooks-installed >/dev/null 2>&1 &
_pids="$_pids $!"
done
_odd=0
for _pid in $_pids; do
wait "$_pid" && _got=0 || _got=$?
[ "$_got" = "$_want" ] || _odd=$((_odd + 1))
done
if [ "$_odd" = 0 ]; then
ok "three runs racing one cold cache each exit $_want, as a run alone does"
else
bad "$_odd of 3 runs racing one cold cache did not exit $_want"
fi
_root="$CLONECACHE/python-sdlc/constitution"
_links=$(find "$_root" -mindepth 1 -maxdepth 1 -type l | wc -l | tr -d ' ')
_dirs=$(find "$_root" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')
_nested=$(find "$_root" -mindepth 2 -name '.v*' | wc -l | tr -d ' ')
if [ "$_links" = 1 ] && [ "$_dirs" = 1 ] && [ "$_nested" = 0 ] \
&& [ -f "$(find "$_root" -mindepth 1 -maxdepth 1 -type l)/constitution.md" ]; then
ok "three runs on a cold cache leave one constitution checkout and one link"
else
bad "concurrent constitution clones left $_links link(s), $_dirs dir(s), $_nested nested"
fi
rm -rf "$CLONESHIM" "$CLONECACHE"

printf '\n'
if [ "$FAILED" -eq 0 ]; then
printf 'host contract PASSED\n'; exit 0
Expand Down
Loading