diff --git a/planning/changes/2026-07-14.04-healthy-gating-flake.md b/planning/changes/2026-07-14.04-healthy-gating-flake.md new file mode 100644 index 0000000..db0a05e --- /dev/null +++ b/planning/changes/2026-07-14.04-healthy-gating-flake.md @@ -0,0 +1,71 @@ +--- +summary: The healthy-gating scenario's postgres healthcheck probes TCP rather than the Unix socket, so the gate now opens only once the dependent's actual transport is accepting connections. +--- + +# Change: Fix the healthy-gating scenario's socket-vs-TCP race + +**Lane:** lightweight — one line in one integration scenario. + +## Goal + +`tests/integration/test_healthy_gating.py` fails intermittently in CI (twice so +far, each time passing on rerun). It is not flaky infrastructure: the scenario +gates on a healthcheck that probes a *different transport* from the one the +dependent service uses. + +- `db`'s healthcheck is `pg_isready -U postgres` — no `-h`, so it probes the + **Unix socket**. +- `app` runs `pg_isready -h 127.0.0.1 -p 5432` — **TCP**. + +The official postgres image runs `initdb`, then starts a **temporary** server to +run its init scripts. That server listens on the Unix socket *only* — and it +announces itself as ready. From its own logs: + +``` +[36] listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" <- temp server, socket ONLY +[36] database system is ready to accept connections <- `pg_isready -U postgres` returns 0 HERE +[37] shutting down <- temp server stops + PostgreSQL init process complete; ready for start up. +[1] listening on IPv4 address "0.0.0.0", port 5432 <- TCP only appears NOW +[1] database system is ready to accept connections <- for real +``` + +There is no IPv4 listener during the init phase, so `pg_isready -U postgres` +(Unix socket) reports ready while `app`'s TCP probe cannot possibly succeed — +and the temp server then shuts down. Measured at 220ms on a developer machine +(21.697 -> 21.917); wider on a slower, contended CI runner, which is why it +fails there and not locally. + +A TCP healthcheck cannot pass during that phase, because there is nothing +listening on TCP for it to pass against. That is what makes the fix +deterministic rather than a widened timeout. + +The 120-second wait budget is untouched and irrelevant — the gate is not timing +out, it is opening on the wrong signal. + +## Approach + +Probe the transport the dependent actually uses: + +```python +"test": ["CMD-SHELL", "pg_isready -U postgres -h 127.0.0.1 -p 5432"] +``` + +This is what the scenario meant to assert all along: `service_healthy` should +mean "the thing `app` depends on is ready", and `app` depends on TCP. It also +makes the scenario a stricter test of compose2pod — the pod's shared-namespace +`127.0.0.1` now has to work for the healthcheck as well as for the target. + +No change to `compose2pod` itself: the tool gated faithfully on the healthcheck +it was handed. The defect is in the scenario's compose document. + +## Files + +- `tests/integration/test_healthy_gating.py` — healthcheck probes TCP + +## Verification + +- [ ] Reproduce the window (socket ready before TCP) against real postgres. +- [ ] Apply the change. +- [ ] Run the scenario repeatedly against real podman — must pass every time. +- [ ] `just test-integration` green; `just test-ci` and `just lint-ci` clean. diff --git a/tests/integration/test_healthy_gating.py b/tests/integration/test_healthy_gating.py index 509c6ab..9ab4f97 100644 --- a/tests/integration/test_healthy_gating.py +++ b/tests/integration/test_healthy_gating.py @@ -12,7 +12,12 @@ def test_healthy_gating_reaches_postgres(run_pod: Callable[..., PodRun]) -> None "image": "postgres:16-alpine", "environment": ["POSTGRES_PASSWORD=pw"], "healthcheck": { - "test": ["CMD-SHELL", "pg_isready -U postgres"], + # Probe TCP, not the Unix socket. The postgres image runs initdb, then a + # temporary server with listen_addresses='' (socket only) for its init + # scripts, and only then restarts on TCP -- so a bare `pg_isready -U postgres` + # reports ready for ~0.5s before 127.0.0.1:5432 accepts anything, opening the + # gate before `app` (which connects over TCP) can reach it. + "test": ["CMD-SHELL", "pg_isready -U postgres -h 127.0.0.1 -p 5432"], "interval": "1s", "timeout": "5s", "retries": 30,