Skip to content

A release builds the same commit twice; the tag event should promote the image that already exists #14

Closed

Description

@fatherlinux

For Scott

What: every release of this image builds the same commit twice -- once on the merge to main, once on the tag -- and publishes whichever build finished last.
Why it matters: it already cost v0.3.5 its image. The two builds raced, the tag run lost, and there is a git tag and a GitHub release for v0.3.5 with no 0.3.5 image behind them. Any repo that had pinned it would have exited 2 with "manifest unknown" and checked nothing.
Recommendation: on a tag push, skip the build and have promote copy the :<sha> image the main-branch run already built and verified. Size S.
Needs from you: nothing.

Where it fits

Feature or program: RT#1552 (python-sdlc fleet conversion), which is why it surfaced -- 26 repos now pin this image by exact release. Related: PR #13, which stopped the race but not the double build.
Why now: the immediate bleeding is stopped. #13 keys the workflow concurrency group on github.sha instead of github.run_id (which was unique per run and therefore grouped nothing), so the two runs serialize and the second verifies the digest it just pushed. The end state is consistent. This issue is the proper fix.

For the agent

Problem: container.yml triggers on push: branches: [main] AND push: tags: ["v*"]. release.yml tags the commit that was just merged, so both events fire for the SAME sha and both run build, which pushes :<sha>. The second push replaces the first manifest. Before #13 the loser's verify-digest step failed with manifest unknown on a digest that no longer existed; v0.3.4 won that race and v0.3.5 lost it.

This contradicts the design stated in the workflow's own header: "the build pushes ONLY the commit SHA tag, the gates address the image by the DIGEST the build reported, and latest plus the semver tags move in a separate promote job that copies that tested digest without rebuilding." A tag event rebuilding the commit is the one step that does not fit.

Also worth noting, raised by Gatehouse on PR #13: a concurrency group holds at most one PENDING run, so a third entrant on the same sha cancels the one waiting. With two runs that cannot happen; removing the second build removes the question.

Files: .github/workflows/container.yml (build, verify-digest, promote, bump-consumers and their needs:/if:), .github/workflows/release.yml (what the tag push is expected to start).

Proposed fix:

  1. Gate build and verify-digest on !startsWith(github.ref, 'refs/tags/v').
  2. On a tag event, have promote resolve the digest from the registry -- skopeo inspect --format "{{.Digest}}" docker://$QUAY:$GITHUB_SHA -- rather than from needs.build.outputs.digest, and keep the existing backwards-move guard and the fixed promote-python-sdlc concurrency group as they are.
  3. Fail loudly if :<sha> is absent: that means the main-branch run never completed, and promoting anything else would publish an untested image.
  4. bump-consumers keeps needs: promote and its tag condition.

Done when: a release produces exactly one Container build; latest, :X.Y.Z, :X.Y and :<sha> all resolve to the same digest after a release; and a tag pushed for a commit whose main build failed does not publish.

Out of scope: the weekly rebuild and pull-request paths, which legitimately build.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions