Skip to content

ci(release-image): build the release image without the gha cache - #785

Merged
robinnsc merged 1 commit into
mainfrom
ci/release-image-no-build-cache
Nov 3, 2026
Merged

robinnsc merged 1 commit into
mainfrom
ci/release-image-no-build-cache

Conversation

@robinnsc

@robinnsc robinnsc commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

What

The build step in release-image.yml no longer reads or writes the GitHub Actions build cache: the cache-from: type=gha and cache-to: type=gha,mode=max lines are removed, and a comment says why. Two other comments are corrected to match:

  • The publish-candidate comment said a failed run could be finished by dispatching the workflow again. Without the cache a new dispatch builds images with different IDs, so the no-overwrite check refuses a candidate an earlier run already pushed. The comment now says to use "Re-run failed jobs" on the failed run while its image artifacts are kept (retention-days: 1), and points to sign-image.yml once they have expired.
  • The build-date comment no longer says a re-run reproduces the same image config. The commit-derived date still keeps the image label and the binary's build time the same in every run of a release.

Why

The image this step builds is the one the workflow scans, pushes and signs, so a layer restored from the Actions cache would ship in a signed release. 8a792cf removed the same cache from release-dev-image.yml, citing CodeQL's actions/cache-poisoning/poisonable-step query. release-image.yml kept it, and no other release workflow uses an Actions cache.

The cache saved little. I read the build logs of every release-image run that still has them (15 of 17: run 32167127911 ran no jobs, and the logs of 32124353804 are gone). The cargo build layer's inputs change with every release, so it was restored only when the same release was built again: the re-dispatches of v0.1.6 (run 32170242786) and v0.1.8 (run 32519595682), the re-run of v0.1.11 (run 34395238814, attempt 2, arm64) and, judging by its 76-85 s build times against 5-8 min uncached, the re-run of v0.1.6 (run 32124353804, attempt 2). In one other build (mongodb v0.1.13, amd64) the WORKDIR, apt, both COPY and first two runtime layers came from the postgres build of the same tag, because the cache scope was per architecture only. Exporting the cache took 52 s to 150 s per uncached build.

The cost is one recovery path. With the cache, a new dispatch of the same tag could rebuild identical image IDs (the v0.1.8 re-dispatch did) and finish a run that had pushed but not signed. After this change, the ways to finish such a run are "Re-run failed jobs" within a day, or sign-image.yml after that.

Testing done

  • The workflow parses as YAML, and actionlint 1.7.7 reports nothing for release-image.yml on this branch or on main (shellcheck and pyflakes integrations off).
  • grep -rn 'cache-from\|cache-to\|type=gha' .github/ matches nothing.
  • The re-run path was checked against the job definitions and past runs. The publish job downloads the image artifacts and runs docker load, and nothing rebuilds or retags them before the match check. Run 31667072227 attempt 2 re-ran only publish-candidate with attempt 1's artifacts. A loaded image's ID equals the build's config digest (run 39892688708). None of the runs with logs took the "already published with the identical artifact" branch. Run 32124353804 attempt 2 may have, but its logs are gone, so that branch is checked by reading the code.
  • The workflow itself has not been run on this branch; it needs a release tag.

Not changed here

  • The no-overwrite check inspects only the multi-arch candidate tag. If an earlier run pushed the per-arch tags and failed before creating the index, a new dispatch overwrites the per-arch tags. That gap predates this change, but rebuilt images now differ where the cache used to make them identical.
  • retention-days: 1 is unchanged. Publish has started 22 minutes to 6 hours after the build in past runs, so the re-run window can be well short of a day in practice.
  • feat(container): publish a Cassandra-backed ExtendDB image #725 (Cassandra image, draft) changes the same cache lines to a per-image scope and will conflict; it should drop them instead.
  • This does not address CodeQL alert perf(gsi): add index on (base_pk, base_sk_*) for cascade deletes #116, which flags the gate's licence step for running code from the checked-out tag.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace): no Rust change
  • Code is formatted (cargo fmt --check): no Rust change
  • Clippy is clean (cargo clippy -- -W clippy::pedantic): no Rust change
  • I have added or updated tests for new functionality: n/a
  • I have updated documentation if behavior changed (workflow comments)
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a

Breaking changes

None for users. For maintainers, a failed release run can no longer be finished by dispatching the workflow again once it has pushed the candidate (see Why).


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

The release image build read and wrote the GitHub Actions build cache
(cache-from/cache-to type=gha). The image that step produces is the
one the workflow scans, pushes and signs, so any layer restored from
the cache would ship in a signed release. 8a792cf removed the same
cache from release-dev-image.yml, citing CodeQL's
actions/cache-poisoning/poisonable-step query; release-image.yml kept
it.

The cache did little for release builds. The cargo build layer's
inputs change with every release, so it was restored only when the
same release was built again: the re-dispatches of v0.1.6 (run
32170242786) and v0.1.8 (run 32519595682), the re-run of v0.1.11
(run 34395238814, attempt 2, arm64) and, judging by its 76-85 s build
times, the re-run of v0.1.6 (run 32124353804, attempt 2, whose logs
are gone). In one other build
(mongodb v0.1.13, amd64) the WORKDIR, apt, both COPY and first two
runtime layers came from the postgres build of the same tag, because
the cache scope was per architecture only. Exporting the cache took
52 s to 150 s per uncached build.

The step now has no cache-from or cache-to, with a comment saying why.

This removes one way to recover a failed release. With the cache, a
new dispatch of the same tag rebuilt the same image IDs (the v0.1.8
re-dispatch did), so the publish-candidate check would accept a
candidate an earlier run had already pushed. Without it, a new
dispatch builds different IDs and that check refuses to overwrite the
candidate. The comment on the check said a failed run could be
finished by re-dispatch; it now says to use "Re-run failed jobs" on
the failed run while its image artifacts are kept (one day), and
points to sign-image.yml once they have expired. The comment on the
commit-derived build date no longer says a re-run reproduces the same
image config.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>

@LeeroyHannigan LeeroyHannigan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR deletes cache-from and cache-to (type=gha) from the build step of release-image.yml, as 8a792cf did for release-dev-image.yml, and corrects the two comments that described rebuilds as reproducible. I checked that nothing under .github/ still uses type=gha, that no other release workflow uses an Actions cache, that actionlint 1.7.12 reports nothing on this branch or on main, that every action pin in the file matches its version tag, and that no permissions change. The build-date comment is correct: BUILD_DATE reaches the binary as EXTENDDB_BUILD_TIME through crates/bin/build.rs. Every RUN step in both Dockerfiles follows the stage's ARG declarations, so a new release missed the cache on all of them, and an uncached rebuild has new image IDs, as the new comment says. The v0.1.8 re-dispatch (run 32519595682, amd64) restored every layer from the cache, including the runtime apt-get upgrade layer, so it reused the first dispatch's package versions. The v0.1.14 build (run 39892685957, amd64) spent 192 s on the cargo build and 134 s exporting the cache. Attempt 2 of run 31667072227 re-ran only publish-candidate and finished with attempt 1's artifacts. The PR adds no tests, and the workflow was not run.

Findings

Minor: .github/workflows/release-image.yml:437. The candidate probe treats any failure of docker buildx imagetools inspect as absence (2>/dev/null), so after a rate limit or a network error the job takes the push branch. A re-dispatch that hit the cache on every layer rebuilt the same image IDs, so that branch re-pushed identical content. Now a new dispatch builds different IDs, and this probe is the only check that stops it from replacing a candidate an earlier run pushed and signed: the Docker Hub tag would move to a new index, and promote-image would then refuse the digest the tester verified. The comment at lines 429 to 431 relies on this check. Please make the probe three-way, as the publish step of release-dev-image.yml, copy-artifact and promote-registry do: capture the error output, require a sha256:<64 hex> value on success, take the push branch only when the error matches manifest unknown|name unknown|not found, and fail on any other error. Optionally, run the same existence and image ID check on ${CANDIDATE}-amd64 and ${CANDIDATE}-arm64 before pushing them, which covers the per-arch case the body lists under "Not changed here".

Optional: .github/workflows/release-image.yml:434-436. "Once the artifacts have expired the run cannot be finished this way" holds for a failed publish-candidate job. A failed propagate job downloads no artifacts, so "Re-run failed jobs" can finish it for as long as GitHub allows re-runs (30 days), and promote-image copies the image and its signature to GHCR and ECR by digest whether or not propagate ran. Suggested text:

          # Once the artifacts have expired, a re-run of publish-candidate
          # fails at the download. sign-image.yml then signs a candidate that
          # was pushed but not signed, and promote-image copies the image and
          # its signature to the mirrors. A failed propagate job reads no
          # artifacts and can still be re-run. "Re-run all jobs" rebuilds the
          # images, which this check refuses once the candidate is pushed.

Optional: .github/workflows/release-image.yml:356. retention-days: 1 is now the deadline for finishing a failed publish from the run itself, and it counts from the upload at the end of the build, before the approval wait. Each tarball is about 44 MB (v0.1.14, amd64). A few days would cover a weekend.

Optional, as a follow-up: .github/workflows/release-image.yml:274. setup-buildx-action creates the builder from moby/buildkit:buildx-stable-1, a tag resolved at run time (BuildKit v0.33.1 in run 39892685957). That daemon produces the signed image, so it is a mutable input to the release build, as the cache was. Suggest driver-opts: image=moby/buildkit:v0.33.1@sha256:<digest> here and in release-dev-image.yml, updated with the other pins.

Not checked

  • A run of the workflow on this branch, which needs a release tag; the next release shows the uncached build time.
  • The release runbook, which is not in the repo. If it says to re-dispatch a failed run, it needs the same change.
  • The run-log figures in the body other than the runs named above.

@robinnsc
robinnsc added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 5a1d4d7 Nov 3, 2026
32 checks passed
@robinnsc
robinnsc deleted the ci/release-image-no-build-cache branch November 3, 2026 18:59
@navareamol navareamol added security v1 Needed for the 1.0 release release-blocker Must ship for its milestone's release labels Nov 3, 2026
@navareamol navareamol added this to the 1.0 milestone Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release security v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants