Repository navigation
ci(release-image): build the release image without the gha cache - #785
Conversation
The release image build read and wrote the GitHub Actions build cache (cache-from/cache-to type=gha). The image that step produces is the one the workflow scans, pushes and signs, so any layer restored from the cache would ship in a signed release. 8a792cf removed the same cache from release-dev-image.yml, citing CodeQL's actions/cache-poisoning/poisonable-step query; release-image.yml kept it. The cache did little for release builds. The cargo build layer's inputs change with every release, so it was restored only when the same release was built again: the re-dispatches of v0.1.6 (run 32170242786) and v0.1.8 (run 32519595682), the re-run of v0.1.11 (run 34395238814, attempt 2, arm64) and, judging by its 76-85 s build times, the re-run of v0.1.6 (run 32124353804, attempt 2, whose logs are gone). In one other build (mongodb v0.1.13, amd64) the WORKDIR, apt, both COPY and first two runtime layers came from the postgres build of the same tag, because the cache scope was per architecture only. Exporting the cache took 52 s to 150 s per uncached build. The step now has no cache-from or cache-to, with a comment saying why. This removes one way to recover a failed release. With the cache, a new dispatch of the same tag rebuilt the same image IDs (the v0.1.8 re-dispatch did), so the publish-candidate check would accept a candidate an earlier run had already pushed. Without it, a new dispatch builds different IDs and that check refuses to overwrite the candidate. The comment on the check said a failed run could be finished by re-dispatch; it now says to use "Re-run failed jobs" on the failed run while its image artifacts are kept (one day), and points to sign-image.yml once they have expired. The comment on the commit-derived build date no longer says a re-run reproduces the same image config. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
LeeroyHannigan
left a comment
There was a problem hiding this comment.
The PR deletes cache-from and cache-to (type=gha) from the build step of release-image.yml, as 8a792cf did for release-dev-image.yml, and corrects the two comments that described rebuilds as reproducible. I checked that nothing under .github/ still uses type=gha, that no other release workflow uses an Actions cache, that actionlint 1.7.12 reports nothing on this branch or on main, that every action pin in the file matches its version tag, and that no permissions change. The build-date comment is correct: BUILD_DATE reaches the binary as EXTENDDB_BUILD_TIME through crates/bin/build.rs. Every RUN step in both Dockerfiles follows the stage's ARG declarations, so a new release missed the cache on all of them, and an uncached rebuild has new image IDs, as the new comment says. The v0.1.8 re-dispatch (run 32519595682, amd64) restored every layer from the cache, including the runtime apt-get upgrade layer, so it reused the first dispatch's package versions. The v0.1.14 build (run 39892685957, amd64) spent 192 s on the cargo build and 134 s exporting the cache. Attempt 2 of run 31667072227 re-ran only publish-candidate and finished with attempt 1's artifacts. The PR adds no tests, and the workflow was not run.
Findings
Minor: .github/workflows/release-image.yml:437. The candidate probe treats any failure of docker buildx imagetools inspect as absence (2>/dev/null), so after a rate limit or a network error the job takes the push branch. A re-dispatch that hit the cache on every layer rebuilt the same image IDs, so that branch re-pushed identical content. Now a new dispatch builds different IDs, and this probe is the only check that stops it from replacing a candidate an earlier run pushed and signed: the Docker Hub tag would move to a new index, and promote-image would then refuse the digest the tester verified. The comment at lines 429 to 431 relies on this check. Please make the probe three-way, as the publish step of release-dev-image.yml, copy-artifact and promote-registry do: capture the error output, require a sha256:<64 hex> value on success, take the push branch only when the error matches manifest unknown|name unknown|not found, and fail on any other error. Optionally, run the same existence and image ID check on ${CANDIDATE}-amd64 and ${CANDIDATE}-arm64 before pushing them, which covers the per-arch case the body lists under "Not changed here".
Optional: .github/workflows/release-image.yml:434-436. "Once the artifacts have expired the run cannot be finished this way" holds for a failed publish-candidate job. A failed propagate job downloads no artifacts, so "Re-run failed jobs" can finish it for as long as GitHub allows re-runs (30 days), and promote-image copies the image and its signature to GHCR and ECR by digest whether or not propagate ran. Suggested text:
# Once the artifacts have expired, a re-run of publish-candidate
# fails at the download. sign-image.yml then signs a candidate that
# was pushed but not signed, and promote-image copies the image and
# its signature to the mirrors. A failed propagate job reads no
# artifacts and can still be re-run. "Re-run all jobs" rebuilds the
# images, which this check refuses once the candidate is pushed.Optional: .github/workflows/release-image.yml:356. retention-days: 1 is now the deadline for finishing a failed publish from the run itself, and it counts from the upload at the end of the build, before the approval wait. Each tarball is about 44 MB (v0.1.14, amd64). A few days would cover a weekend.
Optional, as a follow-up: .github/workflows/release-image.yml:274. setup-buildx-action creates the builder from moby/buildkit:buildx-stable-1, a tag resolved at run time (BuildKit v0.33.1 in run 39892685957). That daemon produces the signed image, so it is a mutable input to the release build, as the cache was. Suggest driver-opts: image=moby/buildkit:v0.33.1@sha256:<digest> here and in release-dev-image.yml, updated with the other pins.
Not checked
- A run of the workflow on this branch, which needs a release tag; the next release shows the uncached build time.
- The release runbook, which is not in the repo. If it says to re-dispatch a failed run, it needs the same change.
- The run-log figures in the body other than the runs named above.
What
The build step in
release-image.ymlno longer reads or writes the GitHub Actions build cache: thecache-from: type=ghaandcache-to: type=gha,mode=maxlines are removed, and a comment says why. Two other comments are corrected to match:retention-days: 1), and points tosign-image.ymlonce they have expired.Why
The image this step builds is the one the workflow scans, pushes and signs, so a layer restored from the Actions cache would ship in a signed release. 8a792cf removed the same cache from
release-dev-image.yml, citing CodeQL'sactions/cache-poisoning/poisonable-stepquery.release-image.ymlkept it, and no other release workflow uses an Actions cache.The cache saved little. I read the build logs of every release-image run that still has them (15 of 17: run 32167127911 ran no jobs, and the logs of 32124353804 are gone). The cargo build layer's inputs change with every release, so it was restored only when the same release was built again: the re-dispatches of v0.1.6 (run 32170242786) and v0.1.8 (run 32519595682), the re-run of v0.1.11 (run 34395238814, attempt 2, arm64) and, judging by its 76-85 s build times against 5-8 min uncached, the re-run of v0.1.6 (run 32124353804, attempt 2). In one other build (mongodb v0.1.13, amd64) the WORKDIR, apt, both COPY and first two runtime layers came from the postgres build of the same tag, because the cache scope was per architecture only. Exporting the cache took 52 s to 150 s per uncached build.
The cost is one recovery path. With the cache, a new dispatch of the same tag could rebuild identical image IDs (the v0.1.8 re-dispatch did) and finish a run that had pushed but not signed. After this change, the ways to finish such a run are "Re-run failed jobs" within a day, or
sign-image.ymlafter that.Testing done
release-image.ymlon this branch or on main (shellcheck and pyflakes integrations off).grep -rn 'cache-from\|cache-to\|type=gha' .github/matches nothing.docker load, and nothing rebuilds or retags them before the match check. Run 31667072227 attempt 2 re-ran only publish-candidate with attempt 1's artifacts. A loaded image's ID equals the build's config digest (run 39892688708). None of the runs with logs took the "already published with the identical artifact" branch. Run 32124353804 attempt 2 may have, but its logs are gone, so that branch is checked by reading the code.Not changed here
retention-days: 1is unchanged. Publish has started 22 minutes to 6 hours after the build in past runs, so the re-run window can be well short of a day in practice.Checklist
cargo test --workspace): no Rust changecargo fmt --check): no Rust changecargo clippy -- -W clippy::pedantic): no Rust changeStoragetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a
Breaking changes
None for users. For maintainers, a failed release run can no longer be finished by dispatching the workflow again once it has pushed the candidate (see Why).
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.