Conversation
The apko build action was pinned to local execution in chainguard-dev#332 to work around chainguard-dev#331. The underlying incompatibility is that apko writes into its --cache-dir even with --offline (it creates directories on cache lookup and expands packages into the cache), while Bazel provides the prepopulated cache tree artifact as a read-only action input, which remote executors enforce strictly. Local sandboxes keep directories writable, so this only surfaced under remote execution. - Copy the cache input into a writable scratch directory inside the action and point --cache-dir at it, so the action works on read-only input trees. - Give HOME a writable fallback when the executor does not provide one. - Bind the apko_image and apko_show_config actions to the resolved apko toolchain so an executor never receives a binary built for a different platform than the action is scheduled on. - Remove the no-remote-exec execution requirement from chainguard-dev#332. - Add an opt-in use_default_shell_env attribute (default True, the previous behavior) so users who want a hermetic action environment can drop host env inheritance per target. Verified locally (all tests, examples, and e2e/smoke) and against a Buildbarn-based remote executor: both e2e/smoke image targets build with the apko actions executing on remote linux-amd64 workers from a macOS host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UgPypJbT7mm4xV1xkE3rqd
apko cross-builds images for any target arch, so the comment's framing about arch mismatches was wrong; the toolchain param needs no explanation beyond what it says. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT4MVMA7GujqmR2YrfqNXM
Mirror the read-only cache tree artifact into the writable scratch directory as real directories containing per-file symlinks rather than cp -RL'ing the whole cache. apko only adds new entries (package expansions) to its --cache-dir and never rewrites existing files, so new writes land in the scratch directories while existing entries are read through the symlinks. This avoids duplicating the full APK cache on every image build. The chmod is no longer needed since apko never writes to the symlinked files. Verified locally (darwin-sandbox) and on a Buildbarn-based remote executor with linux-amd64 workers; layer digests are identical to the copy-based version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT4MVMA7GujqmR2YrfqNXM
Adds an x86_64-only workflow that starts actiond, a local hermetic Remote Execution API worker that runs Bazel actions inside a Linux VM, and points Bazel at it to build //examples/wolfi-base. x86_64 only: GitHub's arm64 Linux runners have no /dev/kvm because the underlying Azure DPDsv6 SKU does not support nested virtualization, and the macOS arm64 runners cannot nest under Apple's Virtualization Framework. actiond's own CI has the same split. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CopyToWorkdir fails at exit 127 before the apko action is ever scheduled, which hides what chainguard-dev#379 changed. Add a non-blocking follow-up build that runs the staging actions locally, so the apko action reaches the VM and its own missing-coreutils failure shows up in the log. Also trigger on actiond** branches so experiments do not have to land on main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
So the workflow triggers on the branch chainguard-dev#379 is developed on, not just on main and actiond** experiment branches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The run_shell actions in copy_to_workdir and apko_image call mkdir, cp, mktemp, find, ln, and rm, all resolved from the executor's PATH. That holds on a conventional RBE image and fails on an executor that ships no userspace: against actiond, a local RE worker that runs each action in an empty chroot, staging dies with "mkdir: command not found" (exit 127) and the apko action with "mktemp: command not found". Take the binary from aspect_bazel_lib's coreutils toolchain and declare it in tools, so it is uploaded as an action input and travels to whichever executor the action lands on. aspect_bazel_lib is already a bazel_dep and registers the toolchain itself, so this adds no dependency and needs no registration. The Linux assets are statically linked against musl, so they need nothing from the executor either. The find/ln mirror collapses into a single cp -Rs, which has the same effect - real directories, per-file symlinks into the read-only input tree - without needing find, which uutils does not ship. These actions still need a shell; making them fully shell-free would mean replacing copy_to_workdir with copy_to_directory and giving the apko action a wrapper binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without a pull_request trigger the job never runs on a PR: a fork PR generates no push event in the upstream repo, so the workflow this PR proposes as a check would not have fired on the PR proposing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At the default spawn strategy an action that opts out of remote execution just runs on the host and the build still passes, so a regression of the kind chainguard-dev#332 introduced would not have been caught. --spawn_strategy=remote with no local fallback turns that into a hard failure. Drop the diagnostic step: it existed to surface the apko action's failure when staging failed first, which no longer happens, and a continue-on-error step reads as a passing check when it is not one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
needs: can only reference jobs in the same workflow, so a separate actiond.yaml could never be a dependency of the summary gate. Moving the job into ci.yaml and adding it to test-roundup's needs means branch protection keeps requiring exactly one context while remote execution becomes part of what that context covers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| exit 1 | ||
|
|
||
| - name: Set up Bazel | ||
| uses: bazel-contrib/setup-bazel@c5acdfb288317d0b5c0bbd7a396a3dc868bb0f86 # v0.19.0 |
|
I like this tweak. It seems solid and worked fine on our internal RBE. One note my 🤖 had was that there is a regression for Bazel 7/8 users. Users can fix this independently by registering the toolchain or it gave the option of bumping the toolchain internally - although this would probably be considered a breaking change/major ver bump so may not be worth it. From 🤖: |
| "trap '\"$COREUTILS\" rm -rf \"$SCRATCH\"' EXIT", | ||
| '"$COREUTILS" mkdir "$SCRATCH/cache" "$SCRATCH/home"', | ||
| 'CACHE_SRC="$(cd {cache_src} && pwd)"', | ||
| '"$COREUTILS" cp -Rs "$CACHE_SRC/." "$SCRATCH/cache/"', |
There was a problem hiding this comment.
please see the conversation on other thread about use of symlinks here.
jeffpignataro
left a comment
There was a problem hiding this comment.
🚀 pending the small tweak to the symlink behavior for efficiency. If you want to pull that commit in then we can just merge this one and I'll close the other. Thanks!!
Summary
Makes the
apko_image/apko_show_configactions runnable on remote executors that don't ship a userspace, and adds a CI job that proves it end to end.The
run_shellactions incopy_to_workdirandapko_imagecallmkdir,cp,mktemp,find,lnandrmresolved from the executor'sPATH. That's fine on a conventional RBE image and fails on a minimal one. This takes the binary fromaspect_bazel_lib's coreutils toolchain and declares it intools, so it's uploaded as an action input and travels to whichever executor the action lands on.aspect_bazel_libis already abazel_dephere and registers that toolchain in its own non-dev section, so this adds no dependency and needs no registration. The Linux assets are statically linked against musl, so they need nothing from the executor either.Please read: this stacks on #379
This branch contains #379 (@jeffpignataro) unchanged —
ddf0245,f1cdd83,f3d1a6d— because the fix here is only reachable once that PR removesno-remote-execfrom the apko action. Credit for the remote-execution work and the read-only---cache-dirdiagnosis is theirs; I've added no changes to those commits.If #379 lands first, I'll rebase and this drops to two
.bzlfiles plus the workflow. Happy to split it into two PRs, or to close this and send the coreutils change against #379's branch instead — whichever is easier to review. I opened jeffpignataro#1 with the workflow alone for the same reason.Evidence
CI job runs actiond, a local Remote Execution API worker that boots a Linux VM on the runner and executes each action inside an empty chroot in it. No account, no secrets, no external service. It's a strict test of the property this PR is about: the chroot ships
/bin/bashand an/usr/bin/envstub, and nothing else.Before this change, both call sites failed at exit 127 — staging with
mkdir: command not found, and the apko action withmktemp: command not found. After:Green run. 5 of 6 processes executed in the VM, nothing local,
--remote_local_fallback=false— a full two-arch apko image build inside an empty chroot.x86_64 only. GitHub's arm64 Linux runners have no
/dev/kvm— the Azure DPDsv6 SKU doesn't support nested virtualization, confirmed by GitHub staff — and macOS arm64 can't nest under Apple's Virtualization Framework. actiond's own CI has the same split.Test plan
bazel test //...— 6/6, including//docs:update_*_test, so stardoc output is unchangede2e/smoke— 4/4, including fix: restore remote execution support for apko_image #379'slock_hermetic_env(use_default_shell_env = False)bazel run //.github/workflows:buildifier.check— cleanNote that
bazel test //...does not exercise this: locally every action runs on a host that has coreutils regardless. Those runs prove no regression, not that the fix works. The actiond job is the only thing that covers it, which is the argument for keeping it.Notes
find … -exec ln -smirror added in fix: restore remote execution support for apko_image #379's last commit collapses to a singlecp -Rs— same effect (real directories, per-file symlinks into the read-only input tree), and it avoidsfind, which uutils doesn't ship. So that PR's optimization survives rather than reverting to a full copy.--remote_default_exec_properties=requires-bash=, a user-side flag. I deliberately did not bakeexec_properties = {"requires-bash": ""}into the rules — it's executor-specific and doesn't belong in a general-purpose ruleset. Going fully shell-free would mean replacingcopy_to_workdirwithcopy_to_directoryand giving the apko action a wrapper binary; happy to do that separately if you want it.sha256sum -conly proves the download is intact. Worth weighing before this becomes a required check.🤖 Generated with Claude Code