Skip to content

fix: carry coreutils as action inputs so apko actions run on minimal executors - #380

Open
xnox wants to merge 12 commits into
chainguard-dev:mainfrom
xnox:actiond-hermetic-tools
Open

xnox wants to merge 12 commits into
chainguard-dev:mainfrom
xnox:actiond-hermetic-tools

Conversation

@xnox

@xnox xnox commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary

Makes the apko_image / apko_show_config actions runnable on remote executors that don't ship a userspace, and adds a CI job that proves it end to end.

The run_shell actions in copy_to_workdir and apko_image call mkdir, cp, mktemp, find, ln and rm resolved from the executor's PATH. That's fine on a conventional RBE image and fails on a minimal one. This takes the binary from aspect_bazel_lib's coreutils toolchain and declares it in tools, so it's uploaded as an action input and travels to whichever executor the action lands on.

aspect_bazel_lib is already a bazel_dep here and registers that toolchain in its own non-dev section, so this adds no dependency and needs no registration. The Linux assets are statically linked against musl, so they need nothing from the executor either.

Please read: this stacks on #379

This branch contains #379 (@jeffpignataro) unchangedddf0245, f1cdd83, f3d1a6d — because the fix here is only reachable once that PR removes no-remote-exec from the apko action. Credit for the remote-execution work and the read-only---cache-dir diagnosis is theirs; I've added no changes to those commits.

If #379 lands first, I'll rebase and this drops to two .bzl files plus the workflow. Happy to split it into two PRs, or to close this and send the coreutils change against #379's branch instead — whichever is easier to review. I opened jeffpignataro#1 with the workflow alone for the same reason.

Evidence

CI job runs actiond, a local Remote Execution API worker that boots a Linux VM on the runner and executes each action inside an empty chroot in it. No account, no secrets, no external service. It's a strict test of the property this PR is about: the chroot ships /bin/bash and an /usr/bin/env stub, and nothing else.

Before this change, both call sites failed at exit 127 — staging with mkdir: command not found, and the apko action with mktemp: command not found. After:

INFO: 6 processes: 1 internal, 5 remote.
INFO: Build completed successfully, 6 total actions

Green run. 5 of 6 processes executed in the VM, nothing local, --remote_local_fallback=false — a full two-arch apko image build inside an empty chroot.

x86_64 only. GitHub's arm64 Linux runners have no /dev/kvm — the Azure DPDsv6 SKU doesn't support nested virtualization, confirmed by GitHub staff — and macOS arm64 can't nest under Apple's Virtualization Framework. actiond's own CI has the same split.

Test plan

  • bazel test //... — 6/6, including //docs:update_*_test, so stardoc output is unchanged
  • e2e/smoke — 4/4, including fix: restore remote execution support for apko_image #379's lock_hermetic_env (use_default_shell_env = False)
  • bazel run //.github/workflows:buildifier.check — clean
  • The actiond job above

Note that bazel test //... does not exercise this: locally every action runs on a host that has coreutils regardless. Those runs prove no regression, not that the fix works. The actiond job is the only thing that covers it, which is the argument for keeping it.

Notes

  • The find … -exec ln -s mirror added in fix: restore remote execution support for apko_image #379's last commit collapses to a single cp -Rs — same effect (real directories, per-file symlinks into the read-only input tree), and it avoids find, which uutils doesn't ship. So that PR's optimization survives rather than reverting to a full copy.
  • These actions still need a shell. On actiond that means --remote_default_exec_properties=requires-bash=, a user-side flag. I deliberately did not bake exec_properties = {"requires-bash": ""} into the rules — it's executor-specific and doesn't belong in a general-purpose ruleset. Going fully shell-free would mean replacing copy_to_workdir with copy_to_directory and giving the apko action a wrapper binary; happy to do that separately if you want it.
  • actiond publishes no build provenance, so the workflow's sha256sum -c only proves the download is intact. Worth weighing before this becomes a required check.

🤖 Generated with Claude Code

jeffpignataro and others added 9 commits August 25, 2026 10:18
The apko build action was pinned to local execution in chainguard-dev#332 to work
around chainguard-dev#331. The underlying incompatibility is that apko writes into
its --cache-dir even with --offline (it creates directories on cache
lookup and expands packages into the cache), while Bazel provides the
prepopulated cache tree artifact as a read-only action input, which
remote executors enforce strictly. Local sandboxes keep directories
writable, so this only surfaced under remote execution.

- Copy the cache input into a writable scratch directory inside the
  action and point --cache-dir at it, so the action works on
  read-only input trees.
- Give HOME a writable fallback when the executor does not provide
  one.
- Bind the apko_image and apko_show_config actions to the resolved
  apko toolchain so an executor never receives a binary built for a
  different platform than the action is scheduled on.
- Remove the no-remote-exec execution requirement from chainguard-dev#332.
- Add an opt-in use_default_shell_env attribute (default True, the
  previous behavior) so users who want a hermetic action environment
  can drop host env inheritance per target.

Verified locally (all tests, examples, and e2e/smoke) and against a
Buildbarn-based remote executor: both e2e/smoke image targets build
with the apko actions executing on remote linux-amd64 workers from a
macOS host.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UgPypJbT7mm4xV1xkE3rqd
apko cross-builds images for any target arch, so the comment's framing
about arch mismatches was wrong; the toolchain param needs no
explanation beyond what it says.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT4MVMA7GujqmR2YrfqNXM
Mirror the read-only cache tree artifact into the writable scratch
directory as real directories containing per-file symlinks rather than
cp -RL'ing the whole cache. apko only adds new entries (package
expansions) to its --cache-dir and never rewrites existing files, so
new writes land in the scratch directories while existing entries are
read through the symlinks. This avoids duplicating the full APK cache
on every image build. The chmod is no longer needed since apko never
writes to the symlinked files.

Verified locally (darwin-sandbox) and on a Buildbarn-based remote
executor with linux-amd64 workers; layer digests are identical to the
copy-based version.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT4MVMA7GujqmR2YrfqNXM
Adds an x86_64-only workflow that starts actiond, a local hermetic
Remote Execution API worker that runs Bazel actions inside a Linux VM,
and points Bazel at it to build //examples/wolfi-base.

x86_64 only: GitHub's arm64 Linux runners have no /dev/kvm because the
underlying Azure DPDsv6 SKU does not support nested virtualization, and
the macOS arm64 runners cannot nest under Apple's Virtualization
Framework. actiond's own CI has the same split.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CopyToWorkdir fails at exit 127 before the apko action is ever
scheduled, which hides what chainguard-dev#379 changed. Add a non-blocking follow-up
build that runs the staging actions locally, so the apko action reaches
the VM and its own missing-coreutils failure shows up in the log.

Also trigger on actiond** branches so experiments do not have to land
on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
So the workflow triggers on the branch chainguard-dev#379 is developed on, not just
on main and actiond** experiment branches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The run_shell actions in copy_to_workdir and apko_image call mkdir, cp,
mktemp, find, ln, and rm, all resolved from the executor's PATH. That
holds on a conventional RBE image and fails on an executor that ships no
userspace: against actiond, a local RE worker that runs each action in
an empty chroot, staging dies with "mkdir: command not found" (exit 127)
and the apko action with "mktemp: command not found".

Take the binary from aspect_bazel_lib's coreutils toolchain and declare
it in tools, so it is uploaded as an action input and travels to
whichever executor the action lands on. aspect_bazel_lib is already a
bazel_dep and registers the toolchain itself, so this adds no dependency
and needs no registration. The Linux assets are statically linked
against musl, so they need nothing from the executor either.

The find/ln mirror collapses into a single cp -Rs, which has the same
effect - real directories, per-file symlinks into the read-only input
tree - without needing find, which uutils does not ship.

These actions still need a shell; making them fully shell-free would
mean replacing copy_to_workdir with copy_to_directory and giving the
apko action a wrapper binary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread .github/workflows/actiond.yaml Fixed
xnox and others added 3 commits August 26, 2026 12:59
Without a pull_request trigger the job never runs on a PR: a fork PR
generates no push event in the upstream repo, so the workflow this PR
proposes as a check would not have fired on the PR proposing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At the default spawn strategy an action that opts out of remote
execution just runs on the host and the build still passes, so a
regression of the kind chainguard-dev#332 introduced would not have been caught.
--spawn_strategy=remote with no local fallback turns that into a hard
failure.

Drop the diagnostic step: it existed to surface the apko action's
failure when staging failed first, which no longer happens, and a
continue-on-error step reads as a passing check when it is not one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
needs: can only reference jobs in the same workflow, so a separate
actiond.yaml could never be a dependency of the summary gate. Moving the
job into ci.yaml and adding it to test-roundup's needs means branch
protection keeps requiring exactly one context while remote execution
becomes part of what that context covers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread .github/workflows/ci.yaml
exit 1

- name: Set up Bazel
uses: bazel-contrib/setup-bazel@c5acdfb288317d0b5c0bbd7a396a3dc868bb0f86 # v0.19.0
@jeffpignataro

Copy link
Copy Markdown

I like this tweak. It seems solid and worked fine on our internal RBE.

One note my 🤖 had was that there is a regression for Bazel 7/8 users. WORKSPACE mode does fail at analysis with No matching toolchains found for @@aspect_bazel_lib//lib:coreutils_toolchain_type because the snippet never registers the coreutils toolchain.

Users can fix this independently by registering the toolchain or it gave the option of bumping the toolchain internally - although this would probably be considered a breaking change/major ver bump so may not be worth it.

From 🤖:
In apko/repositories.bzl, bump the pinned aspect_bazel_lib from 1.34.52.22.5, and have rules_apko_dependencies() itself call register_coreutils_toolchains() — registering inside the existing function means existing users' WORKSPACE files keep working with zero edits when they upgrade. (Registration can't literally live inside the same .bzl without a load-ordering issue — @aspect_bazel_lib must be fetched first — so in practice it's a second load + call in the published snippet, which is the one edit users would need if it can't be folded in.)

"trap '\"$COREUTILS\" rm -rf \"$SCRATCH\"' EXIT",
'"$COREUTILS" mkdir "$SCRATCH/cache" "$SCRATCH/home"',
'CACHE_SRC="$(cd {cache_src} && pwd)"',
'"$COREUTILS" cp -Rs "$CACHE_SRC/." "$SCRATCH/cache/"',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see the conversation on other thread about use of symlinks here.

@jeffpignataro jeffpignataro left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 pending the small tweak to the symlink behavior for efficiency. If you want to pull that commit in then we can just merge this one and I'll close the other. Thanks!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants