Skip to content

ateapi: let actors reference ExternalVolumes - #11

Open
hajiler wants to merge 4 commits into
shared-volumes/03-external-volumefrom
shared-volumes/04-external-volume-refs
Open

hajiler wants to merge 4 commits into
shared-volumes/03-external-volumefrom
shared-volumes/04-external-volume-refs

Conversation

@hajiler

@hajiler hajiler commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Summary

Completes Milestone 1 of go/substrate-shared-volumes: sequential sharing of an ExternalVolume between actors.

  • Template: Volume.external_volume_ref { name } mounts an existing ExternalVolume. name may be empty, in which case the actor binds the slot at creation.
  • Per-actor binding (doc Proposal 2): Actor.external_volume_bindings (immutable map of template volume name → ExternalVolume name). Create validates that every unnamed slot is bound and that no binding targets a named ref or unknown volume. kubectl ate create actor --volume shared=ctx-42.
  • References: every actor referencing a volume appears in status.refs, with active true only while resumed.
    • Create records the mount on the actor and an inactive ref (a miss here is self-healed by resume).
    • Resume activates the ref; a READ_WRITE_ONCE volume with another active holder is refused with an error naming the holder. Inactive (paused/suspended) holders do not block.
    • Pause and suspend mark the ref inactive after detach.
    • Crash leaves the ref untouched.
    • Delete removes the ref, then deletes the volume if its trigger is LAST_ACTOR and no refs remain (idempotent on retry).
    • DeleteExternalVolume is refused while any ref exists, including inactive ones.
  • Golden actors never reference a volume: they skip binding validation and mount an empty_dir placeholder at the same path (needs atelet: add the empty_dir volume source #9).
  • Borrowed volumes are never provisioned or deleted by the per-actor volume code.
  • Docs: new "Shared External Volumes" section in docs/csi-volumes.md.

M2 TODOs are left inline: READ_WRITE_MANY claims, passing access mode to CSI, moving refs out of the volume row.

PR 4 of 4, stacked on #10 (shared-volumes/03-external-volume) and merges #9 (shared-volumes/02-empty-dir-volume); the atelet commit shows in this diff until #9 lands.

Test plan

  • go test ./cmd/ateapi/internal/controlapi/ (new external_volume_refs_test.go, workload spec and volume tests)
  • go test ./cmd/ateapi/internal/controlapi/functionaltest/ (new TestExternalVolume_SequentialHandoff, TestExternalVolume_HeldByAnotherActor, TestDeleteExternalVolume_Referenced, TestCreateActor_ExternalVolumeBindings)
  • go test ./cmd/kubectl-ate/... ./cmd/atelet/... ./internal/ocispec/...
  • cd tools/apitool && go run . validate, hack/verify/{codegen,gofmt,boilerplate,golangci-lint}.sh
  • e2e TestSharedExternalVolumeHandoff / TestSharedExternalVolumeMissing in internal/e2e/suites/combinedvolumes (compiles; not run locally, needs a cluster)
  • hack/verify/proto-fmt.sh (clang-format not installed locally)

🤖 Generated with Claude Code

hajiler and others added 4 commits October 9, 2026 12:04
An EmptyDirVolume is an empty host directory at the same path an external
volume would be published to, mounted into the sandbox the same way. It lets a
golden actor warm up against a placeholder for an external volume it must not
mount, so the snapshot's mount table matches the real actor's on both gVisor
and microVM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A template volume with external_volume_ref mounts an ExternalVolume instead of
provisioning one per actor. The ref names the volume, or leaves the name empty
so each actor binds it at creation through Actor.external_volume_bindings.

Every actor referencing a volume is recorded in the volume's refs, with active
set while the actor is resumed. Resume refuses a READ_WRITE_ONCE volume another
actor holds active, pause and suspend mark the reference inactive, crash leaves
it as is, and delete removes it and deletes a LAST_ACTOR volume that no actor
references any more. Golden actors never reference a volume; they warm up
against an empty directory at the same mount path.

Depends on the empty_dir volume source and the ExternalVolume resource.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The golden actor warms up against a local empty directory in place of a
shared volume, but runsc derives the disable_file_handle_sharing mount option
from the bind source's filesystem and refuses a restore whose mount options
differ from the checkpoint's. On NFS the first resume of every such actor
therefore failed with "Mounts" does not match across checkpoint restore.

Actors created from a template with an external_volume_ref no longer seed
from the golden tag: the first resume boots from the image and later resumes
restore the actor's own snapshot, which was taken over the real volume. An
explicit source tag is refused for these templates. Golden restores for
shared volumes remain follow-up work, tracked by TODO(shared-volumes).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@hajiler
hajiler force-pushed the shared-volumes/03-external-volume branch from b68483b to 1c24dc1 Compare October 13, 2026 15:59
@hajiler
hajiler force-pushed the shared-volumes/04-external-volume-refs branch from 7f19b87 to 41f90e9 Compare October 13, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant