Skip to content

Bind golden snapshot warmup and publication to Actor revisions - #2554

Draft
Luomu (cloudonly) wants to merge 2 commits into
agent-substrate:mainfrom
cloudonly:fix/golden-capture-consistency
Draft

Luomu (cloudonly) wants to merge 2 commits into
agent-substrate:mainfrom
cloudonly:fix/golden-capture-consistency

Conversation

@cloudonly

@cloudonly Luomu (cloudonly) commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Draft direction

This PR is no longer a prerequisite for #2543.

Following the review of #2543, the smaller proposal under discussion would add no public Status fields and protect only the window between snapshot capture and tag publication. Cross-reconcile warmup binding would be outside that scope.

The current diff still contains the broader implementation below, including two GoldenSnapshotStatus fields. It has not been redesigned yet; this draft is awaiting confirmation that the smaller follow-up is worth pursuing.

Current implementation

  • Binds warmup to the observed RUNNING Actor's UID and version using two server-owned Status fields.
  • Checks internal Actor revision preconditions during Suspend and CreateTag.
  • Publishes only a capture approved in the current reconciliation attempt; interrupted attempts restart warmup, while completed tags are recovered before cleanup.

Prior validation

These records apply to the broader implementation and earlier revisions. They do not validate the proposed smaller design.

Recorded validation history

Final sync checks at 0b708e3c:

  • The final main sync changed upstream atelet/imagecache code and two API-guide rows. Control API, validation, protobuf sources, and the API test dependency closure are unchanged from the fully tested revision below.
  • Race tests for the changed atelet/imagecache paths passed on the final Run golden snapshots in the template's atespace #2543 stack. Linux static analysis and formatting passed; metrics and shell verification were rerun on both final branches.
  • The incremental race run skipped 5 root-gated test entries; these require the separate privileged test job.

Complete validation before the final main sync, at 35fbd982:

  • Linux/amd64, Go 1.27.1, UID 65534, CI=true: related race tests passed across 3 packages, with 461 top-level passes (2001 total test entries), no failures or skips. The PostgreSQL and Kubernetes envtest functional suites passed.
  • All 16 repository verification scripts passed across macOS and Linux. Static analysis used GOOS=linux; Weaver and shellcheck used the repository's Docker fallback on Linux.
  • Real runtime E2E was not rerun for this rebase.

Earlier validation before this rebase:

  • go test -race -count=1 ./... with CI=true passed on Linux/amd64 with Go 1.27.1: 154 test-bearing packages and 9,065 tests, including the PostgreSQL and Kubernetes envtest functional suites.
  • hack/verify-all.sh, ateapi/atelet/kubectl-ate builds, and tools/apitool race tests passed.
  • Controller regressions cover interrupted capture, slow Resume, hidden lifecycle cycles, Actor replacement, both publication windows, failed copy, and completed-tag recovery.

Functional tests use a fake atelet. Root-gated tests and real gVisor/micro-VM E2E were not run.

AI assistance: Codex helped implement, review, and validate the original change; Claude Code rebased it and folded in review cleanups. Codex updated this draft's scope and dependency description.

@cloudonly
Luomu (cloudonly) force-pushed the fix/golden-capture-consistency branch 8 times, most recently from 0b708e3 to a8e491d Compare October 16, 2026 05:07
@cloudonly
Luomu (cloudonly) marked this pull request as draft October 16, 2026 07:52
@cloudonly

Luomu (cloudonly) commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Zoe Zhao (@zoez7) Before I redesign this draft: is a narrow follow-up worth pursuing? The concrete case and proposed shape are below. I won't change code until you confirm.

Problem after #2543. Golden actors are now ordinary actors in the template's atespace, so tenants can suspend them. The reconciler publishes any completed snapshot it finds, whether or not it took that snapshot:

Scenario: a tenant calls SuspendActor on the golden actor during the 20s warmup. The next pass publishes that pre-warmup snapshot as the golden tag, and every Actor created from the template starts from it.

Proposed scope

  • Publish only a snapshot captured in the current reconcile attempt. Approval is held in memory. A pass that finds a snapshot it did not capture, including after a controller restart, resumes the actor and restarts warmup.
  • Immediately before and after CreateTag, compare the golden actor's UID, metadata.version and latest Snapshot.uuid with what the reconciler observed after its own SuspendActor. Any change means no publication: the tag is not created, or it is deleted, and warmup restarts. Every Actor update increments metadata.version, so an intervening lifecycle operation cannot go unnoticed.
  • No new API or Status fields and no new preconditions. workflow_tag.go and public RPCs stay unchanged; the change is confined to template_reconciler.go.

Trade-offs. A controller crash between capture and tagging costs one more warmup instead of publishing directly. An update that lands after CreateTag has read the actor causes one unnecessary retry, never a wrong publication.

Not covered

Tests. Controller tests with the lifecycle fake: tenant suspend during warmup, SUSPENDING observed at the start of a pass, interference before and after CreateTag, a crash between capture and tagging, and unchanged completed-tag recovery. Plus one functional test that suspends the golden actor through the public RPC.

If this isn't worth the extra code, I'll close #2554 and track the gap in #2552.

A template whose preferred fidelity is VOLUMES but that mounts no
volume a snapshot can capture (durable-dir, or a CSI volume on the
micro-VM class) fails at runtime on every pause and suspend. Reject it
at template admission instead, in the custom validation that already
checks volume references and the ROOTFS sandbox class.

Document the requirement next to the ROOTFS rule in the API guide,
including that an external volume's data stays on the volume rather
than in the snapshot.
Warmup could outlive the Actor revision it started on, and recovery
assumed that an existing durable snapshot had been approved for
publication. Bind the warmup timer to the observed RUNNING Actor's UID
and version, and publish only a capture completed during the current
reconciliation attempt.

Suspend checks the warmed revision under the actor lease. CreateTag
checks the revision Suspend returned under its own lease, so an
intervening lifecycle operation aborts publication rather than tagging
an unapproved snapshot. Approval lives only in the reconciler's memory:
an interrupted capture or failed copy resumes the existing guest state
and starts another warmup, while a completed tag is recovered before
retrying cleanup.

Adds the server-owned warmup_actor_uid and warmup_actor_version fields
to GoldenSnapshotStatus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants