Run golden snapshots in the template's atespace by cloudonly · Pull Request #2543 · agent-substrate/substrate

Luomu (cloudonly) · 2026-10-13T01:47:02Z

Closes #2533.

Move Golden actors and tags from the shared ate-golden atespace into their ActorTemplate's atespace, named after the template UID. Golden resources use the tenant's ordinary authorization, lifecycle policies, and storage grants. Remove the special handling for the dedicated system atespace.

Golden capture follows the template's preferredFidelity. MEMORY preserves process state. VOLUMES and ROOTFS restore persisted data and start new processes. Durable snapshot metrics use ate.snapshot.kind=latest, including Golden snapshots.

Tests cover tenant storage ownership, snapshot inheritance, restore, and cleanup. The VOLUMES Golden fixture matches the template's fidelity.

Validation

Validated 204b8fb3, rebased onto main 7abc3172, on Linux with Go 1.27.2. The tested source matches the PR's 13,851 tracked files.

Passed go test -race -count=1 ./...: 166 packages passed, including PostgreSQL/envtest functional and authorization tests.
Passed builds of ateapi, atelet, and kubectl-ate, race-enabled compilation of the demo E2E binary, and tools/apitool race tests. Local validation and actor-attribute race tests also passed.
Passed hack/verify-all.sh, including code generation, lint, modules, licenses, metrics, migrations, and formatting. Regeneration left the source unchanged.
Redeployed the dedicated kind cluster and passed real gVisor TestGoldenSnapshotLifecycle and TestDurableDirLifecycle: Golden MEMORY/VOLUMES creation, tenant storage ownership, inheritance, resume, template cleanup, and durable-directory lifecycle. JUnit reports zero failures and two ROOTFS skips.

All Go compilation and tests ran as a non-root user. The runtime copy uses a host-only Linux 5.14 OverlayFS fallback; source checks use the unmodified PR source. Micro-VM E2E and root-gated tests were not run.

Operator notes

Follow the pre-1.0 reinstall procedure in the upgrade guide. Existing ate-golden resources may still own snapshots borrowed by actors.
Object storage grants must cover actor and tag prefixes in each template's atespace.
Golden actors appear in ordinary Actor lists. Lifecycle and ingress operations can interrupt a Golden build. Deleting a published Golden tag invalidates the template's recorded reference.
Update dashboards that select ate.snapshot.kind=golden. The latest series now includes Golden and other durable snapshots.

AI assistance: Codex helped implement and validate this change.

Luomu (cloudonly) · 2026-10-14T09:25:29Z

Zoe Zhao (@zoez7), could you review this PR? Real gVisor E2E verification is complete, and the results and upgrade notes are in the PR description.

Zoe Zhao (zoez7) · 2026-10-15T19:04:09Z

Sorry for the delay, will review today

Zoe Zhao (zoez7) · 2026-10-16T04:48:41Z

+  // warmup_actor_uid identifies the Actor whose RUNNING state starts the timer.
+  // +k8s:optional
+  // +k8s:format=k8s-uuid
+  string warmup_actor_uid = 4;


Let's keep the public API surface minimal and drop the revision-binding changes (warmup_actor_uid, warmup_actor_version, and the revision preconditions) from this PR. We don't need revision binding here.

Removed UID/version binding and its added preconditions.

Zoe Zhao (zoez7) · 2026-10-16T04:49:14Z

+  // A different revision starts a fresh timer, even if the Actor is RUNNING again.
+  // +k8s:optional
+  // +k8s:minimum=1
+  int64 warmup_actor_version = 5;


Same concern as above. This is controller-internal state, but once it is in GoldenSnapshotStatus it becomes public API that clients can read and start depending on. I'd prefer not to expose it.

Removed the added fields from GoldenSnapshotStatus.

Zoe Zhao (zoez7) · 2026-10-16T04:49:48Z

-	// actors live in.
-	GoldenActorAtespace = "ate-golden"
+	// InternalResourceNamePrefix reserves actor and tag names for controllers.
+	InternalResourceNamePrefix = "ate-internal-"


To keep the API surface minimal, can we avoid reserving special names? A reserved prefix is a new rule every client has to learn, and it adds special cases to CreateActor, CreateTag, and authz.

Removed the reserved prefix; Golden resources use the template UID.

Zoe Zhao (zoez7) · 2026-10-16T04:52:17Z


 You need git, go and kubectl, and crane for prebuilt images. Agree on a time with actor owners: their running actors get `SIGTERM` in step 3.

+### Golden snapshots from older releases


🤖 should-fix 🟡 – The table at the top of this runbook says pre-1.0 minor upgrades are not supported and require a reinstall, and AGENTS.md rules out compatibility work before v1.0.0. This release-specific migration guide contradicts both and will go stale in a generic runbook. Please move it to the PR description or release notes, and drop the new Confirm before you begin heading with it.

Moved the notes to the PR description and restored this file to main.

Zoe Zhao (zoez7) · 2026-10-16T04:54:14Z

 	UpdateTag(ctx context.Context, tagRef resources.TagRef, precondition store.Precondition, mutate func(toUpdate *ateapipb.Tag) error) (*ateapipb.Tag, error)
 	DeleteTag(ctx context.Context, tagRef resources.TagRef, precondition store.DeletePreconditions) (*ateapipb.Tag, error)
 	GetActorTemplate(ctx context.Context, templateRef resources.ActorTemplateRef) (*ateapipb.ActorTemplate, error)
+	ListActorTemplates(ctx context.Context, atespace string, opts store.ListOptions) (store.ListResponse[*ateapipb.ActorTemplate], error)


Once acquireTagTemplateLease is removed from DeleteTag, actorWorkflowStore no longer needs ListActorTemplates. Let's remove this from actorWorkflowStore to keep this PR minimal.

Removed ListActorTemplates from actorWorkflowStore.

Zoe Zhao (zoez7) · 2026-10-16T05:09:14Z

+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package controlapi
+
+import (
+	"context"
+
+	"github.com/agent-substrate/substrate/internal/resources"
+)
+
+type reservedCreationKey struct{}
+type heldTemplateLeaseKey struct{}
+
+// withReservedResourceCreation marks ctx as permitted to create reserved
+// names; only controller-internal callers set it.
+func withReservedResourceCreation(ctx context.Context) context.Context {
+	return context.WithValue(ctx, reservedCreationKey{}, true)
+}
+
+func reservedResourceCreationAllowed(ctx context.Context) bool {
+	allowed, _ := ctx.Value(reservedCreationKey{}).(bool)
+	return allowed
+}
+
+// withTemplateLease records that the caller holds the named template lease for
+// the lifetime of ctx.
+func withTemplateLease(ctx context.Context, ref resources.ActorTemplateRef) context.Context {
+	return context.WithValue(ctx, heldTemplateLeaseKey{}, ref)
+}
+
+func holdsTemplateLease(ctx context.Context, ref resources.ActorTemplateRef) bool {
+	held, ok := ctx.Value(heldTemplateLeaseKey{}).(resources.ActorTemplateRef)
+	return ok && held == ref
+}


Please remove this entire file (cmd/ateapi/internal/controlapi/internal_call.go); both context keys in it are unnecessary. Context-value flags hide which callers may skip checks: without reserved names, withReservedResourceCreation / reservedResourceCreationAllowed have nothing to unlock, and withTemplateLease / holdsTemplateLease only exist so DeleteTag can skip re-acquiring the template lease, which DeleteTag should not acquire in the first place.

Deleted internal_call.go and both context flags.

Zoe Zhao (zoez7) · 2026-10-16T05:11:03Z

+// acquireTagTemplateLease serializes deletion with publication by the source
+// template. It protects references regardless of the tag's name.
+func (w *ActorWorkflow) acquireTagTemplateLease(ctx context.Context, tag *ateapipb.Tag, deletingTemplate resources.ActorTemplateRef) (context.Context, *store.Lease, error) {
+	tagRef := resources.TagRefFromTag(tag)
+	var owner *ateapipb.ActorTemplate
+	pageToken := ""
+	for {
+		page, err := w.store.ListActorTemplates(ctx, tagRef.Atespace, store.ListOptions{PageSize: 100, PageToken: pageToken})
+		if err != nil {
+			return ctx, nil, fmt.Errorf("while checking template references to tag %s: %w", tagRef, err)
+		}
+		for _, tmpl := range page.Items {
+			ref := resources.ActorTemplateRefFromActorTemplate(tmpl)
+			if golden := tmpl.GetStatus().GetGoldenSnapshotStatus().GetGoldenTag(); golden != nil && resources.TagRefFromObjectRef(golden) == tagRef && ref != deletingTemplate {
+				return ctx, nil, apierror.FailedPrecondition("Tag %s is referenced by ActorTemplate %s", tagRef, ref)
+			}
+			if tmpl.GetMetadata().GetUid() == tag.GetStatus().GetActorTemplateUid() {
+				owner = tmpl
+			}
+		}
+		if page.NextPageToken == "" {
+			break
+		}
+		pageToken = page.NextPageToken
+	}
+	if owner == nil || owner.GetSnapshotConfig().GetGoldenSnapshotConfig().GetMode() == ateapipb.GoldenSnapshotMode_GOLDEN_SNAPSHOT_MODE_DISABLED || goldenSnapshotDone(owner.GetStatus().GetGoldenSnapshotStatus()) {
+		return ctx, nil, nil
+	}
+	ref := resources.ActorTemplateRefFromActorTemplate(owner)
+	var lease *store.Lease
+	if !holdsTemplateLease(ctx, ref) {
+		var err error
+		ctx, lease, err = acquireLease(ctx, w.store, "lease:actortemplate:"+ref.Atespace+":"+ref.Name, "ActorTemplate "+ref.String())
+		if err != nil {
+			return ctx, nil, err
+		}
+	}
+	current, err := w.store.GetActorTemplate(ctx, ref)
+	if errors.Is(err, store.ErrNotFound) {
+		return ctx, lease, nil
+	}
+	if err != nil {
+		if lease != nil {
+			lease.Close()
+		}
+		return ctx, nil, err
+	}
+	if golden := current.GetStatus().GetGoldenSnapshotStatus().GetGoldenTag(); golden != nil && resources.TagRefFromObjectRef(golden) == tagRef && ref != deletingTemplate {
+		if lease != nil {
+			lease.Close()
+		}
+		return ctx, nil, apierror.FailedPrecondition("Tag %s is referenced by ActorTemplate %s", tagRef, ref)
+	}
+	return ctx, lease, nil
+}


Please revert all changes to cmd/ateapi/internal/controlapi/workflow_tag.go. We don't need the revision-binding precondition on TagActorSnapshot / loadActorForTag, and we don't need acquireTagTemplateLease in DeleteTag (golden tag deletion protection can be done in a separate follow-up PR).

Restored workflow_tag.go to main; tag deletion protection is outside this PR.

Luomu (cloudonly) · 2026-10-16T07:23:48Z

Zoe Zhao (@zoez7) I've simplified the PR as requested and completed validation, including gVisor E2E; please take another look.

Zoe Zhao (zoez7) · 2026-10-16T21:50:29Z

+		declared[vol.GetName()] = vol
 	}
+	microVM := value.GetSandboxConfig().GetSandboxClass() == string(atev1alpha1.SandboxClassMicroVM)
+	hasSnapshotVolume := false


The changes in this file is not related to moving the golden actor's atespace, let's remove it for now.

Removed the VOLUMES validation commit and rebased onto main.

Zoe Zhao (zoez7) · 2026-10-16T21:54:39Z

+		SourceActor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: created.GetMetadata().GetUid()},
 		Scope:       ateapipb.TagScope_TAG_SCOPE_PUBLISHED,
 		Status: &ateapipb.TagStatus{
 			Snapshot:         newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, created.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t)),


nit: createDataFidelityTemplate sets PreferredFidelity: SNAPSHOT_FIDELITY_VOLUMES on the template, and now that preferredFidelity() in workflow_suspend.go no longer forces golden snapshots to SNAPSHOT_FIDELITY_MEMORY, a VOLUMES template's golden tag has SNAPSHOT_FIDELITY_VOLUMES fidelity. Seed the golden tag here with SNAPSHOT_FIDELITY_VOLUMES so the fixture matches what the reconciler produces.

Suggested change

Snapshot: newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, created.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t)),

Snapshot: newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, created.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t)),

Updated the Golden fixture to VOLUMES to match the template.

Luomu (cloudonly) · 2026-10-17T01:20:34Z

Zoe Zhao (@zoez7) Please re-review the updated PR with both comments addressed and full race, verify-all, and gVisor E2E passing.

Place generated actors and published tags beside their templates, using template UIDs as resource names. Remove the system-atespace exceptions for lifecycle operations, snapshot fidelity, authorization, and metrics. Keep the reconciler's existing RPCs and state machine, and cover tenant storage ownership, snapshot inheritance, restore, and cleanup.

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch from 872b864 to 100eadf Compare October 13, 2026 09:46

Luomu (cloudonly) mentioned this pull request Oct 13, 2026

Bind golden snapshot warmup and publication to Actor revisions #2554

Draft

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch 7 times, most recently from f2af7af to a3ae7df Compare October 14, 2026 06:55

Luomu (cloudonly) marked this pull request as ready for review October 14, 2026 07:04

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch 3 times, most recently from 5764167 to 5e9cfd7 Compare October 15, 2026 06:58

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch 3 times, most recently from 14fffe7 to 1abd2d4 Compare October 16, 2026 05:07

Zoe Zhao (zoez7) reviewed Oct 16, 2026

View reviewed changes

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch from 1abd2d4 to eb2be87 Compare October 16, 2026 05:58

Luomu (cloudonly) requested a review from Zoe Zhao (zoez7) October 16, 2026 07:22

Zoe Zhao (zoez7) requested changes Oct 16, 2026

View reviewed changes

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch from 260e824 to 204b8fb Compare October 17, 2026 00:50

Luomu (cloudonly) requested a review from Zoe Zhao (zoez7) October 17, 2026 01:23

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch 2 times, most recently from 6576a2d to 88e5ae8 Compare October 18, 2026 00:28

Zoe Zhao (zoez7) previously approved these changes Oct 18, 2026

View reviewed changes

Zoe Zhao (zoez7) enabled auto-merge October 18, 2026 00:45

Zoe Zhao (zoez7) added this pull request to the merge queue Oct 18, 2026

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 18, 2026

Luomu (cloudonly) added 3 commits October 18, 2026 09:17

Remove unused test imports after dropping Golden checks

f992531

Match the Golden fixture to the template's VOLUMES fidelity

703cf68

Luomu (cloudonly) dismissed Zoe Zhao (zoez7)’s stale review via 703cf68 October 18, 2026 01:23

Luomu (cloudonly) force-pushed the fix/golden-template-atespace branch from 88e5ae8 to 703cf68 Compare October 18, 2026 01:23

Luomu (cloudonly) requested a review from Zoe Zhao (zoez7) October 18, 2026 02:03

Luomu (cloudonly) mentioned this pull request Oct 18, 2026

ateapi: Cover Golden snapshot preferred fidelity #2821

Closed

Zoe Zhao (zoez7) enabled auto-merge October 18, 2026 04:16

Zoe Zhao (zoez7) approved these changes Oct 18, 2026

View reviewed changes

Zoe Zhao (zoez7) added this pull request to the merge queue Oct 18, 2026

Merged via the queue into agent-substrate:main with commit d2b9750 Oct 18, 2026
15 checks passed

This was referenced Oct 18, 2026

Use a preferred_fidelity's valie when golden snapshots is scheduled #2383

Closed

ateapi: reject VOLUMES snapshots without supported volume mounts #1972

Open


		You need git, go and kubectl, and crane for prebuilt images. Agree on a time with actor owners: their running actors get `SIGTERM` in step 3.

		### Golden snapshots from older releases

	Snapshot: newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, created.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t)),
	Snapshot: newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_TAG, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, created.GetMetadata().GetUid(), goldenSnapshotName, goldenSnapshotURI(t)),

Conversation

Luomu (cloudonly) commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Validation

Operator notes

Uh oh!

Luomu (cloudonly) commented Oct 14, 2026

Uh oh!

Zoe Zhao (zoez7) commented Oct 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Luomu (cloudonly) commented Oct 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Luomu (cloudonly) commented Oct 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Luomu (cloudonly) commented Oct 13, 2026 •

edited

Loading