Repository navigation
Snapshots: record the Hardware of the worker that captured them #2249
Description
ExternalSnapshot and LocalSnapshot record only the URI/name, scope, and template UID. The scheduler cannot match a snapshot to a worker without knowing which hardware it was taken on.
Proposal
message ExternalSnapshot {
string snapshot_uri = 1;
SnapshotContentScope content_scope = 2; // becomes SnapshotScope under #2102
string actor_template_uid = 3;
// Hardware the snapshot was captured on.
// +k8s:optional
Hardware hardware = 4;
}
message LocalSnapshot {
repeated string node_vms_with_local_snapshots = 1;
string snapshot_name = 2;
SnapshotContentScope content_scope = 3;
// +k8s:optional
Hardware hardware = 4;
}The control plane copies the Hardware stamp from the assigned Worker's recorded WorkerStatus.hardware when the pause or suspend workflow captures the snapshot. Stamping LocalSnapshot.hardware at pause time ensures that a subsequent paused-origin suspend (isPausedOriginSuspend in workflow_suspend.go, where
WorkerAssignment was already cleared by PauseActor) can propagate LocalSnapshot.hardware onto the promoted ExternalSnapshot. That keeps the stamp identical to what placement matches against and needs no new atelet response fields.
Activity
shrutiyam-glitch commentedon Oct 14, 2026
Updating the issue after the recent API changes (#2289) and re-design in #2348
Part of #2246 (and #2069).
Problem
A memory checkpoint (
SNAPSHOT_FIDELITY_MEMORY) is only restorable on a worker whose sandbox class, host hardware, and runtime assets match the environment that produced the checkpoint. While workers now self-report theirdefault_runtimeandrestorable_runtimesduring registration (#2247), snapshots and active actor assignments do not yet record whichSandboxRuntimean actor is running under or which runtime produced a given snapshot.Additionally, during a rolling runtime upgrade, a worker may update its
default_runtimeto a newer version (v2) while existing actors on that worker continue running under an older version (v1inrestorable_runtimes). Reading a worker's livedefault_runtimeat checkpoint time would mis-stamp snapshots taken from those already-runningv1actors asv2.Proposal
Track the
SandboxRuntimeacross the actor lifecycle by pinning it on the actor's worker assignment when the actor starts or resumes, and propagating that stamp to every snapshot derived from it:WorkerAssignment:When an actor is placed on a worker during
ResumeActor, record the specificSandboxRuntimeserving that actor onWorkerAssignment.sandbox_runtime(either the worker'sdefault_runtimeon a cold start / fallback, or the specific entry fromdefault_runtime/restorable_runtimesthat matched the snapshot being restored).PauseActor): Stamp the node-local checkpoint'sSnapshot.sandbox_runtimefromWorkerAssignment.sandbox_runtime.RUNNING(SuspendActor): Stamp the durable snapshot'sSnapshot.sandbox_runtimefromWorkerAssignment.sandbox_runtime.PAUSED(SuspendActor): When uploading a node-local pause checkpoint after the worker has already been released, preserve theSnapshot.sandbox_runtimealready recorded on the local pause snapshot.CreateTag): Copy the source actor snapshot'ssandbox_runtimeonto the tag's snapshot.CreateActor): PreserveSnapshot.sandbox_runtimewhen initializing an actor from a usersource_tagor a template's golden snapshot.cc: Zoe Zhao (@zoez7)
Stamp snapshots with SandboxRuntime and schedule restores on runtime-…