Repository navigation
Scheduler: place resumes on workers matching the snapshot's Hardware, cold boot otherwise #2250
Description
With Hardware stamps on both workers and snapshots, the scheduler can avoid placing a memory restore on hardware that cannot load it.
Proposal
- Add a
Hardwarerequirement toscheduling.Constraintsand evaluate it inApplies, soSchedule, theBindActorToWorkeradmit re-check, andvalidateAssignedWorkerall honor it. schedulingConstraintsinworkflow_resume.goderives it from the actor's latest snapshot when restoring memory (FULL/MEMORYscope). A snapshot without a stamp imposes no hardware constraint.- Match on
architecture,cpu_vendor, andcpu_model. - When no compatible worker exists for a memory restore and the template's minimum scope allows it (ActorTemplate: replace on_pause/on_commit/on_resume with capture_scope and minimum_scope #2102), schedule with only the
architectureconstraint and restore at the lower scope (or cold-boot viaatelet.Runif the actor only held a borrowed golden snapshot with no durable volumes). Otherwise fail as today. - Emit a metric for fallback resumes so fleet operators see hardware fragmentation.
** Open question** - Exact cpu_model equality may be too strict: same instance type can differ in stepping. Proposal is to record full family/model/stepping in cpu_model (so no information is lost in stored snapshots) while matching on family/model until data says otherwise. ?
Activity
shrutiyam-glitch commentedon Oct 14, 2026
Updating the issue after the recent API changes (#2289) and re-design in #2348
Part of #2246 (and #2069). Depends on #2247 and #2249.
When resuming an actor from a
SNAPSHOT_FIDELITY_MEMORYcheckpoint, placing the actor onto a worker with an incompatible CPU microarchitecture or sandbox runtime version causes the guest memory restore (runsc restoreor microVM snapshot load) to fail and crash the actor. Conversely, if all workers advertising the snapshot's exact runtime are at capacity or have been drained after a runtime rollout, failing the resume outright prevents the actor from making progress even though its durable volume state can still be restored on a newer worker.Proposal
Make
ResumeActorplacement and restore fidelity aware ofSnapshot.sandbox_runtime:SNAPSHOT_FIDELITY_MEMORYsnapshots restored under the actor's current template require runtime compatibility (RequiredRuntime = snapshot.sandbox_runtime).SNAPSHOT_FIDELITY_VOLUMESandSNAPSHOT_FIDELITY_ROOTFS), and actors whose template has been updated since the snapshot was taken do not restore guest memory and may be placed on any worker matching the template'sSandboxClass, selectors, and resource limits.default_runtimeor an entry inrestorable_runtimesmatches the snapshot'ssandbox_runtime(exact match onsandbox_class,version.schema_version, and allversion.attributeskey-value pairs; the informationalnamefield is ignored).SNAPSHOT_FIDELITY_MEMORYand record the matched runtime entry onWorkerAssignment.sandbox_runtime.SandboxClassand selectors. At restore time, when the assigned worker's runtime does not match the snapshot'ssandbox_runtime, discard the incompatible memory image and downgrade the restore toSNAPSHOT_FIDELITY_VOLUMESunder the new worker'sdefault_runtime. When the actor next suspends, its new snapshot is stamped with the new worker's runtime.PAUSEDactor's checkpoint resides on its pinned node (AssignedNode), do not relax the runtime constraint while a local checkpoint is held; if no compatible worker is available on that node, surfaceResourceExhausted.cc: Zoe Zhao (@zoez7)
Stamp snapshots with SandboxRuntime and schedule restores on runtime-…
shrutiyam-glitch commentedon Oct 20, 2026
#2536 closes this.