Repository navigation
api: report worker runtimes per sandbox class (WorkerSandboxClass, WorkerSandboxRuntime, ActorSandboxRuntime) #3008
Copy link
Copy link
Closed
Labels
Description
Context & Related Work
- Parent / Original Design: Worker Self-reports Snapshot Compatibility #2348 (Worker Self-Reported Sandbox & Hardware Compatibility design doc) & How to patch CVEs and upgrade gVisor / microVM version used by SandboxConfig? #587
- Related PRs:
- Update default runsc flags configuration #2361, atelet prepares each local worker's sandbox assets and versions file #2521, atelet, ateworker: probe CPU features and watch sandbox-runtimes.json for dynamic worker registration #2660 (Worker runtime registration,
SandboxConfigbundle hashing, hardware probing, and snapshot runtime matching) - Renumber proto fields consecutively, with ResourceMetadata as field 1 #2888 (Remove
ActorTemplate.sandbox_class) - ateapi: Remove sandbox_class from Worker spec #2922 (Remove
Worker.sandbox_class, leavingWorkerStatusas the sole record of a worker's sandbox class)
- Update default runsc flags configuration #2361, atelet prepares each local worker's sandbox assets and versions file #2521, atelet, ateworker: probe CPU features and watch sandbox-runtimes.json for dynamic worker registration #2660 (Worker runtime registration,
Problem Statement
Under the initial schema introduced in #2348, a Worker reports its runtime capabilities through two flat fields on WorkerStatus and RegisterWorkerRequest (default_runtime and restorable_runtimes), and RegisterWorker rejects any restorable_runtimes entry whose sandbox_class differs from default_runtime.sandbox_class. This causes two structural limitations that would require breaking API and wire changes to fix after GA:
- Flat worker fields can only represent multiple versions of a single sandbox class:
While theWorkerPoolCRD already models sandbox classes as a keyed list (WorkerPool.spec.sandboxClassesandWorkerPool.status.sandboxClasses, withMaxItems=1today) so a worker pod can support more than one sandbox class in the future, a singularWorkerStatus.default_runtimecan only ever describe one class. Replacing a singulardefault_runtimewith a per-class list after v1.0.0 would be a breaking change. - One
SandboxRuntimemessage conflates worker-installed runtimes with actor/snapshot stamps:
previously,SandboxRuntimewas shared between:- What a worker installs for a class, which is already scoped to a class (making
sandbox_classredundant inside a per-class entry) and will gain post-GA fields such asrepeated VersionedSandboxCompat compatible_versionsand an informationalSandboxConfigversionname. - What an actor's assignment (
WorkerAssignment.sandbox_runtime), snapshot (Snapshot.sandbox_runtime), andateletexecution request (RunRequest/RestoreRequest) are stamped with, which always records one concretesandbox_classand one primaryVersionedSandboxCompatversion, and never carriescompatible_versions.
- What a worker installs for a class, which is already scoped to a class (making
Doing this refactor now—while pre-v1.0.0 stored-data breaks are still permitted alongside #2888 and #2922—aligns the control-plane API with WorkerPool.spec.sandboxClasses and cleanly separates the worker and actor runtime types.
Proposed Protobuf Schema
Public Control-Plane API (pkg/proto/ateapipb/ateapi.proto)
message WorkerStatus {
// ... fields 1..4 unchanged ...
// [Removed] SandboxRuntime default_runtime = 5;
// [Removed] repeated SandboxRuntime restorable_runtimes = 6;
// The sandbox classes this Worker runs and the runtimes it runs them
// under, as reported by the Worker at registration; empty until then, and
// nothing is placed on the Worker until then. The first registration fixes
// the set of classes; a later report naming other classes is refused. One
// entry per class the pool declares: one today.
//
// +k8s:optional
// +k8s:maxItems=1 # keep in sync with WorkerPoolSpec.sandboxClasses
// +k8s:listType=map
// +k8s:listMapKey=sandbox_class
repeated WorkerSandboxClass sandbox_classes = 5;
}
// WorkerSandboxClass is one sandbox class a Worker runs, with the runtimes it
// runs it under. A Worker has one entry per WorkerPool sandboxClasses entry,
// so exactly one today.
message WorkerSandboxClass {
// e.g. "gvisor", "microvm", as on WorkerPool.spec.sandboxClasses[].name.
//
// +k8s:required
// +k8s:maxLength=63
// +k8s:customValidation # gvisor or microvm
string sandbox_class = 1;
// Runtime that new and cold-booted actors of this class start under.
// Snapshots matching its version can also warm-boot on this worker.
//
// +k8s:required
WorkerSandboxRuntime default_runtime = 2;
// Additional runtimes of this class (excluding default_runtime) that this
// worker can restore snapshots from right now.
//
// +k8s:optional
// +k8s:maxItems=32
// +k8s:listType=atomic
repeated WorkerSandboxRuntime restorable_runtimes = 3;
}
// One runtime installed on the worker for that class.
message WorkerSandboxRuntime {
// Primary compatibility version stamped onto an actor's snapshot when a
// checkpoint is taken on this runtime. Snapshots matching `version`
// can warm-boot on this runtime.
//
// +k8s:required
VersionedSandboxCompat version = 1;
// TODO: Add an informational SandboxConfig version name (e.g. "gvisor-2")
// once workers report one.
//
// [Post-GA] Additional compatibility versions (excluding `version`) that can
// warm-boot directly onto this runtime. When the actor suspends again, its
// new snapshot is stamped with the primary `version`.
// repeated VersionedSandboxCompat compatible_versions = 3;
}
// Concrete runtime and compatibility identity recorded on an actor's
// assignment or snapshot.
message ActorSandboxRuntime {
// e.g. "gvisor", "microvm"
//
// +k8s:required
// +k8s:maxLength=63
// +k8s:customValidation # gvisor or microvm
string sandbox_class = 1;
// Compatibility version stamped onto an actor's assignment or snapshot.
//
// +k8s:required
VersionedSandboxCompat version = 2;
}
message RegisterWorkerRequest {
ObjectRef worker = 1;
WorkerResources capacity = 2;
// [Removed] SandboxRuntime default_runtime = 3;
// [Removed] repeated SandboxRuntime restorable_runtimes = 4;
// +k8s:required
// +k8s:minItems=1
// +k8s:maxItems=1 # keep in sync with WorkerPoolSpec.sandboxClasses
// +k8s:listType=map
// +k8s:listMapKey=sandbox_class
repeated WorkerSandboxClass sandbox_classes = 3;
}Reactions are currently unavailable
Activity
shrutiyam-glitch commentedon Oct 20, 2026
cc: Zoe Zhao (@zoez7)
atelet, ateapi: report worker runtimes per sandbox class (agent-subst…