fix(sandbox): bound and place Kubernetes session Pods, pre-pull the image

[yannickmonney]

On Kubernetes, session Pods declared no node-disk (ephemeral-storage) requests or limits. The DinD inner Docker store reused the 4Gi workspace size, so a single large docker pull got the whole session evicted. Meanwhile sessions requesting 0 disk could push a node into DiskPressure, and the kubelet would evict platform Pods to relieve it. Session Pods also had no placement controls, so untrusted (possibly privileged DinD) sessions scheduled next to the database and platform Pods. And the first session on each new node pulled the ~2 GB runtime image inside its 180 s create budget.

Changes

fix(sandbox): bound the node disk of Kubernetes session Pods

Session Pod containers declared only cpu and memory, and a DinD agent's
inner Docker store borrowed the 4 GiB workspace size. One pull of a large
build image unpacks to several GiB, so an emptyDir past its sizeLimit got
the whole session evicted; meanwhile a session requesting no ephemeral
storage could write until the node hit DiskPressure and the kubelet
evicted platform Pods beside it.

The runner now requests 256Mi of ephemeral-storage and is limited to a
2Gi headroom for its root filesystem and logs (the kubelet rotates logs
at 10Mi x 5 by default) plus every disk-backed emptyDir it mounts, since
the kubelet evicts on writable layers + logs + emptyDirs against the sum
of container limits: 2Gi for an agent (its workspace is a PVC), 6Gi for
a crawler render (4Gi workspace emptyDir), 22Gi for a DinD agent. The
inner Docker store gets its own SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT
(default 20Gi). The egress sidecar declares 16Mi/128Mi so a quota on
ephemeral storage still admits the Pod and the Pod-level sum stays
honest.

SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST / _LIMIT override the request and
the headroom; they and the store and workspace sizes are validated as
Kubernetes quantities at boot, and the sizes and limit refuse zero. The
workspace size is now validated too, because the Pod sums it.

feat(sandbox): place Kubernetes session Pods by selector and priority

Session Pods carried no nodeSelector, tolerations or priorityClassName,
so untrusted sessions (privileged under runc DinD) scheduled on any node,
beside the database and platform Pods, and competed with them for CPU,
memory and disk at equal priority.

SANDBOX_K8S_NODE_SELECTOR (a JSON object of node labels),
SANDBOX_K8S_TOLERATIONS (a JSON array of Pod tolerations) and
SANDBOX_K8S_PRIORITY_CLASS now set those fields on every session Pod,
crawler renders included; unset, they are omitted as before. The values
are held to the apiserver's rules at boot (label key and value syntax,
toleration fields, operator, effect, Exists without a value, a keyless
toleration only with Exists, tolerationSeconds only for NoExecute, a
DNS-subdomain class name), so a typo stops the spawner with a message
naming the variable instead of failing every create with a 422.

The install guide shows the label, taint, env and a PriorityClass at
value -10 with preemptionPolicy Never: the scheduler may preempt a
session for a platform Pod, never the reverse. No RBAC change: the
spawner's Pod create needs no extra verb for these fields.

docs(sandbox): ship an optional image pre-pull DaemonSet for Kubernetes

On Kubernetes the spawner's warmImage is a no-op and the kubelet pulls
the runtime image per Pod, so the first session on each node downloads
it (about 2 GB compressed, 5.1-5.5 GB unpacked as measured in CI) inside
its 180 s create budget, and kubelet image garbage collection can remove
it again once no Pod uses it.

The Kubernetes install guide (EN/DE/FR) now carries an optional
45-sandbox-prepull.yaml: a DaemonSet whose init containers run the
runtime, egress and gateway images with a no-op sh -c 'exit 0'
behind a pause container (1m CPU / 4Mi requests each, non-root,
read-only, all capabilities dropped), so the images stay referenced on
every node while the Pod exists. Its tags follow ${VERSION} like the
other manifests; the guide says to add it to the apply loop, to bump a
separately applied copy with each release, to apply it first on an
upgrade, and to give it the session placement from
SANDBOX_K8S_NODE_SELECTOR / SANDBOX_K8S_TOLERATIONS. The sandbox
Kubernetes contract points at it, and the guide's verified-scope note
says the node-disk limits, the placement settings and this DaemonSet
were not part of the recorded cluster run.

docs(sandbox): state the kubelet's disk-pressure eviction order exactly

The node-disk and placement notes said the kubelet ranks Pods by priority first. Under node pressure it first takes Pods whose use exceeds their request, then orders those by priority and by how far they exceed it; a session's 256Mi ephemeral-storage request and a lower PriorityClass both work through that order, so the contract and the pod-spec comment now say so.

fix(sandbox): zero the egress sidecar's disk request with the runner's

The egress sidecar requested a fixed 16Mi of ephemeral-storage whatever
SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST said. The documented fix for nodes
that report no ephemeral-storage capacity (a kubelet with
localStorageCapacityIsolation: false, as on some rootless clusters) is a
request of 0, but with transparent egress on, the default on runc,
sysbox and kata, every non-DinD session Pod still asked for 16Mi. The
scheduler and the kubelet's admission check refuse any request above 0
on such a node, so each create waited out its 180 s budget and failed
with no IP; before the disk bounds these Pods requested no disk and
scheduled.

A zero runner request now zeroes the sidecar's too, as an explicit '0':
a container with a limit and no request gets its limit (128Mi) as the
request. Any other request keeps the sidecar at 16Mi.

fix(sandbox): name the scheduler's reason when a session Pod stays pending

The placement and disk settings make it easy to leave a session Pod
unschedulable: a selector no node matches, a taint the tolerations miss,
or an ephemeral-storage request on a node that reports no capacity. The
create then polled only status.podIP, waited out its whole budget
(SANDBOX_SESSION_CREATE_TIMEOUT_MS, 180 s by default) and failed with a
generic error, so every turn burned three minutes and the cause showed
only in kubectl describe pod.

The wait now reads the PodScheduled condition of each Pod it polls. When
the scheduler reports Unschedulable, the spawner logs its message once,
and the error the wait ends with (its own deadline or the create budget's
abort) carries the message of the last read. The create still waits the
full budget, because a cluster autoscaler can add a node meanwhile.

docs(sandbox): apply the image pre-pull before the rest on every upgrade

The Kubernetes guide said both to add 45-sandbox-prepull.yaml to the
apply loop and to apply it first and wait for its rollout on an upgrade.
The loop applies its files in order with no wait, so the DaemonSet landed
after 40-sandbox.yaml: the spawner rolled to the new runtime tag and
created sessions while the DaemonSet was still pulling, and those first
sessions pulled the ~2 GB image inside their 180 s create budget, which
is the cost the DaemonSet exists to remove.

The guide (EN/DE/FR) now shows the exact sequence: apply the namespace
file (the DaemonSet's namespace must exist on a first install, and it is
unchanged on an upgrade), apply the pre-pull through envsubst, wait with
kubectl -n tale rollout status ds/sandbox-image-prepull --timeout=15m,
then run the existing loop. The sandbox Kubernetes contract says the
same.

fix(sandbox): warn when the workspace size no longer bounds the DinD store

The Kubernetes docs used to say SANDBOX_K8S_WORKSPACE_SIZE_LIMIT also
bounds a DinD session's inner Docker store, and the Pod used it as that
emptyDir's sizeLimit. The store now has its own setting,
SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT, defaulting to 20Gi. An operator
who lowered the workspace size to keep DinD stores small on small node
disks would silently get 20Gi stores and a 22Gi Pod ephemeral-storage
limit after the upgrade, with the request still 256Mi.

loadConfig now logs one line at boot when the backend is kubernetes,
sessions can run Docker inside (DinD on and SANDBOX_DOCKER_WORKLOADS not
none), the workspace size is set and the store size is not; the line
names the variable that sizes the store and its default, taken from the
pod spec's own constant. The install guide (EN/DE/FR) and the sandbox
Kubernetes contract carry the matching upgrade note.

fix(sandbox): let a Docker spawner boot past stray Kubernetes settings

The new Kubernetes session settings (workspace, ephemeral-storage and
inner-store sizes, node selector, tolerations, priority class) were
parsed and refused at boot on every backend. A Docker deployment whose
shared env file carried an inert, malformed value, such as a workspace
size of 4g that main accepted as any string, would no longer start.
Only the Kubernetes backend, which reads them, now refuses an unreadable
value; any other backend warns and ignores it, as it already did for a
Kubernetes setting it does not use.

A toleration's empty operator or effect now counts as omitted, the
apiserver's own reading (Equal, every effect), so a toleration copied
from an existing manifest no longer stops the boot. The pod-spec comment
on container log rotation now states the kubelet default: 10 MiB per
file, five files.

Verification

• services/sandbox: config.test.ts and src/backend/kubernetes/ pass (175 tests). That covers node-disk bounds per profile, an egress sidecar request of 0 when the runner's is 0, placement on every profile, the scheduler's reason on a pending Pod, the inner-store warning, and a Docker spawner booting past an unreadable Kubernetes-only setting.
• server.test.ts and import-footprint.test.ts pass, @tale/sandbox typecheck and lint pass, the docs suite passes (685 tests, including EN/DE/FR locale checks), and lint:manual and lint:links pass.
• Adversarial review found one major issue, now fixed: the egress sidecar's fixed 16Mi request kept Pods unschedulable on nodes that report no ephemeral-storage capacity, even with the documented 0 workaround. Its minors and nits are fixed too: the scheduler's reason now shows in the create error, the pre-pull upgrade steps were corrected, there's a boot warning when the workspace size no longer bounds the DinD store, and the toleration parsing and Docker-backend leniency nits.
• Not run on a cluster. The kubelet accounting and eviction behaviour come from Kubernetes semantics, not an observed run.

Upgrade notes

• New session Pods carry ephemeral-storage requests and limits. Nodes that report no ephemeral-storage capacity need SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST=0. A namespace LimitRange whose ephemeral-storage max is below 22Gi rejects DinD agent Pods.
• The DinD inner Docker store is now sized by SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT, default 20Gi (previously the 4Gi workspace size). The boot log says so when only the workspace size is set.
• All placement settings are opt-in: SANDBOX_K8S_NODE_SELECTOR, SANDBOX_K8S_TOLERATIONS and SANDBOX_K8S_PRIORITY_CLASS.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).