diff --git a/.env.example b/.env.example index 40635cca90..8fc0b1dd07 100644 --- a/.env.example +++ b/.env.example @@ -480,6 +480,21 @@ TALE_AUDIT_PEPPER=9c1e5b7d3a2f8064e1c7a9d5b3f2e8c604a7d1b9e5c3f8a2d6b0e4c7a1f9d3 # (read by sandbox-egress; recreate it to apply). A higher value needs the # egress container's pids and open-file limits raised with it. # SANDBOX_EGRESS_MAX_CLIENTS=2000 +# Kubernetes only: a session Pod's node disk. The runner requests REQUEST and +# may write LIMIT outside its sized volumes (root filesystem, logs); the Pod's +# limit adds a crawler render's workspace and a Docker-inside agent's inner +# Docker store (DOCKER_STORAGE_SIZE_LIMIT), so a session past it is evicted +# on its own instead of filling the node. +# SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST=256Mi +# SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT=2Gi +# SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT=20Gi +# Kubernetes only: where session Pods run. Node labels as a JSON object, Pod +# tolerations as a JSON array and a PriorityClass name, applied to every +# session Pod, crawler renders included; unset, the scheduler places them on +# any node. A malformed value stops the spawner at start. +# SANDBOX_K8S_NODE_SELECTOR='{"tale.dev/sandbox":"true"}' +# SANDBOX_K8S_TOLERATIONS='[{"key":"tale.dev/sandbox","operator":"Exists","effect":"NoSchedule"}]' +# SANDBOX_K8S_PRIORITY_CLASS=tale-sandbox-session # ============================================================================ # Object store — the deployment's blob backend diff --git a/docs/de/self-hosted/configuration/environment-reference.md b/docs/de/self-hosted/configuration/environment-reference.md index 6809bb3629..a02957e45c 100644 --- a/docs/de/self-hosted/configuration/environment-reference.md +++ b/docs/de/self-hosted/configuration/environment-reference.md @@ -274,6 +274,10 @@ Der Sandbox-Spawner liest die folgenden Einstellungen. Übergib sie seiner Umgeb | `SANDBOX_SESSION_RELEASED_IDLE_MS` | `300000` (5 Min.) | Leerlauffenster für eine freigegebene Session, also eine, deren Aufgabe oder Lauf beendet ist und die keine andere Arbeit mehr belegt. Eine nicht angepinnte, freigegebene Session stoppt nach dieser Leerlaufzeit, statt `SANDBOX_SESSION_MAX_IDLE_MS` abzuwarten; die nächste Runde setzt sie auf ihrem erhaltenen Arbeitsverzeichnis fort. Agent-Sessions mit Docker in der Sandbox behalten das volle Leerlauffenster, weil ihr innerer Docker-Daemon bei jeder Fortsetzung mit einem leeren Image-Speicher startet. | | `SANDBOX_RUNTIME_IMAGE` | `tale-sandbox-runtime:latest` | **Optional, vom Spawner gelesen.** Das Image, aus dem jeder Session-Container entsteht. Der Default ist der Tag, den der Entwicklungs-Stack lokal baut; ein Host, der seine Images zieht, setzt deshalb den Registry-Namen: `ghcr.io/tale-project/tale/tale-sandbox-runtime:`, passend zum Rest des Stacks. `tale deploy` setzt ihn für dich. | | `SANDBOX_DIND_INNER_POOL` | nicht gesetzt (automatisch) | Optionaler Adresspool für den inneren Docker-Daemon in Agent-Sessions auf Docker oder Kubernetes. Verwende ein kanonisches privates IPv4-`/16` nach RFC1918 außerhalb deiner Pod-, Service- und VPC-Netze. Die Runtime lehnt Überschneidungen mit erkannten Netzen und Adressen ab. | +| `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` | `20Gi` | **Nur Kubernetes.** Die Größe des Speichers für das innere Docker einer Agent-Session, die Docker in ihrer Sandbox ausführt. Wächst eine Session darüber hinaus, räumt Kubernetes ihren Pod (Eviction); bemiss den Wert deshalb nach den größten Images, die deine Agents laden und bauen. Die übrigen Kubernetes-Einstellungen findest du unter [Auf Kubernetes bereitstellen](/de/self-hosted/install/kubernetes). | +| `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` | `256Mi` / `2Gi` | **Nur Kubernetes.** Wie viel Platz auf der Festplatte des Nodes ein Session-Pod anfordert und wie viel er dort außerhalb seines temporären Workspaces und des Speichers für das innere Docker schreiben darf. Die Grenze des Pods rechnet diese Speicher hinzu; überschreitet eine Session sie, räumt Kubernetes nur ihren Pod, statt dass sie den Node füllt. | +| `SANDBOX_K8S_NODE_SELECTOR` / `SANDBOX_K8S_TOLERATIONS` | nicht gesetzt | **Nur Kubernetes.** Die Node-Labels, die jeder Session-Pod erfüllen muss, als JSON-Objekt wie `{"tale.dev/sandbox":"true"}`, und die Taints, die er toleriert, als JSON-Array von Pod-Tolerations. So laufen Sessions auf eigenen Nodes statt neben der Datenbank und der Plattform. Ein fehlerhafter Wert verhindert den Start des Spawners. | +| `SANDBOX_K8S_PRIORITY_CLASS` | nicht gesetzt | **Nur Kubernetes.** Die PriorityClass jedes Session-Pods. Liegt sie unter der Klasse der Plattform, darf der Scheduler Sessions verdrängen, um Pods der Plattform unterzubringen. | | `SANDBOX_EGRESS_MAX_CLIENTS` | `2000` | **Vom Dienst `sandbox-egress` gelesen**, nicht vom Spawner: Erstelle diesen Dienst nach einer Änderung neu. Verbindungen, die der Egress-Proxy für alle Sessions zusammen gleichzeitig bedient; mit transparentem Egress läuft jede ausgehende Verbindung einer Session, ihrer Builds und ihrer Paketinstallationen über ihn. Jenseits der Grenze lehnt der Proxy neue Verbindungen ab, und Installationen und Seitenaufrufe brechen mit zurückgesetzten Verbindungen ab. Jede Verbindung belegt einen Thread und zwei offene Dateien, deshalb bemessen die mitgelieferten Compose-Dateien die Prozess- und Dateigrenzen des Proxys für den Default; ein höherer Wert braucht auch höhere Grenzen, und der Proxy warnt beim Start, wenn seine Grenze für offene Dateien zu niedrig ist. Beim Start hebt der Proxy seine Grenze für offene Dateien auf das an, was die Verbindungen brauchen, so weit die harte Grenze des Containers es erlaubt. Unter Kubernetes setzt du ihn am Egress-Deployment; eine Pod-Spezifikation setzt keine Prozessgrenze, daher muss das `podPidsLimit` des Nodes Platz für einen Thread je Verbindung lassen. | Die Backend-Einstellungen oben müssen API und Worker erreichen. Erstelle beide Dienste nach einer Änderung neu. Der Spawner prüft freien Arbeitsspeicher und Plattenplatz auch beim erneuten Belegen einer freigegebenen Sitzung; daher kann auch ein warmer Arbeitsbereich auf Kapazität warten. diff --git a/docs/de/self-hosted/install/kubernetes.md b/docs/de/self-hosted/install/kubernetes.md index d0a39f0703..3de1c9507e 100644 --- a/docs/de/self-hosted/install/kubernetes.md +++ b/docs/de/self-hosted/install/kubernetes.md @@ -643,14 +643,113 @@ spec: | `SANDBOX_K8S_NAMESPACE` | Der Namespace, in dem Sitzungs-Pods, Secrets und Workspace-Claims entstehen; in dieser Aufteilung der Namespace des Spawners selbst. Standard `tale-sandbox`. | | `SANDBOX_RUNTIME_IMAGE` | Das passende Tale-Sandbox-Runtime-Image, für jeden Node verfügbar. | | `NODE_EXTRA_CA_CERTS` | Die CA-Datei des Clusters, im Spawner normalerweise `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`. Sie ist der einzige CA-Vertrauensweg, den der Spawner beachtet; lass die TLS-Prüfung eingeschaltet. | -| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | Die Größe des `/agent`-Workspace-Claims jeder Agent-Session, Standard `4Gi`; begrenzt auch den temporären Workspace eines Crawler-Renders, der ohne Claim auskommt, weil er nie fortgesetzt wird, und bei aktiviertem innerem Docker dessen temporären Speicher. | +| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | Die Größe des `/agent`-Workspace-Claims jeder Agent-Session, Standard `4Gi`; begrenzt auch den temporären Workspace eines Crawler-Renders, der ohne Claim auskommt, weil er nie fortgesetzt wird. | +| `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` | Die Größe des temporären Speichers für das innere Docker einer Agent-Session, die Docker in ihrer Sandbox ausführt, Standard `20Gi`. Wächst eine Session darüber hinaus, räumt Kubernetes ihren Pod (Eviction); bemiss den Wert deshalb nach den größten Images, die deine Agents laden und bauen. Releases vor dieser Einstellung haben den Speicher mit `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` bemessen: Hast du diesen Wert gesenkt, um den Speicher klein zu halten, setze beim Upgrade auch diese Variable, sonst wächst der Speicher auf `20Gi`. Der Spawner warnt beim Start, wenn `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` ohne sie gesetzt ist. | +| `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` | Wie viel Platz auf der Festplatte des Nodes ein Session-Pod anfordert, Standard `256Mi`, und wie viel er dort außerhalb seines temporären Workspaces und Docker-Speichers schreiben darf, Standard `2Gi`. Die Grenze eines Pods rechnet diese Speicher hinzu: `2Gi` für einen Agent, `6Gi` für einen Crawler-Render, `22Gi` mit Docker in der Sandbox. Überschreitet eine Agent-Session ihre Grenze, räumt Kubernetes nur ihren Pod, und die nächste Aufgabe setzt sie mit unversehrtem Workspace fort, statt dass sie den Node füllt, bis Kubernetes die Pods der Plattform räumt. Melden deine Nodes keine Kapazität für Ephemeral Storage, etwa in manchen Rootless-Clustern, setze die Anforderung auf `0`. | | `SANDBOX_K8S_CPU_REQUEST` / `SANDBOX_K8S_MEMORY_REQUEST` | Was jeder Session-Pod beim Scheduler anfordert, als Kubernetes-Mengenangaben. Ohne Angabe fordert ein Agent-Pod `250m` und `512Mi` an (`1Gi` mit Docker in der Sandbox), ein Crawler-Render `250m` und `512Mi`; eine Anforderung liegt nie über der Grenze des Pods. Erhöhe die Werte, wenn Agents auf deinen Nodes schwerere Builds ausführen, damit der Scheduler nicht mehr Sessions auf einen Node legt, als er tragen kann. | +| `SANDBOX_K8S_NODE_SELECTOR` / `SANDBOX_K8S_TOLERATIONS` | Optional. Die Node-Labels, die jeder Session-Pod erfüllen muss, als JSON-Objekt, und die Taints, die er toleriert, als JSON-Array von Pod-Tolerations. Ohne Angabe kann eine Session auf jedem Node laufen; siehe unten. | +| `SANDBOX_K8S_PRIORITY_CLASS` | Optional. Die PriorityClass jedes Session-Pods. | | `SANDBOX_K8S_CACHE_STORAGECLASS` | Die StorageClass der Workspace-Claims; ohne Wert gilt der Cluster-Standard. | | `SANDBOX_RUNTIME` / `SANDBOX_RUNTIME_CLASS` | Eine unterstützte Laufzeitstufe und bei Bedarf der Name der installierten RuntimeClass. | | `SANDBOX_EGRESS_PROXY` | Der Egress-Service, den die Sitzungen nutzen, Standard `http://sandbox-egress:3128`. | Der Spawner skaliert horizontal. Jedes Replikat findet eine Sitzung, die es nicht selbst angelegt hat, über den deterministischen Pod-Namen und übernimmt sie; exec, Stopp und Zerstören funktionieren daher über jedes Replikat, das der Service auswählt. `SANDBOX_MAX_SESSIONS` zählt den Namespace, gleichzeitige Aufnahmen auf mehreren Replikaten können den Wert aber kurz überschreiten; eine ResourceQuota liefert die harte Grenze. Der [Kubernetes-Vertrag der Sandbox](https://github.com/tale-project/tale/blob/main/services/sandbox/docs/kubernetes.md) dokumentiert die Pod-Form und die Laufzeitdetails. +### Sessions von den Nodes der Plattform fernhalten + +Session-Pods führen den Code aus, den deine Agents schreiben, und mit Docker in der Sandbox laufen sie auf der Stufe `runc` privilegiert. Ohne Platzierungseinstellungen legt der Scheduler sie auf einen beliebigen Node, neben Postgres und die Anwendungsrollen, wo eine ausgelastete Session um dieselbe CPU, denselben Arbeitsspeicher und dieselbe Festplatte konkurriert. Damit Sessions eigene Nodes bekommen, versiehst du diese Nodes mit einem Label und einem Taint: + +```bash +kubectl label node tale.dev/sandbox=true +kubectl taint node tale.dev/sandbox=true:NoSchedule +``` + +Ergänze dann die passenden Einstellungen in der `env` des Spawners in `40-sandbox.yaml`: + +```yaml + - { name: SANDBOX_K8S_NODE_SELECTOR, value: '{"tale.dev/sandbox":"true"}' } + - { name: SANDBOX_K8S_TOLERATIONS, value: '[{"key":"tale.dev/sandbox","operator":"Exists","effect":"NoSchedule"}]' } + - { name: SANDBOX_K8S_PRIORITY_CLASS, value: tale-sandbox-session } +``` + +Der Taint hält andere Pods von diesen Nodes fern, der Selektor hält die Sessions dort. Die PriorityClass stuft Sessions unter der Plattform ein: Der Scheduler darf eine Session verdrängen, um einen Pod der Plattform unterzubringen, und wenn einem Node Arbeitsspeicher oder Festplatte ausgehen, berücksichtigt das Kubelet die Priorität und räumt Sessions früher. Mit `preemptionPolicy: Never` verdrängt eine Session selbst nie einen anderen Pod. Ein Cluster-Administrator legt die Klasse einmal an; der Spawner braucht dafür keine zusätzliche Berechtigung: + +```yaml +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: { name: tale-sandbox-session } +value: -10 +preemptionPolicy: Never +globalDefault: false +description: Tale sandbox sessions yield to the platform. +``` + +Die Einstellungen gelten für jeden Session-Pod, auch für Crawler-Renders, und zwar für die Sessions, die der Spawner nach seinem Neustart anlegt; laufende Sessions behalten ihre Platzierung. Ein fehlerhafter Wert verhindert den Start des Spawners. Passt ein gültiger Selektor auf keinen Node oder fehlt die Toleration für einen Taint, bleibt ein Session-Pod ausstehend: Der Spawner protokolliert den Grund des Schedulers, und das Anlegen schlägt mit diesem Grund fehl, sobald sein Startbudget abgelaufen ist. Bei Node-lokalem Speicher bleibt der Workspace-Claim einer gestoppten Session auf ihrem Node; behalte diesen Node im Selektor, sonst lässt sich die Session nicht fortsetzen. + +### Sandbox-Images vorab laden + +Auf Kubernetes lädt nicht der Spawner die Images, sondern das Kubelet für jeden Pod. Die erste Session auf einem Node lädt deshalb das Runtime-Image, einen Download von etwa 2 GB, innerhalb ihres Startbudgets (`SANDBOX_SESSION_CREATE_TIMEOUT_MS`, standardmäßig 180 Sekunden), und die Image-Garbage-Collection des Kubelets kann das Image wieder entfernen, sobald kein Pod es nutzt. Dieses optionale DaemonSet lädt die Sandbox-Images vor der ersten Session auf jeden Node und hält sie dort: Jeder Init-Container startet ein Image und beendet sich sofort, der Pause-Container hält den Pod am Leben, und solange der Pod besteht, gelten diese Images für das Kubelet als in Gebrauch. Jeder Container fordert 1m CPU und 4 MiB Arbeitsspeicher an. + +```yaml +# 45-sandbox-prepull.yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: { name: sandbox-image-prepull, namespace: tale } +spec: + selector: { matchLabels: { app: sandbox-image-prepull } } + updateStrategy: { rollingUpdate: { maxUnavailable: 25% } } + template: + metadata: { labels: { app: sandbox-image-prepull } } + spec: + enableServiceLinks: false + automountServiceAccountToken: false + terminationGracePeriodSeconds: 0 + # Setzt du SANDBOX_K8S_NODE_SELECTOR und SANDBOX_K8S_TOLERATIONS, übernimm hier dieselben Werte: + # nodeSelector: { tale.dev/sandbox: 'true' } + # tolerations: [{ key: tale.dev/sandbox, operator: Exists, effect: NoSchedule }] + securityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: { type: RuntimeDefault } + initContainers: + - name: runtime + image: ghcr.io/tale-project/tale/tale-sandbox-runtime:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + - name: egress + image: ghcr.io/tale-project/tale/tale-sandbox-egress:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + - name: gateway + image: ghcr.io/tale-project/tale/tale-sandbox-llm-gateway:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + containers: + - name: pause + image: registry.k8s.io/pause:3.10 + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 16Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } +``` + +Entscheidend ist das Runtime-Image: Jeder Session-Pod und sein Egress-Sidecar starten daraus. Die Images von Egress und Gateway ersparen einen Download, wenn diese Deployments auf einen anderen Node wechseln; entferne ihre Init-Container, wenn Sessions auf eigenen Nodes laufen, die diese Deployments nie nutzen. Setzt du `SANDBOX_K8S_NODE_SELECTOR` und `SANDBOX_K8S_TOLERATIONS`, gib dem DaemonSet dieselben Werte, damit es genau auf den Session-Nodes läuft. + +Die Image-Tags folgen wie in jeder anderen Datei `${VERSION}`. Wende diese Datei separat vor der Schleife an und warte, bis jeder Node die Images geladen hat; bei einem Upgrade liegt das neue Runtime-Image dann bereit, bevor der Spawner darauf umstellt und Sessions daraus anlegt. Der erste Befehl legt bei einer Erstinstallation den Namespace an und ändert bei einem Upgrade nichts: + +```bash +envsubst '${VERSION}' < 00-namespace.yaml | kubectl apply -f - +envsubst '${VERSION}' < 45-sandbox-prepull.yaml | kubectl apply -f - +kubectl -n tale rollout status ds/sandbox-image-prepull --timeout=15m +for f in 00-namespace.yaml 10-stores.yaml 20-application.yaml 30-proxy.yaml 40-sandbox.yaml; do + envsubst '${VERSION}' < "$f" | kubectl apply -f - +done +``` + +Führe bei jedem Upgrade dieselben Befehle mit der neuen, exportierten `VERSION` aus. Eine Kopie, die du ohne `envsubst` anwendest oder bei einem Upgrade auslässt, behält ihre alten Tags: Sie hält alte Images auf den Nodes, und die erste Session auf jedem Node lädt das neue Image wieder selbst. + ### Was der Spawner durchsetzt Beim Start legt der Spawner die NetworkPolicy `tale-sandbox-session-egress` an: Sitzungs-Pods erreichen DNS und die Pods ihres eigenen Namespace und sonst nichts. Der Cloud-Metadatendienst, die Nodes und andere Namespaces bleiben damit auch für einen Prozess unerreichbar, der `HTTP_PROXY` ignoriert. Öffentliche Ziele laufen über `sandbox-egress`. Eine fehlende Berechtigung für `networkpolicies` wird protokolliert und stoppt den Spawner nicht; prüfe, dass die Richtlinie existiert, bevor du Arbeit zulässt. @@ -703,4 +802,4 @@ Snapshots, Blue-Green-Wechsel und Rollback-Prüfungen der CLI laufen auf Kuberne ## Geprüfter Umfang -Diese fünf Dateien wurden bis auf die Secret-Werte unverändert auf einem frischen kind-Cluster mit einem Node, Kubernetes 1.36, kube-network-policies, der StorageClass local-path und Tale 0.5.31 angewendet: Start und Migrationen, der öffentliche Zugang, die Einrichtung des ersten Inhabers samt Sandboxes-Karte, eine Agentenaufgabe mit Ergebnis, der Sitzungslebenszyklus einschließlich Leerlaufstopp, Fortsetzung und replikatübergreifendem Zugriff, die Sperrprüfungen oben und ein Rolling Restart der API mit zwei Replikaten. Mehrere Nodes mit `ReadWriteMany`-Konfigurationsspeicher, Docker in Sitzungen auf einer sysbox- oder kata-RuntimeClass, ein Ingress mit `TLS_MODE=external` und hochverfügbare Speicher waren nicht Teil dieses Laufs. +Diese fünf Dateien wurden bis auf die Secret-Werte unverändert auf einem frischen kind-Cluster mit einem Node, Kubernetes 1.36, kube-network-policies, der StorageClass local-path und Tale 0.5.31 angewendet: Start und Migrationen, der öffentliche Zugang, die Einrichtung des ersten Inhabers samt Sandboxes-Karte, eine Agentenaufgabe mit Ergebnis, der Sitzungslebenszyklus einschließlich Leerlaufstopp, Fortsetzung und replikatübergreifendem Zugriff, die Sperrprüfungen oben und ein Rolling Restart der API mit zwei Replikaten. Mehrere Nodes mit `ReadWriteMany`-Konfigurationsspeicher, Docker in Sitzungen auf einer sysbox- oder kata-RuntimeClass, ein Ingress mit `TLS_MODE=external` und hochverfügbare Speicher waren nicht Teil dieses Laufs. Die Festplattengrenzen und Platzierungseinstellungen der Session-Pods sowie das optionale Pre-Pull-DaemonSet kamen später hinzu und waren ebenfalls nicht Teil davon. diff --git a/docs/en/self-hosted/configuration/environment-reference.md b/docs/en/self-hosted/configuration/environment-reference.md index cc68009a4b..42510d19b3 100644 --- a/docs/en/self-hosted/configuration/environment-reference.md +++ b/docs/en/self-hosted/configuration/environment-reference.md @@ -274,6 +274,10 @@ The sandbox spawner reads the settings below. Pass them into its environment and | `SANDBOX_SESSION_RELEASED_IDLE_MS` | `300000` (5 min) | Idle window for a released session: one whose task or run has finished and that no other work holds. An unpinned released session idle this long is stopped instead of waiting out `SANDBOX_SESSION_MAX_IDLE_MS`; the next turn resumes it on its preserved workspace. Agent sessions with Docker inside the sandbox keep the full idle window, because each resume starts their inner Docker daemon with an empty image store. | | `SANDBOX_RUNTIME_IMAGE` | `tale-sandbox-runtime:latest` | **Optional, read by the spawner.** The image every session container is created from. The default is the tag the development stack builds locally, so a host that pulls its images sets the registry one: `ghcr.io/tale-project/tale/tale-sandbox-runtime:`, matching the rest of the stack. `tale deploy` sets it for you. | | `SANDBOX_DIND_INNER_POOL` | unset (automatic) | Optional inner Docker address pool for agent sessions on Docker or Kubernetes. Use a canonical RFC1918 IPv4 `/16` outside your Pod, Service and VPC networks. The runtime rejects overlaps with networks and addresses it discovers. | +| `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` | `20Gi` | **Kubernetes only.** Size of the inner Docker store of an agent session that runs Docker inside its sandbox. A session that outgrows it is evicted, so size it for the largest images your agents pull and build. The other Kubernetes settings are listed in [Deploy on Kubernetes](/self-hosted/install/kubernetes). | +| `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` | `256Mi` / `2Gi` | **Kubernetes only.** How much of the node's disk a session Pod requests, and how much it may write there outside its temporary workspace and inner Docker store. The Pod's limit adds those stores, so a session past it is evicted on its own instead of filling the node. | +| `SANDBOX_K8S_NODE_SELECTOR` / `SANDBOX_K8S_TOLERATIONS` | unset | **Kubernetes only.** The node labels every session Pod must match, as a JSON object such as `{"tale.dev/sandbox":"true"}`, and the taints it tolerates, as a JSON array of Pod tolerations, so sessions run on nodes of their own instead of beside the database and the platform. A malformed value stops the spawner at start. | +| `SANDBOX_K8S_PRIORITY_CLASS` | unset | **Kubernetes only.** The PriorityClass of every session Pod. A class below the platform's lets the scheduler preempt sessions to place platform Pods. | | `SANDBOX_EGRESS_MAX_CLIENTS` | `2000` | **Read by the `sandbox-egress` service**, not the spawner: recreate that service after a change. Connections the egress proxy serves at once for all sessions together; with transparent egress, every outbound connection of a session, its builds and its package installs goes through it. Past the limit the proxy refuses new connections, and installs and page loads fail with connection resets. Each connection takes a thread and two open files, so the shipped compose files size the proxy's process and open-file limits for the default; a higher value needs them raised too, and the proxy warns at start when its open-file limit is too low. At start the proxy raises its open-file limit to what the connections need, as far as the container's hard limit allows. On Kubernetes, set it on the egress Deployment; a Pod spec sets no process limit, so the node's `podPidsLimit` must leave room for one thread per connection. | The backend settings above must reach both API and worker services. Recreate them after a change. The spawner checks memory and disk headroom when a released session is acquired again, so a warm workspace can wait for capacity too. diff --git a/docs/en/self-hosted/install/kubernetes.md b/docs/en/self-hosted/install/kubernetes.md index 7631f3b3f1..1770858480 100644 --- a/docs/en/self-hosted/install/kubernetes.md +++ b/docs/en/self-hosted/install/kubernetes.md @@ -643,14 +643,113 @@ spec: | `SANDBOX_K8S_NAMESPACE` | The namespace the session Pods, Secrets, and workspace claims are created in; the spawner's own namespace in this layout. Defaults to `tale-sandbox`. | | `SANDBOX_RUNTIME_IMAGE` | The matching Tale sandbox runtime image, available to every node. | | `NODE_EXTRA_CA_CERTS` | The cluster CA file, normally `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` inside the spawner. It is the only CA-trust mechanism the spawner honors; keep TLS verification on. | -| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | The size of each agent session's `/agent` workspace claim, default `4Gi`; also bounds a crawler render's temporary workspace, which needs no claim because it is never resumed, and the inner Docker temporary store when that is enabled. | +| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | The size of each agent session's `/agent` workspace claim, default `4Gi`; also bounds a crawler render's temporary workspace, which needs no claim because it is never resumed. | +| `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` | The size of the inner Docker temporary store of an agent session that runs Docker inside its sandbox, default `20Gi`. A session that outgrows it is evicted, so size it for the largest images your agents pull and build. Releases before this setting sized the store with `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT`: if you lowered that to keep the store small, set this variable as well when you upgrade, or the store grows to `20Gi`. The spawner warns at start when `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` is set without it. | +| `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` | How much of the node's disk a session Pod requests, default `256Mi`, and how much it may write there outside its temporary workspace and Docker store, default `2Gi`. A Pod's limit adds those stores: `2Gi` for an agent, `6Gi` for a crawler render, `22Gi` with Docker inside the sandbox. An agent session past its limit is evicted on its own and the next task resumes it with its workspace intact, instead of filling the node until Kubernetes evicts the platform's Pods. On nodes that report no ephemeral-storage capacity, as on some rootless clusters, set the request to `0`. | | `SANDBOX_K8S_CPU_REQUEST` / `SANDBOX_K8S_MEMORY_REQUEST` | What every session Pod requests from the scheduler, as Kubernetes quantities. Unset, an agent Pod requests `250m` and `512Mi` (`1Gi` with Docker inside the sandbox) and a crawler render `250m` and `512Mi`; a request never exceeds the Pod's limit. Raise them when agents on your nodes run heavier builds than this, so the scheduler does not place more sessions than a node can hold. | +| `SANDBOX_K8S_NODE_SELECTOR` / `SANDBOX_K8S_TOLERATIONS` | Optional. The node labels every session Pod must match, as a JSON object, and the taints it tolerates, as a JSON array of Pod tolerations. Unset, sessions can run on any node; see below. | +| `SANDBOX_K8S_PRIORITY_CLASS` | Optional. The PriorityClass of every session Pod. | | `SANDBOX_K8S_CACHE_STORAGECLASS` | The StorageClass for workspace claims; unset uses the cluster default. | | `SANDBOX_RUNTIME` / `SANDBOX_RUNTIME_CLASS` | A supported runtime tier and, when needed, the installed RuntimeClass name. | | `SANDBOX_EGRESS_PROXY` | The egress Service the sessions use, default `http://sandbox-egress:3128`. | The spawner scales horizontally. Any replica resolves a session it did not create by its deterministic Pod name and adopts it, so exec, stop, and destroy work through whichever replica the Service picks. `SANDBOX_MAX_SESSIONS` counts the namespace, but simultaneous admissions on several replicas can exceed it briefly; use a ResourceQuota for a hard bound. The [sandbox Kubernetes contract](https://github.com/tale-project/tale/blob/main/services/sandbox/docs/kubernetes.md) documents the Pod shape and runtime details. +### Keep sessions off the platform's nodes + +Session Pods run the code your agents write, and with Docker inside the sandbox on the `runc` tier they run privileged. Without placement settings, the scheduler puts them on any node, next to Postgres and the application roles, where a busy session competes for the same CPU, memory, and disk. To give sessions nodes of their own, label and taint those nodes: + +```bash +kubectl label node tale.dev/sandbox=true +kubectl taint node tale.dev/sandbox=true:NoSchedule +``` + +Then add the matching settings to the spawner's `env` in `40-sandbox.yaml`: + +```yaml + - { name: SANDBOX_K8S_NODE_SELECTOR, value: '{"tale.dev/sandbox":"true"}' } + - { name: SANDBOX_K8S_TOLERATIONS, value: '[{"key":"tale.dev/sandbox","operator":"Exists","effect":"NoSchedule"}]' } + - { name: SANDBOX_K8S_PRIORITY_CLASS, value: tale-sandbox-session } +``` + +The taint keeps other Pods off those nodes, and the selector keeps sessions on them. The PriorityClass ranks sessions below the platform: the scheduler may preempt a session to place a platform Pod, and when a node runs short of memory or disk, the kubelet takes priority into account and evicts sessions sooner. With `preemptionPolicy: Never`, a session never preempts another Pod itself. A cluster administrator creates the class once; the spawner needs no extra permission: + +```yaml +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: { name: tale-sandbox-session } +value: -10 +preemptionPolicy: Never +globalDefault: false +description: Tale sandbox sessions yield to the platform. +``` + +The settings apply to every session Pod, crawler renders included, for the sessions the spawner creates after its restart; running sessions keep their placement. A malformed value stops the spawner at start. A valid selector that no node matches, or a taint the tolerations miss, leaves a session Pod pending: the spawner logs the scheduler's reason, and the create fails with that reason once its startup budget runs out. With node-local storage, a stopped session's workspace claim stays on its node, so keep that node inside the selector or the session cannot resume. + +### Pre-pull the sandbox images + +The spawner does not pull images on Kubernetes; the kubelet does, per Pod. The first session on a node therefore pulls the runtime image, a download of about 2 GB, inside its startup budget (`SANDBOX_SESSION_CREATE_TIMEOUT_MS`, 180 seconds by default), and the kubelet's image garbage collection can remove the image again once no Pod uses it. This optional DaemonSet pulls the sandbox images onto every node before the first session and keeps them there: each init container starts one image and exits at once, the pause container keeps the Pod alive, and while the Pod exists the kubelet counts those images as in use. Each container requests 1m of CPU and 4 MiB of memory. + +```yaml +# 45-sandbox-prepull.yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: { name: sandbox-image-prepull, namespace: tale } +spec: + selector: { matchLabels: { app: sandbox-image-prepull } } + updateStrategy: { rollingUpdate: { maxUnavailable: 25% } } + template: + metadata: { labels: { app: sandbox-image-prepull } } + spec: + enableServiceLinks: false + automountServiceAccountToken: false + terminationGracePeriodSeconds: 0 + # Match SANDBOX_K8S_NODE_SELECTOR and SANDBOX_K8S_TOLERATIONS when you set them: + # nodeSelector: { tale.dev/sandbox: 'true' } + # tolerations: [{ key: tale.dev/sandbox, operator: Exists, effect: NoSchedule }] + securityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: { type: RuntimeDefault } + initContainers: + - name: runtime + image: ghcr.io/tale-project/tale/tale-sandbox-runtime:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + - name: egress + image: ghcr.io/tale-project/tale/tale-sandbox-egress:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + - name: gateway + image: ghcr.io/tale-project/tale/tale-sandbox-llm-gateway:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + containers: + - name: pause + image: registry.k8s.io/pause:3.10 + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 16Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } +``` + +The runtime image is the one that matters: every session Pod and its egress sidecar start from it. The egress and gateway images spare a pull when those Deployments move to another node; remove their init containers when sessions run on dedicated nodes those Deployments never use. If you set `SANDBOX_K8S_NODE_SELECTOR` and `SANDBOX_K8S_TOLERATIONS`, give the DaemonSet the same values so it runs on exactly the session nodes. + +The image tags follow `${VERSION}` like every other file. Apply this file on its own, before the loop, and wait until every node has pulled the images; on an upgrade, the new runtime image is then in place before the spawner rolls to it and creates sessions from it. The first command creates the namespace on a first install and changes nothing on an upgrade: + +```bash +envsubst '${VERSION}' < 00-namespace.yaml | kubectl apply -f - +envsubst '${VERSION}' < 45-sandbox-prepull.yaml | kubectl apply -f - +kubectl -n tale rollout status ds/sandbox-image-prepull --timeout=15m +for f in 00-namespace.yaml 10-stores.yaml 20-application.yaml 30-proxy.yaml 40-sandbox.yaml; do + envsubst '${VERSION}' < "$f" | kubectl apply -f - +done +``` + +Run the same commands for every upgrade, with the new `VERSION` exported. A copy applied without `envsubst`, or left out of an upgrade, keeps its old tags: it holds old images on the nodes, and the first session on each node pulls the new one again. + ### What the spawner enforces At start the spawner applies the NetworkPolicy `tale-sandbox-session-egress`: session Pods may reach DNS and the Pods of their own namespace and nothing else, so the cloud metadata service, the nodes, and other namespaces stay unreachable even for a process that ignores `HTTP_PROXY`. Public destinations pass through `sandbox-egress`. A missing `networkpolicies` permission is logged and does not stop the spawner; verify that the policy exists before you admit workloads. @@ -703,4 +802,4 @@ The CLI's snapshots, blue-green flips, and rollback checks do not run on Kuberne ## Verified scope -These five files were applied unchanged, apart from the Secret values, to a fresh single-node kind cluster with Kubernetes 1.36, kube-network-policies, the local-path StorageClass, and Tale 0.5.31: boot and migrations, the public edge, the first-owner setup and the Sandboxes card, an agent task with a deliverable, the session lifecycle including idle stop, resume, and cross-replica access, the fence probes above, and a rolling restart of the API with two replicas. Multi-node scheduling with `ReadWriteMany` configuration storage, Docker inside sessions on a sysbox or kata RuntimeClass, an Ingress with `TLS_MODE=external`, and highly available stores were not part of that run. +These five files were applied unchanged, apart from the Secret values, to a fresh single-node kind cluster with Kubernetes 1.36, kube-network-policies, the local-path StorageClass, and Tale 0.5.31: boot and migrations, the public edge, the first-owner setup and the Sandboxes card, an agent task with a deliverable, the session lifecycle including idle stop, resume, and cross-replica access, the fence probes above, and a rolling restart of the API with two replicas. Multi-node scheduling with `ReadWriteMany` configuration storage, Docker inside sessions on a sysbox or kata RuntimeClass, an Ingress with `TLS_MODE=external`, and highly available stores were not part of that run. The node-disk limits and placement settings of session Pods and the optional pre-pull DaemonSet came later and were not part of it either. diff --git a/docs/fr/self-hosted/configuration/environment-reference.md b/docs/fr/self-hosted/configuration/environment-reference.md index 2ebacb4ba4..a074fedee2 100644 --- a/docs/fr/self-hosted/configuration/environment-reference.md +++ b/docs/fr/self-hosted/configuration/environment-reference.md @@ -274,6 +274,10 @@ Le spawner sandbox lit les paramètres ci-dessous. Transmets-les dans son enviro | `SANDBOX_SESSION_RELEASED_IDLE_MS` | `300000` (5 min) | Délai d’inactivité d’une session libérée, c’est-à-dire dont la tâche ou l’exécution est terminée et qu’aucun autre travail n’occupe plus. Une session libérée et non épinglée, inactive pendant ce délai, est arrêtée sans attendre `SANDBOX_SESSION_MAX_IDLE_MS` ; le tour suivant la reprend sur son espace de travail conservé. Les sessions d’agent avec Docker dans la sandbox gardent le délai complet, car à chaque reprise leur daemon Docker interne repart d’un stockage d’images vide. | | `SANDBOX_RUNTIME_IMAGE` | `tale-sandbox-runtime:latest` | **Optionnel, lu par le spawner.** L'image dont sort chaque conteneur de session. Le défaut est le tag que la stack de développement construit localement ; un hôte qui tire ses images pose donc le nom de la registry : `ghcr.io/tale-project/tale/tale-sandbox-runtime:`, aligné sur le reste de la stack. `tale deploy` le pose pour toi. | | `SANDBOX_DIND_INNER_POOL` | non défini (automatique) | Pool d’adresses optionnel du daemon Docker interne des sessions d’agent, sur Docker ou Kubernetes. Choisis un `/16` IPv4 privé RFC1918 sous forme canonique, hors de tes réseaux de Pods, de Services et de VPC. Le runtime refuse tout chevauchement avec les réseaux et adresses qu’elle détecte. | +| `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` | `20Gi` | **Kubernetes uniquement.** La taille du stockage du Docker interne d’une session d’agent qui exécute Docker dans sa sandbox. Une session qui le dépasse est évincée ; dimensionne-le donc pour les plus grosses images que tes agents téléchargent et construisent. Les autres paramètres Kubernetes figurent dans [Déployer sur Kubernetes](/fr/self-hosted/install/kubernetes). | +| `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` | `256Mi` / `2Gi` | **Kubernetes uniquement.** L’espace disque du nœud qu’un Pod de session demande, et ce qu’il peut y écrire en dehors de son workspace temporaire et du stockage du Docker interne. La limite du Pod y ajoute ces stockages : une session qui la dépasse est évincée seule au lieu de remplir le nœud. | +| `SANDBOX_K8S_NODE_SELECTOR` / `SANDBOX_K8S_TOLERATIONS` | non défini | **Kubernetes uniquement.** Les labels de nœud auxquels chaque Pod de session doit correspondre, sous forme d’objet JSON comme `{"tale.dev/sandbox":"true"}`, et les taints qu’il tolère, sous forme de tableau JSON de tolérances de Pod : les sessions tournent ainsi sur leurs propres nœuds plutôt qu’à côté de la base de données et de la plateforme. Une valeur mal formée empêche le spawner de démarrer. | +| `SANDBOX_K8S_PRIORITY_CLASS` | non défini | **Kubernetes uniquement.** La PriorityClass de chaque Pod de session. Une classe inférieure à celle de la plateforme permet au planificateur de préempter des sessions pour placer les Pods de la plateforme. | | `SANDBOX_EGRESS_MAX_CLIENTS` | `2000` | **Lu par le service `sandbox-egress`**, pas par le spawner : recrée ce service après un changement. Connexions que le proxy egress sert en même temps pour toutes les sessions réunies ; avec la sortie transparente, chaque connexion sortante d’une session, de ses builds et de ses installations de paquets passe par lui. Au-delà de la limite, le proxy refuse les nouvelles connexions, et les installations et chargements de pages échouent sur des connexions réinitialisées. Chaque connexion occupe un thread et deux fichiers ouverts, donc les fichiers Compose fournis dimensionnent les limites de processus et de fichiers ouverts du proxy pour la valeur par défaut ; une valeur plus haute demande aussi des limites plus hautes, et le proxy avertit au démarrage quand sa limite de fichiers ouverts est trop basse. Au démarrage, le proxy relève sa limite de fichiers ouverts à ce dont les connexions ont besoin, dans la mesure où la limite stricte du conteneur le permet. Sous Kubernetes, pose-la sur le Deployment egress ; une spécification de Pod ne fixe aucune limite de processus, donc le `podPidsLimit` du nœud doit laisser la place d’un thread par connexion. | Les réglages backend ci-dessus doivent parvenir à l’API et au worker. Recrée ces services après une modification. Le spawner vérifie aussi la mémoire et le disque lors de la reprise d’une session libérée : un espace déjà démarré peut donc attendre de la capacité. diff --git a/docs/fr/self-hosted/install/kubernetes.md b/docs/fr/self-hosted/install/kubernetes.md index cd46f153d8..52c74eded7 100644 --- a/docs/fr/self-hosted/install/kubernetes.md +++ b/docs/fr/self-hosted/install/kubernetes.md @@ -643,14 +643,113 @@ spec: | `SANDBOX_K8S_NAMESPACE` | Le namespace où sont créés les Pods de session, les Secrets et les claims de workspace ; dans cette organisation, celui du spawner lui-même. `tale-sandbox` par défaut. | | `SANDBOX_RUNTIME_IMAGE` | L’image du runtime sandbox Tale correspondante, accessible à chaque nœud. | | `NODE_EXTRA_CA_CERTS` | Le fichier CA du cluster, généralement `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` dans le spawner. C’est le seul mécanisme de confiance CA que le spawner respecte ; garde la vérification TLS active. | -| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | La taille du claim de workspace `/agent` de chaque session d’agent, `4Gi` par défaut ; limite aussi le workspace temporaire d’un rendu du crawler, qui se passe de claim puisqu’il n’est jamais repris, et le stockage temporaire du Docker interne lorsqu’il est activé. | +| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | La taille du claim de workspace `/agent` de chaque session d’agent, `4Gi` par défaut ; limite aussi le workspace temporaire d’un rendu du crawler, qui se passe de claim puisqu’il n’est jamais repris. | +| `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` | La taille du stockage temporaire du Docker interne d’une session d’agent qui exécute Docker dans sa sandbox, `20Gi` par défaut. Une session qui le dépasse est évincée par Kubernetes ; dimensionne-le donc pour les plus grosses images que tes agents téléchargent et construisent. Les versions antérieures à ce paramètre dimensionnaient ce stockage avec `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` : si tu l’avais réduit pour garder ce stockage petit, définis aussi cette variable lors de la mise à niveau, sinon le stockage passe à `20Gi`. Le spawner émet un avertissement au démarrage quand `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` est défini sans elle. | +| `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` | L’espace disque du nœud qu’un Pod de session demande, `256Mi` par défaut, et ce qu’il peut y écrire en dehors de son workspace temporaire et de son stockage Docker, `2Gi` par défaut. La limite d’un Pod y ajoute ces stockages : `2Gi` pour un agent, `6Gi` pour un rendu du crawler, `22Gi` avec Docker dans la sandbox. Une session d’agent qui dépasse sa limite est évincée seule, et la tâche suivante la reprend avec son workspace intact, au lieu de remplir le nœud jusqu’à ce que Kubernetes évince les Pods de la plateforme. Si tes nœuds ne déclarent aucune capacité de stockage éphémère, comme sur certains clusters rootless, mets la demande à `0`. | | `SANDBOX_K8S_CPU_REQUEST` / `SANDBOX_K8S_MEMORY_REQUEST` | Ce que chaque Pod de session demande au planificateur, en quantités Kubernetes. Sans valeur, un Pod d’agent demande `250m` et `512Mi` (`1Gi` avec Docker dans la sandbox) et un rendu du crawler `250m` et `512Mi` ; une demande ne dépasse jamais la limite du Pod. Augmente-les quand les agents de tes nodes lancent des builds plus lourds, pour que le planificateur ne place pas plus de sessions qu’un node ne peut en porter. | +| `SANDBOX_K8S_NODE_SELECTOR` / `SANDBOX_K8S_TOLERATIONS` | Facultatif. Les labels de nœud auxquels chaque Pod de session doit correspondre, sous forme d’objet JSON, et les taints qu’il tolère, sous forme de tableau JSON de tolérances de Pod. Sans valeur, une session peut tourner sur n’importe quel nœud ; voir plus bas. | +| `SANDBOX_K8S_PRIORITY_CLASS` | Facultatif. La PriorityClass de chaque Pod de session. | | `SANDBOX_K8S_CACHE_STORAGECLASS` | La StorageClass des claims de workspace ; sans valeur, celle du cluster s’applique. | | `SANDBOX_RUNTIME` / `SANDBOX_RUNTIME_CLASS` | Un niveau de runtime pris en charge et, si nécessaire, le nom de la RuntimeClass installée. | | `SANDBOX_EGRESS_PROXY` | Le Service de sortie utilisé par les sessions, `http://sandbox-egress:3128` par défaut. | Le spawner se met à l’échelle horizontalement. Chaque réplica retrouve une session qu’il n’a pas créée grâce au nom déterministe de son Pod et l’adopte ; exec, arrêt et destruction fonctionnent donc via le réplica que le Service choisit. `SANDBOX_MAX_SESSIONS` compte le namespace, mais des admissions simultanées sur plusieurs réplicas peuvent le dépasser brièvement ; une ResourceQuota fournit la limite stricte. Le [contrat Kubernetes des sandboxes](https://github.com/tale-project/tale/blob/main/services/sandbox/docs/kubernetes.md) documente la forme du Pod et les détails du runtime. +### Tenir les sessions à l’écart des nœuds de la plateforme + +Les Pods de session exécutent le code qu’écrivent tes agents, et avec Docker dans la sandbox au niveau `runc`, ils tournent en mode privilégié. Sans réglage de placement, le planificateur les place sur n’importe quel nœud, à côté de Postgres et des rôles applicatifs, où une session chargée se dispute le même processeur, la même mémoire et le même disque. Pour réserver des nœuds aux sessions, ajoute-leur un label et un taint : + +```bash +kubectl label node tale.dev/sandbox=true +kubectl taint node tale.dev/sandbox=true:NoSchedule +``` + +Ajoute ensuite les paramètres correspondants à l’`env` du spawner dans `40-sandbox.yaml` : + +```yaml + - { name: SANDBOX_K8S_NODE_SELECTOR, value: '{"tale.dev/sandbox":"true"}' } + - { name: SANDBOX_K8S_TOLERATIONS, value: '[{"key":"tale.dev/sandbox","operator":"Exists","effect":"NoSchedule"}]' } + - { name: SANDBOX_K8S_PRIORITY_CLASS, value: tale-sandbox-session } +``` + +Le taint écarte les autres Pods de ces nœuds, et le sélecteur y maintient les sessions. La PriorityClass classe les sessions sous la plateforme : le planificateur peut préempter une session pour placer un Pod de la plateforme, et quand un nœud manque de mémoire ou de disque, le kubelet tient compte de la priorité et évince les sessions plus tôt. Avec `preemptionPolicy: Never`, une session ne préempte jamais elle-même un autre Pod. Un administrateur du cluster crée la classe une fois ; le spawner n’a besoin d’aucune permission supplémentaire : + +```yaml +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: { name: tale-sandbox-session } +value: -10 +preemptionPolicy: Never +globalDefault: false +description: Tale sandbox sessions yield to the platform. +``` + +Ces paramètres s’appliquent à chaque Pod de session, rendus du crawler compris, pour les sessions que le spawner crée après son redémarrage ; les sessions en cours gardent leur placement. Une valeur mal formée empêche le spawner de démarrer. Un sélecteur valide qu’aucun nœud ne satisfait, ou un taint que les tolérances ne couvrent pas, laisse un Pod de session en attente : le spawner journalise la raison donnée par le scheduler, et la création échoue avec cette raison une fois son budget de démarrage épuisé. Avec un stockage local au nœud, le claim de workspace d’une session arrêtée reste sur son nœud : garde ce nœud dans le sélecteur, sinon la session ne peut pas reprendre. + +### Précharger les images de la sandbox + +Sur Kubernetes, ce n’est pas le spawner qui télécharge les images, mais le kubelet, Pod par Pod. La première session sur un nœud télécharge donc l’image du runtime, environ 2 Go, dans son budget de démarrage (`SANDBOX_SESSION_CREATE_TIMEOUT_MS`, 180 secondes par défaut), et le ramasse-miettes d’images du kubelet peut la supprimer de nouveau dès qu’aucun Pod ne l’utilise. Ce DaemonSet facultatif télécharge les images de la sandbox sur chaque nœud avant la première session et les y garde : chaque conteneur d’initialisation démarre une image et se termine aussitôt, le conteneur pause maintient le Pod en vie, et tant que le Pod existe, le kubelet considère ces images comme utilisées. Chaque conteneur demande 1m de CPU et 4 Mio de mémoire. + +```yaml +# 45-sandbox-prepull.yaml +apiVersion: apps/v1 +kind: DaemonSet +metadata: { name: sandbox-image-prepull, namespace: tale } +spec: + selector: { matchLabels: { app: sandbox-image-prepull } } + updateStrategy: { rollingUpdate: { maxUnavailable: 25% } } + template: + metadata: { labels: { app: sandbox-image-prepull } } + spec: + enableServiceLinks: false + automountServiceAccountToken: false + terminationGracePeriodSeconds: 0 + # Si tu définis SANDBOX_K8S_NODE_SELECTOR et SANDBOX_K8S_TOLERATIONS, reprends ici les mêmes valeurs : + # nodeSelector: { tale.dev/sandbox: 'true' } + # tolerations: [{ key: tale.dev/sandbox, operator: Exists, effect: NoSchedule }] + securityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: { type: RuntimeDefault } + initContainers: + - name: runtime + image: ghcr.io/tale-project/tale/tale-sandbox-runtime:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + - name: egress + image: ghcr.io/tale-project/tale/tale-sandbox-egress:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + - name: gateway + image: ghcr.io/tale-project/tale/tale-sandbox-llm-gateway:${VERSION} + command: [sh, -c, 'exit 0'] + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 32Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + containers: + - name: pause + image: registry.k8s.io/pause:3.10 + resources: { requests: { cpu: 1m, memory: 4Mi }, limits: { memory: 16Mi } } + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } +``` + +C’est l’image du runtime qui compte : chaque Pod de session et son sidecar de sortie démarrent à partir d’elle. Les images du proxy de sortie et de la passerelle évitent un téléchargement quand ces Deployments changent de nœud ; retire leurs conteneurs d’initialisation si les sessions tournent sur des nœuds dédiés que ces Deployments n’utilisent jamais. Si tu définis `SANDBOX_K8S_NODE_SELECTOR` et `SANDBOX_K8S_TOLERATIONS`, donne les mêmes valeurs au DaemonSet pour qu’il tourne exactement sur les nœuds des sessions. + +Les tags des images suivent `${VERSION}` comme dans tous les autres fichiers. Applique ce fichier à part, avant la boucle, et attends que chaque nœud ait téléchargé les images ; lors d’une mise à niveau, la nouvelle image du runtime est alors en place avant que le spawner bascule dessus et crée des sessions à partir d’elle. La première commande crée le namespace lors d’une première installation et ne change rien lors d’une mise à niveau : + +```bash +envsubst '${VERSION}' < 00-namespace.yaml | kubectl apply -f - +envsubst '${VERSION}' < 45-sandbox-prepull.yaml | kubectl apply -f - +kubectl -n tale rollout status ds/sandbox-image-prepull --timeout=15m +for f in 00-namespace.yaml 10-stores.yaml 20-application.yaml 30-proxy.yaml 40-sandbox.yaml; do + envsubst '${VERSION}' < "$f" | kubectl apply -f - +done +``` + +Exécute les mêmes commandes à chaque mise à niveau, avec la nouvelle `VERSION` exportée. Une copie appliquée sans `envsubst`, ou oubliée lors d’une mise à niveau, garde ses anciens tags : elle maintient les anciennes images sur les nœuds, et la première session sur chaque nœud retélécharge la nouvelle. + ### Ce que le spawner impose Au démarrage, le spawner applique la NetworkPolicy `tale-sandbox-session-egress` : les Pods de session peuvent joindre le DNS et les Pods de leur propre namespace, rien d’autre. Le service de métadonnées du cloud, les nœuds et les autres namespaces restent ainsi inaccessibles, même pour un processus qui ignore `HTTP_PROXY`. Les destinations publiques passent par `sandbox-egress`. Une permission `networkpolicies` absente est journalisée sans arrêter le spawner ; vérifie que la politique existe avant d’admettre des traitements. @@ -703,4 +802,4 @@ Les snapshots, bascules bleu-vert et contrôles de rollback de la CLI ne s’ex ## Périmètre vérifié -Ces cinq fichiers ont été appliqués tels quels, à l’exception des valeurs du Secret, sur un cluster kind neuf à un seul nœud, avec Kubernetes 1.36, kube-network-policies, la StorageClass local-path et Tale 0.5.31 : démarrage et migrations, point d’entrée public, création du premier propriétaire et carte Sandboxes, tâche d’agent avec livrable, cycle de vie des sessions avec arrêt sur inactivité, reprise et accès entre réplicas, les sondes de barrière ci-dessus et un redémarrage progressif de l’API à deux réplicas. La planification sur plusieurs nœuds avec un stockage de configuration `ReadWriteMany`, Docker dans les sessions sur une RuntimeClass sysbox ou kata, un Ingress avec `TLS_MODE=external` et des stockages hautement disponibles ne faisaient pas partie de cet essai. +Ces cinq fichiers ont été appliqués tels quels, à l’exception des valeurs du Secret, sur un cluster kind neuf à un seul nœud, avec Kubernetes 1.36, kube-network-policies, la StorageClass local-path et Tale 0.5.31 : démarrage et migrations, point d’entrée public, création du premier propriétaire et carte Sandboxes, tâche d’agent avec livrable, cycle de vie des sessions avec arrêt sur inactivité, reprise et accès entre réplicas, les sondes de barrière ci-dessus et un redémarrage progressif de l’API à deux réplicas. La planification sur plusieurs nœuds avec un stockage de configuration `ReadWriteMany`, Docker dans les sessions sur une RuntimeClass sysbox ou kata, un Ingress avec `TLS_MODE=external` et des stockages hautement disponibles ne faisaient pas partie de cet essai. Les limites de disque et les réglages de placement des Pods de session, ainsi que le DaemonSet facultatif de préchargement, sont venus plus tard et n’en faisaient pas partie non plus. diff --git a/services/platform/tests/manual/reference/automation/tasks.md b/services/platform/tests/manual/reference/automation/tasks.md index e5aca4a312..09ab2f13c0 100644 --- a/services/platform/tests/manual/reference/automation/tasks.md +++ b/services/platform/tests/manual/reference/automation/tasks.md @@ -49,3 +49,5 @@ Rows for [`tasks`](../../suites/tasks.md) boxes that moved out of the suite. Rea | [tasks](../../suites/tasks.md) | Task-detail screenshot readiness waits for the independent activity response in EN/DE/FR after the dialog, brief and comment appear; an activity heading outside the task dialog cannot admit the capture. | ✅ browser | `tests/e2e/specs/docs-screenshot-seed.spec.ts` (controlled response and synthetic DOM; actual localized capture playback remains visual review) | | [tasks](../../suites/tasks.md) / [automations](../../suites/automations.md) | The Gemini and Pi wrappers become their CLI (exec, same pid) instead of waiting on it, so no Python process stays resident for a turn: the CLI reads the prompt on stdin (an anonymous temporary file, any size, a leading dash included) in the exec's workdir, its exit status is the exec's, and a process-group SIGTERM ends it by that signal; what a wrapper staged (Gemini's context file under `~/.gemini/`, Pi's config dir in `$TMPDIR`) carries the exec's pid, stays while that exec runs and is removed by the next wrapper once it has ended; a CLI that cannot start fails the exec with nothing left staged | ✅ automated | `services/sandbox-runtime/daemon/src/harness-wrapper-exec.test.ts` (each real wrapper against a fake CLI on PATH), `services/sandbox-runtime/daemon/src/gemini-settings.test.ts` (the context file named for the exec, the next exec removing a failed one's), `tests/integration/container-sandbox-runtime-test.ts` (the pinned Gemini CLI reading its prompt and appended context through `tale-gemini-run`); the pinned Pi CLI doing so in a built image stays a deployment check | | [tasks](../../suites/tasks.md) / [automations](../../suites/automations.md) | An organization's pip, npm and bun package-cache volumes are removed once no session container of the organization exists and their last recorded use is older than `SANDBOX_PACKAGE_CACHE_RETENTION` (14 days by default; `off` keeps them); caches without a recorded use get one at first sight; a create holds its organization's caches until its container exists, and a create during a removal waits for it. | ✅ unit + fake docker | `services/sandbox/src/package-cache-retention.test.ts`, `services/sandbox/src/backend/docker/docker-session-cache-volumes.test.ts`, `services/sandbox/src/config.test.ts`; real volume sizes on production hosts remain a deployment observation | +| [settings](../../suites/settings.md) | Kubernetes session Pods bound their node disk: the runner requests `256Mi` of `ephemeral-storage` and is limited to a `2Gi` headroom plus every disk-backed `emptyDir` it mounts (a render's workspace, a DinD agent's inner Docker store sized on its own, `20Gi` by default), the request clamped to the limit; the egress sidecar declares its own, zero when the runner's request is zero; `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `_LIMIT`, `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` and `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` are Kubernetes quantities refused at boot when malformed, and the sizes and the limit also when zero; where sessions can run Docker inside, a `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` set without `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` warns at boot that it does not size the store | ✅ automated | `services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.test.ts` (`node disk (ephemeral-storage)`), `services/sandbox/src/config.test.ts` (`session Pod disk bounds`, `a workspace size set alone warns…`); the kubelet's eviction on a real node stays a deployment check | +| [settings](../../suites/settings.md) | Kubernetes session Pods carry the operator's placement: `SANDBOX_K8S_NODE_SELECTOR` and `SANDBOX_K8S_TOLERATIONS` (JSON) and `SANDBOX_K8S_PRIORITY_CLASS` become the `nodeSelector`, `tolerations` and `priorityClassName` of every profile's Pod (DinD agent, lightweight agent, crawler render), each Pod with its own copy, all omitted when unset; the spawner refuses to start, naming the variable, on malformed JSON, a label key or value the apiserver rejects, a toleration with an unknown field, operator or effect, a value under `Exists`, no key without `Exists`, `tolerationSeconds` that is fractional or not under `NoExecute`, or an invalid class name; a Pod the scheduler reports `Unschedulable` keeps its create waiting, the scheduler's message logged once and carried by the error the create fails with | ✅ automated | `services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.test.ts` (`placement: …`), `services/sandbox/src/config.test.ts` (`session Pod placement`), `k8s-session-backend.test.ts` (`a Pod the scheduler cannot place`); scheduling onto labelled and tainted nodes and preemption by priority stay deployment checks | diff --git a/services/sandbox/README.md b/services/sandbox/README.md index cf536140c1..9a7d0e80e8 100644 --- a/services/sandbox/README.md +++ b/services/sandbox/README.md @@ -316,6 +316,24 @@ forwarding protection, including when adopting an older runtime image. Generated Docker containers explicitly disable IPv6 so IPv4-only deployments do not rely on host IPv6 firewall support. See the [operator environment reference](../../docs/en/self-hosted/configuration/environment-reference.md#sandbox-infrastructure). +## Kubernetes session Pods + +Each session Pod's runner requests 256 MiB of node disk (`ephemeral-storage`) +and may write 2 GiB outside its sized volumes (root filesystem, logs). Its +limit adds the disk-backed `emptyDir` volumes it mounts: a crawler render's +workspace (`SANDBOX_K8S_WORKSPACE_SIZE_LIMIT`, 4 GiB) and a DinD agent's inner +Docker store (`SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT`, 20 GiB). A session past +its limit is evicted on its own instead of filling the node until the kubelet +evicts platform Pods; `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` and +`SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` override the request and that headroom. + +Unset, session Pods schedule on any node. `SANDBOX_K8S_NODE_SELECTOR` (a JSON +object of node labels), `SANDBOX_K8S_TOLERATIONS` (a JSON array of Pod +tolerations) and `SANDBOX_K8S_PRIORITY_CLASS` place every session Pod, +crawler renders included, on dedicated nodes and below the platform's +priority; the spawner refuses to start on a value the apiserver would reject. +See the [Kubernetes deployment contract](docs/kubernetes.md). + ## Inner Docker networking An enabled agent session starts its inner engine on the first ordinary Docker diff --git a/services/sandbox/docs/docker-in-container.md b/services/sandbox/docs/docker-in-container.md index 6b7afd8f4c..909ec0dfb1 100644 --- a/services/sandbox/docs/docker-in-container.md +++ b/services/sandbox/docs/docker-in-container.md @@ -213,7 +213,9 @@ override it with `SANDBOX_RUNTIME_CLASS`. the Docker data-root on XFS alone does not assign a project quota to each named volume. Tale does not configure those quotas. A fixed-size filesystem for the entire data-root bounds aggregate use, not individual sessions. - On K8s `emptyDir.sizeLimit` is enforced by eviction, which can lag writes. + On K8s the store's `emptyDir.sizeLimit` (`SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT`, + default `20Gi`) and the Pod's `ephemeral-storage` limit are enforced by + eviction, which can lag writes. **Set a quota before exposing this to untrusted tenants** — an unbounded `docker build` loop can fill shared storage. Docker admission also observes the workspace filesystem and, where the spawner's hostname bind verifies diff --git a/services/sandbox/docs/kubernetes.md b/services/sandbox/docs/kubernetes.md index 885369e582..5194852b49 100644 --- a/services/sandbox/docs/kubernetes.md +++ b/services/sandbox/docs/kubernetes.md @@ -80,11 +80,16 @@ Pod `250m` / `512Mi` (`1Gi` under DinD), a crawler render `250m` / `512Mi`, overridable for every Pod by `SANDBOX_K8S_CPU_REQUEST` / `SANDBOX_K8S_MEMORY_REQUEST` and never above the limit (an idle session uses ~60 MB; the old flat `500m` / `1Gi` capped a node's sessions by CPU they -never used). The transparent-egress sidecar requests `10m` / `16Mi` (limit -`250m` / `64Mi`), so a namespace ResourceQuota admits the Pod. The workspace +never used). The transparent-egress sidecar requests `10m` / `16Mi` / `16Mi` +of ephemeral storage (limit `250m` / `64Mi` / `128Mi`), so a namespace +ResourceQuota admits the Pod; a zero runner `ephemeral-storage` request zeroes +the sidecar's too. The workspace PVC of an agent session is sized by `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` -(default `4Gi`), which under DinD also bounds the inner-docker `emptyDir`; a -crawler render, never resumed, gets a sized `emptyDir` instead of a PVC. +(default `4Gi`); a crawler render, never resumed, gets an `emptyDir` of that +size instead of a PVC. Under DinD the inner Docker store is an `emptyDir` +sized by `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` (default `20Gi`): one pull of +a large build image unpacks to several GiB, and an `emptyDir` past its +`sizeLimit` evicts the whole session. `fsGroupChangePolicy: OnRootMismatch` keeps the kubelet from re-chowning a whole workspace on every resume. `SANDBOX_RUNTIME` selects the RuntimeClass per tier (gVisor / sysbox / kata; @@ -92,6 +97,56 @@ runc omits the field). DinD is an agent-profile capability: a `default`-profile Pod (run_code, crawler renders) stays fully hardened whatever the deployment flags say. +**Node disk:** the runner requests `256Mi` of `ephemeral-storage` +(`SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST`) and is limited to a headroom of +`2Gi` (`SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT`) for its writable root +filesystem and logs, plus every disk-backed `emptyDir` it mounts: the kubelet +evicts a Pod whose writable layers, logs and `emptyDir` volumes together +exceed the sum of its containers' limits. An agent Pod is limited to `2Gi` +(its workspace is a PVC, which does not count), a crawler render to `6Gi`, a +DinD agent to `22Gi`; the request never exceeds the limit. A session past its +limit is evicted on its own instead of filling the node until DiskPressure +evicts the platform's Pods beside it. Under node disk pressure the kubelet +evicts Pods using more than they request first, by priority, then by how far +their use exceeds their request. The memory-backed `/tmp` and `/dev/shm` count against memory +instead. An evicted agent session's Pod stays `Failed` until its next resume +removes it and recreates the session on the intact workspace PVC. + +**Placement:** `SANDBOX_K8S_NODE_SELECTOR` (a JSON object of node labels), +`SANDBOX_K8S_TOLERATIONS` (a JSON array of Pod tolerations) and +`SANDBOX_K8S_PRIORITY_CLASS` set `nodeSelector`, `tolerations` and +`priorityClassName` on every session Pod, crawler renders included. Unset, +the fields are omitted, and sessions (privileged under runc DinD) can +schedule beside the database and platform Pods and starve them. Label and +taint dedicated nodes, select and tolerate them here, and give sessions a +PriorityClass below the platform's with `preemptionPolicy: Never`: the +scheduler may then preempt a session to place a platform Pod, never the +reverse, and among Pods over their requests the kubelet evicts sessions +first. The values are +held to the apiserver's rules at boot (label keys and values, a toleration's +operator, effect and `tolerationSeconds`, a DNS-subdomain class name), so a +typo stops the spawner instead of failing every create. A valid selector no +node matches, a missing toleration or a disk request on a node with no +capacity leaves the Pod `Unschedulable`; the create keeps waiting, since an +autoscaler can still add a node, logs the scheduler's message once and, +when its budget runs out, fails with that message. The spawner needs no +extra RBAC; a cluster administrator creates the PriorityClass once. A +changed selector reaches new Pods only, and with node-local storage a +stopped session resumes only on the node that holds its workspace PVC. + +**Image pre-pull:** `warmImage` is a no-op on this backend; the kubelet +pulls the runtime image per Pod. The first session on a node pulls it +(about 2 GB compressed) inside its create budget, and kubelet image garbage +collection can remove it again once no Pod uses it. The +[Kubernetes install guide](../../../docs/en/self-hosted/install/kubernetes.md) +ships an optional DaemonSet, `45-sandbox-prepull.yaml`: no-op init +containers of the runtime, egress and gateway images behind a pause +container (`1m` / `4Mi` requests), on the same `${VERSION}` tags as the +rest of the manifests and with the session placement as a commented-out +selector and toleration. The guide applies it and waits for its rollout +before the other manifests, so an upgrade's runtime image is on every node +before the spawner rolls to it. + ## RBAC (namespaced Role — no cluster scope, no `pods/exec`) The spawner Deployment's ServiceAccount needs a Role in the sandbox namespace: @@ -147,7 +202,11 @@ stream. Keep it out so a stray exec call fails closed. | `SANDBOX_RUNTIME` | optional | Runtime tier (`runc` default, `gvisor`/`runsc`, `sysbox`, `kata`); sets the Pod `runtimeClassName`, overridable via `SANDBOX_RUNTIME_CLASS` for a non-runc tier. | | `SANDBOX_DIND_INNER_POOL` | optional | Canonical RFC1918 IPv4 `/16` for the agent's inner Docker daemon. Unset selects automatically; configure a range outside the cluster's Pod, Service and VPC CIDRs. Known overlap remains an error. See [Inner Docker networking](#inner-docker-networking). | | `SANDBOX_K8S_CPU_REQUEST` / `SANDBOX_K8S_MEMORY_REQUEST` | optional | What every session Pod requests (K8s quantities), overriding the per-profile defaults; clamped to the Pod's limit. A malformed value fails the spawner at boot. | -| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | optional | Size of the per-session `/agent` workspace PVC (default `4Gi`) and, under DinD, the `sizeLimit` of the inner-docker `emptyDir`. Bounds deps + temp + outputs. | +| `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT` | optional | Size of the per-session `/agent` workspace PVC (default `4Gi`) and of a crawler render's workspace `emptyDir`. Bounds deps + temp + outputs. A malformed or zero value fails the spawner at boot. | +| `SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT` | optional | `sizeLimit` of a DinD agent's inner Docker store `emptyDir` (default `20Gi`); also added to that Pod's ephemeral-storage limit. A malformed or zero value fails the spawner at boot. Releases before it sized the store with `SANDBOX_K8S_WORKSPACE_SIZE_LIMIT`; when sessions can run DinD and that is set without this one, the spawner warns at boot that the store now takes this default. | +| `SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST` / `SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT` | optional | The runner's `ephemeral-storage` request (default `256Mi`, clamped to the limit) and its headroom for the writable root filesystem and logs (default `2Gi`); the Pod's limit is the headroom plus its disk-backed `emptyDir` sizes. A malformed value, or a zero limit, fails the spawner at boot. Set the request to `0` on nodes that report no `ephemeral-storage` capacity (a kubelet with `localStorageCapacityIsolation: false`, as on some rootless clusters), where any request leaves the Pod unschedulable. | +| `SANDBOX_K8S_NODE_SELECTOR` / `SANDBOX_K8S_TOLERATIONS` | optional | JSON: an object of node labels every session Pod must match (`{"tale.dev/sandbox":"true"}`) and an array of Pod tolerations (`[{"key":"tale.dev/sandbox","operator":"Exists","effect":"NoSchedule"}]`). Unset ⇒ omitted. Malformed JSON, label or toleration fails the spawner at boot. See **Placement** above. | +| `SANDBOX_K8S_PRIORITY_CLASS` | optional | `priorityClassName` of every session Pod. Unset ⇒ omitted (the namespace default priority). An invalid name fails the spawner at boot. | | `SANDBOX_K8S_CACHE_STORAGECLASS` | optional | StorageClass for the workspace PVCs (`ReadWriteOnce`). Unset ⇒ the cluster default. On a multi-node cluster use a class whose volumes can re-bind where a resume Pod schedules. | | `SANDBOX_EGRESS_PROXY` | optional | The runner's `HTTPS_PROXY`/`HTTP_PROXY` (default `http://sandbox-egress:3128`); also what the transparent-egress sidecar tunnels to. | | `SANDBOX_K8S_SERVER` / `SANDBOX_K8S_TOKEN` / `SANDBOX_K8S_CAFILE` | dev only | Explicit bearer-token kubeconfig for local Bun dev (kind's client-cert kubeconfig auths as `system:anonymous` under Bun). In-cluster uses the projected SA token automatically. | diff --git a/services/sandbox/src/backend/kubernetes/k8s-session-backend.test.ts b/services/sandbox/src/backend/kubernetes/k8s-session-backend.test.ts index 3772b76048..ff19edbd8a 100644 --- a/services/sandbox/src/backend/kubernetes/k8s-session-backend.test.ts +++ b/services/sandbox/src/backend/kubernetes/k8s-session-backend.test.ts @@ -1,7 +1,7 @@ // Kubernetes lifecycle ownership and workspace preservation are exercised // through a stub CoreV1Api, without requiring a cluster. -import { describe, expect, test } from 'bun:test'; +import { describe, expect, spyOn, test } from 'bun:test'; import type { CoreV1Api, @@ -1084,6 +1084,66 @@ describe('KubernetesSessionBackend.createSession — an orphaned Secret or a Pod }); }); +describe('KubernetesSessionBackend.createSession — a Pod the scheduler cannot place', () => { + const quick = { + ...cfg, + session: { ...cfg.session, createHealthTimeoutMs: 200 }, + }; + const reason = + "0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector."; + + test("the create's error carries the scheduler's reason, logged once", async () => { + const { client } = stub( + () => Promise.resolve({}), + () => Promise.resolve({}), + { + pod: { + status: { + phase: 'Pending', + conditions: [ + { + type: 'PodScheduled', + status: 'False', + reason: 'Unschedulable', + message: reason, + }, + ], + }, + }, + }, + ); + const warn = spyOn(console, 'warn').mockImplementation(() => {}); + try { + const err = await rejection( + new KubernetesSessionBackend(quick, client).createSession(spec), + ); + expect(err?.message).toBe( + `session sess_c4 pod never got an IP: pod unschedulable: ${reason}`, + ); + const logged = warn.mock.calls.filter((call) => + String(call[0]).includes('pod unschedulable'), + ); + expect(logged).toHaveLength(1); + expect(String(logged[0]?.[0])).toContain(reason); + } finally { + warn.mockRestore(); + } + }); + + test('a Pod merely pending says nothing about the scheduler', async () => { + const { client } = stub( + () => Promise.resolve({}), + () => Promise.resolve({}), + { pod: { status: { phase: 'Pending' } } }, + ); + const err = await rejection( + new KubernetesSessionBackend(quick, client).createSession(spec), + ); + expect(err).not.toBeNull(); + expect(err?.message).not.toMatch(/unschedulable/); + }); +}); + // REGRESSION: destroySession swallowed a failed PVC delete, so the route // answered destroyed:true while the user's data (and its storage) survived // with no retry and no sweeper. diff --git a/services/sandbox/src/backend/kubernetes/k8s-session-backend.ts b/services/sandbox/src/backend/kubernetes/k8s-session-backend.ts index b4b7990ec4..5a6cb786d7 100644 --- a/services/sandbox/src/backend/kubernetes/k8s-session-backend.ts +++ b/services/sandbox/src/backend/kubernetes/k8s-session-backend.ts @@ -78,6 +78,18 @@ function conflictError( ); } +/** The scheduler's reason a Pod has no node yet, when it reported one: its + * PodScheduled condition, False with reason Unschedulable. */ +function unschedulableReason(pod: V1Pod | undefined): string | undefined { + const scheduled = pod?.status?.conditions?.find( + (condition) => condition.type === 'PodScheduled', + ); + if (scheduled?.status !== 'False' || scheduled.reason !== 'Unschedulable') { + return undefined; + } + return scheduled.message?.trim() || 'no reason given'; +} + /** Margin past a create's budget before a Pod-less Secret counts as an * orphan rather than a peer replica's create in flight. */ const ORPHAN_SECRET_SLACK_MS = 60_000; @@ -579,7 +591,6 @@ export class KubernetesSessionBackend implements SessionBackend { } } - /** Read the Pod until status.podIP is assigned, then return the runnerd URL. */ /** Watch the session's Pod while a create waits on it: `failed` rejects * with the reason once a container cannot start, and `signal` aborts then * or at stop(), so the waits beside it end too. A read that fails says @@ -627,20 +638,44 @@ export class KubernetesSessionBackend implements SessionBackend { }; } + /** Read the Pod until status.podIP is assigned, then return the runnerd URL. + * A Pod the scheduler cannot place (a selector no node matches, a taint it + * does not tolerate, a disk request on a node with no capacity) keeps + * waiting, since an autoscaler can still add a node; but the scheduler's + * reason is logged once and carried by the error the wait ends with, so an + * operator need not reach for `kubectl describe pod`. */ private async waitForEndpoint( sessionId: string, deadlineMs: number, giveUp?: AbortSignal, ): Promise { - for (;;) { - operationSignal()?.throwIfAborted(); - giveUp?.throwIfAborted(); - const ip = (await this.readPod(sessionId))?.status?.podIP; - if (ip) return `http://${ip}:${RUNNERD_PORT}`; - if (Date.now() > deadlineMs) { - throw new Error(`session ${sessionId} pod never got an IP`); + let unschedulable: string | undefined; + let logged = false; + try { + for (;;) { + operationSignal()?.throwIfAborted(); + giveUp?.throwIfAborted(); + const pod = await this.readPod(sessionId); + const ip = pod?.status?.podIP; + if (ip) return `http://${ip}:${RUNNERD_PORT}`; + unschedulable = unschedulableReason(pod); + if (unschedulable !== undefined && !logged) { + logged = true; + console.warn( + `[sandbox.session] session ${sessionId} pod unschedulable: ${unschedulable}`, + ); + } + if (Date.now() > deadlineMs) { + throw new Error(`session ${sessionId} pod never got an IP`); + } + await new Promise((r) => setTimeout(r, 500)); } - await new Promise((r) => setTimeout(r, 500)); + } catch (error) { + if (unschedulable === undefined) throw error; + throw new Error( + `session ${sessionId} pod never got an IP: pod unschedulable: ${unschedulable}`, + { cause: error }, + ); } } diff --git a/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.test.ts b/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.test.ts index 07039bdef2..e5ed4c9a82 100644 --- a/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.test.ts +++ b/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.test.ts @@ -291,10 +291,11 @@ describe('buildSessionPod', () => { expect(runner(buildSessionPod(cfg, input))?.requests).toEqual({ cpu: '250m', memory: '512Mi', + 'ephemeral-storage': '256Mi', }); expect( runner(buildSessionPod(cfg, { ...input, profile: 'default' }))?.requests, - ).toEqual({ cpu: '250m', memory: '512Mi' }); + ).toEqual({ cpu: '250m', memory: '512Mi', 'ephemeral-storage': '256Mi' }); // An operator override applies to every Pod, clamped to each limit. const tuned = { ...cfg, @@ -303,8 +304,8 @@ describe('buildSessionPod', () => { expect( runner(buildSessionPod(tuned, { ...input, profile: 'default' })), ).toEqual({ - requests: { cpu: '1', memory: '1500Mi' }, - limits: { cpu: '1', memory: '1500Mi' }, + requests: { cpu: '1', memory: '1500Mi', 'ephemeral-storage': '256Mi' }, + limits: { cpu: '1', memory: '1500Mi', 'ephemeral-storage': '10Gi' }, }); const small = { ...cfg, @@ -318,6 +319,110 @@ describe('buildSessionPod', () => { ); }); + describe('node disk (ephemeral-storage)', () => { + const disk = (pod: ReturnType) => ({ + request: + pod.spec?.containers[0]?.resources?.requests?.['ephemeral-storage'], + limit: pod.spec?.containers[0]?.resources?.limits?.['ephemeral-storage'], + }); + const dindCfg: SpawnerConfig = { + ...cfg, + runtimeTier: 'sysbox', + dockerInContainer: true, + k8s: { ...cfg.k8s, runtimeClassName: 'sysbox-runc' }, + }; + + test('every profile requests a little and is bounded by headroom plus its scratch volumes', () => { + // An agent's workspace is a claim, which the kubelet does not count. + expect(disk(buildSessionPod(cfg, input))).toEqual({ + request: '256Mi', + limit: '2Gi', + }); + // A render's workspace emptyDir (8Gi here) counts toward the Pod. + expect( + disk(buildSessionPod(cfg, { ...input, profile: 'default' })), + ).toEqual({ request: '256Mi', limit: '10Gi' }); + // DinD: the inner Docker store (20Gi by default) counts toward the Pod. + expect(disk(buildSessionPod(dindCfg, input))).toEqual({ + request: '256Mi', + limit: '22Gi', + }); + // A render never runs DinD, so its limit ignores the Docker store. + expect( + disk(buildSessionPod(dindCfg, { ...input, profile: 'default' })).limit, + ).toBe('10Gi'); + }); + + test('operator overrides size the Docker store, the headroom and the request', () => { + const tuned: SpawnerConfig = { + ...dindCfg, + k8s: { + ...dindCfg.k8s, + dockerStorageSizeLimit: '50Gi', + ephemeralStorageLimit: '512Mi', + ephemeralStorageRequest: '4Gi', + }, + }; + const pod = buildSessionPod(tuned, input); + expect( + pod.spec?.volumes?.find((v) => v.name === 'docker-storage')?.emptyDir + ?.sizeLimit, + ).toBe('50Gi'); + expect(disk(pod)).toEqual({ request: '4Gi', limit: '51712Mi' }); + // A request above the limit is clamped to it. + expect(disk(buildSessionPod(tuned, { ...input, docker: false }))).toEqual( + { request: '512Mi', limit: '512Mi' }, + ); + }); + + test('an unreadable size fails the Pod build, not every create at the apiserver', () => { + expect(() => + buildSessionPod( + { ...cfg, k8s: { ...cfg.k8s, workspaceSizeLimit: 'lots' } }, + { ...input, profile: 'default' }, + ), + ).toThrow(/unreadable storage quantity/); + }); + }); + + test('placement: unset leaves the scheduler free; set, every profile carries it', () => { + const free = buildSessionPod(cfg, input).spec; + expect(free?.nodeSelector).toBeUndefined(); + expect(free?.tolerations).toBeUndefined(); + expect(free?.priorityClassName).toBeUndefined(); + const placed: SpawnerConfig = { + ...cfg, + runtimeTier: 'sysbox', + dockerInContainer: true, + k8s: { + ...cfg.k8s, + runtimeClassName: 'sysbox-runc', + nodeSelector: { 'tale.dev/sandbox': 'true' }, + tolerations: [ + { key: 'tale.dev/sandbox', operator: 'Exists', effect: 'NoSchedule' }, + ], + priorityClassName: 'tale-sandbox-session', + }, + }; + // Agent (DinD here), lightweight agent and crawler render alike. + for (const session of [ + input, + { ...input, profile: 'agent-light' as const }, + { ...input, profile: 'default' as const }, + ]) { + const spec = buildSessionPod(placed, session).spec; + expect(spec?.nodeSelector).toEqual({ 'tale.dev/sandbox': 'true' }); + expect(spec?.tolerations).toEqual([ + { key: 'tale.dev/sandbox', operator: 'Exists', effect: 'NoSchedule' }, + ]); + expect(spec?.priorityClassName).toBe('tale-sandbox-session'); + } + // Each Pod gets its own copy, so one Pod's object never aliases the config. + const spec = buildSessionPod(placed, input).spec; + expect(spec?.nodeSelector).not.toBe(placed.k8s.nodeSelector); + expect(spec?.tolerations?.[0]).not.toBe(placed.k8s.tolerations?.[0]); + }); + test('a resume does not re-chown the whole workspace', () => { expect( buildSessionPod(cfg, input).spec?.securityContext?.fsGroupChangePolicy, @@ -365,9 +470,10 @@ describe('buildSessionPod', () => { expect(sc?.allowPrivilegeEscalation).toBe(true); expect(sc?.capabilities?.drop).toBeUndefined(); expect(sc?.seccompProfile?.type).toBe('Unconfined'); - // Inner docker store: ephemeral, size-bounded emptyDir at /var/lib/docker. + // Inner docker store: ephemeral, size-bounded emptyDir at /var/lib/docker, + // sized on its own rather than like the workspace. const ds = pod.spec?.volumes?.find((v) => v.name === 'docker-storage'); - expect(ds?.emptyDir?.sizeLimit).toBe(cfg.k8s.workspaceSizeLimit); + expect(ds?.emptyDir?.sizeLimit).toBe('20Gi'); expect(ds?.persistentVolumeClaim).toBeUndefined(); expect( pod.spec?.containers[0]?.volumeMounts?.some( @@ -491,8 +597,10 @@ describe('buildSessionPod', () => { expect(egress?.resources?.requests).toEqual({ cpu: '10m', memory: '16Mi', + 'ephemeral-storage': '16Mi', }); expect(egress?.resources?.limits?.memory).toBe('64Mi'); + expect(egress?.resources?.limits?.['ephemeral-storage']).toBe('128Mi'); // NET_ADMIN lives ONLY in the sidecar (which runs only redsocks). expect(egress?.securityContext?.runAsUser).toBe(0); expect(egress?.securityContext?.capabilities?.add).toEqual([ @@ -514,6 +622,26 @@ describe('buildSessionPod', () => { ).toBe(false); }); + test('a zero disk request reaches the sidecar too, for nodes that report no ephemeral-storage capacity', () => { + const zero: SpawnerConfig = { + ...runcCfg, + transparentEgress: true, + k8s: { ...runcCfg.k8s, ephemeralStorageRequest: '0' }, + }; + for (const profile of ['agent', 'default'] as const) { + const spec = buildSessionPod(zero, { ...input, profile }).spec; + const requests = [ + ...(spec?.initContainers ?? []), + ...(spec?.containers ?? []), + ].map((c) => c.resources?.requests?.['ephemeral-storage']); + // Explicit zeros: an absent request would default to the limit. + expect(requests).toEqual(['0', '0']); + expect( + spec?.initContainers?.[0]?.resources?.limits?.['ephemeral-storage'], + ).toBe('128Mi'); + } + }); + test('on but gvisor tier: no sidecar (runsc netstack), falls back to env', () => { const pod = buildSessionPod({ ...cfg, transparentEgress: true }, input); expect(pod.spec?.initContainers).toBeUndefined(); diff --git a/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.ts b/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.ts index b5fba84525..cd04ecc3d4 100644 --- a/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.ts +++ b/services/sandbox/src/backend/kubernetes/k8s-session-pod-spec.ts @@ -113,6 +113,45 @@ function memoryBytes(quantity: string): number { return Number(m[1]) * (MEMORY_UNITS[m[2] ?? ''] ?? Number.NaN); } +/** What a session's runner asks of the node's disk (`ephemeral-storage`), and + * what it may write there outside its sized volumes: the writable root + * filesystem (an inner dockerd's under DinD) and the container logs, which + * the kubelet rotates at 10 MiB, keeping five files, by default. Without + * them a session can fill + * a node until DiskPressure evicts the platform's Pods beside it; with them + * the kubelet evicts the session alone, and under node disk pressure it + * evicts Pods using more than they request first, by priority, then by how + * far their use exceeds their request. */ +const EPHEMERAL_STORAGE = { request: '256Mi', headroom: '2Gi' } as const; + +/** sizeLimit of a DinD session's inner Docker store (`/var/lib/docker`). A + * single pull of a large build image unpacks to several GiB, and an emptyDir + * past its sizeLimit gets the whole session evicted, so the store is sized on + * its own rather than like the workspace. */ +export const DOCKER_STORAGE_SIZE_LIMIT = '20Gi'; + +const BINARY_UNITS = [ + ['Ti', 2 ** 40], + ['Gi', 2 ** 30], + ['Mi', 2 ** 20], + ['Ki', 2 ** 10], +] as const; + +/** The sum of storage quantities, in the largest binary unit that holds it + * exactly ('2Gi' + '20Gi' → '22Gi'). */ +function sumStorage(quantities: readonly string[]): string { + const bytes = Math.ceil( + quantities.reduce((total, quantity) => total + memoryBytes(quantity), 0), + ); + if (!Number.isFinite(bytes)) { + throw new Error( + `k8s-session-pod-spec: unreadable storage quantity in ${JSON.stringify(quantities)}`, + ); + } + const unit = BINARY_UNITS.find(([, size]) => bytes % size === 0); + return unit ? `${bytes / unit[1]}${unit[0]}` : String(bytes); +} + /** The request, unless it would exceed the limit (the apiserver refuses a * request above its limit, which failed every create). */ function notAbove( @@ -199,6 +238,21 @@ export function buildSessionPod( // a provisioned volume (a CSI create/attach/delete per batch). Agent // sessions keep their PVC across stop and resume. const durableWorkspace = isAgentSessionProfile(inp.profile); + const dockerStorageSizeLimit = + cfg.k8s.dockerStorageSizeLimit ?? DOCKER_STORAGE_SIZE_LIMIT; + // The kubelet evicts a Pod whose disk use (writable layers, logs and + // disk-backed emptyDirs) exceeds the sum of its containers' limits, so the + // runner's limit is its headroom plus every sized scratch volume it mounts. + const ephemeralLimit = sumStorage([ + cfg.k8s.ephemeralStorageLimit ?? EPHEMERAL_STORAGE.headroom, + ...(durableWorkspace ? [] : [cfg.k8s.workspaceSizeLimit]), + ...(dind ? [dockerStorageSizeLimit] : []), + ]); + const ephemeralRequest = notAbove( + cfg.k8s.ephemeralStorageRequest ?? EPHEMERAL_STORAGE.request, + ephemeralLimit, + memoryBytes, + ); // Transparent egress (non-DinD, supported tier). A native sidecar (an init // container with restartPolicy: Always — K8s 1.28+) holds NET_ADMIN, installs @@ -217,6 +271,12 @@ export function buildSessionPod( // REDIRECT inline (signalled via TALE_TRANSPARENT_EGRESS in the runner env // below), exactly like the docker DinD path. const transparentEgressSidecar = transparentEgress && !dind; + // On a node that reports no ephemeral-storage capacity any request at all + // leaves the Pod unschedulable, so a zero runner request zeroes the + // sidecar's too. It stays explicit: a container with a limit and no + // request is given its limit as the request. + const egressEphemeralRequest = + memoryBytes(ephemeralRequest) === 0 ? '0' : '16Mi'; const egressSidecars = transparentEgressSidecar ? [ { @@ -228,12 +288,20 @@ export function buildSessionPod( args: ['egress-sidecar'], // Native sidecar: started (and kept running) before the runner. restartPolicy: 'Always', - // redsocks idles at ~2 MB. Explicit resources also let the Pod - // through a namespace ResourceQuota, which refuses any container - // without them. + // redsocks idles at ~2 MB and logs only errors. Explicit resources + // also let the Pod through a namespace ResourceQuota, which refuses + // any container without them. resources: { - requests: { cpu: '10m', memory: '16Mi' }, - limits: { cpu: '250m', memory: '64Mi' }, + requests: { + cpu: '10m', + memory: '16Mi', + 'ephemeral-storage': egressEphemeralRequest, + }, + limits: { + cpu: '250m', + memory: '64Mi', + 'ephemeral-storage': '128Mi', + }, }, env: [ // redsocks resolves the egress proxy endpoint from these. @@ -309,6 +377,21 @@ export function buildSessionPod( ...(cfg.k8s.runtimeClassName !== null && { runtimeClassName: cfg.k8s.runtimeClassName, }), + // Operator placement, for every profile: untrusted (under runc DinD, + // privileged) sessions kept to their own nodes, and ranked below the + // platform when the scheduler preempts or the kubelet evicts. Unset, + // the fields are omitted and the scheduler places sessions anywhere. + ...(cfg.k8s.nodeSelector !== undefined && { + nodeSelector: { ...cfg.k8s.nodeSelector }, + }), + ...(cfg.k8s.tolerations !== undefined && { + tolerations: cfg.k8s.tolerations.map((toleration) => ({ + ...toleration, + })), + }), + ...(cfg.k8s.priorityClassName !== undefined && { + priorityClassName: cfg.k8s.priorityClassName, + }), securityContext: { fsGroup: gid, // Without it the kubelet re-chowns the whole workspace (repositories, @@ -339,7 +422,7 @@ export function buildSessionPod( ? [ { name: 'docker-storage', - emptyDir: { sizeLimit: cfg.k8s.workspaceSizeLimit }, + emptyDir: { sizeLimit: dockerStorageSizeLimit }, }, ] : []), @@ -416,8 +499,12 @@ export function buildSessionPod( failureThreshold: 6, }, resources: { - requests, - limits: { cpu: cpuLimit, memory: memLimit }, + requests: { ...requests, 'ephemeral-storage': ephemeralRequest }, + limits: { + cpu: cpuLimit, + memory: memLimit, + 'ephemeral-storage': ephemeralLimit, + }, }, securityContext: runnerSecurityContext, volumeMounts: [ diff --git a/services/sandbox/src/config.test.ts b/services/sandbox/src/config.test.ts index 6abe1337d2..5b1c02fc8a 100644 --- a/services/sandbox/src/config.test.ts +++ b/services/sandbox/src/config.test.ts @@ -1,4 +1,4 @@ -import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { afterEach, beforeEach, describe, expect, spyOn, test } from 'bun:test'; import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; @@ -28,6 +28,13 @@ const KEYS = [ 'SANDBOX_MAX_SESSIONS_PER_ORG', 'SANDBOX_K8S_CPU_REQUEST', 'SANDBOX_K8S_MEMORY_REQUEST', + 'SANDBOX_K8S_WORKSPACE_SIZE_LIMIT', + 'SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST', + 'SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT', + 'SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT', + 'SANDBOX_K8S_NODE_SELECTOR', + 'SANDBOX_K8S_TOLERATIONS', + 'SANDBOX_K8S_PRIORITY_CLASS', 'SANDBOX_BUILDKITD_CPUS', 'SANDBOX_BUILDKITD_PROVISION_TIMEOUT_MS', 'SANDBOX_BUILDKITD_MEMORY', @@ -94,6 +101,191 @@ test('session Pod requests: absent by default, read as Kubernetes quantities, re expect(() => loadConfig()).toThrow(/SANDBOX_K8S_CPU_REQUEST/); }); +test('session Pod disk bounds: pod-spec defaults unless set, refused when malformed or zero', () => { + const k8s = loadConfig().k8s; + expect(k8s.workspaceSizeLimit).toBe('4Gi'); + expect(k8s).not.toHaveProperty('ephemeralStorageRequest'); + expect(k8s).not.toHaveProperty('ephemeralStorageLimit'); + expect(k8s).not.toHaveProperty('dockerStorageSizeLimit'); + process.env.SANDBOX_K8S_WORKSPACE_SIZE_LIMIT = ' '; + expect(loadConfig().k8s.workspaceSizeLimit).toBe('4Gi'); + process.env.SANDBOX_K8S_WORKSPACE_SIZE_LIMIT = '8Gi'; + process.env.SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST = '0'; + process.env.SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT = ' 3Gi '; + process.env.SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT = '40Gi'; + expect(loadConfig().k8s).toMatchObject({ + workspaceSizeLimit: '8Gi', + ephemeralStorageRequest: '0', + ephemeralStorageLimit: '3Gi', + dockerStorageSizeLimit: '40Gi', + }); + // Refused at boot by the backend that reads them. + process.env.SANDBOX_BACKEND = 'kubernetes'; + for (const name of [ + 'SANDBOX_K8S_WORKSPACE_SIZE_LIMIT', + 'SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST', + 'SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT', + 'SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT', + ]) { + const kept = process.env[name]; + process.env[name] = '20 GB'; + expect(() => loadConfig()).toThrow(name); + process.env[name] = kept; + } + // A zero limit or store size would evict the Pod on its first write. + for (const name of [ + 'SANDBOX_K8S_WORKSPACE_SIZE_LIMIT', + 'SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT', + 'SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT', + ]) { + const kept = process.env[name]; + process.env[name] = '0Gi'; + expect(() => loadConfig()).toThrow(`${name} must be above zero`); + process.env[name] = kept; + } +}); + +test('a workspace size set alone warns that it does not size the inner Docker store', () => { + const warn = spyOn(console, 'warn').mockImplementation(() => {}); + const storeWarnings = () => + warn.mock.calls.filter((call) => + String(call[0]).includes( + 'SANDBOX_K8S_WORKSPACE_SIZE_LIMIT no longer sizes the inner Docker store', + ), + ); + try { + process.env.SANDBOX_BACKEND = 'kubernetes'; + process.env.SANDBOX_K8S_WORKSPACE_SIZE_LIMIT = '2Gi'; + // Docker inside on by default (sysbox) or by choice (runc). + process.env.SANDBOX_RUNTIME = 'sysbox'; + loadConfig(); + process.env.SANDBOX_RUNTIME = 'runc'; + process.env.SANDBOX_DOCKER_IN_CONTAINER = 'true'; + loadConfig(); + expect(storeWarnings()).toHaveLength(2); + expect(String(storeWarnings()[0]?.[0])).toContain( + 'SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT does (default 20Gi)', + ); + // Quiet when the store is sized, no session can run Docker inside, the + // backend is Docker, or the workspace size is left alone. + for (const [name, value] of [ + ['SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT', '8Gi'], + ['SANDBOX_DOCKER_IN_CONTAINER', 'false'], + ['SANDBOX_DOCKER_WORKLOADS', 'none'], + ['SANDBOX_BACKEND', 'docker'], + ['SANDBOX_K8S_WORKSPACE_SIZE_LIMIT', ' '], + ] as const) { + const kept = process.env[name]; + process.env[name] = value; + loadConfig(); + expect(storeWarnings(), `${name}=${value}`).toHaveLength(2); + if (kept === undefined) delete process.env[name]; + else process.env[name] = kept; + } + } finally { + warn.mockRestore(); + } +}); + +test('session Pod placement: absent by default, read from JSON, refused with a clear error', () => { + const k8s = loadConfig().k8s; + expect(k8s).not.toHaveProperty('nodeSelector'); + expect(k8s).not.toHaveProperty('tolerations'); + expect(k8s).not.toHaveProperty('priorityClassName'); + process.env.SANDBOX_K8S_NODE_SELECTOR = + ' {"tale.dev/sandbox":"true","pool":"sandbox-1"} '; + process.env.SANDBOX_K8S_TOLERATIONS = JSON.stringify([ + { key: 'tale.dev/sandbox', operator: 'Exists', effect: 'NoSchedule' }, + { + key: 'pool', + value: 'sandbox', + effect: 'NoExecute', + tolerationSeconds: 60, + }, + { operator: 'Exists' }, + ]); + process.env.SANDBOX_K8S_PRIORITY_CLASS = ' tale-sandbox-session '; + expect(loadConfig().k8s).toMatchObject({ + nodeSelector: { 'tale.dev/sandbox': 'true', pool: 'sandbox-1' }, + tolerations: [ + { key: 'tale.dev/sandbox', operator: 'Exists', effect: 'NoSchedule' }, + { + key: 'pool', + value: 'sandbox', + effect: 'NoExecute', + tolerationSeconds: 60, + }, + { operator: 'Exists' }, + ], + priorityClassName: 'tale-sandbox-session', + }); + // An empty operator reads as Equal and an empty effect as every effect, + // the apiserver's own reading, so manifests that spell them out still boot. + process.env.SANDBOX_K8S_TOLERATIONS = JSON.stringify([ + { key: 'pool', operator: '', value: 'sandbox', effect: '' }, + ]); + expect(loadConfig().k8s.tolerations).toEqual([ + { key: 'pool', value: 'sandbox' }, + ]); + // An empty object or array places nothing. + process.env.SANDBOX_K8S_NODE_SELECTOR = '{}'; + process.env.SANDBOX_K8S_TOLERATIONS = '[]'; + expect(loadConfig().k8s).not.toHaveProperty('nodeSelector'); + expect(loadConfig().k8s).not.toHaveProperty('tolerations'); + delete process.env.SANDBOX_K8S_TOLERATIONS; + + const refused: [string, string, RegExp][] = [ + ['SANDBOX_K8S_NODE_SELECTOR', 'tale.dev/sandbox=true', /not valid JSON/], + ['SANDBOX_K8S_NODE_SELECTOR', '["sandbox"]', /JSON object of node labels/], + ['SANDBOX_K8S_NODE_SELECTOR', '{"bad key":"x"}', /no Kubernetes label key/], + ['SANDBOX_K8S_NODE_SELECTOR', '{"a/b/c":"x"}', /no Kubernetes label key/], + ['SANDBOX_K8S_NODE_SELECTOR', '{"pool":true}', /no Kubernetes label value/], + ['SANDBOX_K8S_TOLERATIONS', '{"key":"pool"}', /JSON array of tolerations/], + ['SANDBOX_K8S_TOLERATIONS', '["pool"]', /\[0\] must be a JSON object/], + [ + 'SANDBOX_K8S_TOLERATIONS', + '[{"key":"pool","efect":"NoSchedule"}]', + /unknown field efect/, + ], + [ + 'SANDBOX_K8S_TOLERATIONS', + '[{"key":"pool","operator":"In"}]', + /Equal or Exists/, + ], + [ + 'SANDBOX_K8S_TOLERATIONS', + '[{"key":"pool","effect":"NoRun"}]', + /effect must be/, + ], + [ + 'SANDBOX_K8S_TOLERATIONS', + '[{"key":"pool","operator":"Exists","value":"x"}]', + /takes no value/, + ], + ['SANDBOX_K8S_TOLERATIONS', '[{"value":"x"}]', /needs operator Exists/], + [ + 'SANDBOX_K8S_TOLERATIONS', + '[{"key":"pool","effect":"NoSchedule","tolerationSeconds":60}]', + /only to effect NoExecute/, + ], + [ + 'SANDBOX_K8S_TOLERATIONS', + '[{"key":"pool","effect":"NoExecute","tolerationSeconds":1.5}]', + /whole number of seconds/, + ], + ['SANDBOX_K8S_PRIORITY_CLASS', 'Tale_Sessions', /not a PriorityClass name/], + ]; + process.env.SANDBOX_BACKEND = 'kubernetes'; + for (const [name, value, error] of refused) { + const kept = process.env[name]; + process.env[name] = value; + expect(() => loadConfig(), `${name}=${value}`).toThrow(error); + expect(() => loadConfig(), `${name}=${value}`).toThrow(name); + if (kept === undefined) delete process.env[name]; + else process.env[name] = kept; + } +}); + test('the builder bounds are optional and validated', () => { expect(loadConfig()).not.toHaveProperty('buildkitdCpus'); expect(loadConfig()).not.toHaveProperty('buildkitdMemoryBytes'); @@ -488,3 +680,21 @@ test('Docker workloads inherit by default and validate an explicit allowlist', ( process.env.SANDBOX_DOCKER_WORKLOADS = 'browser'; expect(() => loadConfig()).toThrow(/SANDBOX_DOCKER_WORKLOADS/); }); + +test('an unreadable Kubernetes-only setting stops only a Kubernetes spawner', () => { + const warn = spyOn(console, 'warn').mockImplementation(() => {}); + try { + process.env.SANDBOX_K8S_WORKSPACE_SIZE_LIMIT = '4g'; + process.env.SANDBOX_K8S_TOLERATIONS = '{"key":"pool"}'; + // The Docker backend ignores both, so a stray value in a shared env file + // warns instead of refusing the boot. + expect(loadConfig().k8s.workspaceSizeLimit).toBe('4Gi'); + expect(warn.mock.calls.map(String).join('\n')).toContain( + 'ignoring SANDBOX_K8S_TOLERATIONS', + ); + process.env.SANDBOX_BACKEND = 'kubernetes'; + expect(() => loadConfig()).toThrow('SANDBOX_K8S_WORKSPACE_SIZE_LIMIT'); + } finally { + warn.mockRestore(); + } +}); diff --git a/services/sandbox/src/config.ts b/services/sandbox/src/config.ts index cfefc83753..16f30c6347 100644 --- a/services/sandbox/src/config.ts +++ b/services/sandbox/src/config.ts @@ -6,6 +6,7 @@ import { dirname, isAbsolute, join } from 'node:path'; import { parse as parseYaml } from 'yaml'; +import { DOCKER_STORAGE_SIZE_LIMIT } from './backend/kubernetes/k8s-session-pod-spec.ts'; import { parseDindInnerPool } from './network-address.ts'; import { dindDefaultEnabled, @@ -18,7 +19,7 @@ import { type RuntimeTier, } from './runtime-tier.ts'; import { RUNNERD_MAX_REQUEST_BODY_BYTES } from './session/runnerd-protocol.ts'; -import type { HubConfig, SpawnerConfig } from './types.ts'; +import type { HubConfig, K8sToleration, SpawnerConfig } from './types.ts'; // Parse a boolean env, returning undefined when UNSET/empty so a caller can // distinguish "operator didn't set it" (apply a default) from an explicit @@ -120,6 +121,223 @@ function k8sQuantityEnv(name: string, re: RegExp): string | undefined { return value; } +/** A storage size from the environment that must hold something: a zero + * sizeLimit or ephemeral-storage limit gets a Pod evicted on its first + * write. */ +function k8sSizeEnv(name: string): string | undefined { + const value = k8sQuantityEnv(name, MEMORY_QUANTITY_RE); + if (value !== undefined && Number.parseFloat(value) <= 0) { + throw new Error(`Env var ${name} must be above zero; got: ${value}`); + } + return value; +} + +/** A Kubernetes-only setting: refused at boot on the Kubernetes backend when + * it cannot be read. Any other backend ignores it, so there an unreadable + * value only warns, and a stray one in a shared env file never stops a + * Docker spawner from starting. */ +function k8sOnlyEnv(name: string, read: () => T): T | undefined { + try { + return read(); + } catch (err) { + if ((process.env.SANDBOX_BACKEND ?? 'docker') === 'kubernetes') throw err; + console.warn( + `[sandbox.config] ignoring ${name}, which only the Kubernetes backend reads:`, + err instanceof Error ? err.message : err, + ); + return undefined; + } +} + +const LABEL_NAME_RE = /^[A-Za-z0-9]([-A-Za-z0-9_.]{0,61}[A-Za-z0-9])?$/; +const DNS_SUBDOMAIN_RE = + /^(?=.{1,253}$)[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$/; + +/** A Kubernetes label key: a name of at most 63 characters, optionally behind + * a DNS-subdomain prefix (`tale.dev/sandbox`). Toleration keys share it. */ +function isLabelKey(key: string): boolean { + const parts = key.split('/'); + const name = parts.at(-1) ?? ''; + if (parts.length > 2 || !LABEL_NAME_RE.test(name)) return false; + return parts.length === 1 || DNS_SUBDOMAIN_RE.test(parts[0] ?? ''); +} + +function isLabelValue(value: string): boolean { + return value === '' || LABEL_NAME_RE.test(value); +} + +/** A JSON value from the environment: undefined when unset, refused at boot + * when it does not parse. */ +function jsonEnv(name: string): unknown { + const raw = process.env[name]?.trim(); + if (raw === undefined || raw === '') return undefined; + try { + return JSON.parse(raw); + } catch (err) { + throw new Error( + `Env var ${name} is not valid JSON: ${JSON.stringify(raw)}`, + { + cause: err, + }, + ); + } +} + +function isPlainObject(value: unknown): value is Record { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} + +/** The node labels a session Pod must land on (SANDBOX_K8S_NODE_SELECTOR, a + * JSON object), checked here so a malformed label fails the boot instead of + * every create at the apiserver. */ +function nodeSelectorEnv(): Record | undefined { + const name = 'SANDBOX_K8S_NODE_SELECTOR'; + const parsed = jsonEnv(name); + if (parsed === undefined) return undefined; + if (!isPlainObject(parsed)) { + throw new Error( + `Env var ${name} must be a JSON object of node labels, such as {"tale.dev/sandbox":"true"}`, + ); + } + const selector: Record = {}; + for (const [key, value] of Object.entries(parsed)) { + if (!isLabelKey(key)) { + throw new Error( + `Env var ${name} has a key that is no Kubernetes label key: ${JSON.stringify(key)}`, + ); + } + if (typeof value !== 'string' || !isLabelValue(value)) { + throw new Error( + `Env var ${name} has a value for ${key} that is no Kubernetes label value: ${JSON.stringify(value)}`, + ); + } + selector[key] = value; + } + return Object.keys(selector).length > 0 ? selector : undefined; +} + +const TOLERATION_FIELDS = [ + 'key', + 'operator', + 'value', + 'effect', + 'tolerationSeconds', +] as const; +const TOLERATION_EFFECTS = [ + 'NoSchedule', + 'PreferNoSchedule', + 'NoExecute', +] as const; + +function isTolerationEffect( + value: unknown, +): value is (typeof TOLERATION_EFFECTS)[number] { + return TOLERATION_EFFECTS.some((effect) => effect === value); +} + +/** One entry of SANDBOX_K8S_TOLERATIONS, held to the apiserver's rules. */ +function tolerationAt(at: string, entry: unknown): K8sToleration { + if (!isPlainObject(entry)) throw new Error(`${at} must be a JSON object`); + const unknown = Object.keys(entry).filter( + (field) => !TOLERATION_FIELDS.some((known) => known === field), + ); + if (unknown.length > 0) { + throw new Error( + `${at} has unknown field ${unknown.join(', ')}; a toleration takes ${TOLERATION_FIELDS.join(', ')}`, + ); + } + const { key, operator, value, effect, tolerationSeconds } = entry; + const toleration: K8sToleration = {}; + if (key !== undefined) { + if (typeof key !== 'string' || (key !== '' && !isLabelKey(key))) { + throw new Error( + `${at}.key is no Kubernetes label key: ${JSON.stringify(key)}`, + ); + } + toleration.key = key; + } + // The apiserver reads an empty operator as Equal and an empty effect as + // every effect, so both count as omitted here too. + if (operator !== undefined && operator !== '') { + if (operator !== 'Equal' && operator !== 'Exists') { + throw new Error( + `${at}.operator must be Equal or Exists; got: ${JSON.stringify(operator)}`, + ); + } + toleration.operator = operator; + } + if (value !== undefined) { + if (typeof value !== 'string' || !isLabelValue(value)) { + throw new Error( + `${at}.value is no Kubernetes label value: ${JSON.stringify(value)}`, + ); + } + toleration.value = value; + } + if (effect !== undefined && effect !== '') { + if (!isTolerationEffect(effect)) { + throw new Error( + `${at}.effect must be ${TOLERATION_EFFECTS.join(', ')}; got: ${JSON.stringify(effect)}`, + ); + } + toleration.effect = effect; + } + if (tolerationSeconds !== undefined) { + if ( + typeof tolerationSeconds !== 'number' || + !Number.isSafeInteger(tolerationSeconds) + ) { + throw new Error( + `${at}.tolerationSeconds must be a whole number of seconds`, + ); + } + toleration.tolerationSeconds = tolerationSeconds; + } + if (toleration.operator === 'Exists' && toleration.value) { + throw new Error(`${at} uses operator Exists, which takes no value`); + } + if (!toleration.key && toleration.operator !== 'Exists') { + throw new Error(`${at} has no key, which needs operator Exists`); + } + if ( + toleration.tolerationSeconds !== undefined && + toleration.effect !== 'NoExecute' + ) { + throw new Error(`${at}.tolerationSeconds applies only to effect NoExecute`); + } + return toleration; +} + +/** The taints a session Pod tolerates (SANDBOX_K8S_TOLERATIONS, a JSON array + * in the Pod spec's own shape). */ +function tolerationsEnv(): K8sToleration[] | undefined { + const name = 'SANDBOX_K8S_TOLERATIONS'; + const parsed = jsonEnv(name); + if (parsed === undefined) return undefined; + if (!Array.isArray(parsed)) { + throw new Error( + `Env var ${name} must be a JSON array of tolerations, such as [{"key":"tale.dev/sandbox","operator":"Exists","effect":"NoSchedule"}]`, + ); + } + const tolerations = parsed.map((entry: unknown, index) => + tolerationAt(`Env var ${name}[${index}]`, entry), + ); + return tolerations.length > 0 ? tolerations : undefined; +} + +/** The PriorityClass of session Pods (SANDBOX_K8S_PRIORITY_CLASS). */ +function priorityClassEnv(): string | undefined { + const name = 'SANDBOX_K8S_PRIORITY_CLASS'; + const value = process.env[name]?.trim(); + if (value === undefined || value === '') return undefined; + if (!DNS_SUBDOMAIN_RE.test(value)) { + throw new Error( + `Env var ${name} is not a PriorityClass name (lowercase letters, digits, '-' and '.'): ${JSON.stringify(value)}`, + ); + } + return value; +} + /** A Docker-style size of memory or disk from the environment ('2g', * '1536m', bytes), undefined when unset, refused at boot when unreadable. */ function sizeEnv(name: string): number | undefined { @@ -297,6 +515,37 @@ export function loadConfig(): SpawnerConfig { 'SANDBOX_K8S_MEMORY_REQUEST', MEMORY_QUANTITY_RE, ); + // A render's workspace emptyDir counts toward its Pod's ephemeral-storage + // limit, so the size must be one the pod spec can add up. + const k8sWorkspaceSizeLimit = + k8sOnlyEnv('SANDBOX_K8S_WORKSPACE_SIZE_LIMIT', () => + k8sSizeEnv('SANDBOX_K8S_WORKSPACE_SIZE_LIMIT'), + ) ?? '4Gi'; + const k8sEphemeralStorageRequest = k8sOnlyEnv( + 'SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST', + () => + k8sQuantityEnv( + 'SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST', + MEMORY_QUANTITY_RE, + ), + ); + const k8sEphemeralStorageLimit = k8sOnlyEnv( + 'SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT', + () => k8sSizeEnv('SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT'), + ); + const k8sDockerStorageSizeLimit = k8sOnlyEnv( + 'SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT', + () => k8sSizeEnv('SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT'), + ); + const k8sNodeSelector = k8sOnlyEnv( + 'SANDBOX_K8S_NODE_SELECTOR', + nodeSelectorEnv, + ); + const k8sTolerations = k8sOnlyEnv('SANDBOX_K8S_TOLERATIONS', tolerationsEnv); + const k8sPriorityClassName = k8sOnlyEnv( + 'SANDBOX_K8S_PRIORITY_CLASS', + priorityClassEnv, + ); const minFreeMemoryBytes = sizeEnv('SANDBOX_MIN_FREE_MEMORY'); const minFreeDiskBytes = sizeEnv('SANDBOX_MIN_FREE_DISK'); const buildkitdMemoryBytes = sizeEnv('SANDBOX_BUILDKITD_MEMORY'); @@ -404,6 +653,12 @@ export function loadConfig(): SpawnerConfig { const K8S_ONLY_ENVS = [ 'SANDBOX_K8S_NAMESPACE', 'SANDBOX_K8S_WORKSPACE_SIZE_LIMIT', + 'SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST', + 'SANDBOX_K8S_EPHEMERAL_STORAGE_LIMIT', + 'SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT', + 'SANDBOX_K8S_NODE_SELECTOR', + 'SANDBOX_K8S_TOLERATIONS', + 'SANDBOX_K8S_PRIORITY_CLASS', 'SANDBOX_K8S_CACHE_STORAGECLASS', 'SANDBOX_K8S_SERVER', 'SANDBOX_K8S_TOKEN', @@ -429,6 +684,25 @@ export function loadConfig(): SpawnerConfig { ); } } + // SANDBOX_K8S_WORKSPACE_SIZE_LIMIT does not size the inner Docker store of + // a session with Docker inside. An operator who set it to keep that store + // small on small node disks, without the store's own setting, gets the + // store's larger default and a Pod limit that grows with it, so the boot + // log names the variable that sizes the store. + if ( + backend === 'kubernetes' && + dockerInContainer && + dockerWorkloads?.length !== 0 && + (process.env.SANDBOX_K8S_WORKSPACE_SIZE_LIMIT?.trim() ?? '') !== '' && + k8sDockerStorageSizeLimit === undefined + ) { + console.warn( + `[sandbox.config] SANDBOX_K8S_WORKSPACE_SIZE_LIMIT no longer sizes the inner Docker store of a ` + + `session with Docker inside: SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT does (default ` + + `${DOCKER_STORAGE_SIZE_LIMIT}), and the session Pod's ephemeral-storage limit grows with it. ` + + `Set SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT to bound the store on small node disks.`, + ); + } // Body cap on every spawner route. /v1/sessions/:id/files/stage takes // INLINE base64 content (bound org skills, useSkills subtrees, steer control // files), so the cap must fit a real skill-bundle chunk plus JSON envelope; @@ -513,11 +787,27 @@ export function loadConfig(): SpawnerConfig { ? null : (process.env.SANDBOX_RUNTIME_CLASS ?? k8sRuntimeClassFor(runtimeTier)), - workspaceSizeLimit: process.env.SANDBOX_K8S_WORKSPACE_SIZE_LIMIT ?? '4Gi', + workspaceSizeLimit: k8sWorkspaceSizeLimit, ...(k8sCpuRequest !== undefined ? { cpuRequest: k8sCpuRequest } : {}), ...(k8sMemoryRequest !== undefined ? { memoryRequest: k8sMemoryRequest } : {}), + ...(k8sEphemeralStorageRequest !== undefined + ? { ephemeralStorageRequest: k8sEphemeralStorageRequest } + : {}), + ...(k8sEphemeralStorageLimit !== undefined + ? { ephemeralStorageLimit: k8sEphemeralStorageLimit } + : {}), + ...(k8sDockerStorageSizeLimit !== undefined + ? { dockerStorageSizeLimit: k8sDockerStorageSizeLimit } + : {}), + ...(k8sNodeSelector !== undefined + ? { nodeSelector: k8sNodeSelector } + : {}), + ...(k8sTolerations !== undefined ? { tolerations: k8sTolerations } : {}), + ...(k8sPriorityClassName !== undefined + ? { priorityClassName: k8sPriorityClassName } + : {}), }, port: numEnv('SANDBOX_PORT', 8003, { min: 1, max: 65535 }), // The shared HMAC secret every state-changing route is verified against diff --git a/services/sandbox/src/types.ts b/services/sandbox/src/types.ts index 01b3836358..699bc08d04 100644 --- a/services/sandbox/src/types.ts +++ b/services/sandbox/src/types.ts @@ -90,17 +90,34 @@ export interface SpawnerConfig { runtimeClassName: string | null; // Size of the per-session /agent workspace PVC (K8s quantity string, env // SANDBOX_K8S_WORKSPACE_SIZE_LIMIT; storage class from - // SANDBOX_K8S_CACHE_STORAGECLASS) and, under DinD, the sizeLimit of the - // inner-docker emptyDir. Everything the session writes — dependency - // installs, temp files, outputs — lands on the workspace, so without a - // bound a runaway session can fill the node disk; the K8s analogue of - // docker's fsize ulimit. + // SANDBOX_K8S_CACHE_STORAGECLASS) and of a crawler render's workspace + // emptyDir. Everything the session writes — dependency installs, temp + // files, outputs — lands on the workspace, so without a bound a runaway + // session can fill the node disk; the K8s analogue of docker's fsize + // ulimit. workspaceSizeLimit: string; // What every session Pod requests from the scheduler, overriding the // per-profile defaults (env SANDBOX_K8S_CPU_REQUEST / // SANDBOX_K8S_MEMORY_REQUEST, K8s quantities). Never above the limit. cpuRequest?: string; memoryRequest?: string; + // The runner's node-disk budget (env SANDBOX_K8S_EPHEMERAL_STORAGE_REQUEST + // / _LIMIT, K8s quantities), overriding the pod-spec defaults. The limit + // covers what the runner writes outside its sized volumes (writable root + // filesystem, logs); the Pod's limit adds those volumes on top. + ephemeralStorageRequest?: string; + ephemeralStorageLimit?: string; + // sizeLimit of a DinD session's inner Docker store emptyDir (env + // SANDBOX_K8S_DOCKER_STORAGE_SIZE_LIMIT); unset, the pod-spec default. + dockerStorageSizeLimit?: string; + // Where session Pods may run and how they rank against other Pods (env + // SANDBOX_K8S_NODE_SELECTOR and SANDBOX_K8S_TOLERATIONS as JSON, + // SANDBOX_K8S_PRIORITY_CLASS), so untrusted sessions can be kept off the + // nodes of the database and platform and yield to them under pressure. + // Unset, the scheduler places them anywhere at the default priority. + nodeSelector?: Readonly>; + tolerations?: readonly K8sToleration[]; + priorityClassName?: string; }; maxTimeoutMs: number; // Single flat host session root. The sandbox tier is one container that rolls @@ -143,6 +160,15 @@ export interface SpawnerConfig { session: SessionConfig; } +/** A Pod toleration, as the apiserver accepts it (validated by loadConfig). */ +export interface K8sToleration { + key?: string; + operator?: 'Equal' | 'Exists'; + value?: string; + effect?: 'NoSchedule' | 'PreferNoSchedule' | 'NoExecute'; + tolerationSeconds?: number; +} + export interface HubConfig { /** Port of the WebSocket door (published by the proxy at /sandbox/tunnel). */ port: number;