perf(sandbox): skip docker inspect when runnerd names its incarnation

services/platform/tests/manual/reference/automation.md

@@ -465,6 +465,7 @@ Legend: ✅ fully automated · 🔶 partially automated · ⛔ manual-only (no s
 | [settings](../suites/settings.md) | The first-come line for host room: a create refused for room waits by session id in the order of its first refusal; freed room goes to the oldest waiters still asking (a slot each, and memory for their working sets, before a newer create — none for a waiter the host could never fit); each 429 carries `queue: { position, waiting }` and a `retry-after` of 5 s plus 5 s a place, up to 60 s; asking again keeps a waiter's place, one that stops asking gives it up after twice its hint and 15 s, a destroy takes it out, and a restart starts with no line | ✅ automated | `services/sandbox/src/session/session-routes.test.ts` (`the first-come line for host room`, and the capacity and memory admission tests) |
 | [settings](../suites/settings.md) | Idle sandbox reclamation under deployment capacity pressure | ✅ automated | `services/sandbox/src/session/session-routes.test.ts` (released compute reclaimed before admission, persistent workspace retained, busy/pinned/unknown/legacy runtime protection, delayed release after reacquire, concurrent creates and stop failure); `services/sandbox-runtime/daemon/src/activity-gate.test.ts` (atomic idle claim, active-operation protection and frozen execution gate); `services/sandbox/src/backend/docker/docker-session-backend.test.ts`, `backend/kubernetes/k8s-session-backend.test.ts` (incarnation-fenced compute removal and retained workspace); `backend/domains/sandbox/idle-release.test.ts` (ticket capture, transactional queueing and committed-state checks), `idle-release.integration.ts` (28 real-Postgres checks: organization/allocation/owner guards, admission and release ordering, atomic cancellation and release-job rollback, post-op retry on an already-stopped workflow); `backend/core/node_only/sandbox/agent_session.test.ts` (`ensureAgentSession` refuses a false allocation resume before acquiring runtime compute) |
 | [settings](../suites/settings.md) | The session sweep frees what nothing uses: a released session stops after the short idle window (Docker-in-sandbox agent sessions keep the full one), a running session whose daemon has not answered five sweeps is stopped (a sweep that cannot probe it starts the count over), ended containers and Pods are removed by their incarnation with the workspace kept, beside the API and several at a time, a create of the id waits for that removal (bounded) and a session registered under the id is neither held up by it nor counted as freed by it, a pass still running is joined, a burst of creates at capacity reclaims one released session each, idle-longest first, a create refused at capacity probes at most eight daemons, sessions last seen released first, and none that answered busy (to a reclaim probe or the sweep) or was acquired within 15 s unless released since, and the refusals in the 3 s after a walk that found nothing are answered at once, the build-cache reconcile after adoption covers agent sessions without holding adoption up, a slow build-cache job never holds the next pass's sweep up (one job at a time, organizations adopted meanwhile reconciled right after it), and a caller hanging up on an attach costs no backend check | ✅ automated | `services/sandbox/src/session/session-routes.test.ts` (`sweep and adoption hygiene`, `creates refused at capacity do not probe every busy session`), `services/sandbox/src/backend/docker/docker-session-backend.test.ts` (a failed inner image volume removal is reported) |
+| [settings](../suites/settings.md) | A turn spends no backend check on a session its runnerd vouches for: a Docker session's container carries its creation stamp to runnerd, which names it in `/healthz` and in every activity answer and refuses with 409 `incarnation_mismatch`, before anything changes, an activity request (release ticket, acquire, release, reclaim, pin) meant for another incarnation; once runnerd has named the registered incarnation (the create's readiness answer included), acquire, release ticket and release skip the backend's existence check (`docker inspect`), a session read and the sweep of a pinned session ask runnerd's `/healthz` instead, and a replacement answering under the name is evicted without one (workspace kept, durable pin untouched); the backend is still asked when runnerd fails or names no incarnation (an older runtime image, or a replacement launched without a stamp, checked after the fact), and always on Kubernetes, where a terminating Pod still answers | ✅ automated | `services/sandbox/src/session/session-routes.test.ts` (`runnerd answers naming the session incarnation`), `services/sandbox/src/session/runnerd-client.test.ts` (`session incarnation`), `services/sandbox/src/session/docker-session-args.test.ts`, `services/sandbox/src/backend/docker/docker-session-create-calls.test.ts` (`the create reports the incarnation its readiness answer named`), `services/sandbox-runtime/daemon/src/incarnation.test.ts`, `services/sandbox-runtime/daemon/src/main.test.ts` (`activity answers name the incarnation`) |
 | [settings](../suites/settings.md) | Failed sandbox starts leave nothing behind: a fresh agent session whose create fails asks the spawner to destroy what it left before its row reads `failed` (best-effort, logged, the create's error kept; a failed resume never destroys the standing workspace), and the sandbox watchdog collects every `failed` row that destroy missed once past its grace — destroyed only when idle, never while a newer or live incarnation carries the reused session id, stamped `destroyed_at_ms` by row id with the status kept `failed` | ✅ backend + real Postgres | `backend/core/node_only/sandbox/agent_session.test.ts`, `backend/domains/sandbox/watchdogs.test.ts`, `backend/integration-check.ts` (`checkWatchdogs`); an empty `docker ps -a --filter name=tale-sbx --filter status=created` on a real host after a failed start remains a deployment check |
 | [settings](../suites/settings.md) / [governance](../suites/governance.md) | Sandbox workspace cleanup (the logic half of `SET-F54` and `SET-F69`–`SET-F72`, `SET-B28`, `GOV-F49`): one retire path re-decides under the organization's admission lock, ends the live incarnation (`expired`, unpinned on both sides, gateway keys revoked) before the spawner deletes and settles only the rows it claimed after — a fresh incarnation started meanwhile stays live; a workspace resumed after it was judged stays; the verdict — a deleted agent's workspace, or one of runs a departed person started, goes even pinned or warm (never a member's or a schedule's), one in use, pinned or with a queued run stays, a live agent's goes once unused past the `sandbox_workspaces` window and never before a full window since that rule took effect (the upgrade, turning it on, a shorter window) or since its unpin, an ended run's after the watchdog's reclaim window, a render's or legacy owner's as an orphan; an organization or custodian hold keeps what it covers; a member's removal (every door, through the membership cascade) and a user's erasure (strict, whatever runs) delete their workspaces; an agent's or project's deletion queues its agents' workspaces in the deleting transaction; an organization's deletion reads its workspaces, gateway keys and devices before the cascade and revokes keys first, releases devices only once every slice's workspaces are gone (a healed project-agent workspace included), then tears the organization down on the spawner; the hourly sweep's inventory deletes stopped workspaces no live row owns, and of those no row names only this deployment's (a render's, ours, a deleted organization's — never another deployment's), with deletions and attempts per pass bounded; a deletion settles (rows destroyed, audited, counted by an erasure) only on an explicit completion — `deletion: done`, or Kubernetes' `handed_off`, recorded in the audit metadata — and one still being deleted, failing to delete, or answered without a deletion state by a spawner older than the contract (19776cf18's) is deferred by the sweep, throws for the owner's and the organization's jobs and fails the erasure's pass (receipt partial, Retry settles it once explicit), and a failing one is audited as a failure; the Sandboxes list dates each unused workspace's deletion. Spawner side: every destroy answers `deletion` — read from the Docker trash (`pending` until the bytes go, `failed` after a failed attempt, never `done` across a restart, a fresh workspace under the id never counted), `handed_off` from Kubernetes — `await_deletion` retrying and waiting a bounded time, and the hub keeping a destroyed session's placement, marked deleting and out of the capacity view, until its device confirms (an older device's answer without a state included), asking the device again itself at most every 5 minutes, least recently asked first, so a placement nobody destroys or creates again ends once its bytes are gone (destroy finalization compares both the captured create generation and immutable placement after pending creates settle, so an ask dispatched before or during a durable create write cannot delete or mark the new route deleting), and a fresh create placed with the normal fallbacks but trying that device first; the workspace inventory in both layouts and on Kubernetes (PVCs joined with Pods, organization annotation), `if_stopped` refusing any compute or a create in flight, destroys of one id serialized and a create waiting for one, a stopped workspace naming its organization (`.owners/` marker), a deleted organization's leftover sessions and stopped workspaces, build helpers (by id, ownership-checked, egress detached before the network goes) and package caches, a create's wait for a destroy bounded | ✅ backend + real Postgres | `backend/domains/sandbox/workspace-cleanup.test.ts`, `backend/domains/sandbox/workspace-cleanup.integration.ts` (`backend:integration`, lane `checkWorkspaceCleanup`), `backend/domains/sandbox/routes.test.ts`, `backend/auth/membership.cascade.test.ts`, `backend/auth/auth.member-removal.test.ts`, `backend/domains/scim/service.test.ts`, `backend/domains/governance/routes.test.ts` (the rule recorded with the policy's save), `backend/domains/projects/service.agents.test.ts`, `backend/domains/projects/service.delete.test.ts`, `backend/domains/organizations/service.test.ts`, `services/sandbox/src/session/session-routes.test.ts`, `services/sandbox/src/session/workspace-inventory.test.ts`, `services/sandbox/src/backend/docker/docker-workspaces.test.ts`, `services/sandbox/src/backend/docker/docker-session-backend.test.ts`, `services/sandbox/src/session/workspace-trash.test.ts`, `services/sandbox/src/devices/hub.test.ts`, `services/sandbox/src/devices/placements.test.ts`, `backend/core/node_only/sandbox/helpers/session_client.test.ts`, `services/sandbox/src/backend/docker/docker-session-build-cache.test.ts`, `services/sandbox/src/backend/kubernetes/k8s-session-backend.test.ts`, `backend/core/sandbox/session_naming.test.ts`, `app/features/settings/sandboxes/workspace-cleanup-editor.test.tsx` (defaults without a policy file, the saved payload, the disabled days field, the schema's exact bounds, one localized failure toast), `app/features/settings/sandboxes/sandboxes-settings.test.tsx` (the dated row, the editor above Workspaces, no editor or policy read for Developers), `app/lib/backend/settings-sandbox.test.ts` (a `sandbox_workspaces` save refreshes the workspace list); a real spawner deleting a directory or PVC, a device coming back online, and the page in a browser stay manual |
 | [settings](../suites/settings.md) | The sandbox TTL expiry never cuts a live turn: a session past its window keeps its status, slot and credentials while one of its `running` ops shows a sign of life inside the recovery staleness window (the re-attach sweeps' rule, `sessionOpLastSignOfLifeMs` against `RECOVERY_STALE_MS`); an op silent past that window or already finished spares nothing, and the first tick after the turn ends expires the session | ✅ backend + real Postgres | `backend/domains/sandbox/watchdogs.test.ts`, `backend/integration-check.ts` (`checkWatchdogs`) |

services/sandbox-runtime/README.md

@@ -424,3 +424,9 @@ checks daemon liveness; `/readyz` also checks requested Docker capability.
 `/healthz` reuses that Docker snapshot and adds bounded, coalesced egress
 diagnostics when configured. Docker-disabled sessions remain ready; health
 failures do not hide activity or prevent file access and exec cancellation.
+A container launched with `TALE_RUNNERD_INCARNATION` (the spawner's creation
+stamp) names it as `incarnation` in `/healthz` and in every activity answer,
+and refuses with `409 incarnation_mismatch` an activity request whose
+`x-tale-runnerd-incarnation` header names another stamp, before anything
+changes. A malformed stamp is never named; without one, answers name none and
+requests are not checked.

services/sandbox-runtime/daemon/src/incarnation.test.ts

@@ -0,0 +1,41 @@
+import { describe, expect, spyOn, test } from 'bun:test';
+
+import { namesOtherIncarnation, readIncarnation } from './incarnation.ts';
+
+describe('readIncarnation', () => {
+  test('a launch without a stamp names no incarnation', () => {
+    expect(readIncarnation(undefined)).toBeUndefined();
+    expect(readIncarnation('')).toBeUndefined();
+  });
+
+  test('a creation stamp is named as given', () => {
+    expect(readIncarnation('1760000000000')).toBe('1760000000000');
+    expect(readIncarnation('0')).toBe('0');
+  });
+
+  test.each(['1760000000000.5', '-1', ' 1760000000000', 'abc', '1'.repeat(17)])(
+    'a malformed stamp %p is never named',
+    (raw) => {
+      const warn = spyOn(console, 'warn').mockImplementation(() => {});
+      try {
+        expect(readIncarnation(raw)).toBeUndefined();
+        expect(warn).toHaveBeenCalledTimes(1);
+      } finally {
+        warn.mockRestore();
+      }
+    },
+  );
+});
+
+describe('namesOtherIncarnation', () => {
+  test('only a request naming another stamp to a daemon that has one is refused', () => {
+    expect(namesOtherIncarnation('1760000000000', '1760000000001')).toBe(true);
+    expect(namesOtherIncarnation('1760000000000', ['1', '1760000000000'])).toBe(
+      true,
+    );
+    expect(namesOtherIncarnation('1760000000000', '1760000000000')).toBe(false);
+    expect(namesOtherIncarnation('1760000000000', undefined)).toBe(false);
+    expect(namesOtherIncarnation('1760000000000', '')).toBe(false);
+    expect(namesOtherIncarnation(undefined, '1760000000001')).toBe(false);
+  });
+});

services/sandbox-runtime/daemon/src/incarnation.ts

@@ -0,0 +1,36 @@
+// The session incarnation this daemon serves: the creation stamp the spawner
+// records on the backend object (Docker's `tale.created` label) and passes in
+// at launch. Answers name it, so the spawner proves which incarnation answered
+// without asking its backend; an activity request meant for another one is
+// refused before it changes anything.
+
+import { RUNNERD_INCARNATION_ENV } from './protocol.ts';
+
+/** A creation stamp is a millisecond epoch, written as a decimal integer. */
+const STAMP_RE = /^[0-9]{1,16}$/;
+
+/** The stamp from the launch environment, or undefined when there is none. A
+ * malformed value proves nothing about the incarnation, so it is never named. */
+export function readIncarnation(raw: string | undefined): string | undefined {
+  if (raw === undefined || raw === '') return undefined;
+  if (STAMP_RE.test(raw)) return raw;
+  console.warn(
+    `[runnerd] ${RUNNERD_INCARNATION_ENV} is not a creation stamp; answers name no incarnation`,
+  );
+  return undefined;
+}
+
+/** Does a request name an incarnation other than the one served? A request
+ * naming none, or a daemon launched without a stamp, cannot tell. */
+export function namesOtherIncarnation(
+  served: string | undefined,
+  named: string | string[] | undefined,
+): boolean {
+  const value = Array.isArray(named) ? named[0] : named;
+  return (
+    served !== undefined &&
+    value !== undefined &&
+    value !== '' &&
+    value !== served
+  );
+}