fix(platform): let agent turns ride out a sandbox spawner restart

[yannickmonney]

Any spawner restart killed every in-flight agent run on the host, even though the CLI kept running in its healthy container: a crash, an OOM kill at its 1 GiB limit, a deploy or a dockerd restart. A short network partition did the same. The platform's exec drain gave up after five failed re-attaches over about 7.5 s. Both drive hosts then settled the run as failed, while the agent in the sandbox kept working until runnerd's 30-minute orphan deadline.

Stacked on #4543, and reuses its transient-connection classifier and retry budget. After #4543 merges, this is retargeted to main and rebased.

Changes

fix(platform): let agent turns ride out a sandbox spawner outage

A spawner restart (a crash, an OOM at its 1 GiB limit, a deploy, a
dockerd restart) or a short partition failed every in-flight task and
automation agent run on the host: the resilient drain allowed five
consecutive failed re-attaches on a linear 0.5..2.5 s backoff, about
7.5 s, and counted a refused connection, a 502/503 and a timeout alike,
so the drive window threw and both hosts cancelled the exec and settled
the run turn_crashed, while runnerd would have kept the CLI working for
its 30-minute sliding orphan window.

The drain now tells transport from verdict. A spawner it cannot reach
(every fetch failure, the refused/reset/unresolved connections the
existing classifier waits out included), a stream that broke mid-read,
a 429/502/503/504 or a timed-out call is an outage: a drive window waits
the spawner's retry-after, or the acquire/create backoff doubling from
250 ms to 5 s (the same SpawnerRetryBudget, bounded by the window), and
attaches again after its cursor without spending its failure budget. A
404, a replay or protocol gap and a stream-reported error stay verdicts,
and drains outside the drive windows keep the old budget.

A window that ends with the stream still lost, or whose checkpoint read
the spawner did not answer, ends running with the outage's start; the
host chains the next window 5 s later with spawnerOutageSince in the
drive payload, so it resumes from the checkpoint and the outage is
measured across windows. Once it reaches 10 minutes (at most a third of
TALE_EXTERNAL_TURN_DEADLINE_MS, so the exec found again is the one left)
the host cancels the exec and settles the run failed through the same
exactly-once fences as a drain failure; the work-turn deadline still
applies at every window. The re-attach watchdog is unchanged: a queued
delayed window counts as a live chain, and it probes before it claims.

fix(sandbox): listen before boot adoption and hold session calls on 503

A restarting spawner opened its port only after the host lock, the boot
sweep, the re-adoption of every running session (a backend listing plus
an endpoint read per session, eight at a time, and runnerd probes) and
the device hub's placement load. Every platform call in that window met
a refused connection, which only the acquire, create and drive-window
drain wait out; on a host with many sessions, or a docker daemon slow
after its own restart, the window ran long enough to fail starts.

The listener now opens once the host lock and the boot sweep have run,
before adoption. Until adoption has run and the hub has loaded its
placements, the calls whose answer depends on them (every session route,
the workspace inventory, the capacity read, an organization teardown and
a device disconnect) answer 503 session_unavailable with retry-after: 1,
the same answer session-routes already gives for a session it cannot yet
place, and never a 404 the platform would take for a lost session. A
restart thus reads as a short "not now" the platform waits out at its
retry-after. /health, /v1/limits and /v1/devices answer as before, and
the signal handlers are installed with the listener, so a SIGTERM during
adoption stops the listener and releases the host lock as after boot.

fix(sandbox): hold the deploy drain calls while boot adoption runs

With the listener open before boot adoption, POST /v1/drain and GET
/v1/drain-status were reachable mid-adoption. A drain latched then made
adoptExisting and adoptSession stop at their isDraining checks, so every
session not yet adopted answered 404, which the platform settles as
session_gone; and drain-status counted only the registry adoption had
filled so far, often 0, so a configuration activation read the spawner
as drained and ran docker restart under live sessions.

Both routes now sit behind the adoption gate and answer 503
session_unavailable with retry-after: 1 until adoption and the hub's
placement load are done. The CLI's control call exits 1 on that answer:
a configuration activation, whose pending intent is written before the
drain, throws and stays pending for its retry, and tale deploy's
best-effort drainSandbox proceeds as it does for an unreachable spawner.

fix(sandbox): answer /health starting until boot adoption has run

Opening the listener before boot adoption also made /health answer 200
during adoption, while before it could not be reached until adoption and
the hub's placement load had finished. Every probe changed meaning: on
the documented Kubernetes Deployment (RollingUpdate, readinessProbe on
/health) a new spawner Pod went Ready mid-adoption and the old Pod was
retired while the new one answered 503 to every session call, and
Docker's healthcheck and the CLI's waitForRuntime passed early enough to
drain a spawner part-way through adoption.

While adoption is pending /health now answers 503 {"status":"starting"}
without probing the backend, so ready again means adopted. The session
routes still answer 503 rather than refuse the connection, and Compose,
which routes by network alias whatever the health, keeps delivering
them during a Docker restart.

refactor(sandbox): lift the boot's adoption order into a tested step

The point of opening the listener before boot adoption lived only in
main(), which no test can run: the router tests stubbed
BootAdoption.prototype.pending, so moving Bun.serve back after adoption,
or ending the gate before the hub had loaded its placements, left every
test green.

main() now runs the listener, adoption and the hub load through the
exported listenThenAdopt(steps, adoption). It begins the gate before the
listener accepts a call, logs an adoption that throws without skipping
the hub load (as main's catch did), and ends the gate however the steps
end. The periodic session sweep moves unchanged into startSessionSweep.
Tests pin the order (listen, adopt, hub, all with the gate held, then
released), a failed adoption and a failed hub load, and drive the
module's own gate through a pending adoption: /health answers starting,
the deploy drain, drain status and a session read answer 503, and drain
status answers 200 once adoption ends. Swapping listen after adopt
fails two of them.

fix(platform): fail a drain on an offline device instead of waiting

The drain that rides out a spawner outage read every 503 as "not now",
the device hub's 503 device_offline included: the refused-answer path
cancelled the body unread, so the x-tale-sandbox-device header and the
device_offline body were never consulted. A turn whose device went away
(a laptop closed) then waited the full 10-minute outage budget instead
of failing in about 7.5 s, and settled saying the sandbox service could
not be reached, though the service had answered.

An exec, attach or checkpoint answer of 503 carrying the device header
is now read, and a device_offline body becomes SandboxDeviceOfflineError,
a verdict: the drain spends its five consecutive failures on it (0.5 to
2.5 s apart, about 7.5 s) and throws it, so the automation host's
failure reason names the device; the checkpoint read retries it like
any 503 and then throws it, and the drive window rethrows it rather than
ending running. A device spawner's own 503 session_unavailable, which
carries the same header, is still ridden out.

docs(platform): say the turn deadline also bounds a spawner outage

TALE_EXTERNAL_TURN_DEADLINE_MS now also sets how long a running agent
turn waits out an unreachable sandbox service before its run fails:
SPAWNER_OUTAGE_BUDGET_MS = min(10 min, deadline / 3), so 10 minutes at
the 30-minute default and shorter whenever the deadline is lowered below
30 minutes. The operator reference in EN, DE and FR and .env.example
described only runnerd's orphan window, leaving nothing to tell an
operator that a run may now wait about 10 minutes before failing, or
that lowering this variable shortens that wait.

Verification

• These services/platform vitest suites pass:
  • session_client (93 tests)
  • external_turn_shared (71)
  • agent_run_host.reap (13)
  • The new agent_host.spawner_outage suite. It runs the real drain and window over a fake transport: the spawner dies mid-stream, then refuses connections. The run continues and the next window attaches from the checkpoint and settles once. Past the budget it settles failed once.
  • The checkpoint, activity, shutdown, retry-resume, steer-fallback and settle-body suites, plus the legacy agent-flow guard.
  • The automation-protocol, task-policies and runner jobs.
• services/sandbox: server.test.ts covers the boot gate. It checks the listen → adopt → hub order with the gate held throughout, and that a failed adoption still ends the gate. While adoption is pending, /health reads starting and drain and drain-status answer 503.
• @tale/platform and @tale/sandbox typecheck and lint pass, and lint:manual is ok.
• Adversarial review found two major issues, and both are fixed here:
  • A deploy drain during adoption could cut adoption short.
  • /health reported ready before adoption had finished, so a Kubernetes rollout could retire the old Pod too early.
• Its three minor findings are fixed too: an offline device fails fast as before, the boot order is tested, and the turn-deadline docs are updated in EN/DE/FR.
• Not exercised against a live spawner restart.

Behaviour changes

• A running turn waits out an unreachable spawner for at most a third of TALE_EXTERNAL_TURN_DEADLINE_MS, which is 10 minutes by default. Past that it fails as before.
• During boot adoption the spawner listens and answers 503 session_unavailable with retry-after: 1 to session calls, instead of refusing connections. /health answers 503 starting until adoption has run.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).