The hub engine bounce kills all in-flight campaign loops — sessions must survive their own deploy step (interrupted-session re-dispatch)

[aarontrowbridge]

What happened (2026-09-24, live fleet evidence)

The user left three research-mode campaigns running hub-side overnight. The next morning, every campaign was frozen at exactly the same instant — ~18:10 EDT — and nothing ran overnight. The histories were intact (all messages persisted); the loops were dead.

Root cause: the sanctioned engine restart (the deploy bounce) killed them. One of the three campaigns — the harness-switcher deploy, our own #1550 — executed the hub-restart.sh runbook faithfully: stage → swap → restart. The restart is the sanctioned single-verb path and it worked exactly as designed: the engine came back up (new binary sha dbf8be49…, confirmed in server.log's spawn cycle), the hub was never down. But session agent loops live in the engine process's memory — SIGTERM = every in-flight loop dead, and no mechanism re-dispatches an interrupted session after the engine returns.

The evidence chain

• The deploying campaign's own last message narrates it: "Staged and swapped — the branch engine image (sha dbf8be49…) is now the live binary… Now the restart — this is the bounce: this chat and all fleet panels drop for a few seconds and reconnect." — mid-turn, by-the-book.
• The Legatissimo CI campaign was mid-poll of a detached test-suite run; the H13 refinement subagents were mid-flight. All three froze within the same minute (session last-message timestamps cluster at the bounce; the service-runner log shows the matching SIGTERM → spawn cycle).
• hub-restart.sh already knows sessions die at restarts — its header cites the 2026-08-30 double incident ("a session hosted ON the hub kills its own runtime mid-command"). Its safety contract covers the hub never stays down. The missing law is the dual: the hub never comes back up without resuming the loops it killed.

Why this matters beyond last night

The bounce is not an edge case — it's the deploy path (ops/hub-restart.sh, driven by the extension's "Restart Hub Server" command and callable by agents hub-side). Every engine upgrade, every staged binary, every ops bounce takes this shape. As long as in-flight campaigns die at every deploy:

• overnight/long campaigns can never be safe (any deploy mid-flight kills them silently),
• an agent deploying a new engine always kills its own session plus every bystander campaign,
• the fleet promise ("agents work while you sleep") fails precisely at the moment we exercise the infrastructure.

The signal is available: in-flight turns are open chat POSTs

A running agent loop is an open POST /session/:id/chat request — the service proxies these to the engine, so the service already sees every in-flight turn. The fix can be entirely in our layer (packages/extension/src/amicode_service/ + amicode_service_runner.ts), no engine-core changes:

1. Track: the service records session IDs with open chat requests (append-on-start, mark-on-end; a write-through sidecar like active-sessions.jsonl also covers engine crashes, not just SIGTERM).
2. Snapshot: on SIGTERM teardown (and equivalently, on the next boot after a crash — reads the sidecar), sessions with start-without-end = interrupted.
3. Re-dispatch: on the next boot after engine health, the runner POSTs one mechanical resume turn per interrupted session via its engine credential — the ledger-re-read nudge ("[auto-resume] the engine restarted mid-loop — re-read your ledger/state and continue"). Capped, never-wedge-boot, one log line per dispatch.
4. The self-restart case resolves naturally: the deploying campaign gets the nudge, verifies its own deploy, finishes.

Acceptance shape

• A SIGTERM bounce with two in-flight chat turns results in both sessions receiving exactly one resume turn after the next engine health ✓, third-party sessions untouched ✓, boot never wedged by a failed resume POST ✓ (logged, moved on).
• An engine crash (no SIGTERM) produces the same resume via the sidecar.
• hub-restart.sh's incident contract gains the dual sentence; the runbook's restart step stays the single verb (the runner owns continuity, not the shell script — the script must stay shell-only).
• Unit tests on the pure derivation (interrupted-set from the journal, resume-message shape, the cap, the never-wedge failure path).

Related

• The hub cutover: main's architecture onto the production hub (spec-20260910-080000) #955 — the service runner (the layer that owns this)
• service: campaign routes — GET /amicode/campaigns + /amicode/campaign (parsed ledger sections) + widget-allowlist entries #658 — the campaign ledger (the resume nudge's ground truth is the ledger grammar)
• Fleet Studio B2b (#1540) — remote peer-studio control + live streaming #1547 — Fleet Studio B2b (hub-active flows are exactly the sessions at risk)
• The separate unknown worth one test: whether a client disconnect (lid-close, tunnel death) also stalls in-flight loops — last night can't answer it because the bounce killed everything first. This issue is the engine-side half; if the disconnect half also kills loops, that's a follow-up with its own evidence.

The frozen campaigns' state (recoverable)

All three campaigns are resumable today by hand — the ledger grammar means a manual nudge continues each from its own state (verified live: the Legatissimo campaign resumed and re-ran its suite after a nudge). The fix makes that resume automatic, so nobody has to know.