Skip to content

fix(service): re-dispatch interrupted sessions after a hub engine bounce (#1552) - #1553

Open
aarontrowbridge wants to merge 1 commit into
mainfrom
slice/interrupted-session-redispatch
Open

aarontrowbridge wants to merge 1 commit into
mainfrom
slice/interrupted-session-redispatch

Conversation

@aarontrowbridge

Copy link
Copy Markdown
Member

Closes #1552. Read that issue first — the evidence chain (the 2026-09-24 bounce that froze three research campaigns mid-loop) and the ratified design are there.

What this does

A running agent loop is an open POST /session/{id}/message (the SDK's session.prompt()) proxied through the service — so the service already sees every in-flight turn. This makes that signal durable and closes the loop on boot:

  • In-flight journal (inflight_journal.ts, new): the dispatch seam (server.ts, the one place that sees every engine-bound request) appends {start, sessionID} on chat-POST entry and {end} on response close — write-through, best-effort, never a request-path dependency. The journal's gaps ARE the crash record: a start without an end survives both SIGTERM teardowns and engine crashes with zero teardown logic.
  • Derivation (pure): interrupted = latest unmatched start within a recency window (default 12h, AMICODE_RESUME_RECENT_MS), capped at 8 freshest.
  • Boot-time re-dispatch (amicode_service_runner.ts): after engine health, the runner fires ONE mechanical resume turn per interrupted session — [auto-resume] The engine restarted while your loop was in flight (issue #1552). Re-read your session ledger/state and continue exactly where you left off. — fire-and-forget (the chat POST holds until the turn completes, which can be tens of minutes; boot never wedges), one log line per outcome, journal cleared after dispatch, AMICODE_RESUME_DISABLED=1 escape hatch.

The deploying campaign that bounces the hub now gets the nudge too: it verifies its own deploy and finishes, exactly as it would have if the bounce hadn't killed it.

Evidence

  • 21/21 new tests (test/amicode_service_inflight_journal.test.ts): derivation group (start-without-end / matched end / re-start / only-ends / stale / cap / corrupt lines / missing file), exact resume text, the tracking seam (chat POST matched; GETs/SSE/other routes not), never-wedge (a never-answering engine POST doesn't block boot; a failed/non-2xx POST is one log line), the disabled hatch.
  • Full amicode_service_* sweep: 257 passed | 3 skipped (17 files), no regression.
  • pnpm typecheck (workspace): all 4 packages clean.
  • Scope: extension src + test only — no app-bundle/overlay/manifest surface, no engine changes, hub-restart.sh untouched (the runner owns continuity; the script's incident-contract comment gets its follow-up sentence in the ops slice).

Route confirmation

Confirmed against the vendored SDK generation at the manifest's exact upstream base (anomalyco/opencode @ 3104c142, v1.18.30): session.prompt() = POST /session/{id}/message, body {parts: [{type: "text", text}]} — the app's real send path (submit.ts → sdk().api.session.prompt) uses this exact route. The 200 response carries the completed turn only when the turn COMPLETES, hence the fire-and-forget discipline.

Known follow-ups (noted for review)

  1. A resumed turn fires directly to the engine, so it does not re-enter the (just-cleared) journal — a SECOND bounce landing mid-resumed-turn would under-resume. One-line follow-up: route the resume through the service origin so the tracking hook journals it.
  2. hub-restart.sh's incident-contract comment deserves the dual sentence: "the hub never comes back up without resuming the loops it killed."

A hub engine restart (hub-restart.sh's sanctioned systemctl bounce) SIGTERMs
the engine and kills every in-flight agent loop — loops live in the engine
process's memory — and nothing re-dispatched them once the engine returned
(2026-09-24: three campaigns frozen at the bounce instant).

The service already sees every in-flight turn (an open chat POST proxied
through the engine seam), so the fix is entirely in the extension layer:

- track (inflight_journal.ts): the service's dispatch path journals every
  proxied POST /session/{id}/message (the SDK's session.prompt — start on
  entry, end on response close, which fires on completion AND on death);
  best-effort appends only, never a request-path dependency
- derive: a pure fold of the journal into the interrupted set (latest
  unmatched start, within a 12h recency window, capped at 8)
- resume: after engine health, the runner fires ONE mechanical resume turn
  per interrupted session (fire-and-forget — the chat POST holds until the
  turn completes; boot never wedges), then clears the journal so a
  subsequent boot cannot double-resume; a failed resume is one log line,
  never a failed boot
- CLI env surface: AMICODE_INFLIGHT_JOURNAL, AMICODE_RESUME_RECENT_MS,
  AMICODE_RESUME_DISABLED (the test/ops escape hatch)
@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: fc5603d2-02fd-420f-a01f-d17f1632a596


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The hub engine bounce kills all in-flight campaign loops — sessions must survive their own deploy step (interrupted-session re-dispatch)

1 participant