fix(service): re-dispatch interrupted sessions after a hub engine bounce (#1552) - #1553
Open
aarontrowbridge wants to merge 1 commit into
Open
aarontrowbridge wants to merge 1 commit into
aarontrowbridge wants to merge 1 commit into
Conversation
A hub engine restart (hub-restart.sh's sanctioned systemctl bounce) SIGTERMs
the engine and kills every in-flight agent loop — loops live in the engine
process's memory — and nothing re-dispatched them once the engine returned
(2026-09-24: three campaigns frozen at the bounce instant).
The service already sees every in-flight turn (an open chat POST proxied
through the engine seam), so the fix is entirely in the extension layer:
- track (inflight_journal.ts): the service's dispatch path journals every
proxied POST /session/{id}/message (the SDK's session.prompt — start on
entry, end on response close, which fires on completion AND on death);
best-effort appends only, never a request-path dependency
- derive: a pure fold of the journal into the interrupted set (latest
unmatched start, within a 12h recency window, capped at 8)
- resume: after engine health, the runner fires ONE mechanical resume turn
per interrupted session (fire-and-forget — the chat POST holds until the
turn completes; boot never wedges), then clears the journal so a
subsequent boot cannot double-resume; a failed resume is one log line,
never a failed boot
- CLI env surface: AMICODE_INFLIGHT_JOURNAL, AMICODE_RESUME_RECENT_MS,
AMICODE_RESUME_DISABLED (the test/ops escape hatch)
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
aarontrowbridge
marked this pull request as ready for review
September 25, 2026 08:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1552. Read that issue first — the evidence chain (the 2026-09-24 bounce that froze three research campaigns mid-loop) and the ratified design are there.
What this does
A running agent loop is an open
POST /session/{id}/message(the SDK'ssession.prompt()) proxied through the service — so the service already sees every in-flight turn. This makes that signal durable and closes the loop on boot:inflight_journal.ts, new): the dispatch seam (server.ts, the one place that sees every engine-bound request) appends{start, sessionID}on chat-POST entry and{end}on response close — write-through, best-effort, never a request-path dependency. The journal's gaps ARE the crash record: a start without an end survives both SIGTERM teardowns and engine crashes with zero teardown logic.AMICODE_RESUME_RECENT_MS), capped at 8 freshest.amicode_service_runner.ts): after engine health, the runner fires ONE mechanical resume turn per interrupted session —[auto-resume] The engine restarted while your loop was in flight (issue #1552). Re-read your session ledger/state and continue exactly where you left off.— fire-and-forget (the chat POST holds until the turn completes, which can be tens of minutes; boot never wedges), one log line per outcome, journal cleared after dispatch,AMICODE_RESUME_DISABLED=1escape hatch.The deploying campaign that bounces the hub now gets the nudge too: it verifies its own deploy and finishes, exactly as it would have if the bounce hadn't killed it.
Evidence
test/amicode_service_inflight_journal.test.ts): derivation group (start-without-end / matched end / re-start / only-ends / stale / cap / corrupt lines / missing file), exact resume text, the tracking seam (chat POST matched; GETs/SSE/other routes not), never-wedge (a never-answering engine POST doesn't block boot; a failed/non-2xx POST is one log line), the disabled hatch.amicode_service_*sweep: 257 passed | 3 skipped (17 files), no regression.pnpm typecheck(workspace): all 4 packages clean.hub-restart.shuntouched (the runner owns continuity; the script's incident-contract comment gets its follow-up sentence in the ops slice).Route confirmation
Confirmed against the vendored SDK generation at the manifest's exact upstream base (anomalyco/opencode @ 3104c142, v1.18.30):
session.prompt()=POST /session/{id}/message, body{parts: [{type: "text", text}]}— the app's real send path (submit.ts→sdk().api.session.prompt) uses this exact route. The 200 response carries the completed turn only when the turn COMPLETES, hence the fire-and-forget discipline.Known follow-ups (noted for review)
hub-restart.sh's incident-contract comment deserves the dual sentence: "the hub never comes back up without resuming the loops it killed."