fix: stop phantom CoS agents from blocking the update page forever - #5624
Merged
Conversation
The Apps > Update page sat on "Update paused — 4 CoS agents running" with an empty agent list underneath and no way out: updating restarts PortOS, and the restart was the only thing that would have cleared the count. Root cause, in four layers: 1. node-pty's Windows backend throws "Signals not supported on windows." for any signal argument, so every TUI kill that passed one was a logged no-op. The codex/claude PTY survived, held its worktree locked, and stayed in the CoS Runner's active set. This hit every TUI termination path — the runner's /terminate, /kill, /pause, /terminate-all and the server's post-finalize tui:kill relay — plus stopRun on a TUI prompt run through the aiToolkit runner, and terminate/kill of a direct-mode TUI agent. 2. The runner kept advertising those survivors in GET /agents. 3. syncRunnerAgents adopted whatever the runner advertised, minting phantom runnerAgents entries that nothing ever pruned. 4. The update gate counted map entries, so the phantoms blocked updates. Each layer is fixed where it belongs: - killProcessTree (and its self-contained aiToolkit mirror) drops the signal for a non-ChildProcess killable on Windows, so a pty kill actually lands. This is the one place that already knew a pty from a spawned child; it just forgot the second half of the rule. - The runner routes every kill through it and shares one force-kill escalation (armForceKill) instead of four hand-rolled copies. The escalation drops the agent from the active set, so a process that outlives its own SIGKILL can no longer be advertised as a live run. - syncRunnerAgents adopts only agents PortOS still has a live record for, and the orphan sweep gained the reverse pass that retires runnerAgents entries with nothing behind them. A failed record READ is kept distinct from an absent record — one leaves live state alone, the other retires it. - The update gate counts only tracked ids whose durable record is still live, so a run PortOS already finalized cannot hold a restart back. Follow-up filed as #5622: the zombie and orphan sweeps still treat the runner's agent list as proof of liveness. Test fixtures that stood in for a spawned child now carry the ChildProcess prototype, so they exercise the spawned-child branch rather than the pty one.
Follow-ups from review of the phantom-agent fix, three of them defects that change only became reachable: - armForceKill refused to arm when the agent already had a killTimer, but /pause leaves one behind that neither clears the handle nor drops the map entry. A terminate on a paused agent therefore skipped its escalation and never removed the entry — stranding the exact phantom the escalation exists to reap. A fresh termination now always re-arms, and every timer clears its own handle when it fires. - The runner's TUI exit handler had no `paused` guard, unlike its CLI sibling, so a paused TUI was finalized as failed and its task retired. Unreachable while the node-pty kill was throwing; live the moment it landed. - A kill relayed for a paused agent reaped its durable runner record — the record a later resume reads. `dropState` is now scoped to an agent the server has actually finalized. killProcessTree no longer drops the signal unconditionally for a non-ChildProcess killable on Windows: it offers the signal first and falls back to the bare kill only for a handle that rejects it. node-pty rejects; the CoS runner's TUI socket proxy does not, and was silently downgrading a force-kill to a graceful one. It also takes an injected platform flag now, so both Windows branches are covered on a Linux runner instead of being skipped there. filterLiveAgentIds now fails CLOSED. It read the records through loadState, which substitutes an empty default state for a corrupt file — that would have judged every tracked agent finalized and let the update gate restart PortOS out from under live agents. It reads through a non-defaulting readAgentsStateForSafetyCheck instead (the sibling of the Persistent Mind gate's own trusted read) and counts every tracked id as live when the records cannot be established. armForceKill moved into its own module so it is exercised for real rather than by grepping index.js — the stale-timer bug above was invisible to a source-inspection test. Test mocks now import the real isLiveAgentRecord instead of re-implementing it.
atomantic
added a commit
that referenced
this pull request
Sep 1, 2026
…nto #5624 still tests the finalized-record gate Bare CLI listings without processActive/liveness now go through the pid probe, so pid 1/2/3 no longer look live. Keep those rows tagged live so the test still covers ignoring finalized/missing records rather than the liveness skip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Apps → Update page sat on "Update paused — 4 CoS agents running" with an
empty agent list underneath, and no way out: updating restarts PortOS, and that
restart was the only thing that would have cleared the count.
The four agents were real processes —
codexTUIs whose PRs had merged hoursearlier — that nobody could kill.
Root cause, in four layers
Signals not supported on windows.forany signal argument. Every TUI kill that passed one was a logged no-op:
❌ TUI termination relay failed: Signals not supported on windows.The PTYsurvived, held its worktree locked (
EBUSYon cleanup), and stayed in theCoS Runner's active set. This hit every TUI termination path — the runner's
/terminate,/kill,/pause,/terminate-all, the server'spost-finalize
tui:killrelay — plusstopRunon a TUI prompt run throughthe vendored aiToolkit runner, and terminate/kill of a direct-mode TUI agent.
GET /agents.syncRunnerAgentsadopted whatever the runner advertised, minting phantomrunnerAgentsentries that nothing ever pruned.Changes
Each layer is fixed where it belongs.
killProcessTree(server/lib/bufferedSpawn.js, mirrored in theself-contained
aiToolkit/runner.js) already knew a node-pty handle from aspawned child — it just forgot the second half of the rule. A non-
ChildProcesskillable is now offered the caller's signal first and falls back to a bare
kill()only when the handle rejects it, so node-pty kills land while the CoSrunner's signal-forwarding socket proxy keeps its force-kill. It takes an
injected platform flag so both Windows branches are covered on a Linux runner.
The CoS runner routes every kill through that helper and shares one
force-kill escalation (
forceKill.js) instead of four hand-rolled copies. Theescalation drops the agent from the active set, so a process that outlives its
own SIGKILL can no longer be advertised as a live run.
syncRunnerAgentsadopts only agents PortOS still holds a live durablerecord for, and the orphan sweep gained the reverse pass that retires
runnerAgentsentries with nothing behind them. A failed record read staysdistinct from an absent record — one leaves live state alone, the other
retires it.
The update gate counts only tracked ids whose durable record is still live,
so a run PortOS already finalized cannot hold a restart back. It fails
closed: the records are read through a non-defaulting
readAgentsStateForSafetyCheck(sibling of the Persistent Mind gate's owntrusted read), because
loadStatesubstitutes an empty state for a corruptfile — which would have handed the gate a confident zero and restarted PortOS
out from under live agents.
Defects the fix exposed, closed in the second commit
armForceKillrefused to arm over a timer/pausehad left behind, soterminating a paused agent skipped its escalation entirely.
pausedguard (its CLI sibling does), soa paused TUI was finalized as failed and its task retired — unreachable
while the kill was throwing, live the moment it landed.
reads.
Test plan
server/cos-runner/forceKill.test.js— new; exercises the escalation for real(drop-on-grace, re-arm over a stale pause timer, handle clearing,
dropStatescoping, throw containment). The stale-timer case fails against the previous
implementation, which is how that regression was caught.
server/lib/bufferedSpawn.test.js— the pty cases now drive an injectedplatform instead of
if (!IS_WIN32) return, so they assert on every runner.filterLiveAgentIds/readAgentRecordOrUnreadableunit tests, theupdate-route gate tests, the runner-sync adoption guard, and the orphan
sweep's reverse prune all pin their contracts and fail if reverted.
isLiveAgentRecordrather thanre-implementing it; fixtures standing in for a spawned child carry the
ChildProcessprototype so they exercise the spawned-child branch.failures on this machine are pre-existing and environmental (missing
playwright-core, no Python).Follow-up filed as #5622: the zombie and orphan sweeps still treat the runner's
agent list as proof of liveness.