Skip to content

fix: stop phantom CoS agents from blocking the update page forever - #5624

Merged
atomantic merged 2 commits into
mainfrom
cos/task-mtiwdio4/agent-d8c5a533
Sep 1, 2026
Merged

fix: stop phantom CoS agents from blocking the update page forever#5624
atomantic merged 2 commits into
mainfrom
cos/task-mtiwdio4/agent-d8c5a533

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

The Apps → Update page sat on "Update paused — 4 CoS agents running" with an
empty agent list underneath, and no way out: updating restarts PortOS, and that
restart was the only thing that would have cleared the count.

The four agents were real processes — codex TUIs whose PRs had merged hours
earlier — that nobody could kill.

Root cause, in four layers

  1. node-pty's Windows backend throws Signals not supported on windows. for
    any signal argument.
    Every TUI kill that passed one was a logged no-op:
    ❌ TUI termination relay failed: Signals not supported on windows. The PTY
    survived, held its worktree locked (EBUSY on cleanup), and stayed in the
    CoS Runner's active set. This hit every TUI termination path — the runner's
    /terminate, /kill, /pause, /terminate-all, the server's
    post-finalize tui:kill relay — plus stopRun on a TUI prompt run through
    the vendored aiToolkit runner, and terminate/kill of a direct-mode TUI agent.
  2. The runner kept advertising the survivors in GET /agents.
  3. syncRunnerAgents adopted whatever the runner advertised, minting phantom
    runnerAgents entries that nothing ever pruned.
  4. The update gate counted map entries, so the phantoms blocked updates forever.

Changes

Each layer is fixed where it belongs.

killProcessTree (server/lib/bufferedSpawn.js, mirrored in the
self-contained aiToolkit/runner.js)
already knew a node-pty handle from a
spawned child — it just forgot the second half of the rule. A non-ChildProcess
killable is now offered the caller's signal first and falls back to a bare
kill() only when the handle rejects it, so node-pty kills land while the CoS
runner's signal-forwarding socket proxy keeps its force-kill. It takes an
injected platform flag so both Windows branches are covered on a Linux runner.

The CoS runner routes every kill through that helper and shares one
force-kill escalation (forceKill.js) instead of four hand-rolled copies. The
escalation drops the agent from the active set, so a process that outlives its
own SIGKILL can no longer be advertised as a live run.

syncRunnerAgents adopts only agents PortOS still holds a live durable
record for, and the orphan sweep gained the reverse pass that retires
runnerAgents entries with nothing behind them. A failed record read stays
distinct from an absent record — one leaves live state alone, the other
retires it.

The update gate counts only tracked ids whose durable record is still live,
so a run PortOS already finalized cannot hold a restart back. It fails
closed
: the records are read through a non-defaulting
readAgentsStateForSafetyCheck (sibling of the Persistent Mind gate's own
trusted read), because loadState substitutes an empty state for a corrupt
file — which would have handed the gate a confident zero and restarted PortOS
out from under live agents.

Defects the fix exposed, closed in the second commit

  • armForceKill refused to arm over a timer /pause had left behind, so
    terminating a paused agent skipped its escalation entirely.
  • The runner's TUI exit handler had no paused guard (its CLI sibling does), so
    a paused TUI was finalized as failed and its task retired — unreachable
    while the kill was throwing, live the moment it landed.
  • A kill relayed for a paused agent reaped the durable record a later resume
    reads.

Test plan

  • server/cos-runner/forceKill.test.js — new; exercises the escalation for real
    (drop-on-grace, re-arm over a stale pause timer, handle clearing, dropState
    scoping, throw containment). The stale-timer case fails against the previous
    implementation, which is how that regression was caught.
  • server/lib/bufferedSpawn.test.js — the pty cases now drive an injected
    platform instead of if (!IS_WIN32) return, so they assert on every runner.
  • filterLiveAgentIds / readAgentRecordOrUnreadable unit tests, the
    update-route gate tests, the runner-sync adoption guard, and the orphan
    sweep's reverse prune all pin their contracts and fail if reverted.
  • Test mocks now import the real isLiveAgentRecord rather than
    re-implementing it; fixtures standing in for a spawned child carry the
    ChildProcess prototype so they exercise the spawned-child branch.
  • Full server suite: 493 affected suites green (11 912 tests). The 15 remaining
    failures on this machine are pre-existing and environmental (missing
    playwright-core, no Python).

Follow-up filed as #5622: the zombie and orphan sweeps still treat the runner's
agent list as proof of liveness.

The Apps > Update page sat on "Update paused — 4 CoS agents running" with an
empty agent list underneath and no way out: updating restarts PortOS, and the
restart was the only thing that would have cleared the count.

Root cause, in four layers:

1. node-pty's Windows backend throws "Signals not supported on windows." for any
   signal argument, so every TUI kill that passed one was a logged no-op. The
   codex/claude PTY survived, held its worktree locked, and stayed in the CoS
   Runner's active set. This hit every TUI termination path — the runner's
   /terminate, /kill, /pause, /terminate-all and the server's post-finalize
   tui:kill relay — plus stopRun on a TUI prompt run through the aiToolkit
   runner, and terminate/kill of a direct-mode TUI agent.
2. The runner kept advertising those survivors in GET /agents.
3. syncRunnerAgents adopted whatever the runner advertised, minting phantom
   runnerAgents entries that nothing ever pruned.
4. The update gate counted map entries, so the phantoms blocked updates.

Each layer is fixed where it belongs:

- killProcessTree (and its self-contained aiToolkit mirror) drops the signal for
  a non-ChildProcess killable on Windows, so a pty kill actually lands. This is
  the one place that already knew a pty from a spawned child; it just forgot the
  second half of the rule.
- The runner routes every kill through it and shares one force-kill escalation
  (armForceKill) instead of four hand-rolled copies. The escalation drops the
  agent from the active set, so a process that outlives its own SIGKILL can no
  longer be advertised as a live run.
- syncRunnerAgents adopts only agents PortOS still has a live record for, and
  the orphan sweep gained the reverse pass that retires runnerAgents entries
  with nothing behind them. A failed record READ is kept distinct from an absent
  record — one leaves live state alone, the other retires it.
- The update gate counts only tracked ids whose durable record is still live, so
  a run PortOS already finalized cannot hold a restart back.

Follow-up filed as #5622: the zombie and orphan sweeps still treat the runner's
agent list as proof of liveness.

Test fixtures that stood in for a spawned child now carry the ChildProcess
prototype, so they exercise the spawned-child branch rather than the pty one.
Follow-ups from review of the phantom-agent fix, three of them defects that
change only became reachable:

- armForceKill refused to arm when the agent already had a killTimer, but
  /pause leaves one behind that neither clears the handle nor drops the map
  entry. A terminate on a paused agent therefore skipped its escalation and
  never removed the entry — stranding the exact phantom the escalation exists
  to reap. A fresh termination now always re-arms, and every timer clears its
  own handle when it fires.
- The runner's TUI exit handler had no `paused` guard, unlike its CLI sibling,
  so a paused TUI was finalized as failed and its task retired. Unreachable
  while the node-pty kill was throwing; live the moment it landed.
- A kill relayed for a paused agent reaped its durable runner record — the
  record a later resume reads. `dropState` is now scoped to an agent the server
  has actually finalized.

killProcessTree no longer drops the signal unconditionally for a
non-ChildProcess killable on Windows: it offers the signal first and falls back
to the bare kill only for a handle that rejects it. node-pty rejects; the CoS
runner's TUI socket proxy does not, and was silently downgrading a force-kill
to a graceful one. It also takes an injected platform flag now, so both Windows
branches are covered on a Linux runner instead of being skipped there.

filterLiveAgentIds now fails CLOSED. It read the records through loadState,
which substitutes an empty default state for a corrupt file — that would have
judged every tracked agent finalized and let the update gate restart PortOS out
from under live agents. It reads through a non-defaulting
readAgentsStateForSafetyCheck instead (the sibling of the Persistent Mind gate's
own trusted read) and counts every tracked id as live when the records cannot be
established.

armForceKill moved into its own module so it is exercised for real rather than
by grepping index.js — the stale-timer bug above was invisible to a
source-inspection test. Test mocks now import the real isLiveAgentRecord instead
of re-implementing it.
@atomantic
atomantic merged commit 68bf5ed into main Sep 1, 2026
7 checks passed
@atomantic
atomantic deleted the cos/task-mtiwdio4/agent-d8c5a533 branch September 1, 2026 18:04
atomantic added a commit that referenced this pull request Sep 1, 2026
…nto #5624 still tests the finalized-record gate

Bare CLI listings without processActive/liveness now go through the pid probe, so pid 1/2/3 no longer look live. Keep those rows tagged live so the test still covers ignoring finalized/missing records rather than the liveness skip.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant