Skip to content

[1/3] fix: bound Windows ACP shutdown and retain failed cleanup - #2

Closed
slashdevcorpse wants to merge 6 commits into
mainfrom
fix/windows-process-tree-cleanup
Closed

[1/3] fix: bound Windows ACP shutdown and retain failed cleanup#2
slashdevcorpse wants to merge 6 commits into
mainfrom
fix/windows-process-tree-cleanup

Conversation

@slashdevcorpse

@slashdevcorpse slashdevcorpse commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Related issue

Refs LodyAI#429

Problem / pressure

Cleanup can discard owned child handles or hide shutdown failures. PID-based taskkill also creates a reuse race between the liveness check and invocation. Pending terminal starts and queued machine requests can be admitted while shutdown tears down their owners, and stalled disposal can consume the forced-cleanup deadline.

Summary

  • Signal Windows children through their retained ChildProcess/libuv handle and bound observed-exit waiting. Remove cached-PID lookup and taskkill entirely. This layer terminates the owned root; PR [3/3] fix: own Windows process trees with native job supervision #4 supplies durable descendant ownership with a private Windows Job Object.
  • Retain failed ACP startup, authentication and sandbox owners until observed exit. Failed cleanup prevents successor startup and successful authentication reporting; later retries retain the exact child handle.
  • Coalesce session termination and upgrade concurrent force requests. Bound terminal disposal and process waits; failed cleanup remains retryable, including after late root-exit events.
  • Close terminal admission synchronously, drain pending starts, then dispose every published handle. Report actual terminal exits and aggregate failures.
  • Stop machine-request admission and reject discarded queued/new requests while preserving already-running handler outcomes.
  • Reserve 15 seconds for graceful shutdown, 15 seconds for forced cleanup, and a separate bounded telemetry flush. Force retained workspace and preparing-session owners concurrently.
  • Independently attempt PTY and local-endpoint teardown. Retain shared cloud/watch dependencies while failed runtimes still own resources, then retry their cleanup without losing ownership.

Stack order

Native GitHub stack LodyAI#6 in slashdevcorpse/Lody, created with gh stack link:

  1. [1/3] fix: bound Windows ACP shutdown and retain failed cleanup #2 — bounded shutdown and retained cleanup ownership; base main.
  2. [2/3] fix: protect background sessions and record resource history #3 — background-aware eviction and bounded resource history; base fix/windows-process-tree-cleanup.
  3. [3/3] fix: own Windows process trees with native job supervision #4 — native Windows Job Object ownership; base fix/background-session-resource-history.

Replaces the closed cross-fork LodyAI#430 and LodyAI#435. Each PR has an incremental diff against its predecessor. GitHub native stacks require all branches in the same repository, so this stack is in the contributor fork. It has not merged or landed upstream; LodyAI#429 remains open.

Landing dependency: Ship the complete three-PR stack for the Windows process-tree fix. This first slice supplies safe retained-root termination; PR #4 supplies required descendant and crash ownership. Do not ship PR #2 alone as complete Windows tree cleanup.

Before / after

Before After
Windows cleanup targets a cached PID after an asynchronous gap Signals the retained OS child handle and observes its exit
Failed startup/authentication cleanup permits a successor or success response Failure propagates and the live child remains owned
Pending terminal starts and queued requests cross shutdown admission Admission closes before draining and discarding queued work
Forced exit or terminal disposal can stall shutdown Bounded waits reserve time for forced cleanup
Manager clears failed owners or terminal release fabricates an exit Exact failed owners stay retryable; exit status is observed

Test plan

  • 156 focused shutdown/lifecycle tests passed after the review fixes, including real Windows retained-handle and failed-spawn tests. The native five-process descendant proof is in PR [3/3] fix: own Windows process trees with native job supervision #4.
  • Focused authentication/recovery regressions passed, including pending/failed cleanup ownership and non-retryable cleanup failures. Four full-file Windows argument/path failures remain previously reproduced baseline failures.
  • CLI typecheck, changed-file formatting, whitespace checks and root check:quick passed (zero lint errors plus boundary guards).
  • GitHub CI at 720e84c passed Static checks, Tests and Desktop E2E; this PR does not claim a green full local repository suite.

Context handoff

Instructions for reviewing agents

  • Review focus: Trace Windows helper failures through Session and SessionManager; verify terminal real-exit reporting, the graceful-to-force deadline, coalesced force upgrades, closed admission, and late-exit retention.
  • Decisions to challenge: Failed runtime ownership retains shared dependencies while independent PTY/endpoint cleanup runs; retained handles authorize termination without a PID lookup.
  • Plausible failures / evidence gaps: The Windows test proves retained-root-handle cleanup. Durable descendant and crash ownership are supplied and tested in PR [3/3] fix: own Windows process trees with native job supervision #4; real-provider residency remains unproven.

Authoring context

  • User goal / directives: Investigate suspected process retention and submit an upstream fix, including bounded disposal, failed-session retention, truthful terminal status, and actual Windows descendant verification.
  • Constraints / non-goals: Preserve unrelated work; no native Codex changes, global process scans, image-name killing, or claims that all subagent MCP retention is fixed.
  • Risk-bearing decisions: Cleanup now rejects instead of fabricating success; manager failures stop shared-resource teardown, retaining state for retry. Timed-out terminal disposal remains pending rather than launching duplicate disposal.
  • Destructive or irreversible behavior: Explicit shutdown terminates owned live roots, including forced escalation. PR [3/3] fix: own Windows process trees with native job supervision #4 adds descendant ownership. Tests terminate only their own synthetic processes.
  • Deliberately not done or tested: This first PR excludes Windows Job Objects and background eviction/resource history, which are separate dependent PRs. No real user sessions were terminated.
  • Unknowns / confidence: The current focused shutdown tests and retained OS-handle proof passed at 720e84c. Exact-head CI passed; provider-specific residency is unproven and crash ownership is handled by PR [3/3] fix: own Windows process trees with native job supervision #4.

Latest review verification

At 720e84c, the stack includes the eight latest PR #2 review resolutions from 720e84c: PTY admission/draining and retry retention, authentication cleanup ownership, startup owner admission, spawn-error handling, explicit Windows termination semantics and non-retryable cleanup failures. PR #4 supplies the requested descendant Job Object ownership. Local composed verification and exact-head GitHub Static checks, Tests and Desktop E2E all passed. The top PR also passed Windows x64, Windows ARM64 and the same-run dual-architecture npm archive verification.

Copilot AI lite review requested due to automatic review settings September 6, 2026 05:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Sep 6, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 413710a9-cdca-4a92-a3da-418b795c6873


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@slashdevcorpse

Copy link
Copy Markdown
Owner Author

Corrected the stack trunk: fork main was 231 upstream commits behind the original patch baseline, which caused unrelated history to appear in this PR. Fast-forwarded fork main to the original baseline 12f919b without changing any implementation branch. The intended bottom-layer range contains four shutdown commits across 24 files (+1892/-150). Parts 2 and 3 retain their existing incremental bases.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 809 files

Note: This PR contains a large number of files. cubic selects up to 200 of the highest-priority eligible files for this review, so some files may not have been reviewed.

Re-trigger cubic

Comment thread .github/scripts/pr-issue-link.mjs
Comment thread apps/cli/src/utils/windows-process-tree.ts Outdated
Comment thread .github/workflows/pr-policy.yml
Comment thread apps/cli/src/agent/deepseek-harness-runtime.ts Outdated
Comment thread apps/cli/src/agent/acp-authentication-output.ts
Comment thread apps/cli/src/lib/machine-runtime.ts
Comment thread apps/cli/src/utils/stream.ts
Comment thread e2e/journeys/author-result.schema.json
Comment thread apps/electron/src/system-language-argument.ts
@slashdevcorpse

Copy link
Copy Markdown
Owner Author

Verified corrected GitHub PR diff: 24 files, +1892/-150, exactly four shutdown commits. GitHub retained the stale base in native stack metadata after main advanced, so the same PRs were unstacked, the bottom base refreshed, and relinked as native stack LodyAI#6. PR numbers and all implementation commit SHAs are unchanged.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 23 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/cli/src/lib/lody-fleet.ts
Comment thread apps/cli/src/agent/acp-authentication.ts
Comment thread apps/cli/src/session/session-sandbox.ts
Comment thread apps/cli/src/utils/windows-child-process.ts
Comment thread apps/cli/src/session/session-sandbox.ts
Comment thread apps/cli/src/session/session.ts
Comment thread apps/cli/tests/lody-fleet-shutdown.test.ts
Comment thread apps/cli/src/agent/acp-authentication.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 17 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/cli/tests/session-terminate-cleanup.test.ts">

<violation number="1" location="apps/cli/tests/session-terminate-cleanup.test.ts:443">
P3: The new serialization test relies on the in-flight createAgent rejecting with any error, but the only reason `first` rejects quickly is that `createProcessHandle` builds a plain EventEmitter child with no `stdout`/`stderr` properties, so `if (!agentProcess.stdout) throw` fires inside `createAgentOnce`. `first` is also left floating (never awaited directly). If the mock is ever made to expose `stdout`/`stdin` streams, `createAcpClient` would drive a real ACP handshake against a silent child and `await rejected` would stall on the ACP startup timeout instead of failing fast. Make the rejection deterministic by supplying `createProcessHandle` streams that timeout/no-op or by aborting via `abortSignal`, and assert the specific error.</violation>
</file>

<file name="apps/cli/src/lib/lody-fleet.ts">

<violation number="1" location="apps/cli/src/lib/lody-fleet.ts:542">
P3: In `forceTerminateSessions`, `terminal-pty-service.closeAll()` throws `AggregateError('Terminal PTY cleanup failed', [e1, e2, ...])` when one or more PTY kills fail. That nested `AggregateError` is pushed as a single element into the outer `AggregateError('Forced fleet process cleanup failed')`, so the resulting `errors` array is `[AggregateError([...]), ...workspaceRejections]` rather than a flat list of individual failure reasons. Any consumer that iterates `.errors` (or the `toMatchObject({ errors: [...] })` assertion in the new test) won't see the underlying PTY kill failures — the test only passes because its mock throws a plain `Error` instead of the real `AggregateError`. Flatten the terminal failure into the outer array so the individual PTY failures surface consistently with the workspace failures.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

args: ['acp'],
} as Parameters<Session['createAgent']>[0];
const first = session.createAgent(callbacks);
const rejected = expect(first).rejects.toThrow();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new serialization test relies on the in-flight createAgent rejecting with any error, but the only reason first rejects quickly is that createProcessHandle builds a plain EventEmitter child with no stdout/stderr properties, so if (!agentProcess.stdout) throw fires inside createAgentOnce. first is also left floating (never awaited directly). If the mock is ever made to expose stdout/stdin streams, createAcpClient would drive a real ACP handshake against a silent child and await rejected would stall on the ACP startup timeout instead of failing fast. Make the rejection deterministic by supplying createProcessHandle streams that timeout/no-op or by aborting via abortSignal, and assert the specific error.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/cli/tests/session-terminate-cleanup.test.ts, line 443:

<comment>The new serialization test relies on the in-flight createAgent rejecting with any error, but the only reason `first` rejects quickly is that `createProcessHandle` builds a plain EventEmitter child with no `stdout`/`stderr` properties, so `if (!agentProcess.stdout) throw` fires inside `createAgentOnce`. `first` is also left floating (never awaited directly). If the mock is ever made to expose `stdout`/`stdin` streams, `createAcpClient` would drive a real ACP handshake against a silent child and `await rejected` would stall on the ACP startup timeout instead of failing fast. Make the rejection deterministic by supplying `createProcessHandle` streams that timeout/no-op or by aborting via `abortSignal`, and assert the specific error.</comment>

<file context>
@@ -401,9 +401,50 @@ it('propagates failed startup cleanup and retains ownership for shutdown', async
+    args: ['acp'],
+  } as Parameters<Session['createAgent']>[0];
+  const first = session.createAgent(callbacks);
+  const rejected = expect(first).rejects.toThrow();
+  await spawning;
+  await expect(session.createAgent(callbacks)).rejects.toThrow('Previous agent ownership');
</file context>

try {
this.terminalPtyService.closeAll();
} catch (error) {
terminalFailures.push(error);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: In forceTerminateSessions, terminal-pty-service.closeAll() throws AggregateError('Terminal PTY cleanup failed', [e1, e2, ...]) when one or more PTY kills fail. That nested AggregateError is pushed as a single element into the outer AggregateError('Forced fleet process cleanup failed'), so the resulting errors array is [AggregateError([...]), ...workspaceRejections] rather than a flat list of individual failure reasons. Any consumer that iterates .errors (or the toMatchObject({ errors: [...] }) assertion in the new test) won't see the underlying PTY kill failures — the test only passes because its mock throws a plain Error instead of the real AggregateError. Flatten the terminal failure into the outer array so the individual PTY failures surface consistently with the workspace failures.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/cli/src/lib/lody-fleet.ts, line 542:

<comment>In `forceTerminateSessions`, `terminal-pty-service.closeAll()` throws `AggregateError('Terminal PTY cleanup failed', [e1, e2, ...])` when one or more PTY kills fail. That nested `AggregateError` is pushed as a single element into the outer `AggregateError('Forced fleet process cleanup failed')`, so the resulting `errors` array is `[AggregateError([...]), ...workspaceRejections]` rather than a flat list of individual failure reasons. Any consumer that iterates `.errors` (or the `toMatchObject({ errors: [...] })` assertion in the new test) won't see the underlying PTY kill failures — the test only passes because its mock throws a plain `Error` instead of the real `AggregateError`. Flatten the terminal failure into the outer array so the individual PTY failures surface consistently with the workspace failures.</comment>

<file context>
@@ -532,14 +532,24 @@ export class LodyFleet {
+    try {
+      this.terminalPtyService.closeAll();
+    } catch (error) {
+      terminalFailures.push(error);
+    }
     const results = await Promise.allSettled(
</file context>

@slashdevcorpse

Copy link
Copy Markdown
Owner Author

Superseded by upstream PR LodyAI#456. The series now lives in LodyAI/Lody as LodyAI#456 -> LodyAI#457 -> LodyAI#458. Source branches and review history are preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants