Skip to content

fix(sandbox): find a leftover mid-exec by reading its environment again - #4531

Merged
larryro merged 1 commit into
mainfrom
fix/reaper-exec-window
Oct 8, 2026
Merged

larryro merged 1 commit into
mainfrom
fix/reaper-exec-window

Conversation

@larryro

@larryro larryro commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Unit (workspaces) failed on #4445's Checks run (37752204357, attempt 1) in services/sandbox-runtime/daemon/src/exec-completion.test.ts: "attach during terminal replay finish cannot rearm the orphan deadline or reap sibling-dependent leftovers", barrier not reached at its final barrier, 5021 ms. That PR touches nothing under services/sandbox-runtime; the cause is in the reaper.

The reaper finds a process that left an exec's group by the exec id in its environment. A process in the middle of an execve has no environment to read: from the moment the new image's memory replaces the old until its arguments and environment are laid out, the kernel answers an empty one. A probe reading /proc/<pid>/environ of a process that execs in a loop got an empty answer in 3,341 of 20,000 reads. A leftover is often right there when a round comes — cmd & and the command's exit are an instant apart, and setsid cmd is two execs — and the scan counted such a process as untagged and skipped it. The SIGTERM round missed it; only the SIGKILL round, SIGKILL_GRACE_MS (5 s) later, ended it. The test's final barrier gave the reaper 5 s, so the SIGKILL round landed a few milliseconds after the test gave up — which is the 5021 ms.

What changes

  • process-reaper.ts: an empty environment is read again, four times ten milliseconds apart, before the process counts as untagged. A process whose environment really is empty (env -i) costs a scan that long. The environment read moved into readEnviron() (unchanged: one read shared by the scans that run at once). Header comment says why.
  • process-reaper.test.ts: a process whose environment reads empty and is laid out during the re-reads is found; one whose environment stays empty counts as untagged after a bounded wait (not the scan's deadline). procTable gained midExec for an environ file with no bytes.
  • exec-completion.test.ts: the final barrier waits for the reaper's own bound — the SIGKILL grace plus the scan deadline, with a margin (8 s, under the test's 15 s) — instead of 5 s. The re-read closes the common case; the SIGKILL round stays the bound the daemon promises.
  • services/sandbox/docs/sessions.md and services/platform/tests/manual/reference/automation.md say so.

Not reproduced locally

40 runs under 14 CPU hogs beside the full daemon suite, and 60 runs pinned to four CPUs with hogs, all passed, as did 900 probe runs of the test's own bash -lc 'setsid sleep 60 … & echo $!' followed by an immediate tag scan. The window is in the kernel's exec path and the probe above shows it exists; on the four-vCPU runner, with turbo --concurrency=2 and @tale/sandbox#test running beside the daemon suite, the scan read the survivor inside it once in the last 100 Checks runs.

Checklist

  • bun run check: not end to end on this machine. Ran on this tree: @tale/sandbox-runtime-daemon typecheck and lint (clean), the full daemon suite (463 pass, 11 fail — the same 11 tale-steer-hook / tale-vision-read-hook tests fail on unmodified main here, jq is not installed), bun run lint:manual, bun run lint:conflicts, oxfmt on the changed files, and the pre-commit lint-staged pass.
  • bun run lint:sast (Opengrep): the pre-commit hook's staged-file pass. I did not run the whole-repository scan.
  • Translations: N/A, no message changed.
  • Docs under docs/: N/A, no user-visible change; the internal services/sandbox/docs/sessions.md is updated.
  • READMEs: N/A.

Test plan

  • bun test src/process-reaper.test.ts src/exec-completion.test.ts in services/sandbox-runtime/daemon: the two new reaper tests (one found by a re-read, one untagged after ~40 ms), and the exec-completion test (passes in ~190 ms: the survivor is ended by the SIGTERM round, not the bound).
  • The kernel behaviour: start sh -c 'exec /bin/sh "$0"' <script that execs itself> and read its /proc/<pid>/environ in a loop — a share of the reads come back empty.
  • CI: Unit (workspaces) on this PR.

@larryro
larryro force-pushed the fix/reaper-exec-window branch 2 times, most recently from d0e833d to f89309e Compare October 8, 2026 14:24
@larryro
larryro enabled auto-merge (squash) October 8, 2026 14:24
@larryro

larryro commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

CI evidence for the mechanism, from the throwaway diagnostic branch (#4561, now closed):

  • The real test, made strict, suite run four times on one runner (Checks run 37796405873, unfixed reaper): in the first run — the one on a cold page cache — the survivor was caught in state D (disk sleep) with PPid 1 while the leftover round was already under way (leftovers: 0), i.e. ~20 ms after bash printed $! the setsid/sleep chain was still executing (loading binary pages from disk). The round that time still reached it through its old group, because it had not called setsid(2) yet. The three warm runs after it took 19–22 ms and signalled it by tag at once.
  • That is the window the two failures sat in (5021 ms and 5046 ms, i.e. the SIGKILL round a few milliseconds after the 5 s barrier): a survivor that has just left the group but is still inside execve reads as an empty environment, so the SIGTERM round neither owns it by tag nor reaches it by group; the SIGKILL round SIGKILL_GRACE_MS later finds it with its environment laid out.
  • 36 iterations of the same scenario in a test file that ran last in the suite (warm cache, both with and without the re-read) never hit the window — which is why it never reproduced locally either: the window is the cold-cache exec on a fresh runner, not CPU load.

A probe of /proc/<pid>/environ on a process that execs in a loop returned an empty read 3,341 times out of 20,000, confirming the kernel behaviour behind it.

The reaper finds a process that left an exec's group by the exec id in its
environment. A process in the middle of an execve has no environment to
read: from the moment the new image's memory replaces the old until its
arguments and environment are laid out, the kernel answers an empty one
(a probe reading /proc/<pid>/environ of a process that execs in a loop got
an empty answer in 3,341 of 20,000 reads). A leftover is often right there
when a round comes: `cmd &` and the command's exit are an instant apart,
and `setsid cmd` is two execs. The SIGTERM round counted such a process as
untagged and skipped it; only the SIGKILL round, SIGKILL_GRACE_MS later,
ended it.

That is how exec-completion.test.ts failed on CI (#4445, Checks run
37752204357): its final barrier gave the reaper five seconds to end the
`setsid sleep 60` a finished exec left, and the SIGKILL round came a few
milliseconds after that. The test now waits for the reaper's own bound (the
SIGKILL grace plus the scan deadline), and the scan reads an empty
environment again, four times ten milliseconds apart, before the process
counts as untagged; a process whose environment really is empty costs a
scan that long.

No user-visible change: no messages, docs under docs/ or READMEs.
@larryro
larryro force-pushed the fix/reaper-exec-window branch from f89309e to ce58778 Compare October 8, 2026 15:32
@larryro
larryro merged commit a7ea4f4 into main Oct 8, 2026
80 checks passed
@larryro
larryro deleted the fix/reaper-exec-window branch October 8, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant