Repository navigation
fix(sandbox): find a leftover mid-exec by reading its environment again - #4531
Merged
Merged
Conversation
larryro
force-pushed
the
fix/reaper-exec-window
branch
2 times, most recently
from
October 8, 2026 14:24
d0e833d to
f89309e
Compare
larryro
enabled auto-merge (squash)
October 8, 2026 14:24
Collaborator
Author
|
CI evidence for the mechanism, from the throwaway diagnostic branch (#4561, now closed):
A probe of |
The reaper finds a process that left an exec's group by the exec id in its environment. A process in the middle of an execve has no environment to read: from the moment the new image's memory replaces the old until its arguments and environment are laid out, the kernel answers an empty one (a probe reading /proc/<pid>/environ of a process that execs in a loop got an empty answer in 3,341 of 20,000 reads). A leftover is often right there when a round comes: `cmd &` and the command's exit are an instant apart, and `setsid cmd` is two execs. The SIGTERM round counted such a process as untagged and skipped it; only the SIGKILL round, SIGKILL_GRACE_MS later, ended it. That is how exec-completion.test.ts failed on CI (#4445, Checks run 37752204357): its final barrier gave the reaper five seconds to end the `setsid sleep 60` a finished exec left, and the SIGKILL round came a few milliseconds after that. The test now waits for the reaper's own bound (the SIGKILL grace plus the scan deadline), and the scan reads an empty environment again, four times ten milliseconds apart, before the process counts as untagged; a process whose environment really is empty costs a scan that long. No user-visible change: no messages, docs under docs/ or READMEs.
larryro
force-pushed
the
fix/reaper-exec-window
branch
from
October 8, 2026 15:32
f89309e to
ce58778
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unit (workspaces)failed on #4445's Checks run (37752204357, attempt 1) inservices/sandbox-runtime/daemon/src/exec-completion.test.ts: "attach during terminal replay finish cannot rearm the orphan deadline or reap sibling-dependent leftovers",barrier not reachedat its final barrier, 5021 ms. That PR touches nothing underservices/sandbox-runtime; the cause is in the reaper.The reaper finds a process that left an exec's group by the exec id in its environment. A process in the middle of an
execvehas no environment to read: from the moment the new image's memory replaces the old until its arguments and environment are laid out, the kernel answers an empty one. A probe reading/proc/<pid>/environof a process that execs in a loop got an empty answer in 3,341 of 20,000 reads. A leftover is often right there when a round comes —cmd &and the command's exit are an instant apart, andsetsid cmdis two execs — and the scan counted such a process as untagged and skipped it. The SIGTERM round missed it; only the SIGKILL round,SIGKILL_GRACE_MS(5 s) later, ended it. The test's final barrier gave the reaper 5 s, so the SIGKILL round landed a few milliseconds after the test gave up — which is the 5021 ms.What changes
process-reaper.ts: an empty environment is read again, four times ten milliseconds apart, before the process counts as untagged. A process whose environment really is empty (env -i) costs a scan that long. The environment read moved intoreadEnviron()(unchanged: one read shared by the scans that run at once). Header comment says why.process-reaper.test.ts: a process whose environment reads empty and is laid out during the re-reads is found; one whose environment stays empty counts as untagged after a bounded wait (not the scan's deadline).procTablegainedmidExecfor an environ file with no bytes.exec-completion.test.ts: the final barrier waits for the reaper's own bound — the SIGKILL grace plus the scan deadline, with a margin (8 s, under the test's 15 s) — instead of 5 s. The re-read closes the common case; the SIGKILL round stays the bound the daemon promises.services/sandbox/docs/sessions.mdandservices/platform/tests/manual/reference/automation.mdsay so.Not reproduced locally
40 runs under 14 CPU hogs beside the full daemon suite, and 60 runs pinned to four CPUs with hogs, all passed, as did 900 probe runs of the test's own
bash -lc 'setsid sleep 60 … & echo $!'followed by an immediate tag scan. The window is in the kernel's exec path and the probe above shows it exists; on the four-vCPU runner, withturbo --concurrency=2and@tale/sandbox#testrunning beside the daemon suite, the scan read the survivor inside it once in the last 100 Checks runs.Checklist
bun run check: not end to end on this machine. Ran on this tree:@tale/sandbox-runtime-daemontypecheckandlint(clean), the full daemon suite (463 pass, 11 fail — the same 11tale-steer-hook/tale-vision-read-hooktests fail on unmodifiedmainhere,jqis not installed),bun run lint:manual,bun run lint:conflicts,oxfmton the changed files, and the pre-commitlint-stagedpass.bun run lint:sast(Opengrep): the pre-commit hook's staged-file pass. I did not run the whole-repository scan.docs/: N/A, no user-visible change; the internalservices/sandbox/docs/sessions.mdis updated.Test plan
bun test src/process-reaper.test.ts src/exec-completion.test.tsinservices/sandbox-runtime/daemon: the two new reaper tests (one found by a re-read, one untagged after ~40 ms), and the exec-completion test (passes in ~190 ms: the survivor is ended by the SIGTERM round, not the bound).sh -c 'exec /bin/sh "$0"' <script that execs itself>and read its/proc/<pid>/environin a loop — a share of the reads come back empty.Unit (workspaces)on this PR.