Repository navigation
fix(sandbox): keep agent workspaces through lost sandboxes - #4560
Merged
Merged
Conversation
yannickmonney
force-pushed
the
fix/sandbox-workspace-recovery
branch
4 times, most recently
from
October 9, 2026 02:40
c529122 to
011aea5
Compare
yannickmonney
enabled auto-merge
October 9, 2026 02:40
yannickmonney
force-pushed
the
fix/sandbox-workspace-recovery
branch
from
October 9, 2026 02:41
011aea5 to
0e93f3b
Compare
A create whose deterministic name was taken by a LIVE session the registry did not hold answered 502 create_failed: Docker threw a plain Error for a surviving `docker run` name conflict, Kubernetes the same for a Pod/Secret 409. The route answered 409 duplicate only from its own registry and creating maps. The platform adopts only on 409; on a 502 for a new row it runs its failed-create cleanup, an if_idle destroy of the id, which removes an idle live session's container and deletes the workspace preserved under it. Both backends now throw SessionExistsError when the holder of the name is a live session, that is a running container labelled with this spawner's instance (re-inspected after the terminal-orphan reap), or a Pod that is neither terminating nor Succeeded/Failed (Pending counts, as a peer is starting it). The route answers it as 409 duplicate, so the platform adopts the session through acquire and ensureRegistered registers it. A holder that is `created`, being removed, another instance's, a terminating or ended Pod, or unreadable stays a 502, and nothing under the name is touched, as before.
Losing a session container could turn into losing its workspace. When the container behind an unpinned agent row was gone (a host reboot, a dockerd restart, an OOM-killed runnerd, a TTL stop), the platform's reconcile settled the row as destroyed although the spawner kept the workspace. The owner lookup only sees creating/active/stopped rows, so the next start took the new-row path, and a create that then failed for anything but a 429 (runnerd not ready within 180 s, a docker run timeout, an unreachable spawner) ran an if_idle destroy of the id. With no container under it, hasLiveExecs answered idle and the destroy deleted the preserved workspace. The watchdog's COLLECT pass repeated that destroy 10 minutes later, and the destroyed heal also dropped the incarnation the harness conversation resume binds to. The reconcile now settles an unpinned agent or agent-light row whose compute is gone as stopped while the spawner's workspace inventory lists the workspace, or cannot be read (read at most once per pass, so a reboot that empties a host costs one inventory, not one per row). It reclaims the credentials and frees the slot as every teardown edge does, and the next turn resumes in place. A render row, or one whose workspace the inventory does not list, still settles as destroyed. A failed create's cleanup and COLLECT of an agent row now remove the compute alone through DELETE /v1/sessions/:id?if_idle=1&keep_workspace=1, mapped to backend.stopSession on both backends; render leftovers are still destroyed whole, and deleting what nothing owns stays the workspace cleanup's. The device hub keeps the placement of a workspace its device kept. The rule is SBX-R17 in the sandbox spec, with the user docs in EN, DE and FR.
runnerd registered handlers for SIGTERM and SIGINT only. It runs on the image's Node 24, whose default turns an unhandled promise rejection into an exit, and it is tini's only child in the session container, so one stray rejection in any request path ended the container and every live exec in it, turns included. An unhandled rejection is now logged and survived, as the spawner does. An uncaught exception leaves the daemon's state unknown, so it is logged and runnerd stops through the same path SIGTERM takes: live execs are told to end, the two-second forced deadline still bounds the stop, and the exit code is 70 (EX_SOFTWARE), apart from a stop's 0, Node's own 1 and 7 and a signal's 128 + N, so a crash is readable from the container state. A second exception during that stop starts nothing new. The test runs the bundled guard under Node as the image does: a stray rejection is survived and a thrown exception exits 70 through the stop.
A Docker session container runs with --oom-score-adj=500, and runnerd and every exec it starts shared that score. When a session reaches its memory limit the kernel's OOM killer picks the process with the highest badness, its RSS plus adj/1000 of the limit, so with equal scores it could pick runnerd, the largest process once its replay buffers grow. runnerd is the container's only child under tini: its end is the container's, and every turn in it. The exec shim now raises its child's /proc/self/oom_score_adj to 900 before exec'ing the command, never lowering a higher score; the command and what it starts inherit it, and the shim keeps runnerd's. The 400 point gap means runnerd is picked only when its RSS exceeds a command's by 40 % of the limit (1.6 GiB of a 4 GiB agent). Raising a score needs no privilege, so it works with cap-drop=ALL; where the file is missing or the write is refused, the command runs with the score it inherited and nothing is said on the status pipe. The Linux shim test reads the command's score against the shim's.
Collecting a failed create of an agent session now stops its compute and keeps its workspace, and stamps the failed row as collected. The wait retention then deleted that row a day later. When it was the only row naming the session id (a member's or agent's first create failed, so no earlier incarnation exists), no row named the kept workspace any more. The hourly inventory keeps a pa- workspace no row names for as long as its agent exists (a restored database may have lost the row), so it escaped the unused-days rule, and the agent-deletion and member-removal jobs, which find workspaces through their rows, could no longer reach it. On Kubernetes that can be a provisioned PVC. The retention now never deletes the newest row naming a project agent's session id; an older failed row goes once a newer row names the id. The kept row is failed, so nothing reconciles, resumes or lists it, and the cleanup's row-based passes find the workspace through it: the unused rule and both removals settle it as destroyed when they delete the workspace. This was chosen over destroying the workspace of a lone failed row in COLLECT and in the failed-create cleanup: it needs no spawner call and no new fence, it also covers a lone row superseded by a refused (429) create, whose collect makes no spawner call, and it keeps the files of a first create after a database restore. Kept rows are bounded by one per project-agent session id (agent x member). An automation run's rows still go after a day: the hourly inventory deletes a run's workspace that no row names. The statement was checked on Postgres in a rolled-back transaction: of ten rows, it deleted the older of two failed rows of one id, a render's and a run's lone row, and a failed row behind an active successor, and kept a lone project-agent row, the newer of the two, a recent one and one whose only other row is destroyed. The integration lane now holds those cases; the unit test pins the statement's shape.
The reconcile asks the spawner's workspace inventory whether an agent session whose compute is gone still has its workspace, and it asks under the session's lifecycle lock. The read had only the client's own 60 s bound and ignored the pass's abort signal, so one slow inventory could hold that lock for a minute and outlive the watchdog pass's 120 s budget or the Sandboxes page probe's 20 s one. The read now takes the caller's signal and the reconcile's own 15 s bound, the same as its other spawner reads (observe, isAlive). An inventory that times out on its own reads as unknown and keeps the workspace, as a failed read did. One that the pass's signal ended throws instead, so a pass out of time settles nothing on a guess and leaves the row to its next visit. The pass's shared read passes the first heal's options through; every heal of a pass carries the same pass signal.
The docs said a sandbox that disappears on its own keeps its workspace, reads Stopped and the next run continues in it. That holds where the heal can see the workspace: it asks the spawner's GET /v1/workspaces, which lists the server's own workspaces only. A device-placed session is never listed there, so a readable inventory heals its row as destroyed. The admin page now says so in EN, DE and FR (on the Tale server; not for an environment on a connected device), and SBX-R17, the service's sessions doc and the reference row say the same. No test changes: the tests that name SBX-R17 cover server-side sessions only.
SBX-R17 sat under Automatic deletion, whose opening says Tale deletes a workspace in two cases, while the rule is about a workspace that is kept when its sandbox disappears. It now opens its own topic, When a sandbox disappears, between Pinning and destroying a workspace (whose SBX-R10 covers a pinned workspace in the same event) and Automatic deletion, and the spec's summary names the topic. The rule keeps its ID and text: the domain-specs guard asks only that a card sit under a topic heading before Not yet, with IDs unique and appended, not that they appear in numeric order.
The failed-create collect lane's opening comment in the integration check had a 122-column line where an agent session's kept workspace was added to it; the paragraph is reflowed to the file's 80 columns, words unchanged.
The automation coverage map now keeps one file per suite, so parallel changes stop appending to the same lines; this change's rows move there, after their own anchor row.
yannickmonney
force-pushed
the
fix/sandbox-workspace-recovery
branch
from
October 9, 2026 02:57
0e93f3b to
15a083e
Compare
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Losing a session container could turn into losing the agent's workspace (repos, uncommitted work, conversation state): the platform healed the row to destroyed, the next start took the new-row path, and a failed create then destroyed the preserved workspace. A name conflict with a live unregistered container also answered 502, which the same failure path treats as a reason to destroy. Separately, one stray promise rejection in runnerd ended the whole session, and runnerd shared the OOM score of the processes it runs.
Changes
fix(sandbox): answer a live unregistered name conflict as a duplicate
A create whose deterministic name was taken by a LIVE session the
registry did not hold answered 502 create_failed: Docker threw a plain
Error for a surviving
docker runname conflict, Kubernetes the samefor a Pod/Secret 409. The route answered 409 duplicate only from its
own registry and creating maps. The platform adopts only on 409; on a
502 for a new row it runs its failed-create cleanup, an if_idle destroy
of the id, which removes an idle live session's container and deletes
the workspace preserved under it.
Both backends now throw SessionExistsError when the holder of the name
is a live session, that is a running container labelled with this
spawner's instance (re-inspected after the terminal-orphan reap), or a
Pod that is neither terminating nor Succeeded/Failed (Pending counts,
as a peer is starting it). The route answers it as 409 duplicate, so
the platform adopts the session through acquire and ensureRegistered
registers it. A holder that is
created, being removed, anotherinstance's, a terminating or ended Pod, or unreadable stays a 502, and
nothing under the name is touched, as before.
fix(sandbox): keep an agent's workspace when its sandbox is lost
Losing a session container could turn into losing its workspace. When
the container behind an unpinned agent row was gone (a host reboot, a
dockerd restart, an OOM-killed runnerd, a TTL stop), the platform's
reconcile settled the row as destroyed although the spawner kept the
workspace. The owner lookup only sees creating/active/stopped rows, so
the next start took the new-row path, and a create that then failed
for anything but a 429 (runnerd not ready within 180 s, a docker run
timeout, an unreachable spawner) ran an if_idle destroy of the id. With
no container under it, hasLiveExecs answered idle and the destroy
deleted the preserved workspace. The watchdog's COLLECT pass repeated
that destroy 10 minutes later, and the destroyed heal also dropped the
incarnation the harness conversation resume binds to.
The reconcile now settles an unpinned agent or agent-light row whose
compute is gone as stopped while the spawner's workspace inventory
lists the workspace, or cannot be read (read at most once per pass, so
a reboot that empties a host costs one inventory, not one per row). It
reclaims the credentials and frees the slot as every teardown edge
does, and the next turn resumes in place. A render row, or one whose
workspace the inventory does not list, still settles as destroyed.
A failed create's cleanup and COLLECT of an agent row now remove the
compute alone through DELETE /v1/sessions/:id?if_idle=1&keep_workspace=1,
mapped to backend.stopSession on both backends; render leftovers are
still destroyed whole, and deleting what nothing owns stays the
workspace cleanup's. The device hub keeps the placement of a workspace
its device kept. The rule is SBX-R17 in the sandbox spec, with the
user docs in EN, DE and FR.
fix(sandbox): keep runnerd alive past an unhandled rejection
runnerd registered handlers for SIGTERM and SIGINT only. It runs on the
image's Node 24, whose default turns an unhandled promise rejection into
an exit, and it is tini's only child in the session container, so one
stray rejection in any request path ended the container and every live
exec in it, turns included.
An unhandled rejection is now logged and survived, as the spawner does.
An uncaught exception leaves the daemon's state unknown, so it is logged
and runnerd stops through the same path SIGTERM takes: live execs are
told to end, the two-second forced deadline still bounds the stop, and
the exit code is 70 (EX_SOFTWARE), apart from a stop's 0, Node's own 1
and 7 and a signal's 128 + N, so a crash is readable from the container
state. A second exception during that stop starts nothing new.
The test runs the bundled guard under Node as the image does: a stray
rejection is survived and a thrown exception exits 70 through the stop.
fix(sandbox): rank an exec's command above runnerd for the OOM killer
A Docker session container runs with --oom-score-adj=500, and runnerd
and every exec it starts shared that score. When a session reaches its
memory limit the kernel's OOM killer picks the process with the highest
badness, its RSS plus adj/1000 of the limit, so with equal scores it
could pick runnerd, the largest process once its replay buffers grow.
runnerd is the container's only child under tini: its end is the
container's, and every turn in it.
The exec shim now raises its child's /proc/self/oom_score_adj to 900
before exec'ing the command, never lowering a higher score; the command
and what it starts inherit it, and the shim keeps runnerd's. The 400
point gap means runnerd is picked only when its RSS exceeds a command's
by 40 % of the limit (1.6 GiB of a 4 GiB agent). Raising a score needs
no privilege, so it works with cap-drop=ALL; where the file is missing
or the write is refused, the command runs with the score it inherited
and nothing is said on the status pipe.
The Linux shim test reads the command's score against the shim's.
Review fixes
An adversarial review of this branch found one major and four minor issues. Each is fixed in its own commit below.
fix(sandbox): keep the row that names a project agent's kept workspace
Collecting a failed create of an agent session now stops its compute and
keeps its workspace, and stamps the failed row as collected. The wait
retention then deleted that row a day later. When it was the only row
naming the session id (a member's or agent's first create failed, so no
earlier incarnation exists), no row named the kept workspace any more.
The hourly inventory keeps a pa- workspace no row names for as long as
its agent exists (a restored database may have lost the row), so it
escaped the unused-days rule, and the agent-deletion and member-removal
jobs, which find workspaces through their rows, could no longer reach
it. On Kubernetes that can be a provisioned PVC.
The retention now never deletes the newest row naming a project agent's
session id; an older failed row goes once a newer row names the id. The
kept row is failed, so nothing reconciles, resumes or lists it, and the
cleanup's row-based passes find the workspace through it: the unused
rule and both removals settle it as destroyed when they delete the
workspace. This was chosen over destroying the workspace of a lone
failed row in COLLECT and in the failed-create cleanup: it needs no
spawner call and no new fence, it also covers a lone row superseded by
a refused (429) create, whose collect makes no spawner call, and it
keeps the files of a first create after a database restore. Kept rows
are bounded by one per project-agent session id (agent x member).
An automation run's rows still go after a day: the hourly inventory
deletes a run's workspace that no row names.
The statement was checked on Postgres in a rolled-back transaction: of
ten rows, it deleted the older of two failed rows of one id, a render's
and a run's lone row, and a failed row behind an active successor, and
kept a lone project-agent row, the newer of the two, a recent one and
one whose only other row is destroyed. The integration lane now holds
those cases; the unit test pins the statement's shape.
fix(sandbox): bound the heal's inventory read by the pass's signal
The reconcile asks the spawner's workspace inventory whether an agent
session whose compute is gone still has its workspace, and it asks
under the session's lifecycle lock. The read had only the client's own
60 s bound and ignored the pass's abort signal, so one slow inventory
could hold that lock for a minute and outlive the watchdog pass's
120 s budget or the Sandboxes page probe's 20 s one.
The read now takes the caller's signal and the reconcile's own 15 s
bound, the same as its other spawner reads (observe, isAlive). An
inventory that times out on its own reads as unknown and keeps the
workspace, as a failed read did. One that the pass's signal ended
throws instead, so a pass out of time settles nothing on a guess and
leaves the row to its next visit. The pass's shared read passes the
first heal's options through; every heal of a pass carries the same
pass signal.
docs(sandbox): limit kept-workspace recovery to the Tale server
The docs said a sandbox that disappears on its own keeps its workspace,
reads Stopped and the next run continues in it. That holds where the
heal can see the workspace: it asks the spawner's GET /v1/workspaces,
which lists the server's own workspaces only. A device-placed session
is never listed there, so a readable inventory heals its row as
destroyed.
The admin page now says so in EN, DE and FR (on the Tale server; not
for an environment on a connected device), and SBX-R17, the service's
sessions doc and the reference row say the same. No test changes: the
tests that name SBX-R17 cover server-side sessions only.
docs(sandbox): give SBX-R17 a topic of its own in the sandbox spec
SBX-R17 sat under Automatic deletion, whose opening says Tale deletes a
workspace in two cases, while the rule is about a workspace that is
kept when its sandbox disappears. It now opens its own topic, When a
sandbox disappears, between Pinning and destroying a workspace (whose
SBX-R10 covers a pinned workspace in the same event) and Automatic
deletion, and the spec's summary names the topic.
The rule keeps its ID and text: the domain-specs guard asks only that a
card sit under a topic heading before Not yet, with IDs unique and
appended, not that they appear in numeric order.
style(platform): wrap the collect lane's comment at 80 columns
The failed-create collect lane's opening comment in the integration
check had a 122-column line where an agent session's kept workspace was
added to it; the paragraph is reflowed to the file's 80 columns, words
unchanged.
Verification
session-routes.test.tsrun on macOS shows only the pre-existingcancelled status probes…hang, identical on main.Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).
After the review fixes:
wait-retention,service,watchdogs,session_client,agent_session, the domain-specs guard and the legacy agent-flow guard (217 + 42 tests).session-routes.test.ts, only the macOS-only "cancelled status probes" hang fails, and it fails the same way on unmodified main.exec-shimandprocess-guardsuites pass (12 tests).@tale/platformand@tale/sandboxtypecheck and lint are clean.