Skip to content

fix(sandbox): keep agent workspaces through lost sandboxes - #4560

Merged
yannickmonney merged 10 commits into
mainfrom
fix/sandbox-workspace-recovery
Oct 9, 2026
Merged

yannickmonney merged 10 commits into
mainfrom
fix/sandbox-workspace-recovery

Conversation

@yannickmonney

@yannickmonney yannickmonney commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Losing a session container could turn into losing the agent's workspace (repos, uncommitted work, conversation state): the platform healed the row to destroyed, the next start took the new-row path, and a failed create then destroyed the preserved workspace. A name conflict with a live unregistered container also answered 502, which the same failure path treats as a reason to destroy. Separately, one stray promise rejection in runnerd ended the whole session, and runnerd shared the OOM score of the processes it runs.

Changes

fix(sandbox): answer a live unregistered name conflict as a duplicate

A create whose deterministic name was taken by a LIVE session the
registry did not hold answered 502 create_failed: Docker threw a plain
Error for a surviving docker run name conflict, Kubernetes the same
for a Pod/Secret 409. The route answered 409 duplicate only from its
own registry and creating maps. The platform adopts only on 409; on a
502 for a new row it runs its failed-create cleanup, an if_idle destroy
of the id, which removes an idle live session's container and deletes
the workspace preserved under it.

Both backends now throw SessionExistsError when the holder of the name
is a live session, that is a running container labelled with this
spawner's instance (re-inspected after the terminal-orphan reap), or a
Pod that is neither terminating nor Succeeded/Failed (Pending counts,
as a peer is starting it). The route answers it as 409 duplicate, so
the platform adopts the session through acquire and ensureRegistered
registers it. A holder that is created, being removed, another
instance's, a terminating or ended Pod, or unreadable stays a 502, and
nothing under the name is touched, as before.

fix(sandbox): keep an agent's workspace when its sandbox is lost

Losing a session container could turn into losing its workspace. When
the container behind an unpinned agent row was gone (a host reboot, a
dockerd restart, an OOM-killed runnerd, a TTL stop), the platform's
reconcile settled the row as destroyed although the spawner kept the
workspace. The owner lookup only sees creating/active/stopped rows, so
the next start took the new-row path, and a create that then failed
for anything but a 429 (runnerd not ready within 180 s, a docker run
timeout, an unreachable spawner) ran an if_idle destroy of the id. With
no container under it, hasLiveExecs answered idle and the destroy
deleted the preserved workspace. The watchdog's COLLECT pass repeated
that destroy 10 minutes later, and the destroyed heal also dropped the
incarnation the harness conversation resume binds to.

The reconcile now settles an unpinned agent or agent-light row whose
compute is gone as stopped while the spawner's workspace inventory
lists the workspace, or cannot be read (read at most once per pass, so
a reboot that empties a host costs one inventory, not one per row). It
reclaims the credentials and frees the slot as every teardown edge
does, and the next turn resumes in place. A render row, or one whose
workspace the inventory does not list, still settles as destroyed.

A failed create's cleanup and COLLECT of an agent row now remove the
compute alone through DELETE /v1/sessions/:id?if_idle=1&keep_workspace=1,
mapped to backend.stopSession on both backends; render leftovers are
still destroyed whole, and deleting what nothing owns stays the
workspace cleanup's. The device hub keeps the placement of a workspace
its device kept. The rule is SBX-R17 in the sandbox spec, with the
user docs in EN, DE and FR.

fix(sandbox): keep runnerd alive past an unhandled rejection

runnerd registered handlers for SIGTERM and SIGINT only. It runs on the
image's Node 24, whose default turns an unhandled promise rejection into
an exit, and it is tini's only child in the session container, so one
stray rejection in any request path ended the container and every live
exec in it, turns included.

An unhandled rejection is now logged and survived, as the spawner does.
An uncaught exception leaves the daemon's state unknown, so it is logged
and runnerd stops through the same path SIGTERM takes: live execs are
told to end, the two-second forced deadline still bounds the stop, and
the exit code is 70 (EX_SOFTWARE), apart from a stop's 0, Node's own 1
and 7 and a signal's 128 + N, so a crash is readable from the container
state. A second exception during that stop starts nothing new.

The test runs the bundled guard under Node as the image does: a stray
rejection is survived and a thrown exception exits 70 through the stop.

fix(sandbox): rank an exec's command above runnerd for the OOM killer

A Docker session container runs with --oom-score-adj=500, and runnerd
and every exec it starts shared that score. When a session reaches its
memory limit the kernel's OOM killer picks the process with the highest
badness, its RSS plus adj/1000 of the limit, so with equal scores it
could pick runnerd, the largest process once its replay buffers grow.
runnerd is the container's only child under tini: its end is the
container's, and every turn in it.

The exec shim now raises its child's /proc/self/oom_score_adj to 900
before exec'ing the command, never lowering a higher score; the command
and what it starts inherit it, and the shim keeps runnerd's. The 400
point gap means runnerd is picked only when its RSS exceeds a command's
by 40 % of the limit (1.6 GiB of a 4 GiB agent). Raising a score needs
no privilege, so it works with cap-drop=ALL; where the file is missing
or the write is refused, the command runs with the score it inherited
and nothing is said on the status pipe.

The Linux shim test reads the command's score against the shim's.

Review fixes

An adversarial review of this branch found one major and four minor issues. Each is fixed in its own commit below.

fix(sandbox): keep the row that names a project agent's kept workspace

Collecting a failed create of an agent session now stops its compute and
keeps its workspace, and stamps the failed row as collected. The wait
retention then deleted that row a day later. When it was the only row
naming the session id (a member's or agent's first create failed, so no
earlier incarnation exists), no row named the kept workspace any more.
The hourly inventory keeps a pa- workspace no row names for as long as
its agent exists (a restored database may have lost the row), so it
escaped the unused-days rule, and the agent-deletion and member-removal
jobs, which find workspaces through their rows, could no longer reach
it. On Kubernetes that can be a provisioned PVC.

The retention now never deletes the newest row naming a project agent's
session id; an older failed row goes once a newer row names the id. The
kept row is failed, so nothing reconciles, resumes or lists it, and the
cleanup's row-based passes find the workspace through it: the unused
rule and both removals settle it as destroyed when they delete the
workspace. This was chosen over destroying the workspace of a lone
failed row in COLLECT and in the failed-create cleanup: it needs no
spawner call and no new fence, it also covers a lone row superseded by
a refused (429) create, whose collect makes no spawner call, and it
keeps the files of a first create after a database restore. Kept rows
are bounded by one per project-agent session id (agent x member).

An automation run's rows still go after a day: the hourly inventory
deletes a run's workspace that no row names.

The statement was checked on Postgres in a rolled-back transaction: of
ten rows, it deleted the older of two failed rows of one id, a render's
and a run's lone row, and a failed row behind an active successor, and
kept a lone project-agent row, the newer of the two, a recent one and
one whose only other row is destroyed. The integration lane now holds
those cases; the unit test pins the statement's shape.

fix(sandbox): bound the heal's inventory read by the pass's signal

The reconcile asks the spawner's workspace inventory whether an agent
session whose compute is gone still has its workspace, and it asks
under the session's lifecycle lock. The read had only the client's own
60 s bound and ignored the pass's abort signal, so one slow inventory
could hold that lock for a minute and outlive the watchdog pass's
120 s budget or the Sandboxes page probe's 20 s one.

The read now takes the caller's signal and the reconcile's own 15 s
bound, the same as its other spawner reads (observe, isAlive). An
inventory that times out on its own reads as unknown and keeps the
workspace, as a failed read did. One that the pass's signal ended
throws instead, so a pass out of time settles nothing on a guess and
leaves the row to its next visit. The pass's shared read passes the
first heal's options through; every heal of a pass carries the same
pass signal.

docs(sandbox): limit kept-workspace recovery to the Tale server

The docs said a sandbox that disappears on its own keeps its workspace,
reads Stopped and the next run continues in it. That holds where the
heal can see the workspace: it asks the spawner's GET /v1/workspaces,
which lists the server's own workspaces only. A device-placed session
is never listed there, so a readable inventory heals its row as
destroyed.

The admin page now says so in EN, DE and FR (on the Tale server; not
for an environment on a connected device), and SBX-R17, the service's
sessions doc and the reference row say the same. No test changes: the
tests that name SBX-R17 cover server-side sessions only.

docs(sandbox): give SBX-R17 a topic of its own in the sandbox spec

SBX-R17 sat under Automatic deletion, whose opening says Tale deletes a
workspace in two cases, while the rule is about a workspace that is
kept when its sandbox disappears. It now opens its own topic, When a
sandbox disappears, between Pinning and destroying a workspace (whose
SBX-R10 covers a pinned workspace in the same event) and Automatic
deletion, and the spec's summary names the topic.

The rule keeps its ID and text: the domain-specs guard asks only that a
card sit under a topic heading before Not yet, with IDs unique and
appended, not that they appear in numeric order.

style(platform): wrap the collect lane's comment at 80 columns

The failed-create collect lane's opening comment in the integration
check had a 122-column line where an agent session's kept workspace was
added to it; the paragraph is reflowed to the file's 80 columns, words
unchanged.

Verification

  • Builder-verified: touched platform vitest suites + domain-spec guard, services/sandbox session-routes/backends/hub tests, runtime daemon main/exec-shim/process-guard tests; typecheck and lint clean in @tale/sandbox, @tale/sandbox-runtime-daemon and @tale/platform.
  • Re-run after rebase: legacy-flow and domain-spec guards (18 pass). A full session-routes.test.ts run on macOS shows only the pre-existing cancelled status probes… hang, identical on main.
  • An independent adversarial review is still pending; this PR merges only after it.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).

After the review fixes:

  • Platform vitest passes: wait-retention, service, watchdogs, session_client, agent_session, the domain-specs guard and the legacy agent-flow guard (217 + 42 tests).
  • services/sandbox: create-cleanup, Kubernetes backend, hub and server suites pass (154 tests). In session-routes.test.ts, only the macOS-only "cancelled status probes" hang fails, and it fails the same way on unmodified main.
  • Daemon exec-shim and process-guard suites pass (12 tests).
  • @tale/platform and @tale/sandbox typecheck and lint are clean.

@yannickmonney
yannickmonney force-pushed the fix/sandbox-workspace-recovery branch 4 times, most recently from c529122 to 011aea5 Compare October 9, 2026 02:40
@yannickmonney
yannickmonney force-pushed the fix/sandbox-workspace-recovery branch from 011aea5 to 0e93f3b Compare October 9, 2026 02:41
A create whose deterministic name was taken by a LIVE session the
registry did not hold answered 502 create_failed: Docker threw a plain
Error for a surviving `docker run` name conflict, Kubernetes the same
for a Pod/Secret 409. The route answered 409 duplicate only from its
own registry and creating maps. The platform adopts only on 409; on a
502 for a new row it runs its failed-create cleanup, an if_idle destroy
of the id, which removes an idle live session's container and deletes
the workspace preserved under it.

Both backends now throw SessionExistsError when the holder of the name
is a live session, that is a running container labelled with this
spawner's instance (re-inspected after the terminal-orphan reap), or a
Pod that is neither terminating nor Succeeded/Failed (Pending counts,
as a peer is starting it). The route answers it as 409 duplicate, so
the platform adopts the session through acquire and ensureRegistered
registers it. A holder that is `created`, being removed, another
instance's, a terminating or ended Pod, or unreadable stays a 502, and
nothing under the name is touched, as before.
Losing a session container could turn into losing its workspace. When
the container behind an unpinned agent row was gone (a host reboot, a
dockerd restart, an OOM-killed runnerd, a TTL stop), the platform's
reconcile settled the row as destroyed although the spawner kept the
workspace. The owner lookup only sees creating/active/stopped rows, so
the next start took the new-row path, and a create that then failed
for anything but a 429 (runnerd not ready within 180 s, a docker run
timeout, an unreachable spawner) ran an if_idle destroy of the id. With
no container under it, hasLiveExecs answered idle and the destroy
deleted the preserved workspace. The watchdog's COLLECT pass repeated
that destroy 10 minutes later, and the destroyed heal also dropped the
incarnation the harness conversation resume binds to.

The reconcile now settles an unpinned agent or agent-light row whose
compute is gone as stopped while the spawner's workspace inventory
lists the workspace, or cannot be read (read at most once per pass, so
a reboot that empties a host costs one inventory, not one per row). It
reclaims the credentials and frees the slot as every teardown edge
does, and the next turn resumes in place. A render row, or one whose
workspace the inventory does not list, still settles as destroyed.

A failed create's cleanup and COLLECT of an agent row now remove the
compute alone through DELETE /v1/sessions/:id?if_idle=1&keep_workspace=1,
mapped to backend.stopSession on both backends; render leftovers are
still destroyed whole, and deleting what nothing owns stays the
workspace cleanup's. The device hub keeps the placement of a workspace
its device kept. The rule is SBX-R17 in the sandbox spec, with the
user docs in EN, DE and FR.
runnerd registered handlers for SIGTERM and SIGINT only. It runs on the
image's Node 24, whose default turns an unhandled promise rejection into
an exit, and it is tini's only child in the session container, so one
stray rejection in any request path ended the container and every live
exec in it, turns included.

An unhandled rejection is now logged and survived, as the spawner does.
An uncaught exception leaves the daemon's state unknown, so it is logged
and runnerd stops through the same path SIGTERM takes: live execs are
told to end, the two-second forced deadline still bounds the stop, and
the exit code is 70 (EX_SOFTWARE), apart from a stop's 0, Node's own 1
and 7 and a signal's 128 + N, so a crash is readable from the container
state. A second exception during that stop starts nothing new.

The test runs the bundled guard under Node as the image does: a stray
rejection is survived and a thrown exception exits 70 through the stop.
A Docker session container runs with --oom-score-adj=500, and runnerd
and every exec it starts shared that score. When a session reaches its
memory limit the kernel's OOM killer picks the process with the highest
badness, its RSS plus adj/1000 of the limit, so with equal scores it
could pick runnerd, the largest process once its replay buffers grow.
runnerd is the container's only child under tini: its end is the
container's, and every turn in it.

The exec shim now raises its child's /proc/self/oom_score_adj to 900
before exec'ing the command, never lowering a higher score; the command
and what it starts inherit it, and the shim keeps runnerd's. The 400
point gap means runnerd is picked only when its RSS exceeds a command's
by 40 % of the limit (1.6 GiB of a 4 GiB agent). Raising a score needs
no privilege, so it works with cap-drop=ALL; where the file is missing
or the write is refused, the command runs with the score it inherited
and nothing is said on the status pipe.

The Linux shim test reads the command's score against the shim's.
Collecting a failed create of an agent session now stops its compute and
keeps its workspace, and stamps the failed row as collected. The wait
retention then deleted that row a day later. When it was the only row
naming the session id (a member's or agent's first create failed, so no
earlier incarnation exists), no row named the kept workspace any more.
The hourly inventory keeps a pa- workspace no row names for as long as
its agent exists (a restored database may have lost the row), so it
escaped the unused-days rule, and the agent-deletion and member-removal
jobs, which find workspaces through their rows, could no longer reach
it. On Kubernetes that can be a provisioned PVC.

The retention now never deletes the newest row naming a project agent's
session id; an older failed row goes once a newer row names the id. The
kept row is failed, so nothing reconciles, resumes or lists it, and the
cleanup's row-based passes find the workspace through it: the unused
rule and both removals settle it as destroyed when they delete the
workspace. This was chosen over destroying the workspace of a lone
failed row in COLLECT and in the failed-create cleanup: it needs no
spawner call and no new fence, it also covers a lone row superseded by
a refused (429) create, whose collect makes no spawner call, and it
keeps the files of a first create after a database restore. Kept rows
are bounded by one per project-agent session id (agent x member).

An automation run's rows still go after a day: the hourly inventory
deletes a run's workspace that no row names.

The statement was checked on Postgres in a rolled-back transaction: of
ten rows, it deleted the older of two failed rows of one id, a render's
and a run's lone row, and a failed row behind an active successor, and
kept a lone project-agent row, the newer of the two, a recent one and
one whose only other row is destroyed. The integration lane now holds
those cases; the unit test pins the statement's shape.
The reconcile asks the spawner's workspace inventory whether an agent
session whose compute is gone still has its workspace, and it asks
under the session's lifecycle lock. The read had only the client's own
60 s bound and ignored the pass's abort signal, so one slow inventory
could hold that lock for a minute and outlive the watchdog pass's
120 s budget or the Sandboxes page probe's 20 s one.

The read now takes the caller's signal and the reconcile's own 15 s
bound, the same as its other spawner reads (observe, isAlive). An
inventory that times out on its own reads as unknown and keeps the
workspace, as a failed read did. One that the pass's signal ended
throws instead, so a pass out of time settles nothing on a guess and
leaves the row to its next visit. The pass's shared read passes the
first heal's options through; every heal of a pass carries the same
pass signal.
The docs said a sandbox that disappears on its own keeps its workspace,
reads Stopped and the next run continues in it. That holds where the
heal can see the workspace: it asks the spawner's GET /v1/workspaces,
which lists the server's own workspaces only. A device-placed session
is never listed there, so a readable inventory heals its row as
destroyed.

The admin page now says so in EN, DE and FR (on the Tale server; not
for an environment on a connected device), and SBX-R17, the service's
sessions doc and the reference row say the same. No test changes: the
tests that name SBX-R17 cover server-side sessions only.
SBX-R17 sat under Automatic deletion, whose opening says Tale deletes a
workspace in two cases, while the rule is about a workspace that is
kept when its sandbox disappears. It now opens its own topic, When a
sandbox disappears, between Pinning and destroying a workspace (whose
SBX-R10 covers a pinned workspace in the same event) and Automatic
deletion, and the spec's summary names the topic.

The rule keeps its ID and text: the domain-specs guard asks only that a
card sit under a topic heading before Not yet, with IDs unique and
appended, not that they appear in numeric order.
The failed-create collect lane's opening comment in the integration
check had a 122-column line where an agent session's kept workspace was
added to it; the paragraph is reflowed to the file's 80 columns, words
unchanged.
The automation coverage map now keeps one file per suite, so parallel changes stop appending to the same lines; this change's rows move there, after their own anchor row.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant