fix(sandbox): keep agent workspaces through lost sandboxes

docs/de/platform/admin/sandboxes.md

@@ -51,7 +51,7 @@ Die Werte aktualisieren sich alle 15 Sekunden. Mit **Aktualisieren** forderst du
 
 ## Belegte und inaktive Workspaces unterscheiden
 
-Inhaber und Admins können **Arbeitsbereiche** prüfen. Jede Zeile nennt den zugehörigen Agenten oder Workflow-Lauf, Laufzeitstatus, Belegungsstatus und laufende Aufgaben. Die Zustände beantworten unterschiedliche Fragen: Ein Container kann für die Wiederverwendung weiterlaufen, obwohl er seinen Organisationsplatz bereits freigegeben hat. Der Arbeitsbereich eines Projekt-Agenten bleibt auch im Leerlauf aufgeführt, als **Gestoppt** mit **Kontingent freigegeben**, bis du ihn löschst oder Tale ihn löscht: wenn ihn niemand [während der von deiner Organisation festgelegten Anzahl Tage](#delete-unused-workspaces-automatically) genutzt hat oder wenn [sein Agent, sein Projekt oder das zugehörige Mitglied entfernt wird](#explain-why-a-workspace-disappeared). Wurde der Agent selbst gelöscht, steht in der Zeile **Gelöschter Agent**, bis Tale den Arbeitsbereich gelöscht hat. Der Arbeitsbereich eines Workflow-Laufs wird kurz nach dem Ende des Laufs zurückgefordert.
+Inhaber und Admins können **Arbeitsbereiche** prüfen. Jede Zeile nennt den zugehörigen Agenten oder Workflow-Lauf, Laufzeitstatus, Belegungsstatus und laufende Aufgaben. Die Zustände beantworten unterschiedliche Fragen: Ein Container kann für die Wiederverwendung weiterlaufen, obwohl er seinen Organisationsplatz bereits freigegeben hat. Der Arbeitsbereich eines Projekt-Agenten bleibt auch im Leerlauf aufgeführt, als **Gestoppt** mit **Kontingent freigegeben**, bis du ihn löschst oder Tale ihn löscht: wenn ihn niemand [während der von deiner Organisation festgelegten Anzahl Tage](#delete-unused-workspaces-automatically) genutzt hat oder wenn [sein Agent, sein Projekt oder das zugehörige Mitglied entfernt wird](#explain-why-a-workspace-disappeared). Wurde der Agent selbst gelöscht, steht in der Zeile **Gelöschter Agent**, bis Tale den Arbeitsbereich gelöscht hat. Der Arbeitsbereich eines Workflow-Laufs wird kurz nach dem Ende des Laufs zurückgefordert. Verschwindet eine Umgebung auf dem Tale-Server von selbst, etwa nach einem Neustart des Hosts oder weil ihr der Speicher ausgegangen ist, behält ihr Arbeitsbereich seine Dateien und steht auf **Gestoppt**; der nächste Lauf arbeitet darin weiter. Für eine Umgebung auf einem verbundenen [Gerät](/de/platform/admin/sandbox-devices) gilt das nicht.
 
 Die Ausgaben enthalten die gemessenen Kosten abgeschlossener Durchläufe. Ein noch laufender Durchlauf wird nach seinem Ende eingerechnet. Vorübergehende Crawler-Umgebungen zählen zur Kapazität, auch ohne eigene dauerhafte Workspace-Zeile.
 

docs/en/platform/admin/sandboxes.md

@@ -51,7 +51,7 @@ Measurements refresh every 15 seconds; **Refresh** requests a new observation. C
 
 ## Explain an allocated or idle workspace
 
-Owners and Admins can inspect **Workspaces**. A row identifies its agent or workflow run, runtime state, allocation state and running tasks. These states answer different questions: a container may remain running for reuse after it has released its organization slot. A project agent’s workspace stays listed while the agent is idle, as **Stopped** with **Quota released**, until you destroy it or Tale deletes it: when nobody has used it for [the number of days your organization sets](#delete-unused-workspaces-automatically), or when [its agent, project or member is removed](#explain-why-a-workspace-disappeared). Once the agent itself is deleted, the row reads **Deleted agent** until Tale has deleted the workspace. A workflow run’s workspace is reclaimed shortly after the run ends.
+Owners and Admins can inspect **Workspaces**. A row identifies its agent or workflow run, runtime state, allocation state and running tasks. These states answer different questions: a container may remain running for reuse after it has released its organization slot. A project agent’s workspace stays listed while the agent is idle, as **Stopped** with **Quota released**, until you destroy it or Tale deletes it: when nobody has used it for [the number of days your organization sets](#delete-unused-workspaces-automatically), or when [its agent, project or member is removed](#explain-why-a-workspace-disappeared). Once the agent itself is deleted, the row reads **Deleted agent** until Tale has deleted the workspace. A workflow run’s workspace is reclaimed shortly after the run ends. If an environment on the Tale server disappears on its own, for example after a host restart or because it ran out of memory, its workspace keeps its files and reads **Stopped**, and the next run continues in it. This does not apply to an environment on a connected [device](/platform/admin/sandbox-devices).
 
 **Spend** adds the metered cost of finished turns. A turn still running is included when it ends. Temporary crawler environments appear in capacity counts even without a standing workspace row.
 

docs/fr/platform/admin/sandboxes.md

@@ -51,7 +51,7 @@ Les mesures s’actualisent toutes les 15 secondes. **Actualiser** demande une n
 
 ## Distinguer allocation et environnement inactif
 
-Les Propriétaires et Admins peuvent examiner les espaces de travail. Chaque ligne identifie l’agent ou l’exécution, l’état de l’environnement, celui de l’allocation et les tâches en cours. Ces états répondent à des questions différentes : un conteneur peut rester actif pour être réutilisé après avoir libéré sa place dans l’organisation. L’espace de travail d’un agent de projet reste affiché tant que l’agent est inactif, avec l’état **Arrêtée** et **Quota libéré**, jusqu’à ce que tu le supprimes ou que Tale le supprime : quand personne ne l’a utilisé pendant [le nombre de jours fixé par ton organisation](#delete-unused-workspaces-automatically), ou quand [son agent, son projet ou le membre concerné est retiré](#explain-why-a-workspace-disappeared). Si l’agent lui-même a été supprimé, la ligne indique **Agent supprimé** jusqu’à ce que Tale ait supprimé l’espace. L’espace d’une exécution de workflow est récupéré peu après la fin de l’exécution.
+Les Propriétaires et Admins peuvent examiner les espaces de travail. Chaque ligne identifie l’agent ou l’exécution, l’état de l’environnement, celui de l’allocation et les tâches en cours. Ces états répondent à des questions différentes : un conteneur peut rester actif pour être réutilisé après avoir libéré sa place dans l’organisation. L’espace de travail d’un agent de projet reste affiché tant que l’agent est inactif, avec l’état **Arrêtée** et **Quota libéré**, jusqu’à ce que tu le supprimes ou que Tale le supprime : quand personne ne l’a utilisé pendant [le nombre de jours fixé par ton organisation](#delete-unused-workspaces-automatically), ou quand [son agent, son projet ou le membre concerné est retiré](#explain-why-a-workspace-disappeared). Si l’agent lui-même a été supprimé, la ligne indique **Agent supprimé** jusqu’à ce que Tale ait supprimé l’espace. L’espace d’une exécution de workflow est récupéré peu après la fin de l’exécution. Si un environnement sur le serveur Tale disparaît de lui-même, par exemple après un redémarrage de l’hôte ou faute de mémoire, son espace de travail garde ses fichiers et passe à l’état **Arrêtée**, et l’exécution suivante s’y poursuit. Cela ne vaut pas pour un environnement sur un [appareil](/fr/platform/admin/sandbox-devices) connecté.
 
 Les dépenses additionnent le coût mesuré des échanges terminés. Un échange en cours est ajouté lorsqu’il se termine. Les environnements temporaires du crawler comptent dans la capacité même sans ligne d’espace de travail permanent.
 

services/platform/backend/core/node_only/sandbox/agent_session.test.ts

@@ -13,7 +13,7 @@ import {
 const runtime = vi.hoisted(() => ({
   sessionCreate: vi.fn(),
   sessionAcquire: vi.fn(),
-  sessionDestroyIfIdle: vi.fn(),
+  sessionStopIfIdle: vi.fn(),
 }));
 vi.mock('./helpers/session_client', async (importOriginal) => ({
   ...(await importOriginal<typeof import('./helpers/session_client')>()),
@@ -71,9 +71,9 @@ function fixture(
   runtime.sessionCreate.mockImplementation(async () => {
     events.push('create');
   });
-  runtime.sessionDestroyIfIdle.mockImplementation(async () => {
-    events.push('destroy');
-    return { destroyed: true, busy: false };
+  runtime.sessionStopIfIdle.mockImplementation(async () => {
+    events.push('stop');
+    return { stopped: true, busy: false, workspaceKept: true };
   });
   return {
     ctx,
@@ -237,7 +237,7 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
       );
       // The standing workspace is the incarnation's state: a failed resume
       // releases the slot and never destroys it.
-      expect(runtime.sessionDestroyIfIdle).not.toHaveBeenCalled();
+      expect(runtime.sessionStopIfIdle).not.toHaveBeenCalled();
     },
   );
 
@@ -284,7 +284,7 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
         status: 'active',
       });
       expect(runtime.sessionAcquire).toHaveBeenCalledTimes(adopt ? 1 : 0);
-      expect(runtime.sessionDestroyIfIdle).not.toHaveBeenCalled();
+      expect(runtime.sessionStopIfIdle).not.toHaveBeenCalled();
     },
   );
 
@@ -294,7 +294,7 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
     await expect(f.ensure()).rejects.toBe(f.mutationError);
 
     expect(runtime.sessionCreate).not.toHaveBeenCalled();
-    expect(runtime.sessionDestroyIfIdle).not.toHaveBeenCalled();
+    expect(runtime.sessionStopIfIdle).not.toHaveBeenCalled();
   });
 
   it('marks a failed fresh create as failed instead of leaving its quota occupied', async () => {
@@ -310,15 +310,14 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
     });
     expect(f.events).toEqual([
       'reserveSessionSlotAndInsert',
-      'destroy',
+      'stop',
       'setSessionStatus',
     ]);
   });
 
   // A refused create (429: the host is full, or short of memory) made
-  // nothing — and a destroy of an id with no compute deletes the preserved
-  // workspace a stopped standing session keeps under it.
-  it('a create the sandbox host refused destroys nothing, and the row reads failed', async () => {
+  // nothing to stop.
+  it('a create the sandbox host refused stops nothing, and the row reads failed', async () => {
     const f = fixture(scenario, null);
     const error = new SpawnerBusyError(15_000);
     runtime.sessionCreate.mockImplementation(async () => {
@@ -328,24 +327,26 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
 
     await expect(f.ensure()).rejects.toBe(error);
 
-    expect(runtime.sessionDestroyIfIdle).not.toHaveBeenCalled();
+    expect(runtime.sessionStopIfIdle).not.toHaveBeenCalled();
     expect(f.events).toEqual([
       'reserveSessionSlotAndInsert',
       'create',
       'setSessionStatus',
     ]);
     // Settled as collected: the COLLECT pass would otherwise run the very
-    // destroy this skipped once the row's grace had passed.
+    // stop this skipped once the row's grace had passed.
     expect(f.ctx.runMutation).toHaveBeenLastCalledWith(expect.anything(), {
       rowId: 'row_1',
       status: 'failed',
       collected: true,
     });
   });
 
-  // The regression (#3494): a create the spawner had started, then failed —
-  // its container stayed in Docker state `created` with no owner to remove it.
-  it('destroys what a created-then-failed session left spawner-side before its row reads failed', async () => {
+  // A create the spawner had started, then failed, left its container in
+  // Docker state `created` with no owner to remove it. The stop releases that
+  // compute and keeps the workspace: the id may name one preserved for the
+  // owner's next turn (its last container lost to a reboot or the OOM killer).
+  it('stops what a created-then-failed session left spawner-side, keeping its workspace, before its row reads failed [SBX-R17]', async () => {
     const f = fixture(scenario, null);
     const error = new Error('runnerd did not become ready');
     runtime.sessionCreate.mockImplementation(async () => {
@@ -355,14 +356,14 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
 
     await expect(f.ensure()).rejects.toBe(error);
 
-    expect(runtime.sessionDestroyIfIdle).toHaveBeenCalledTimes(1);
-    expect(runtime.sessionDestroyIfIdle).toHaveBeenCalledWith('session_1');
+    expect(runtime.sessionStopIfIdle).toHaveBeenCalledTimes(1);
+    expect(runtime.sessionStopIfIdle).toHaveBeenCalledWith('session_1');
     // While the row is still `creating` it holds the owner's slot, so no
-    // fresh create of the same id can start under the destroy.
+    // fresh create of the same id can start under the stop.
     expect(f.events).toEqual([
       'reserveSessionSlotAndInsert',
       'create',
-      'destroy',
+      'stop',
       'setSessionStatus',
     ]);
     expect(f.ctx.runMutation).toHaveBeenLastCalledWith(expect.anything(), {
@@ -371,7 +372,7 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
     });
   });
 
-  it('destroys an adopted orphan that turned out to be gone', async () => {
+  it('stops an adopted orphan that turned out to be gone', async () => {
     const f = fixture(scenario, null);
     runtime.sessionCreate.mockImplementation(async () => {
       f.events.push('create');
@@ -381,27 +382,27 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
 
     await expect(f.ensure()).rejects.toBeInstanceOf(SessionNotFoundError);
 
-    expect(runtime.sessionDestroyIfIdle).toHaveBeenCalledWith('session_1');
+    expect(runtime.sessionStopIfIdle).toHaveBeenCalledWith('session_1');
     expect(f.events).toEqual([
       'reserveSessionSlotAndInsert',
       'create',
-      'destroy',
+      'stop',
       'setSessionStatus',
     ]);
   });
 
-  it('a destroy that fails is logged, never thrown: the row still reads failed and the create error survives', async () => {
+  it('a stop that fails is logged, never thrown: the row still reads failed and the create error survives', async () => {
     const f = fixture(scenario, null);
     const error = new Error('container create failed');
     runtime.sessionCreate.mockRejectedValue(error);
-    const destroyError = new Error('sandbox session destroy failed (502)');
-    runtime.sessionDestroyIfIdle.mockRejectedValue(destroyError);
+    const stopError = new Error('sandbox session stop failed (502)');
+    runtime.sessionStopIfIdle.mockRejectedValue(stopError);
 
     await expect(f.ensure()).rejects.toBe(error);
 
     expect(console.warn).toHaveBeenCalledWith(
-      expect.stringContaining('destroy after failed create of session_1'),
-      destroyError,
+      expect.stringContaining('stop after failed create of session_1'),
+      stopError,
     );
     expect(f.ctx.runMutation).toHaveBeenLastCalledWith(expect.anything(), {
       rowId: 'row_1',
@@ -416,14 +417,15 @@ describe.each(scenarios)('ensureAgentSession ($owner.type)', (scenario) => {
     const f = fixture(scenario, null);
     const error = new Error('runnerd did not become ready');
     runtime.sessionCreate.mockRejectedValue(error);
-    runtime.sessionDestroyIfIdle.mockResolvedValue({
-      destroyed: false,
+    runtime.sessionStopIfIdle.mockResolvedValue({
+      stopped: false,
       busy: true,
+      workspaceKept: true,
     });
 
     await expect(f.ensure()).rejects.toBe(error);
 
-    expect(runtime.sessionDestroyIfIdle).toHaveBeenCalledWith('session_1');
+    expect(runtime.sessionStopIfIdle).toHaveBeenCalledWith('session_1');
     expect(console.warn).toHaveBeenCalledWith(
       expect.stringContaining("runs a sibling turn's exec"),
     );

services/platform/backend/core/node_only/sandbox/agent_session.ts

@@ -12,7 +12,7 @@ import {
   SpawnerBusyError,
   sessionAcquire,
   sessionCreate,
-  sessionDestroyIfIdle,
+  sessionStopIfIdle,
 } from './helpers/session_client';
 
 type SessionContext = Pick<ActionCtx, 'runQuery' | 'runMutation'>;
@@ -61,10 +61,11 @@ function ownerPolicy(
  * /agent, including the harness conversation store, survives a stop on both
  * backends. A new row returns no previous stamp, even when adopting an orphan,
  * so a caller cannot resume a conversation from an unproven incarnation.
- * A new row whose create fails first asks the spawner to destroy whatever it
- * holds under the id unless a sibling turn is executing in it, then reads
- * `failed`; the sandbox watchdog collects what that best-effort destroy could
- * not.
+ * A new row whose create fails first asks the spawner to stop whatever it
+ * holds under the id unless a sibling turn is executing in it — compute
+ * only: the workspace stays, since the id may name one preserved for the
+ * owner's next turn — then reads `failed`; the sandbox watchdog collects
+ * what that best-effort stop could not.
  */
 export async function ensureAgentSession(
   ctx: SessionContext,
@@ -152,36 +153,40 @@ export async function ensureAgentSession(
   } catch (error) {
     // The spawner may already hold what this create made (one cut short
     // between Docker's create and start stays `created`), and a `failed` row
-    // is never reconciled, resumed or listed. Destroy while this row still
+    // is never reconciled, resumed or listed. Stop it while this row still
     // holds the owner's slot: once it reads `failed`, a fresh create of the
-    // same deterministic id may start, and a later destroy would hit that one.
-    // Only an idle session goes: a sibling turn of the same owner can resume
-    // this still-`creating` row and create or adopt the session itself, and
-    // its running exec must never die for this turn's failure. A container
-    // that never started runs no exec, so it is idle. A spawner that refused
-    // the create for want of room (429) made nothing to destroy — and a
-    // destroy of an id with no compute deletes its preserved workspace, which
-    // a stopped standing session's id still names. Its row is settled as
-    // collected, too: the watchdog's COLLECT pass would otherwise run that
-    // very destroy once the row's grace had passed.
+    // same deterministic id may start, and a later stop would hit that one.
+    // Only an idle session's compute goes: a sibling turn of the same owner
+    // can resume this still-`creating` row and create or adopt the session
+    // itself, and its running exec must never die for this turn's failure. A
+    // container that never started runs no exec, so it is idle. The workspace
+    // always stays: this row is new because the last one was settled, not
+    // because nothing is kept under the id — a container lost to a host
+    // reboot or the OOM killer leaves its workspace for the next turn, and a
+    // create that then fails (runnerd slow to start, an unreachable spawner)
+    // must not turn that loss into the loss of the files. Deleting what
+    // nothing owns is the workspace cleanup's. A spawner that refused the
+    // create for want of room (429) made nothing to stop, and its row is
+    // settled as collected: the watchdog's COLLECT pass would otherwise run
+    // that very stop once the row's grace had passed.
     if (error instanceof SpawnerBusyError) {
       await ctx.runMutation(
         internal.sandbox.session_mutations.setSessionStatus,
         { rowId, status: 'failed', collected: true },
       );
       throw error;
     }
-    await sessionDestroyIfIdle(sessionId)
+    await sessionStopIfIdle(sessionId)
       .then(({ busy }) => {
         if (busy)
           console.warn(
             `[sandbox.session] ${sessionId} runs a sibling turn's exec after this failed create; the watchdog collects it once idle`,
           );
       })
-      .catch((destroyError: unknown) => {
+      .catch((stopError: unknown) => {
         console.warn(
-          `[sandbox.session] destroy after failed create of ${sessionId} failed (the watchdog collects it):`,
-          destroyError,
+          `[sandbox.session] stop after failed create of ${sessionId} failed (the watchdog collects it):`,
+          stopError,
         );
       });
     await ctx.runMutation(internal.sandbox.session_mutations.setSessionStatus, {