Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/verify-sandbox-asleep/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,12 @@ Synced with commit `0ca8daad`. Paths are under `packages/trpc/src/`.
| `ProvisionDone` | `provisionCloudWorkspace` in `router/cloud-workspace/provision.ts`: `provisionSandbox`, the transition to `awake`, then `recordSandboxAwake` |
| `WakeResume` / `WakeRecord` | `wakeCloudWorkspace` in `router/cloud-workspace/wake.ts`: `wakeSandbox` in `lib/sandbox/vercel.ts`, then `recordSandboxAwake` in `router/cloud-workspace/sandbox-expiry.ts`. A wake that read the box running and resumed it after a timeout stop is the same `WakeResume`: the box was stopped when the call resumed it |
| `TopUpExtend` / `TopUpRecord` | `sandbox.keepalive` in `router/sandbox/sandbox.ts`: `topUpSandboxSession`, then `recordSandboxAwake` |
| `StopBox` / `StopRecord` | `stopAndSnapshot` or `stopSandbox`, then `recordSandboxAsleep` in `router/cloud-workspace/sandbox-sleep.ts` (most sites call it through `recordBoxStopped`, which first writes the boot's end reason, a write the model leaves out), from `cloudWorkspace.sleep` (`cloud-workspace.ts`), `restartCloudWorkspace` and `wakeAfterStop` (`wake.ts`; `environment.promote` calls `wakeAfterStop` after the snapshot in `promoteSandboxToEnvironment` stopped the source, so `StopBox` there is the snapshot), a refused `wakeCloudWorkspace` whose own call resumed the box (it records that session awake first, a `WakeRecord`, then stops it; a refusal for a box another wake resumed stops nothing), and the spend-cap stops in `meter.ts` (`settleBoot`, `settleOrganizationNow`, the refused adopt in `adoptRunningBoxes`) |
| `StopBox` / `StopRecord` | `stopAndSnapshot` or `stopSandbox`, then `recordSandboxAsleep` in `router/cloud-workspace/sandbox-sleep.ts` (most sites call it through `recordBoxStopped`, which first writes the boot's end reason, a write the model leaves out), from `cloudWorkspace.sleep` (`cloud-workspace.ts`), `restartCloudWorkspace` and `wakeAfterStop` (`wake.ts`; `environment.promote` calls `wakeAfterStop` after the snapshot in `promoteSandboxToEnvironment` stopped the source, so `StopBox` there is the snapshot), a create whose drive refused (`provision.ts`, after its transition to `awake`), a refused `wakeCloudWorkspace` whose own call resumed the box (it records that session awake first, a `WakeRecord`, then stops it; a refusal for a box another wake resumed stops nothing), and the spend-cap stops in `meter.ts` (`settleBoot`, `settleOrganizationNow`, the refused adopt in `adoptRunningBoxes`) |
| `WatchFire` | `watchSandboxExpiry` in `sandbox-expiry.ts`, queued by `queueWatch` through `publishCloudWorkspaceJob` (`jobs.ts`), served by `apps/api/src/app/api/cloud-workspaces/sandbox-expiry/route.ts` |
| `ReconcileList` / `ReconcileApply` | `reconcileCloudWorkspaces` in `sandbox-expiry.ts`: `listActiveWorkspaceSandboxes`, then `syncSandboxExpiries` against each listing's `currentSessionId`; a stopped listing goes through `recordStopSeen` in `sandbox-sleep.ts` (`recordSandboxAsleep`, with the end reason first when the recorded expiry has passed) |
| `Archive` | `cloudWorkspace.delete` in `cloud-workspace.ts`: the transition to `archived`, then `queueReap` |
| `Reap` | the stop stage of `reapArchivedCloudWorkspace` in `reap.ts`, and the `queueReap` fallback that stops at once when the stop cannot be queued. Both call `recordSandboxAsleep`, which on an archived row writes only the expiry and session id |
| `Tick` / `Crash` | Vercel ends a session at its timeout, or a session fails on its own |
| `Tick` / `Crash` | Vercel ends a session at its timeout, or a session fails on its own. A wake that stops a box whose drive refused (`wakeSandbox`) writes no status, so the row sees it as a crash |
| `CAS` | `stillStoppedSession` in `sandbox-sleep.ts` (asleep writes) and the new-session rule in `markAwake` (awake writes) |

When you add, remove or reorder a writer, change its action here and in `Asleep.tla`, run both
Expand Down
50 changes: 49 additions & 1 deletion .github/workflows/sandbox.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,24 @@ on:
- real-sandbox
- production-release
- inventory
- drive-sweep
default: real-sandbox
real_sandbox_drive:
description: "real-sandbox only: also run the drive-mode pass (a throwaway golden drive, a box on a fork of it, a stop and a wake)"
type: boolean
default: false
real_sandbox_keep:
description: "real-sandbox only: keep the boxes (stopped) and the box drive for inspection instead of deleting them"
type: boolean
default: false
drive_sweep_delete:
description: "drive-sweep only: delete the orphaned drives it lists (off: a dry run)"
type: boolean
default: false
real_sandbox_work_gb:
description: "real-sandbox drive pass only: GB written under /workspace on the drive box and the plain box"
type: string
default: "4"

jobs:
boot-twice:
Expand Down Expand Up @@ -119,7 +136,7 @@ jobs:
name: Real sandbox
if: github.event_name == 'workflow_dispatch' && inputs.run == 'real-sandbox'
runs-on: ubuntu-latest
timeout-minutes: 30
timeout-minutes: 90
steps:
- name: Check out code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
Expand All @@ -146,6 +163,9 @@ jobs:
CDN_R2_ENDPOINT: ${{ secrets.CDN_R2_ENDPOINT }}
CDN_R2_BUCKET: ${{ secrets.CDN_R2_BUCKET }}
CDN_URL: ${{ secrets.CDN_URL }}
REAL_SANDBOX_DRIVE: ${{ inputs.real_sandbox_drive && '1' || '' }}
REAL_SANDBOX_KEEP: ${{ inputs.real_sandbox_keep && '1' || '' }}
REAL_SANDBOX_WORK_GB: ${{ inputs.real_sandbox_work_gb }}
run: bun run src/real-sandbox.ts

inventory:
Expand Down Expand Up @@ -173,6 +193,34 @@ jobs:
VERCEL_SANDBOX_PROJECT_ID: ${{ secrets.VERCEL_SANDBOX_PROJECT_ID }}
run: bun run inventory

drive-sweep:
name: Drive sweep
if: github.event_name == 'workflow_dispatch' && inputs.run == 'drive-sweep'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Check out code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1

- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: .bun-version

- name: Install dependencies
run: bun install --frozen --ignore-scripts

- name: List, or delete, drives nothing names
working-directory: packages/sandbox
env:
NEON_API_KEY: ${{ secrets.NEON_API_KEY }}
NEON_PROJECT_ID: ${{ secrets.NEON_PROJECT_ID }}
VERCEL_SANDBOX_TOKEN: ${{ secrets.VERCEL_SANDBOX_TOKEN }}
VERCEL_SANDBOX_TEAM_ID: ${{ secrets.VERCEL_SANDBOX_TEAM_ID }}
VERCEL_SANDBOX_PROJECT_ID: ${{ secrets.VERCEL_SANDBOX_PROJECT_ID }}
DRIVE_SWEEP_DELETE: ${{ inputs.drive_sweep_delete && '--delete' || '' }}
run: bun run drive-sweep --production $DRIVE_SWEEP_DELETE

production-release:
name: Production release
# A merge that touches the image or the bundle releases it, and the nightly
Expand Down
1 change: 1 addition & 0 deletions apps/api/src/app/api/cloud-workspaces/provision/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ const payloadSchema = z
/** Absent when the user typed a name, which the row already holds. */
namingPrompt: z.string().max(CLOUD_AGENT_PROMPT_MAX_LENGTH).optional(),
launch: cloudAgentLaunchSchema.optional(),
lockRetries: z.number().int().min(0).optional(),
})
.strict();

Expand Down
34 changes: 34 additions & 0 deletions apps/api/src/app/api/cloud-workspaces/teardown/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import { teardownSandbox } from "@superset/trpc/cloud-workspace-teardown";
import { z } from "zod";
import { verifyQstashRequest } from "@/lib/verifyQstash";

export const dynamic = "force-dynamic";
export const maxDuration = 60;

const payloadSchema = z
.object({
providerSandboxId: z.string().min(1),
driveName: z.string().min(1).nullable(),
attempt: z.number().int().min(0),
keepDrive: z.boolean().optional(),
})
.strict();

/** Stops a box, then deletes it and its drive once the drive has detached; requeues itself until then. */
export async function POST(request: Request): Promise<Response> {
const body = await request.text();
const rejected = await verifyQstashRequest(
request,
body,
"/api/cloud-workspaces/teardown",
);
if (rejected) return rejected;

const parsed = payloadSchema.safeParse(JSON.parse(body));
if (!parsed.success) {
console.error("[cloud-workspaces/teardown] invalid payload", parsed.error);
return Response.json({ error: "Invalid payload" }, { status: 400 });
}
const outcome = await teardownSandbox(parsed.data);
return Response.json({ ok: true, outcome });
}
63 changes: 62 additions & 1 deletion docs/cloud-sandbox-mismatches.md
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,56 @@ socket, all describing processes that no longer exist. The boot runner
deletes and recreates it first thing; anything that reads a flag there must
tolerate a stale one for the milliseconds before that.

**A workspace drive is not a disk the box owns.** In drive mode
(`environments.golden_drive_name` set), `/workspace` and large home subtrees
live on a Vercel drive at `/mnt/superset-drive`, so they never enter the
box's snapshot. Three things a machine someone owns never makes you think
about:

- *Bind mounts end with the session.* Vercel attaches the drive on every
session (the control plane passes `mounts` on every create and wake), but
the `mount --bind` of `/workspace` and each home subtree is gone after a
stop. `superset-boot` lays them out again on every boot, before
host-service and the checkout read `/workspace` (`drive.layout` in the
boot log). Anything that runs before that sees the root disk's empty
`/workspace`.
- *Hardlinks must stay inside `/workspace`.* Each bind mount is its own
`vfsmount`, and `link(2)` across two of them fails with `EXDEV` even on the
same filesystem. bun and pnpm then copy every package instead of linking
it. So their stores are inside the `/workspace` mount
(`BUN_INSTALL_CACHE_DIR=/workspace/.superset/stores/bun`,
`npm_config_store_dir=/workspace/.superset/stores/pnpm`, in host-service's
env and the login profile), not under `~/.bun` or `~/.cache`.
- *A drive needs detach before the next mount.* After a stop the drive stays
attached while the session flushes what it wrote (6 s after 1 GB, 44 s
after 12 GB), and a mount taken meanwhile reads a corrupt or empty
filesystem. `wakeSandbox` waits for `Drive.get(...).currentSandboxName` to
clear, at most 2 minutes, and fails the wake rather than mount. A sandbox
deleted while it still holds its drive leaves the drive attached for
minutes, so every delete stops the box first (`deleteSandboxAndDrive`).

A drive has no discard either: deleting a file on it does not lower its
billed size.
A drive that did not mount leaves the root disk's empty `/workspace` in its
place, which would look like the user's work is gone. So the identity file
names the drive (`SUPERSET_SANDBOX_DRIVE`), the drive names its owner
(`/mnt/superset-drive/state/workspace-id`), and `superset-boot` refuses to
continue (`drive.missing`, `drive.foreign`, `drive.unlaid`) before the bundle,
host-service and the checkout. The control plane spots the refusal in this
boot's lines of the boot log within about 2 s and answers "still starting";
the next wake mounts the drive afresh, and a create records the box asleep.
A refusal no remount can change (another workspace's drive, a laid-out drive
that mounts empty, a boot script that predates the drive) ends after 3 boots
in a row as `failed`, with the box and the drive kept. A boot that answers
without `drive.layout` is refused the same way.
A box on an image whose runner predates the guard gets it after the reexec
into the bundle's runner, still before host-service.

*The root disk and the drive can disagree.* A stop whose snapshot does not
land rolls the root back and not the drive. So in drive mode the checkout
markers and `host.db` live on the drive, and the checkout never replaces a
non-empty checkout there.

**The control plane's command runs from the image's workdir, which the
checkout replaces.** The first boot cloned `/workspace` by deleting and
recreating it, and every child of the boot command — host-service, git,
Expand Down Expand Up @@ -961,7 +1011,9 @@ OpenCode and Pi update in place, beside their binary. On a box they run from
a read-only mount and are pinned by the bundle. `/etc/profile.d/superset.sh`
turns off their update checks (`OPENCODE_DISABLE_AUTOUPDATE`,
`PI_SKIP_VERSION_CHECK`), the same way `DISABLE_AUTOUPDATER` does for
Claude. They keep their state under `~/.local/share/opencode`,
Claude. `superset-boot` also puts all three in host-service's own env: a chat
agent gets host-service's env, not a terminal's, whenever host-service's
login-shell probe fails. They keep their state under `~/.local/share/opencode`,
`~/.config/opencode` and `~/.pi/agent`, as they do on a laptop.

**A read-only snapshot mount taken while a writer holds the drive is corrupt.**
Expand Down Expand Up @@ -1076,3 +1128,12 @@ doesn't re-spend a session on it; the next thing to try is attaching to the
`win.setSize()` directly rather than simulating it from outside, or watching
`app.on("child-process-gone")` / `render-process-gone` across a CDP-reload
loop to catch the actual GPU-process death this is presumably downstream of.

**`sudo` cannot resolve the box's hostname.** A machine someone owns has its
hostname in `/etc/hosts`; a Vercel box does not, so every `sudo` (including
`runCommand({ sudo: true })`) writes `sudo: unable to resolve host <name>:
Name or service not known` to stderr. The command still runs. A check that
reads `stdout + stderr` then sees an extra line and fails a comparison that
was correct; the first real drive check failed 7 expectations that way.
**What we did:** the real-sandbox drive pass compares stdout only and shows
stderr as detail. **Open:** the hostname is still missing from `/etc/hosts`.
3 changes: 3 additions & 0 deletions packages/i18n/locales/cs/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Tímto zrušíte přípravu a vrátíte všechny připravené změny. P
msgid "This workspace"
msgstr "Tento pracovní prostor"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Tento pracovní prostor se nepodařilo uložit jako prostředí. Zkuste to později."

msgid "This workspace has no project to branch from"
msgstr "Tento pracovní prostor nemá projekt, ze kterého by šlo větvit"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/de/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Damit werden alle gestageden Änderungen aus dem Index genommen und zur
msgid "This workspace"
msgstr "Dieser Arbeitsbereich"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Dieser Arbeitsbereich konnte nicht als Umgebung gespeichert werden. Versuche es später erneut."

msgid "This workspace has no project to branch from"
msgstr "Dieser Workspace hat kein Projekt, von dem aus verzweigt werden kann"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/en/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "This will unstage and revert all staged changes. Staged new files will b
msgid "This workspace"
msgstr "This workspace"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "This workspace could not be saved as an environment. Try again later."

msgid "This workspace has no project to branch from"
msgstr "This workspace has no project to branch from"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/es/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Esto quitará de preparados y revertirá todos los cambios preparados. L
msgid "This workspace"
msgstr "Este espacio de trabajo"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "No se pudo guardar este espacio de trabajo como entorno. Inténtalo de nuevo más tarde."

msgid "This workspace has no project to branch from"
msgstr "Este espacio de trabajo no tiene un proyecto desde el que bifurcar"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/fr/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Cette action désindexera et annulera toutes les modifications indexées
msgid "This workspace"
msgstr "Cet espace de travail"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Impossible d’enregistrer cet espace de travail comme environnement. Réessayez plus tard."

msgid "This workspace has no project to branch from"
msgstr "Cet espace de travail n'a aucun projet depuis lequel dupliquer"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/id/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Ini akan meng-unstage dan mengembalikan semua perubahan yang di-stage. F
msgid "This workspace"
msgstr "Workspace ini"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Workspace ini tidak dapat disimpan sebagai lingkungan. Coba lagi nanti."

msgid "This workspace has no project to branch from"
msgstr "Ruang kerja ini tidak memiliki proyek untuk dicabangkan"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/it/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Questo toglierà dallo stage e annullerà tutte le modifiche in stage. I
msgid "This workspace"
msgstr "Questo workspace"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Impossibile salvare questo workspace come ambiente. Riprova più tardi."

msgid "This workspace has no project to branch from"
msgstr "Questo workspace non ha un progetto da cui diramare"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/ja/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "ステージ済みの変更をすべてステージ解除して元に戻
msgid "This workspace"
msgstr "このワークスペース"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "このワークスペースを環境として保存できませんでした。しばらくしてからもう一度お試しください。"

msgid "This workspace has no project to branch from"
msgstr "このワークスペースには分岐元のプロジェクトがありません"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/ko/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "스테이징된 변경 사항을 모두 스테이징 해제하고 되돌
msgid "This workspace"
msgstr "이 워크스페이스"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "이 워크스페이스를 환경으로 저장할 수 없습니다. 나중에 다시 시도하세요."

msgid "This workspace has no project to branch from"
msgstr "이 워크스페이스에는 분기할 프로젝트가 없습니다"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/nl/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Hiermee worden alle gestagede wijzigingen ge-unstaged en teruggedraaid.
msgid "This workspace"
msgstr "Deze werkruimte"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Deze werkruimte kon niet als omgeving worden opgeslagen. Probeer het later opnieuw."

msgid "This workspace has no project to branch from"
msgstr "Deze workspace heeft geen project om vanaf af te splitsen"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/pl/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Usunie to z indeksu i cofnie wszystkie zaindeksowane zmiany. Zaindeksowa
msgid "This workspace"
msgstr "Ten obszar roboczy"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Nie udało się zapisać tego obszaru roboczego jako środowiska. Spróbuj ponownie później."

msgid "This workspace has no project to branch from"
msgstr "Ten obszar roboczy nie ma projektu, z którego można się rozgałęzić"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/pt-BR/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Isso remove do stage e reverte todas as alterações no stage. Arquivos
msgid "This workspace"
msgstr "Esta área de trabalho"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Não foi possível salvar esta área de trabalho como ambiente. Tente novamente mais tarde."

msgid "This workspace has no project to branch from"
msgstr "Este workspace não tem projeto do qual ramificar"

Expand Down
3 changes: 3 additions & 0 deletions packages/i18n/locales/ru/messages.po
Original file line number Diff line number Diff line change
Expand Up @@ -20867,6 +20867,9 @@ msgstr "Это снимет индексацию и откатит все про
msgid "This workspace"
msgstr "Эта рабочая область"

msgid "This workspace could not be saved as an environment. Try again later."
msgstr "Не удалось сохранить это рабочее пространство как окружение. Повторите попытку позже."

msgid "This workspace has no project to branch from"
msgstr "В этом рабочем пространстве нет проекта для ответвления"

Expand Down
Loading
Loading