From f9c6868fb8b0b59a78c0aa13381227128aa4843a Mon Sep 17 00:00:00 2001 From: Satya Patel <14907857+saddlepaddle@users.noreply.github.com> Date: Wed, 18 Nov 2026 22:12:21 +0000 Subject: [PATCH 1/3] feat(sandbox): per-workspace Vercel drive behind cloud-workspace-drives The workspace drive work of #9847 as one commit on main after 78a831b9 (#9915, migration 0199), resolved by hand against main's M5b failure reasons, M7s logging, M8 metering, M9 non-resuming handles, and the agent-tools drive. The previous 33 commits are kept locally as backup/9847-pre-rebase. Merged with main: - wakeSandbox keeps the resume lock, the drive detach, the composed mounts and withLiveSession, and takes main's onlyIfRunning and its non-resuming handles; a session withLiveSession resumes mid-wake is reported to beforeResume as resumedSession before it boots, so it is metered as M9 meters it. - promoteSandboxToEnvironment takes main's shape: the snapshot ends the source's session and the caller's wakeAfterStop starts it again; the drive fork and the golden drive's strip stay. - provision fails through failCloudWorkspace, which now takes onBox and answers whether it moved the row, so a provision still fails only its own row. - A delete or a failed create still retires the names (retireAndTearDown). --- .agents/skills/verify-sandbox-asleep/SKILL.md | 4 +- .github/workflows/sandbox.yml | 50 +- .../api/cloud-workspaces/provision/route.ts | 1 + .../api/cloud-workspaces/teardown/route.ts | 34 + docs/cloud-sandbox-mismatches.md | 59 ++ packages/i18n/locales/cs/messages.po | 3 + packages/i18n/locales/de/messages.po | 3 + packages/i18n/locales/en/messages.po | 3 + packages/i18n/locales/es/messages.po | 3 + packages/i18n/locales/fr/messages.po | 3 + packages/i18n/locales/id/messages.po | 3 + packages/i18n/locales/it/messages.po | 3 + packages/i18n/locales/ja/messages.po | 3 + packages/i18n/locales/ko/messages.po | 3 + packages/i18n/locales/nl/messages.po | 3 + packages/i18n/locales/pl/messages.po | 3 + packages/i18n/locales/pt-BR/messages.po | 3 + packages/i18n/locales/ru/messages.po | 3 + packages/i18n/locales/tr/messages.po | 3 + packages/i18n/locales/uk/messages.po | 3 + packages/i18n/locales/vi/messages.po | 3 + packages/i18n/locales/zh-CN/messages.po | 3 + packages/i18n/locales/zh-TW/messages.po | 3 + packages/i18n/src/server-errors.ts | 7 + packages/sandbox/README.md | 2 +- .../bundle/rootfs/etc/profile.d/superset.sh | 2 + .../bundle/rootfs/usr/local/bin/superset-boot | 179 +++- packages/sandbox/package.json | 1 + packages/sandbox/src/boot-twice.ts | 594 +++++++++++- packages/sandbox/src/drive-comparison.test.ts | 64 ++ packages/sandbox/src/drive-comparison.ts | 204 ++++ packages/sandbox/src/drive-sweep.test.ts | 80 ++ packages/sandbox/src/drive-sweep.ts | 199 ++++ packages/sandbox/src/environments/probe.ts | 28 + .../src/environments/production-database.ts | 37 + packages/sandbox/src/environments/release.ts | 98 +- packages/sandbox/src/real-sandbox.ts | 685 ++++++++++++- packages/shared/src/constants.ts | 7 + packages/shared/src/sandbox/contract.ts | 57 ++ packages/trpc/package.json | 4 + packages/trpc/src/lib/sandbox/claim.test.ts | 112 +++ packages/trpc/src/lib/sandbox/claim.ts | 20 + packages/trpc/src/lib/sandbox/drives.test.ts | 212 +++++ packages/trpc/src/lib/sandbox/drives.ts | 250 +++++ .../trpc/src/lib/sandbox/golden-drives.ts | 24 + packages/trpc/src/lib/sandbox/index.ts | 17 + .../src/lib/sandbox/vercel-credentials.ts | 9 + packages/trpc/src/lib/sandbox/vercel.test.ts | 897 +++++++++++++++++- packages/trpc/src/lib/sandbox/vercel.ts | 800 ++++++++++++---- .../cloud-workspace/client-status.test.ts | 14 +- .../router/cloud-workspace/client-status.ts | 12 +- .../router/cloud-workspace/cloud-workspace.ts | 10 +- .../router/cloud-workspace/drive-laid-out.ts | 18 + .../cloud-workspace/provision-drive.test.ts | 288 ++++++ .../src/router/cloud-workspace/provision.ts | 260 +++-- .../src/router/cloud-workspace/reap.test.ts | 48 + .../trpc/src/router/cloud-workspace/reap.ts | 14 +- .../src/router/cloud-workspace/resume-lock.ts | 41 + .../router/cloud-workspace/sandbox-name.ts | 10 + .../router/cloud-workspace/teardown-guard.ts | 57 ++ .../router/cloud-workspace/teardown.test.ts | 229 +++++ .../src/router/cloud-workspace/teardown.ts | 167 ++++ .../src/router/cloud-workspace/transition.ts | 12 +- .../src/router/cloud-workspace/wake.test.ts | 81 ++ .../trpc/src/router/cloud-workspace/wake.ts | 24 +- .../router/environment/environment.test.ts | 75 +- .../src/router/environment/environment.ts | 85 +- .../router/environment/resolve-environment.ts | 3 + plans/20261116-workspace-drive.md | 590 ++++++++++++ 69 files changed, 6466 insertions(+), 363 deletions(-) create mode 100644 apps/api/src/app/api/cloud-workspaces/teardown/route.ts create mode 100644 packages/sandbox/src/drive-comparison.test.ts create mode 100644 packages/sandbox/src/drive-comparison.ts create mode 100644 packages/sandbox/src/drive-sweep.test.ts create mode 100644 packages/sandbox/src/drive-sweep.ts create mode 100644 packages/sandbox/src/environments/production-database.ts create mode 100644 packages/trpc/src/lib/sandbox/claim.test.ts create mode 100644 packages/trpc/src/lib/sandbox/drives.test.ts create mode 100644 packages/trpc/src/lib/sandbox/drives.ts create mode 100644 packages/trpc/src/lib/sandbox/golden-drives.ts create mode 100644 packages/trpc/src/lib/sandbox/vercel-credentials.ts create mode 100644 packages/trpc/src/router/cloud-workspace/drive-laid-out.ts create mode 100644 packages/trpc/src/router/cloud-workspace/provision-drive.test.ts create mode 100644 packages/trpc/src/router/cloud-workspace/resume-lock.ts create mode 100644 packages/trpc/src/router/cloud-workspace/sandbox-name.ts create mode 100644 packages/trpc/src/router/cloud-workspace/teardown-guard.ts create mode 100644 packages/trpc/src/router/cloud-workspace/teardown.test.ts create mode 100644 packages/trpc/src/router/cloud-workspace/teardown.ts create mode 100644 plans/20261116-workspace-drive.md diff --git a/.agents/skills/verify-sandbox-asleep/SKILL.md b/.agents/skills/verify-sandbox-asleep/SKILL.md index 18fe8968e98..f3b41e9df58 100644 --- a/.agents/skills/verify-sandbox-asleep/SKILL.md +++ b/.agents/skills/verify-sandbox-asleep/SKILL.md @@ -38,12 +38,12 @@ Synced with commit `0ca8daad`. Paths are under `packages/trpc/src/`. | `ProvisionDone` | `provisionCloudWorkspace` in `router/cloud-workspace/provision.ts`: `provisionSandbox`, the transition to `awake`, then `recordSandboxAwake` | | `WakeResume` / `WakeRecord` | `wakeCloudWorkspace` in `router/cloud-workspace/wake.ts`: `wakeSandbox` in `lib/sandbox/vercel.ts`, then `recordSandboxAwake` in `router/cloud-workspace/sandbox-expiry.ts`. A wake that read the box running and resumed it after a timeout stop is the same `WakeResume`: the box was stopped when the call resumed it | | `TopUpExtend` / `TopUpRecord` | `sandbox.keepalive` in `router/sandbox/sandbox.ts`: `topUpSandboxSession`, then `recordSandboxAwake` | -| `StopBox` / `StopRecord` | `stopAndSnapshot` or `stopSandbox`, then `recordSandboxAsleep` in `router/cloud-workspace/sandbox-sleep.ts` (most sites call it through `recordBoxStopped`, which first writes the boot's end reason, a write the model leaves out), from `cloudWorkspace.sleep` (`cloud-workspace.ts`), `restartCloudWorkspace` and `wakeAfterStop` (`wake.ts`; `environment.promote` calls `wakeAfterStop` after the snapshot in `promoteSandboxToEnvironment` stopped the source, so `StopBox` there is the snapshot), a refused `wakeCloudWorkspace` whose own call resumed the box (it records that session awake first, a `WakeRecord`, then stops it; a refusal for a box another wake resumed stops nothing), and the spend-cap stops in `meter.ts` (`settleBoot`, `settleOrganizationNow`, the refused adopt in `adoptRunningBoxes`) | +| `StopBox` / `StopRecord` | `stopAndSnapshot` or `stopSandbox`, then `recordSandboxAsleep` in `router/cloud-workspace/sandbox-sleep.ts` (most sites call it through `recordBoxStopped`, which first writes the boot's end reason, a write the model leaves out), from `cloudWorkspace.sleep` (`cloud-workspace.ts`), `restartCloudWorkspace` and `wakeAfterStop` (`wake.ts`; `environment.promote` calls `wakeAfterStop` after the snapshot in `promoteSandboxToEnvironment` stopped the source, so `StopBox` there is the snapshot), a create whose drive refused (`provision.ts`, after its transition to `awake`), a refused `wakeCloudWorkspace` whose own call resumed the box (it records that session awake first, a `WakeRecord`, then stops it; a refusal for a box another wake resumed stops nothing), and the spend-cap stops in `meter.ts` (`settleBoot`, `settleOrganizationNow`, the refused adopt in `adoptRunningBoxes`) | | `WatchFire` | `watchSandboxExpiry` in `sandbox-expiry.ts`, queued by `queueWatch` through `publishCloudWorkspaceJob` (`jobs.ts`), served by `apps/api/src/app/api/cloud-workspaces/sandbox-expiry/route.ts` | | `ReconcileList` / `ReconcileApply` | `reconcileCloudWorkspaces` in `sandbox-expiry.ts`: `listActiveWorkspaceSandboxes`, then `syncSandboxExpiries` against each listing's `currentSessionId`; a stopped listing goes through `recordStopSeen` in `sandbox-sleep.ts` (`recordSandboxAsleep`, with the end reason first when the recorded expiry has passed) | | `Archive` | `cloudWorkspace.delete` in `cloud-workspace.ts`: the transition to `archived`, then `queueReap` | | `Reap` | the stop stage of `reapArchivedCloudWorkspace` in `reap.ts`, and the `queueReap` fallback that stops at once when the stop cannot be queued. Both call `recordSandboxAsleep`, which on an archived row writes only the expiry and session id | -| `Tick` / `Crash` | Vercel ends a session at its timeout, or a session fails on its own | +| `Tick` / `Crash` | Vercel ends a session at its timeout, or a session fails on its own. A wake that stops a box whose drive refused (`wakeSandbox`) writes no status, so the row sees it as a crash | | `CAS` | `stillStoppedSession` in `sandbox-sleep.ts` (asleep writes) and the new-session rule in `markAwake` (awake writes) | When you add, remove or reorder a writer, change its action here and in `Asleep.tla`, run both diff --git a/.github/workflows/sandbox.yml b/.github/workflows/sandbox.yml index 31d4d56969c..ad0708d52c9 100644 --- a/.github/workflows/sandbox.yml +++ b/.github/workflows/sandbox.yml @@ -24,7 +24,24 @@ on: - real-sandbox - production-release - inventory + - drive-sweep default: real-sandbox + real_sandbox_drive: + description: "real-sandbox only: also run the drive-mode pass (a throwaway golden drive, a box on a fork of it, a stop and a wake)" + type: boolean + default: false + real_sandbox_keep: + description: "real-sandbox only: keep the boxes (stopped) and the box drive for inspection instead of deleting them" + type: boolean + default: false + drive_sweep_delete: + description: "drive-sweep only: delete the orphaned drives it lists (off: a dry run)" + type: boolean + default: false + real_sandbox_work_gb: + description: "real-sandbox drive pass only: GB written under /workspace on the drive box and the plain box" + type: string + default: "4" jobs: boot-twice: @@ -119,7 +136,7 @@ jobs: name: Real sandbox if: github.event_name == 'workflow_dispatch' && inputs.run == 'real-sandbox' runs-on: ubuntu-latest - timeout-minutes: 30 + timeout-minutes: 90 steps: - name: Check out code uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 @@ -146,6 +163,9 @@ jobs: CDN_R2_ENDPOINT: ${{ secrets.CDN_R2_ENDPOINT }} CDN_R2_BUCKET: ${{ secrets.CDN_R2_BUCKET }} CDN_URL: ${{ secrets.CDN_URL }} + REAL_SANDBOX_DRIVE: ${{ inputs.real_sandbox_drive && '1' || '' }} + REAL_SANDBOX_KEEP: ${{ inputs.real_sandbox_keep && '1' || '' }} + REAL_SANDBOX_WORK_GB: ${{ inputs.real_sandbox_work_gb }} run: bun run src/real-sandbox.ts inventory: @@ -173,6 +193,34 @@ jobs: VERCEL_SANDBOX_PROJECT_ID: ${{ secrets.VERCEL_SANDBOX_PROJECT_ID }} run: bun run inventory + drive-sweep: + name: Drive sweep + if: github.event_name == 'workflow_dispatch' && inputs.run == 'drive-sweep' + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Setup Bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 + with: + bun-version-file: .bun-version + + - name: Install dependencies + run: bun install --frozen --ignore-scripts + + - name: List, or delete, drives nothing names + working-directory: packages/sandbox + env: + NEON_API_KEY: ${{ secrets.NEON_API_KEY }} + NEON_PROJECT_ID: ${{ secrets.NEON_PROJECT_ID }} + VERCEL_SANDBOX_TOKEN: ${{ secrets.VERCEL_SANDBOX_TOKEN }} + VERCEL_SANDBOX_TEAM_ID: ${{ secrets.VERCEL_SANDBOX_TEAM_ID }} + VERCEL_SANDBOX_PROJECT_ID: ${{ secrets.VERCEL_SANDBOX_PROJECT_ID }} + DRIVE_SWEEP_DELETE: ${{ inputs.drive_sweep_delete && '--delete' || '' }} + run: bun run drive-sweep --production $DRIVE_SWEEP_DELETE + production-release: name: Production release # A merge that touches the image or the bundle releases it, and the nightly diff --git a/apps/api/src/app/api/cloud-workspaces/provision/route.ts b/apps/api/src/app/api/cloud-workspaces/provision/route.ts index afefc8333f3..3175e8ae40e 100644 --- a/apps/api/src/app/api/cloud-workspaces/provision/route.ts +++ b/apps/api/src/app/api/cloud-workspaces/provision/route.ts @@ -18,6 +18,7 @@ const payloadSchema = z /** Absent when the user typed a name, which the row already holds. */ namingPrompt: z.string().max(CLOUD_AGENT_PROMPT_MAX_LENGTH).optional(), launch: cloudAgentLaunchSchema.optional(), + lockRetries: z.number().int().min(0).optional(), }) .strict(); diff --git a/apps/api/src/app/api/cloud-workspaces/teardown/route.ts b/apps/api/src/app/api/cloud-workspaces/teardown/route.ts new file mode 100644 index 00000000000..ead4e9cbc8f --- /dev/null +++ b/apps/api/src/app/api/cloud-workspaces/teardown/route.ts @@ -0,0 +1,34 @@ +import { teardownSandbox } from "@superset/trpc/cloud-workspace-teardown"; +import { z } from "zod"; +import { verifyQstashRequest } from "@/lib/verifyQstash"; + +export const dynamic = "force-dynamic"; +export const maxDuration = 60; + +const payloadSchema = z + .object({ + providerSandboxId: z.string().min(1), + driveName: z.string().min(1).nullable(), + attempt: z.number().int().min(0), + keepDrive: z.boolean().optional(), + }) + .strict(); + +/** Stops a box, then deletes it and its drive once the drive has detached; requeues itself until then. */ +export async function POST(request: Request): Promise { + const body = await request.text(); + const rejected = await verifyQstashRequest( + request, + body, + "/api/cloud-workspaces/teardown", + ); + if (rejected) return rejected; + + const parsed = payloadSchema.safeParse(JSON.parse(body)); + if (!parsed.success) { + console.error("[cloud-workspaces/teardown] invalid payload", parsed.error); + return Response.json({ error: "Invalid payload" }, { status: 400 }); + } + const outcome = await teardownSandbox(parsed.data); + return Response.json({ ok: true, outcome }); +} diff --git a/docs/cloud-sandbox-mismatches.md b/docs/cloud-sandbox-mismatches.md index 803c699ddca..6385138d963 100644 --- a/docs/cloud-sandbox-mismatches.md +++ b/docs/cloud-sandbox-mismatches.md @@ -699,6 +699,56 @@ socket, all describing processes that no longer exist. The boot runner deletes and recreates it first thing; anything that reads a flag there must tolerate a stale one for the milliseconds before that. +**A workspace drive is not a disk the box owns.** In drive mode +(`environments.golden_drive_name` set), `/workspace` and large home subtrees +live on a Vercel drive at `/mnt/superset-drive`, so they never enter the +box's snapshot. Three things a machine someone owns never makes you think +about: + +- *Bind mounts end with the session.* Vercel attaches the drive on every + session (the control plane passes `mounts` on every create and wake), but + the `mount --bind` of `/workspace` and each home subtree is gone after a + stop. `superset-boot` lays them out again on every boot, before + host-service and the checkout read `/workspace` (`drive.layout` in the + boot log). Anything that runs before that sees the root disk's empty + `/workspace`. +- *Hardlinks must stay inside `/workspace`.* Each bind mount is its own + `vfsmount`, and `link(2)` across two of them fails with `EXDEV` even on the + same filesystem. bun and pnpm then copy every package instead of linking + it. So their stores are inside the `/workspace` mount + (`BUN_INSTALL_CACHE_DIR=/workspace/.superset/stores/bun`, + `npm_config_store_dir=/workspace/.superset/stores/pnpm`, in host-service's + env and the login profile), not under `~/.bun` or `~/.cache`. +- *A drive needs detach before the next mount.* After a stop the drive stays + attached while the session flushes what it wrote (6 s after 1 GB, 44 s + after 12 GB), and a mount taken meanwhile reads a corrupt or empty + filesystem. `wakeSandbox` waits for `Drive.get(...).currentSandboxName` to + clear, at most 2 minutes, and fails the wake rather than mount. A sandbox + deleted while it still holds its drive leaves the drive attached for + minutes, so every delete stops the box first (`deleteSandboxAndDrive`). + +A drive has no discard either: deleting a file on it does not lower its +billed size. +A drive that did not mount leaves the root disk's empty `/workspace` in its +place, which would look like the user's work is gone. So the identity file +names the drive (`SUPERSET_SANDBOX_DRIVE`), the drive names its owner +(`/mnt/superset-drive/state/workspace-id`), and `superset-boot` refuses to +continue (`drive.missing`, `drive.foreign`, `drive.unlaid`) before the bundle, +host-service and the checkout. The control plane spots the refusal in this +boot's lines of the boot log within about 2 s and answers "still starting"; +the next wake mounts the drive afresh, and a create records the box asleep. +A refusal no remount can change (another workspace's drive, a laid-out drive +that mounts empty, a boot script that predates the drive) ends after 3 boots +in a row as `failed`, with the box and the drive kept. A boot that answers +without `drive.layout` is refused the same way. +A box on an image whose runner predates the guard gets it after the reexec +into the bundle's runner, still before host-service. + +*The root disk and the drive can disagree.* A stop whose snapshot does not +land rolls the root back and not the drive. So in drive mode the checkout +markers and `host.db` live on the drive, and the checkout never replaces a +non-empty checkout there. + **The control plane's command runs from the image's workdir, which the checkout replaces.** The first boot cloned `/workspace` by deleting and recreating it, and every child of the boot command — host-service, git, @@ -1076,3 +1126,12 @@ doesn't re-spend a session on it; the next thing to try is attaching to the `win.setSize()` directly rather than simulating it from outside, or watching `app.on("child-process-gone")` / `render-process-gone` across a CDP-reload loop to catch the actual GPU-process death this is presumably downstream of. + +**`sudo` cannot resolve the box's hostname.** A machine someone owns has its +hostname in `/etc/hosts`; a Vercel box does not, so every `sudo` (including +`runCommand({ sudo: true })`) writes `sudo: unable to resolve host : +Name or service not known` to stderr. The command still runs. A check that +reads `stdout + stderr` then sees an extra line and fails a comparison that +was correct; the first real drive check failed 7 expectations that way. +**What we did:** the real-sandbox drive pass compares stdout only and shows +stderr as detail. **Open:** the hostname is still missing from `/etc/hosts`. diff --git a/packages/i18n/locales/cs/messages.po b/packages/i18n/locales/cs/messages.po index 09933a46950..372594aaa63 100644 --- a/packages/i18n/locales/cs/messages.po +++ b/packages/i18n/locales/cs/messages.po @@ -22633,6 +22633,9 @@ msgstr "Pracovní prostor vytvořen" msgid "Workspace details" msgstr "Podrobnosti pracovního prostoru" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Disky pracovních prostorů jsou pro tuto organizaci vypnuté, takže pracovní prostor, jehož soubory jsou na disku, se nemůže stát prostředím." + msgid "Workspace ID copied" msgstr "ID pracovního prostoru zkopírováno" diff --git a/packages/i18n/locales/de/messages.po b/packages/i18n/locales/de/messages.po index 6a48fd54ac7..6143d0ca70f 100644 --- a/packages/i18n/locales/de/messages.po +++ b/packages/i18n/locales/de/messages.po @@ -22633,6 +22633,9 @@ msgstr "Arbeitsbereich erstellt" msgid "Workspace details" msgstr "Arbeitsbereichsdetails" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Arbeitsbereich-Laufwerke sind für diese Organisation ausgeschaltet, daher kann ein Arbeitsbereich, dessen Dateien auf einem Laufwerk liegen, nicht zu einer Umgebung werden." + msgid "Workspace ID copied" msgstr "Arbeitsbereich-ID kopiert" diff --git a/packages/i18n/locales/en/messages.po b/packages/i18n/locales/en/messages.po index 22835bd432c..496dbf02e99 100644 --- a/packages/i18n/locales/en/messages.po +++ b/packages/i18n/locales/en/messages.po @@ -22633,6 +22633,9 @@ msgstr "Workspace created" msgid "Workspace details" msgstr "Workspace details" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." + msgid "Workspace ID copied" msgstr "Workspace ID copied" diff --git a/packages/i18n/locales/es/messages.po b/packages/i18n/locales/es/messages.po index 4bb6cf105a8..1c3ab41db32 100644 --- a/packages/i18n/locales/es/messages.po +++ b/packages/i18n/locales/es/messages.po @@ -22633,6 +22633,9 @@ msgstr "Espacio de trabajo creado" msgid "Workspace details" msgstr "Detalles del espacio de trabajo" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Las unidades de espacio de trabajo están desactivadas para esta organización, así que un espacio de trabajo cuyos archivos están en una unidad no puede convertirse en un entorno." + msgid "Workspace ID copied" msgstr "ID del espacio de trabajo copiado" diff --git a/packages/i18n/locales/fr/messages.po b/packages/i18n/locales/fr/messages.po index 1f42b4bba60..19ca75f319e 100644 --- a/packages/i18n/locales/fr/messages.po +++ b/packages/i18n/locales/fr/messages.po @@ -22633,6 +22633,9 @@ msgstr "Espace de travail créé" msgid "Workspace details" msgstr "Détails de l'espace de travail" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Les disques d'espace de travail sont désactivés pour cette organisation : un espace de travail dont les fichiers sont sur un disque ne peut pas devenir un environnement." + msgid "Workspace ID copied" msgstr "ID de l'espace de travail copié" diff --git a/packages/i18n/locales/id/messages.po b/packages/i18n/locales/id/messages.po index 82c2d66b65d..e3638f0c5bd 100644 --- a/packages/i18n/locales/id/messages.po +++ b/packages/i18n/locales/id/messages.po @@ -22633,6 +22633,9 @@ msgstr "Workspace dibuat" msgid "Workspace details" msgstr "Detail workspace" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Drive workspace dinonaktifkan untuk organisasi ini, jadi workspace yang file-nya ada di drive tidak bisa menjadi lingkungan." + msgid "Workspace ID copied" msgstr "ID workspace disalin" diff --git a/packages/i18n/locales/it/messages.po b/packages/i18n/locales/it/messages.po index 629470d66f5..1e85d757a96 100644 --- a/packages/i18n/locales/it/messages.po +++ b/packages/i18n/locales/it/messages.po @@ -22633,6 +22633,9 @@ msgstr "Workspace creato" msgid "Workspace details" msgstr "Dettagli del workspace" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "I dischi dei workspace sono disattivati per questa organizzazione, quindi un workspace i cui file si trovano su un disco non può diventare un ambiente." + msgid "Workspace ID copied" msgstr "ID del workspace copiato" diff --git a/packages/i18n/locales/ja/messages.po b/packages/i18n/locales/ja/messages.po index 4b765ee5866..efe1e6e7083 100644 --- a/packages/i18n/locales/ja/messages.po +++ b/packages/i18n/locales/ja/messages.po @@ -22633,6 +22633,9 @@ msgstr "ワークスペースを作成しました" msgid "Workspace details" msgstr "ワークスペースの詳細" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "この組織ではワークスペースのドライブが無効になっているため、ファイルがドライブ上にあるワークスペースは環境にできません。" + msgid "Workspace ID copied" msgstr "ワークスペースIDをコピーしました" diff --git a/packages/i18n/locales/ko/messages.po b/packages/i18n/locales/ko/messages.po index 062b60ed8a5..2cf53230988 100644 --- a/packages/i18n/locales/ko/messages.po +++ b/packages/i18n/locales/ko/messages.po @@ -22633,6 +22633,9 @@ msgstr "워크스페이스를 만들었습니다" msgid "Workspace details" msgstr "워크스페이스 세부 정보" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "이 조직에서는 워크스페이스 드라이브가 꺼져 있어, 파일이 드라이브에 있는 워크스페이스는 환경이 될 수 없습니다." + msgid "Workspace ID copied" msgstr "워크스페이스 ID 복사됨" diff --git a/packages/i18n/locales/nl/messages.po b/packages/i18n/locales/nl/messages.po index e551554bd86..ae8abdb4d9e 100644 --- a/packages/i18n/locales/nl/messages.po +++ b/packages/i18n/locales/nl/messages.po @@ -22633,6 +22633,9 @@ msgstr "Werkruimte aangemaakt" msgid "Workspace details" msgstr "Werkruimtedetails" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Werkruimteschijven zijn uitgeschakeld voor deze organisatie, dus een werkruimte waarvan de bestanden op een schijf staan, kan geen omgeving worden." + msgid "Workspace ID copied" msgstr "Werkruimte-ID gekopieerd" diff --git a/packages/i18n/locales/pl/messages.po b/packages/i18n/locales/pl/messages.po index 411313bc257..581122cab21 100644 --- a/packages/i18n/locales/pl/messages.po +++ b/packages/i18n/locales/pl/messages.po @@ -22633,6 +22633,9 @@ msgstr "Obszar roboczy utworzony" msgid "Workspace details" msgstr "Szczegóły obszaru roboczego" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Dyski obszarów roboczych są wyłączone w tej organizacji, więc obszar roboczy, którego pliki są na dysku, nie może stać się środowiskiem." + msgid "Workspace ID copied" msgstr "Skopiowano ID obszaru roboczego" diff --git a/packages/i18n/locales/pt-BR/messages.po b/packages/i18n/locales/pt-BR/messages.po index 80174655017..28cad998528 100644 --- a/packages/i18n/locales/pt-BR/messages.po +++ b/packages/i18n/locales/pt-BR/messages.po @@ -22633,6 +22633,9 @@ msgstr "Área de trabalho criada" msgid "Workspace details" msgstr "Detalhes da área de trabalho" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Os discos de área de trabalho estão desativados para esta organização, então uma área de trabalho cujos arquivos estão em um disco não pode virar um ambiente." + msgid "Workspace ID copied" msgstr "ID da área de trabalho copiado" diff --git a/packages/i18n/locales/ru/messages.po b/packages/i18n/locales/ru/messages.po index 814cd29b4b7..b489ee5c6ac 100644 --- a/packages/i18n/locales/ru/messages.po +++ b/packages/i18n/locales/ru/messages.po @@ -22633,6 +22633,9 @@ msgstr "Рабочая область создана" msgid "Workspace details" msgstr "Сведения о рабочей области" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Диски рабочих областей отключены для этой организации, поэтому рабочая область, файлы которой лежат на диске, не может стать окружением." + msgid "Workspace ID copied" msgstr "ID рабочей области скопирован" diff --git a/packages/i18n/locales/tr/messages.po b/packages/i18n/locales/tr/messages.po index b8e7b7af79a..632c9ba7412 100644 --- a/packages/i18n/locales/tr/messages.po +++ b/packages/i18n/locales/tr/messages.po @@ -22633,6 +22633,9 @@ msgstr "Çalışma alanı oluşturuldu" msgid "Workspace details" msgstr "Çalışma alanı ayrıntıları" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Bu kuruluş için çalışma alanı sürücüleri kapalı; bu yüzden dosyaları bir sürücüde olan bir çalışma alanı ortama dönüştürülemez." + msgid "Workspace ID copied" msgstr "Çalışma alanı kimliği kopyalandı" diff --git a/packages/i18n/locales/uk/messages.po b/packages/i18n/locales/uk/messages.po index e0cd327ba64..d00c564ad93 100644 --- a/packages/i18n/locales/uk/messages.po +++ b/packages/i18n/locales/uk/messages.po @@ -22627,6 +22627,9 @@ msgstr "Робочий простір створено" msgid "Workspace details" msgstr "Відомості про робочий простір" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Диски робочих просторів вимкнено для цієї організації, тому робочий простір, файли якого на диску, не може стати середовищем." + msgid "Workspace ID copied" msgstr "ID робочого простору скопійовано" diff --git a/packages/i18n/locales/vi/messages.po b/packages/i18n/locales/vi/messages.po index cd39fe440dd..08ecef9adb0 100644 --- a/packages/i18n/locales/vi/messages.po +++ b/packages/i18n/locales/vi/messages.po @@ -22633,6 +22633,9 @@ msgstr "Đã tạo workspace" msgid "Workspace details" msgstr "Chi tiết workspace" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "Ổ đĩa workspace đã bị tắt cho tổ chức này, nên một workspace có tệp nằm trên ổ đĩa không thể trở thành môi trường." + msgid "Workspace ID copied" msgstr "Đã sao chép ID workspace" diff --git a/packages/i18n/locales/zh-CN/messages.po b/packages/i18n/locales/zh-CN/messages.po index ff17e6a7592..58d4baf1e82 100644 --- a/packages/i18n/locales/zh-CN/messages.po +++ b/packages/i18n/locales/zh-CN/messages.po @@ -22633,6 +22633,9 @@ msgstr "工作区已创建" msgid "Workspace details" msgstr "工作区详情" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "此组织已关闭工作区驱动器,因此文件位于驱动器上的工作区无法成为环境。" + msgid "Workspace ID copied" msgstr "已复制工作区ID" diff --git a/packages/i18n/locales/zh-TW/messages.po b/packages/i18n/locales/zh-TW/messages.po index b750ae9f547..e8db5cba7d3 100644 --- a/packages/i18n/locales/zh-TW/messages.po +++ b/packages/i18n/locales/zh-TW/messages.po @@ -22633,6 +22633,9 @@ msgstr "工作區已建立" msgid "Workspace details" msgstr "工作區詳細資料" +msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." +msgstr "此組織已關閉工作區磁碟,因此檔案位於磁碟上的工作區無法成為環境。" + msgid "Workspace ID copied" msgstr "已複製工作區ID" diff --git a/packages/i18n/src/server-errors.ts b/packages/i18n/src/server-errors.ts index 2ef4684e458..81cc3b2e08e 100644 --- a/packages/i18n/src/server-errors.ts +++ b/packages/i18n/src/server-errors.ts @@ -508,6 +508,13 @@ export const serverErrorMessages: Record< "This environment's repositories are fixed; promote a workspace again to change them", }), ), + "serverError.environment.drivesOff": () => + i18n._( + msg({ + message: + "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment.", + }), + ), "serverError.environment.couldNotRecord": () => i18n._( msg({ diff --git a/packages/sandbox/README.md b/packages/sandbox/README.md index 7785deac5fb..1d64303fa9a 100644 --- a/packages/sandbox/README.md +++ b/packages/sandbox/README.md @@ -53,7 +53,7 @@ The rule of thumb: if you can answer "where is it on the box", put it there. | `bun run build` | Hashes `bundle/`, renders `contract.sh` from `@superset/shared/sandbox/contract`, derives the step versions, writes `dist/bundle//` and the tarball. Deterministic: the same tree is the same sha on any machine. | | `bun run build --publish` | Also uploads the bundle and any asset the bucket lacks, from the local cache or else its row's `source`, checked against the row's sha256. Refuses to publish a bundle that references an asset the bucket does not have. | | `bun run image` | Builds the image: OS, apt lists, vendor tools, the `ubuntu` user, then installs the bundle at its sha and runs every pass once, so a fresh box finds every hash sidecar and step marker matching. A push logs Docker into the registry itself with a credential minted from `VERCEL_SANDBOX_TOKEN`, so there is no login to renew. `--local` builds `superset-sandbox:local` for the boot-twice test; `--dry` prints the Dockerfile. | -| boot (`superset-boot`) | On the box, as root, run by the control plane on every create and wake, after it has written `sandbox.conf`, with the host secret in its env: clears `/run/superset`, installs the pinned bundle if the hash differs, runs `apply-rootfs` and the steps marked `beforeHost` in `steps.json` (each a hash compare), starts host-service as `ubuntu` with the secret from the boot env, writes `resources.env`, starts `superset-memguard` (ranks every process for the OOM killer and reaps the largest before the kernel has to). When `sandbox.conf` has `SUPERSET_SANDBOX_BOX_LOGS=1`, it also starts `superset-logship`, which sends the boot, memguard, desktop and host-service logs to Sentry Logs from a byte offset saved on disk. Then, detached: `sync-assets` and the other steps, then the desktop, with the checkout beside them. `checkout.ready` waits for both, and the stamp `steps.end` says the steps are done. Once host-service answers, the environment has been pushed and the checkout is ready, the runner asks host-service to run the repo's `start` hook with that environment. | +| boot (`superset-boot`) | On the box, as root, run by the control plane on every create and wake, after it has written `sandbox.conf`, with the host secret in its env: clears `/run/superset`, lays out the workspace drive when one is mounted at `/mnt/superset-drive` (`plans/20261116-workspace-drive.md`), installs the pinned bundle if the hash differs, runs `apply-rootfs` and the steps marked `beforeHost` in `steps.json` (each a hash compare), starts host-service as `ubuntu` with the secret from the boot env, writes `resources.env`, starts `superset-memguard` (ranks every process for the OOM killer and reaps the largest before the kernel has to). When `sandbox.conf` has `SUPERSET_SANDBOX_BOX_LOGS=1`, it also starts `superset-logship`, which sends the boot, memguard, desktop and host-service logs to Sentry Logs from a byte offset saved on disk. Then, detached: `sync-assets` and the other steps, then the desktop, with the checkout beside them. `checkout.ready` waits for both, and the stamp `steps.end` says the steps are done. Once host-service answers, the environment has been pushed and the checkout is ready, the runner asks host-service to run the repo's `start` hook with that environment. | | `bun run release` | Rewrites the host-service and CLI assets from this checkout and the claude, codex, Chrome, opencode and pi rows to their latest upstream release, publishes the bundle, pushes the image under a candidate tag `release-` (unless `--skip-image`), builds the internal golden from it with the environment's `setup` hook, probes a fork of it (`src/environments/probe.ts`) including a stop and wake, and only after the probe passes moves `latest` to the candidate and writes the environment rows. Every image environment resolves `superset-hostsvc` to `latest`, so a failed release changes no box. | | `bun run agent-tools --bundle ` | Writes that bundle's agent CLIs to `agent-tools--`, a drive boxes mount read-only at `/opt/agents` when their organization has the `sandbox-agent-tools-drive` flag. A drive is written once and never again; it starts as a fork of the region's newest earlier drive, so only changed agents are fetched, and an agent that fails its download or `--version` check keeps the earlier drive's version or is left out. The drive carries claude, codex and pdev (also in the image) and opencode and pi (only on the drive). `bun run release` does this for every region, six at a time, before it records the bundle. It fails when a region with an environment has no drive, or a drive without claude, codex or pdev. Afterwards a `--production` release deletes the drives that neither an environment row (archived ones too) nor one of the 3 newest released bundles names; one Vercel project holds every database's drives, so no other release deletes any. | | `bun run src/real-sandbox.ts` | The Sandbox workflow dispatched with `real-sandbox`: one real sandbox from the registry image on this checkout's bundle and its agents drive in the default region, the same probe, a stop and a wake. | diff --git a/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh b/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh index 6eae7dfec23..32c4b127ad6 100644 --- a/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh +++ b/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh @@ -9,6 +9,8 @@ export DISPLAY="${DISPLAY:-${SUPERSET_DISPLAY:-:1}}" [ -r "${SUPERSET_RUN_DIR:-/run/superset}/desktop.env" ] && . "${SUPERSET_RUN_DIR:-/run/superset}/desktop.env" # The box's memory ceiling and CPUs, written by superset-boot from the cgroup. [ -r "${SUPERSET_RUN_DIR:-/run/superset}/resources.env" ] && . "${SUPERSET_RUN_DIR:-/run/superset}/resources.env" +# The package stores on the workspace drive, written by superset-boot in drive mode. +[ -r "${SUPERSET_RUN_DIR:-/run/superset}/drive.env" ] && . "${SUPERSET_RUN_DIR:-/run/superset}/drive.env" case ":$PATH:" in *":/usr/local/go/bin:"*) ;; *) export PATH="$PATH:/usr/local/go/bin" ;; esac # The agent CLIs are pinned by the bundle (install-agent-clis) or run from the # read-only agents drive; a self-update would drift from the pin and only diff --git a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot index 3077db2b102..eee15fe1aa5 100755 --- a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot +++ b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot @@ -6,6 +6,8 @@ # # clear /run/superset facts about a boot that is over # read the contract and identity /etc/superset/{contract.sh,sandbox.conf} +# lay out the workspace drive only when the control plane mounted one; +# refuse when sandbox.conf names a drive that is not mounted # install the pinned bundle only when its hash differs from current # apply-rootfs, and the steps each a hash compare per file and step; # marked beforeHost the only steps host-service waits for @@ -18,6 +20,9 @@ # the log on its health route so a boot is measurable from outside. set -uo pipefail . /etc/superset/contract.sh +# superset-boot drive-contract: 1 +# A boot that has laid a drive out re-executes only into a script with the +# line above; one without it predates the drive and would clone over it. # The control plane runs this from the image's workdir, which the checkout # below may replace; every child would inherit a deleted cwd. @@ -51,7 +56,7 @@ if [ -f "${SUPERSET_RUN_DIR}/host-service.pid" ] && kill -0 "$(cat "${SUPERSET_R exit 0 fi -stamp "boot.start pid=$$ bundle=$(cat "${SUPERSET_BUNDLE_ROOT}/current.bundle-hash" 2>/dev/null | cut -c1-12)" +stamp "boot.start pid=$$ bundle=$(cat "${SUPERSET_BUNDLE_ROOT}/current.bundle-hash" 2>/dev/null | cut -c1-12) boot=${SUPERSET_BOOT_ID:-}" # --- per-boot state: a restored disk carries the previous session's pids # and ready flags, which describe processes that no longer exist. @@ -59,6 +64,82 @@ rm -rf "$SUPERSET_RUN_DIR" install -d -m 0755 -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$SUPERSET_RUN_DIR" stamp "run.cleared" +# --- the workspace drive, when the environment is in drive mode: /workspace +# and the large home state live on it, so they never enter the box's +# snapshot. Bind mounts end with the session, so every boot does this, before +# anything reads /workspace. A bind mount, not a symlink: paths must not move. +DRIVE_MODE="" +user_path() { + local dir="$1" part IFS=/ + for part in $2; do + dir="${dir}/${part}" + [ -d "$dir" ] || install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$dir" + done +} +bind_into() { + mountpoint -q "$2" || mount --bind "$1" "$2" || refuse "cannot bind ${1} onto ${2}" +} +# In drive mode, what describes the drive's contents lives on the drive. +use_drive_state() { + SUPERSET_HOST_DB="${SUPERSET_DRIVE_STATE_DIR}/host.db" + SUPERSET_CHECKOUTS_DIR="${SUPERSET_DRIVE_STATE_DIR}/checkouts" +} +# A subtree the drive lacks is seeded from what the image left there, under a +# temporary name, so a boot that dies halfway never leaves a partial copy. +seed_home_subtree() { + local sub="$1" target="${drive}/home/$1" tmp="${drive}/home/$1.seeding" + [ "$sub" = "${sub%/*}" ] || user_path "${drive}/home" "${sub%/*}" + rm -rf "$tmp" + install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$tmp" + if [ -d "${SUPERSET_USER_HOME}/${sub}" ]; then + cp -a "${SUPERSET_USER_HOME}/${sub}/." "${tmp}/" || refuse "cannot seed ${target}" + fi + mv -T "$tmp" "$target" || refuse "cannot seed ${target}" +} +layout_drive() { + local drive="${SUPERSET_DRIVE_DIR:-}" started sub owner sentinel + if [ -z "$drive" ] || ! mountpoint -q "$drive"; then + # Without its drive the box would serve the root disk's empty /workspace as the user's work. + [ -z "${SUPERSET_SANDBOX_DRIVE:-}" ] && return 0 + stamp "drive.missing ${SUPERSET_SANDBOX_DRIVE} is not mounted at ${drive:-the drive path}" + refuse "the workspace drive ${SUPERSET_SANDBOX_DRIVE} is not mounted" + fi + : "${SUPERSET_DRIVE_STATE_DIR:=${drive}/state}" + sentinel="${SUPERSET_DRIVE_STATE_DIR}/workspace-id" + owner="$(cat "$sentinel" 2>/dev/null || true)" + if [ -n "$owner" ] && [ "$owner" != "$SUPERSET_SANDBOX_WORKSPACE_ID" ]; then + stamp "drive.foreign ${SUPERSET_SANDBOX_DRIVE:-} belongs to workspace ${owner}" + refuse "the drive at ${drive} belongs to workspace ${owner}" + fi + if [ -z "$owner" ] && [ -n "${SUPERSET_SANDBOX_DRIVE_LAID_OUT:-}" ]; then + stamp "drive.unlaid ${SUPERSET_SANDBOX_DRIVE:-} carries no workspace id though it was laid out before" + refuse "the drive at ${drive} is not this workspace's laid-out drive" + fi + started="$(date +%s%3N)" + user_path "$drive" workspace + user_path "$drive" home + user_path "$drive" state + install -d -m 0710 "${drive}/docker" + install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$SUPERSET_WORKSPACE_DIR" + bind_into "${drive}/workspace" "$SUPERSET_WORKSPACE_DIR" + for sub in ${SUPERSET_DRIVE_HOME_SUBTREES:-}; do + [ -d "${drive}/home/${sub}" ] || seed_home_subtree "$sub" + user_path "$SUPERSET_USER_HOME" "$sub" + bind_into "${drive}/home/${sub}" "${SUPERSET_USER_HOME}/${sub}" + done + if [ -z "$owner" ]; then + # A fresh fork: its checkout is no one's work until the first checkout has run, in this boot or a later one. + : > "${SUPERSET_DRIVE_STATE_DIR}/fresh" && + printf '%s\n' "$SUPERSET_SANDBOX_WORKSPACE_ID" > "${sentinel}.tmp" && mv -T "${sentinel}.tmp" "$sentinel" && + sync "${SUPERSET_DRIVE_STATE_DIR}/fresh" "$sentinel" "$SUPERSET_DRIVE_STATE_DIR" || refuse "cannot write ${sentinel}" + fi + DRIVE_MODE=1 + use_drive_state + printf 'export BUN_INSTALL_CACHE_DIR=%s npm_config_store_dir=%s\n' "$SUPERSET_BUN_STORE" "$SUPERSET_PNPM_STORE" > "${SUPERSET_RUN_DIR}/drive.env" + stamp "drive.layout ms=$(( $(date +%s%3N) - started ))" +} +layout_drive + # --- the bundle the environment pins. A box with no pin would silently keep # whatever its image or golden baked, for as long as it lives. [ -n "${SUPERSET_BUNDLE_SHA:-}" ] || refuse "SUPERSET_BUNDLE_SHA missing from ${SUPERSET_CONF}; the control plane pins every box" @@ -107,17 +188,35 @@ BUNDLE="${SUPERSET_BUNDLE_ROOT}/current" stamp "rootfs.start"; "$BUNDLE/setup" apply-rootfs >> "$SUPERSET_BOOT_LOG" 2>&1; stamp "rootfs.end" # apply-rootfs may have replaced the contract; re-read it. . /etc/superset/contract.sh +[ -z "$DRIVE_MODE" ] || use_drive_state # It may also have replaced this script. The shell is running the copy the # image baked, so without this a bundle's change to the boot script only # takes effect on the boot after the one that installed it — which is how an # older image's checkout raced a newer bundle's. Once per boot: the guard # stops a bundle whose script never settles from spinning here. +if [ -n "$DRIVE_MODE" ] && ! grep -q '^# superset-boot drive-contract: [1-9]' "$0"; then + stamp "drive.downgrade the installed boot script predates the workspace drive" + refuse "the pinned bundle's boot script does not know the workspace drive" +fi if [ -z "${SUPERSET_BOOT_REEXEC:-}" ] && [ "$(self_hash)" != "$BOOT_HASH_BEFORE" ]; then stamp "boot.reexec into the installed bundle's script" SUPERSET_BOOT_REEXEC=1 exec "$0" "$@" fi stamp "steps.host.start"; "$BUNDLE/setup" run-steps before-host >> "$SUPERSET_BOOT_LOG" 2>&1; stamp "steps.host.end" +# In drive mode, Docker's images and layers go on the drive too. apply-rootfs +# compares only the hash sidecar, so this rewrite stays until the bundle +# ships a new daemon.json, and then it is merged again right here. +configure_docker() { + [ -n "$DRIVE_MODE" ] || return 0 + local want + want="$(jq -c --arg root "${SUPERSET_DRIVE_DIR}/docker" '. + {"data-root": $root}' "${BUNDLE}/rootfs/etc/docker/daemon.json")" || + { stamp "docker.config.failed"; return 0; } + [ "$(jq -c . /etc/docker/daemon.json 2>/dev/null)" = "$want" ] || printf '%s\n' "$want" > /etc/docker/daemon.json + stamp "docker.data-root ${SUPERSET_DRIVE_DIR}/docker" +} +configure_docker + # --- the agent CLIs from the region's agent-tools drive, when the control # plane mounted one. Without it the image's agents stay on PATH. Mounts are # per session, so a link into a mount this session lacks points nowhere. @@ -195,6 +294,7 @@ host_env=( "SUPERSET_RUN_DIR=${SUPERSET_RUN_DIR}" "SUPERSET_SANDBOX_BOOT_LOG=${SUPERSET_BOOT_LOG}" "HOST_SERVICE_SECRET=${HOST_SERVICE_SECRET}" ) +[ -n "$DRIVE_MODE" ] && host_env+=("BUN_INSTALL_CACHE_DIR=${SUPERSET_BUN_STORE}" "npm_config_store_dir=${SUPERSET_PNPM_STORE}") for key in HOST_SERVICE_SENTRY_DSN HOST_SERVICE_SENTRY_ENVIRONMENT SUPERSET_SANDBOX_CREATOR_USER_ID \ SUPERSET_SANDBOX_PLUGINS \ SUPERSET_SANDBOX_AGENT SUPERSET_SANDBOX_AGENT_PROMPT SUPERSET_SANDBOX_AGENT_MODEL SUPERSET_SANDBOX_AGENT_EFFORT SUPERSET_SANDBOX_AGENT_MODE SUPERSET_SANDBOX_AGENT_ATTACHMENTS SUPERSET_SANDBOX_AGENT_SURFACE SUPERSET_SANDBOX_AGENT_AUTOMATION_RUN_ID SUPERSET_SANDBOX_AGENT_TERMINAL_ID \ @@ -281,22 +381,64 @@ start_hook() { # path is ".". A marker per path under the state directory says it is done; # a golden has the markers stripped so a fork fetches its branch into the # checkout it inherited. +# A drive laid out in an earlier boot holds the user's work: nothing on it is replaced. +drive_has_work() { [ -n "$DRIVE_MODE" ] && [ ! -e "${SUPERSET_DRIVE_STATE_DIR}/fresh" ]; } +# Still exactly the golden's copy: HEAD at the commit the fresh fork had, nothing changed, and the +# branch absent or at that commit. Then no user work can be in it, and it may be fetched. +untouched_golden() { + local ws="$1" recorded="$2" branch="$3" head + [ -f "$recorded" ] || return 1 + head="$(cat "$recorded")" + [ "$(as_user git -C "$ws" rev-parse HEAD 2>/dev/null)" = "$head" ] || return 1 + [ -z "$(as_user git -C "$ws" status --porcelain 2>/dev/null)" ] || return 1 + case "$(as_user git -C "$ws" rev-parse -q --verify "refs/heads/${branch}" 2>/dev/null || echo none)" in + "$head"|none) return 0 ;; + *) return 1 ;; + esac +} checkout_one() { local url="$1" branch="$2" path="$3" base="$4" [ -n "$base" ] || base="$branch" local ws="${SUPERSET_WORKSPACE_DIR}/${path}" marker="${SUPERSET_CHECKOUTS_DIR}/${path}" if [ "$path" = "." ]; then ws="$SUPERSET_WORKSPACE_DIR"; marker="${SUPERSET_CHECKOUTS_DIR}/@root"; fi if [ -f "$marker" ]; then stamp "checkout.skipped ${path} already done"; return 0; fi - local baked + local baked recorded untouched="" + recorded="${SUPERSET_DRIVE_STATE_DIR:-}/golden-head/${path//\//%}" + [ "$path" = "." ] && recorded="${SUPERSET_DRIVE_STATE_DIR:-}/golden-head/@root" baked="$(as_user git -C "$ws" remote get-url origin 2>/dev/null || true)" if [ "$baked" = "$url" ] && [ -d "$ws/.git" ]; then - if as_user git -C "$ws" fetch -q --depth 1 origin "$base" && as_user git -C "$ws" checkout -q -B "$branch" FETCH_HEAD; then - touch "$marker"; stamp "checkout.fetched ${path} ${branch} from ${base}"; return 0 + if [ -n "$DRIVE_MODE" ] && ! drive_has_work && [ ! -f "$recorded" ]; then + install -d "${recorded%/*}" + as_user git -C "$ws" rev-parse HEAD > "$recorded" 2>/dev/null && sync "$recorded" || rm -f "$recorded" + fi + drive_has_work && untouched_golden "$ws" "$recorded" "$branch" && untouched=1 + # On a drive laid out before, a checkout that already has the branch is the user's work. + if drive_has_work && [ -z "$untouched" ] && as_user git -C "$ws" rev-parse -q --verify "refs/heads/${branch}" > /dev/null; then + touch "$marker"; stamp "checkout.kept ${path} ${branch} is already on the drive"; return 0 + fi + # A fresh fork tries a few times, then keeps the golden's copy: never a reclone over its dependencies. + local attempts=1 attempt + [ -n "$DRIVE_MODE" ] && ! drive_has_work && attempts=3 + for attempt in $(seq 1 "$attempts"); do + if as_user git -C "$ws" fetch -q --depth 1 origin "$base" && as_user git -C "$ws" checkout -q -B "$branch" FETCH_HEAD; then + touch "$marker"; rm -f "$recorded" + stamp "checkout.fetched ${path} ${branch} from ${base}${untouched:+ (an untouched golden copy)}"; return 0 + fi + [ "$attempt" -lt "$attempts" ] && sleep $(( attempt * 2 )) + done + if [ "$attempts" -gt 1 ]; then + stamp "checkout.fresh.failed ${path} ${branch}: the golden's checkout is kept; a later boot fetches it while it is untouched"; return 0 + fi + if drive_has_work; then + stamp "checkout.kept ${path} fetch failed; the checkout on the drive is left as it is"; return 1 fi # A clone a previous session interrupted leaves a repository that can # fetch nothing; start over rather than fail every boot. stamp "checkout.fetch.failed ${path} ${branch}; recloning" fi + if drive_has_work && [ -n "$(find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' ! -name '.superset' -print -quit 2>/dev/null)" ]; then + stamp "checkout.kept ${path} ${ws} is not empty; nothing on the drive is replaced"; return 1 + fi # Clone beside the target and move the result in, the worktree first and # .git last, so the directory becomes a repository in one rename: host-service # seeds the workspace row before the checkout lands and would otherwise find @@ -309,10 +451,19 @@ checkout_one() { || { find "$tmp" -mindepth 1 -delete; as_user git clone -q --depth 1 "$url" "$tmp"; }; then as_user git -C "$tmp" checkout -q -B "$branch" install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$ws" - find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' -exec rm -rf {} + - ( shopt -s dotglob nullglob; for entry in "$tmp"/*; do [ "${entry##*/}" = ".git" ] || mv "$entry" "$ws"/; done ) + if [ -n "$DRIVE_MODE" ]; then + # A fresh fork's golden checkout goes; the package stores stay, and a repository's own .superset/ merges into them. + find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' ! -name '.superset' -exec rm -rf {} + + ( shopt -s dotglob nullglob; for entry in "$tmp"/*; do + [ "${entry##*/}" = ".git" ] && continue + if [ -d "${ws}/${entry##*/}" ]; then cp -a "$entry/." "${ws}/${entry##*/}/" && rm -rf "$entry"; else mv "$entry" "$ws"/; fi + done ) + else + find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' -exec rm -rf {} + + ( shopt -s dotglob nullglob; for entry in "$tmp"/*; do [ "${entry##*/}" = ".git" ] || mv "$entry" "$ws"/; done ) + fi mv "$tmp/.git" "$ws/.git"; rmdir "$tmp" - touch "$marker"; stamp "checkout.cloned ${path} ${url}"; return 0 + touch "$marker"; rm -f "$recorded"; stamp "checkout.cloned ${path} ${url}"; return 0 fi rm -rf "$tmp"; stamp "checkout.clone.failed ${path}"; return 1 } @@ -325,8 +476,22 @@ checkout() { [ -n "$path" ] || continue checkout_one "$url" "$branch" "$path" "$base" || failed=1 done < <(printf '%s' "$SUPERSET_SANDBOX_REPOSITORIES" | jq -r '.[] | [.url, .branch, .path, (.baseBranch // "")] | @tsv') + # The box was usable once this checkout ran, failed or not: from now on what is on the drive may be the user's. + if [ -n "$DRIVE_MODE" ] && [ -e "${SUPERSET_DRIVE_STATE_DIR}/fresh" ]; then + rm -f "${SUPERSET_DRIVE_STATE_DIR}/fresh" && sync "$SUPERSET_DRIVE_STATE_DIR" && stamp "checkout.fresh.done" + fi + exclude_stores return "$failed" } +# The package stores sit inside a root checkout in drive mode; git must not +# see them. Only stores/: a repository may track a .superset/ of its own. +exclude_stores() { + local exclude="${SUPERSET_WORKSPACE_DIR}/.git/info/exclude" + [ -n "$DRIVE_MODE" ] && [ -d "${SUPERSET_WORKSPACE_DIR}/.git" ] || return 0 + grep -qxF '/.superset/stores/' "$exclude" 2>/dev/null && return 0 + as_user mkdir -p "${exclude%/*}" + printf '/.superset/stores/\n' | as_user tee -a "$exclude" > /dev/null +} # --- the rest of the passes, then the desktop, beside the checkout. # checkout.ready waits for the steps too: the first agent launch, the start diff --git a/packages/sandbox/package.json b/packages/sandbox/package.json index e00d676269f..7fea8d12a3b 100644 --- a/packages/sandbox/package.json +++ b/packages/sandbox/package.json @@ -9,6 +9,7 @@ "pin": "bun run src/pin.ts", "assets": "bun run src/assets/produce.ts", "inventory": "bun run src/inventory.ts", + "drive-sweep": "bun run src/drive-sweep.ts", "release": "bun run src/environments/release.ts", "agent-tools": "bun run src/agent-tools.ts", "record-bundle": "bun run src/environments/record-bundle.ts", diff --git a/packages/sandbox/src/boot-twice.ts b/packages/sandbox/src/boot-twice.ts index 64b37c49a05..b0b73fa2435 100644 --- a/packages/sandbox/src/boot-twice.ts +++ b/packages/sandbox/src/boot-twice.ts @@ -9,6 +9,9 @@ * * bun run src/boot-twice.ts expects superset-sandbox:local * SANDBOX_IMAGE=... bun run src/boot-twice.ts + * BOOT_TWICE_CA_FILE=/etc/ssl/certs/ca-certificates.crt bun run src/boot-twice.ts + * in a cloud workspace, whose egress firewall re-signs TLS: without + * its CA the checkout cannot reach GitHub * * Exits non-zero on any failed expectation, with the boot log printed. */ @@ -17,14 +20,24 @@ import { readFileSync } from "node:fs"; import { join } from "node:path"; import { renderSandboxConf, + SANDBOX_DRIVE_HOME_SUBTREES, + SANDBOX_DRIVE_STORES, SANDBOX_PATHS, SANDBOX_PORTS, + type SandboxIdentity, } from "@superset/shared/sandbox/contract"; import { BUNDLE_SRC, PACKAGE_ROOT } from "./build"; import { discardingSteps, loadAssets, loadSteps } from "./manifest"; const IMAGE = process.env.SANDBOX_IMAGE ?? "superset-sandbox:local"; const NAME = `superset-boot-twice-${Date.now().toString(36)}`; +const DRIVE_NAME = `${NAME}-drive`; +const CA_MOUNT = process.env.BOOT_TWICE_CA_FILE + ? [ + "-v", + `${process.env.BOOT_TWICE_CA_FILE}:/etc/ssl/certs/ca-certificates.crt:ro`, + ] + : []; const bundleSha = readFileSync( join(PACKAGE_ROOT, "dist", "bundle.sha256"), "utf8", @@ -47,9 +60,19 @@ function docker( } return out; } -const exec = (cmd: string, opts: { user?: string; check?: boolean } = {}) => +const exec = ( + cmd: string, + opts: { user?: string; check?: boolean; on?: string } = {}, +) => docker( - ["exec", ...(opts.user ? ["-u", opts.user] : []), NAME, "bash", "-lc", cmd], + [ + "exec", + ...(opts.user ? ["-u", opts.user] : []), + opts.on ?? NAME, + "bash", + "-lc", + cmd, + ], { check: opts.check }, ); @@ -57,7 +80,7 @@ const exec = (cmd: string, opts: { user?: string; check?: boolean } = {}) => * The way the control plane starts boot: as the image user, through sudo * with the secret preserved from the command's env, never on the argv. */ -function bootAsUser(): string { +function bootAsUser(on = NAME): string { const proc = Bun.spawnSync( [ "docker", @@ -66,7 +89,7 @@ function bootAsUser(): string { "ubuntu", "-e", "HOST_SERVICE_SECRET=boot-twice-secret", - NAME, + on, "sudo", "--preserve-env=HOST_SERVICE_SECRET", "/usr/local/bin/superset-boot", @@ -87,7 +110,7 @@ function expect(label: string, ok: boolean, detail = ""): void { if (!ok) failures.push(label); } -const identity = renderSandboxConf({ +const identityFields: SandboxIdentity = { SUPERSET_SANDBOX_CONTRACT: "1", SUPERSET_BUNDLE_SHA: bundleSha, SUPERSET_API_URL: "https://api.example.invalid", @@ -116,9 +139,561 @@ const identity = renderSandboxConf({ enabled: true, }, ]), -}); +}; +const identity = renderSandboxConf(identityFields); -docker(["rm", "-f", NAME], { check: false }); +/** + * Drive mode, with a tmpfs standing in for the Vercel drive at its mount + * path: the layout, the store variables and Docker's data-root, then a second boot after the session's bind mounts are gone. + */ +function drivePass(): void { + const drive = SANDBOX_PATHS.drive; + const on = DRIVE_NAME; + docker([ + "run", + "-d", + "--name", + on, + "--platform", + "linux/amd64", + "--privileged", + "--user", + "root", + ...CA_MOUNT, + "--tmpfs", + `${drive}:exec,mode=0755`, + IMAGE, + "sleep", + "infinity", + ]); + const driveIdentity: SandboxIdentity = { + ...identityFields, + SUPERSET_SANDBOX_DRIVE: DRIVE_NAME, + }; + const writeConf = (fields: SandboxIdentity) => + docker(["exec", "-i", on, "bash", "-c", `cat > ${SANDBOX_PATHS.conf}`], { + stdin: renderSandboxConf(fields), + }); + writeConf(driveIdentity); + const bootDrive = () => { + bootAsUser(on); + exec( + `for i in $(seq 1 600); do [ -f ${SANDBOX_PATHS.run}/host-service.ready ] && exit 0; sleep 0.1; done; exit 1`, + { check: false, on }, + ); + }; + const homeMounts = SANDBOX_DRIVE_HOME_SUBTREES.map( + (sub) => `${SANDBOX_PATHS.home}/${sub}`, + ); + const allMounted = () => + exec( + [SANDBOX_PATHS.workspace, ...homeMounts] + .map((path) => `mountpoint -q ${path} || echo "not mounted ${path}"`) + .join("; "), + { check: false, on }, + ).trim(); + + bootDrive(); + const log = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive: layout stamped with its milliseconds", + /drive\.layout ms=\d+/.test(log), + log.match(/drive\.layout.*/)?.[0], + ); + expect( + "drive: /workspace and every home subtree are bind mounts", + allMounted() === "", + allMounted(), + ); + expect( + "drive: /workspace is backed by the drive", + /\/workspace/.test( + exec(`findmnt -no FSROOT ${SANDBOX_PATHS.workspace}`, { + check: false, + on, + }), + ), + exec(`findmnt -no SOURCE,FSROOT ${SANDBOX_PATHS.workspace}`, { + check: false, + on, + }).trim(), + ); + expect( + "drive: its directories belong to the sandbox user", + exec( + `stat -c %U ${drive}/workspace ${drive}/home ${drive}/home/.local ${SANDBOX_PATHS.home}/.local ${SANDBOX_PATHS.home}/.cargo`, + { check: false, on }, + ).trim() === Array(5).fill("ubuntu").join("\n"), + ); + const hostEnv = exec( + `tr '\\0' '\\n' < /proc/$(cat ${SANDBOX_PATHS.run}/host-service.pid)/environ`, + { check: false, on }, + ); + expect( + "drive: host-service gets the stores inside /workspace", + hostEnv.includes(`BUN_INSTALL_CACHE_DIR=${SANDBOX_DRIVE_STORES.bun}`) && + hostEnv.includes(`npm_config_store_dir=${SANDBOX_DRIVE_STORES.pnpm}`), + ); + expect( + "drive: a login shell gets them too", + exec("echo $BUN_INSTALL_CACHE_DIR", { + user: "ubuntu", + check: false, + on, + }).trim() === SANDBOX_DRIVE_STORES.bun, + ); + expect( + "drive: Docker's data-root is on the drive, storage driver kept", + /"data-root":"\/mnt\/superset-drive\/docker"/.test( + exec("jq -c . /etc/docker/daemon.json", { check: false, on }), + ) && + /fuse-overlayfs/.test( + exec("cat /etc/docker/daemon.json", { check: false, on }), + ), + exec("cat /etc/docker/daemon.json", { check: false, on }).trim(), + ); + const checkedOut = /ready/.test( + exec( + `for i in $(seq 1 1200); do [ -f ${SANDBOX_PATHS.run}/checkout.ready ] && echo ready && exit 0; sleep 0.1; done; echo missing`, + { check: false, on }, + ), + ); + expect("drive: checkout done", checkedOut); + expect( + "drive: the checkout landed on the drive", + /ok/.test( + exec(`test -d ${drive}/workspace/.git && echo ok`, { check: false, on }), + ), + ); + expect( + "drive: git ignores the stores of a root checkout", + /\/\.superset\/stores\//.test( + exec(`cat ${SANDBOX_PATHS.workspace}/.git/info/exclude`, { + check: false, + on, + }), + ), + ); + + // A new session: no processes and no bind mounts survive; the drive does. + const endSession = () => + exec( + `pkill -u ubuntu -TERM || true; pkill -x superset-memgua || true; sleep 1; pkill -u ubuntu -KILL || true; pkill -x Xvnc || true; rm -rf ${SANDBOX_PATHS.run}; for m in $(findmnt -rn -o TARGET | grep -E '^(${SANDBOX_PATHS.workspace}|${SANDBOX_PATHS.home}/)' | sort -r); do umount -l "$m"; done; : > ${SANDBOX_PATHS.bootLog}`, + { check: false, on }, + ); + endSession(); + bootDrive(); + const log2 = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive, second boot: layout redone", + /drive\.layout ms=\d+/.test(log2), + ); + expect( + "drive, second boot: every bind mount back", + allMounted() === "", + allMounted(), + ); + expect( + "drive, second boot: nothing installed", + /apply-rootfs installed=0/.test(log2), + ); + expect( + "drive, second boot: the checkout is still there", + /checkout\.(skipped|fetched)/.test( + exec( + `for i in $(seq 1 600); do grep -qE 'checkout.(skipped|fetched|cloned|clone)' ${SANDBOX_PATHS.bootLog} && break; sleep 0.1; done; cat ${SANDBOX_PATHS.bootLog}`, + { check: false, on }, + ), + ) || !checkedOut, + log2.match(/checkout\..*/)?.[0], + ); + console.log(`\nboot log (drive, second boot):\n${log2}`); + + const state = SANDBOX_PATHS.driveState; + expect( + "drive: the drive names its workspace", + exec(`cat ${state}/workspace-id`, { check: false, on }).trim() === + identityFields.SUPERSET_SANDBOX_WORKSPACE_ID, + ); + expect( + "drive: host.db and the checkout markers are on the drive, not the root disk", + exec( + `test -f ${state}/host.db && test -f ${state}/checkouts/@root && ! test -e ${SANDBOX_PATHS.checkouts}/@root && echo ok`, + { check: false, on }, + ).trim() === "ok", + exec(`ls ${state} ${state}/checkouts ${SANDBOX_PATHS.state}`, { + check: false, + on, + }), + ); + + // The user's work: a local commit and an uncommitted file. + const ws = SANDBOX_PATHS.workspace; + exec( + `git -C ${ws} -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m user-work && echo user > ${ws}/user-file.txt`, + { user: "ubuntu", on }, + ); + const head = () => + exec(`git -C ${ws} rev-parse HEAD`, { + check: false, + user: "ubuntu", + on, + }).trim(); + const work = head(); + const bootAndCheckout = () => { + bootDrive(); + return exec( + `for i in $(seq 1 1200); do grep -q 'checkout.end' ${SANDBOX_PATHS.bootLog} && break; sleep 0.1; done; cat ${SANDBOX_PATHS.bootLog}`, + { check: false, on }, + ); + }; + const workKept = () => + head() === work && + exec(`cat ${ws}/user-file.txt`, { check: false, on }).trim() === "user"; + + // The root disk rolled back to before the checkout; the drive did not. + endSession(); + exec( + `rm -rf ${SANDBOX_PATHS.checkouts} ${SANDBOX_PATHS.hostDb} ${SANDBOX_PATHS.hostDb}-wal ${SANDBOX_PATHS.hostDb}-shm`, + { on }, + ); + const rolledBack = bootAndCheckout(); + expect( + "drive, root rolled back: checkout skipped, the work kept", + /checkout\.skipped \. /.test(rolledBack) && workKept(), + rolledBack.match(/checkout\..*/g)?.join(" | "), + ); + + // Even with its marker gone, a checkout that has the branch is the user's. + endSession(); + exec(`rm -rf ${state}/checkouts`, { on }); + const markerLost = bootAndCheckout(); + expect( + "drive, marker lost: the branch is kept, not reset", + /checkout\.kept \. main is already on the drive/.test(markerLost) && + workKept(), + markerLost.match(/checkout\..*/g)?.join(" | "), + ); + + // A fetch that fails must never turn into a reclone over the drive. + endSession(); + exec(`rm -rf ${state}/checkouts`, { on }); + exec(`git -C ${drive}/workspace branch -q -m main user-main`, { + user: "ubuntu", + on, + }); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { + url: "https://github.com/paradevai/paradev.git", + branch: "superset/boot-twice-no-such-branch", + baseBranch: "superset/boot-twice-no-such-branch", + path: ".", + }, + ]), + }); + const fetchFailed = bootAndCheckout(); + expect( + "drive, fetch failed: nothing on the drive is replaced", + /checkout\.kept \. fetch failed/.test(fetchFailed) && + !/checkout\.cloned/.test(fetchFailed) && + workKept(), + fetchFailed.match(/checkout\..*/g)?.join(" | "), + ); + exec(`git -C ${drive}/workspace branch -q -m user-main main`, { + user: "ubuntu", + on, + }); + writeConf(driveIdentity); + + // A pinned bundle whose boot script predates the drive: it must never run over the laid-out drive. + endSession(); + const installedBoot = "/usr/local/bin/superset-boot"; + const bundledBoot = `${SANDBOX_PATHS.bundleRoot}/current/rootfs${installedBoot}`; + exec( + `cp ${bundledBoot} /tmp/superset-boot.keep && sed -i '/^# superset-boot drive-contract:/d' ${bundledBoot} && rm -f ${installedBoot}.hash`, + { on }, + ); + const downgraded = bootAsUser(on); + const downgradeLog = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive, older boot script: refused before the reexec, nothing checked out", + /exit=[1-9]/.test(downgraded) && + /drive\.downgrade /.test(downgradeLog) && + !/boot\.reexec|checkout\.start|host\.exec/.test(downgradeLog) && + workKept(), + downgradeLog.match(/drive\.downgrade.*/)?.[0] ?? downgraded.slice(-200), + ); + exec( + `cp /tmp/superset-boot.keep ${bundledBoot} && rm -f ${installedBoot}.hash`, + { + on, + }, + ); + + // A fresh fork of a golden drive: promote stripped its state, and a new workspace boots on it. + const repo = "https://github.com/paradevai/paradev.git"; + const currentBranch = () => + exec(`git -C ${ws} rev-parse --abbrev-ref HEAD`, { + check: false, + user: "ubuntu", + on, + }).trim(); + const forkBoot = (branch: string, baseBranch: string) => { + endSession(); + exec(`rm -rf ${state}`, { on }); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: randomUUID(), + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { url: repo, branch, baseBranch, path: "." }, + ]), + }); + return bootAndCheckout(); + }; + const newBranch = forkBoot("superset/boot-twice-fork", "main"); + expect( + "fresh fork, new branch: fetched from its base, on the branch", + /checkout\.fetched \. superset\/boot-twice-fork from main/.test( + newBranch, + ) && currentBranch() === "superset/boot-twice-fork", + newBranch.match(/checkout\..*/g)?.join(" | "), + ); + const sameBranch = forkBoot("main", "main"); + expect( + "fresh fork, a branch the golden has: reset to its base, not kept", + /checkout\.fetched \. main from main/.test(sameBranch) && + currentBranch() === "main" && + head() !== work, + sameBranch.match(/checkout\..*/g)?.join(" | "), + ); + const fetchFails = forkBoot( + "superset/boot-twice-fork-2", + "superset/boot-twice-no-such-branch", + ); + expect( + "fresh fork, fetch failed: never recloned; the golden's checkout is kept and the box is ready", + /checkout\.fresh\.failed \. superset\/boot-twice-fork-2/.test(fetchFails) && + !/checkout\.cloned/.test(fetchFails) && + currentBranch() === "main" && + exec( + `for i in $(seq 1 600); do [ -f ${SANDBOX_PATHS.run}/checkout.ready ] && echo ready && exit 0; sleep 0.1; done; echo missing`, + { check: false, on }, + ).trim() === "ready", + fetchFails.match(/checkout\..*/g)?.join(" | "), + ); + // A fork of a golden whose checkout carries its installed dependencies, on a new branch. + endSession(); + exec( + `git -C ${drive}/workspace checkout -q main && mkdir -p ${drive}/workspace/node_modules/golden-dep && echo installed > ${drive}/workspace/node_modules/golden-dep/index.js`, + { user: "ubuntu", on }, + ); + exec(`rm -rf ${state}`, { on }); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: randomUUID(), + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { + url: repo, + branch: "superset/boot-twice-fork-deps", + baseBranch: "main", + path: ".", + }, + ]), + }); + const withDeps = bootAndCheckout(); + expect( + "fresh fork of a golden with node_modules, new branch: the dependencies survive", + /checkout\.fetched \. superset\/boot-twice-fork-deps from main/.test( + withDeps, + ) && + exec(`cat ${ws}/node_modules/golden-dep/index.js`, { + check: false, + on, + }).trim() === "installed", + withDeps.match(/checkout\..*/g)?.join(" | "), + ); + + // A fresh fork whose first session ended between its layout and its checkout. + endSession(); + const interrupted = randomUUID(); + exec( + `git -C ${drive}/workspace checkout -q main && git -C ${drive}/workspace -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m stale-golden-copy`, + { user: "ubuntu", on }, + ); + const staleCopy = exec(`git -C ${drive}/workspace rev-parse main`, { + user: "ubuntu", + on, + }).trim(); + exec( + `rm -rf ${state} && mkdir -p ${state} && : > ${state}/fresh && echo ${interrupted} > ${state}/workspace-id && chown -R ubuntu:ubuntu ${state}`, + { on }, + ); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: interrupted, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { url: repo, branch: "main", baseBranch: "main", path: "." }, + ]), + }); + const resumedFresh = bootAndCheckout(); + expect( + "fresh fork, first session ended before its checkout: the next boot still fetches", + /checkout\.fetched \. main from main/.test(resumedFresh) && + head() !== staleCopy && + exec(`test -e ${state}/fresh && echo still || echo gone`, { + check: false, + on, + }).trim() === "gone", + resumedFresh.match(/checkout\..*/g)?.join(" | "), + ); + + // A fresh fork whose first fetch and clone both fail: a later boot still fetches the untouched golden copy. + endSession(); + const failedFirst = randomUUID(); + const unreachable = "https://example.invalid/paradev.git"; + // The golden is clean: the earlier cases' user file is not part of it. + exec( + `rm -f ${drive}/workspace/user-file.txt && git -C ${drive}/workspace checkout -q main && git -C ${drive}/workspace -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m stale-golden-copy-2 && git -C ${drive}/workspace remote set-url origin ${unreachable}`, + { user: "ubuntu", on }, + ); + const staleCopy2 = exec(`git -C ${drive}/workspace rev-parse main`, { + user: "ubuntu", + on, + }).trim(); + exec(`rm -rf ${state}`, { on }); + const forkOn = (url: string) => + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: failedFirst, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { url, branch: "main", baseBranch: "main", path: "." }, + ]), + }); + forkOn(unreachable); + const firstAttempt = bootAndCheckout(); + endSession(); + exec(`git -C ${drive}/workspace remote set-url origin ${repo}`, { + user: "ubuntu", + on, + }); + forkOn(repo); + const statusAfterFailure = exec( + `git -C ${drive}/workspace status --porcelain 2>&1 | head -10`, + { check: false, user: "ubuntu", on }, + ).trim(); + const depsAfterFailure = exec( + `cat ${drive}/workspace/node_modules/golden-dep/index.js`, + { check: false, on }, + ).trim(); + const laterBoot = bootAndCheckout(); + expect( + "fresh fork of a golden with node_modules, first fetch failed: the dependencies survive and a later boot fetches", + /checkout\.fresh\.failed/.test(firstAttempt) && + !/checkout\.cloned/.test(firstAttempt) && + depsAfterFailure === "installed" && + exec(`cat ${ws}/node_modules/golden-dep/index.js`, { + check: false, + on, + }).trim() === "installed" && + /checkout\.fetched \. main from main \(an untouched golden copy\)/.test( + laterBoot, + ) && + head() !== staleCopy2, + `${firstAttempt.match(/checkout\..*/g)?.join(" | ")} || ${laterBoot.match(/checkout\..*/g)?.join(" | ")} || status after the failure: ${statusAfterFailure || "clean"}`, + ); + + // The same, but the user committed after the failed first fetch: that copy is theirs now, kept. + endSession(); + exec(`git -C ${drive}/workspace remote set-url origin ${unreachable}`, { + user: "ubuntu", + on, + }); + exec(`rm -rf ${state}`, { on }); + forkOn(unreachable); + bootAndCheckout(); + endSession(); + exec( + `git -C ${drive}/workspace -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m user-work-after-failed-fetch && git -C ${drive}/workspace remote set-url origin ${repo}`, + { user: "ubuntu", on }, + ); + const usersCommit = exec(`git -C ${drive}/workspace rev-parse HEAD`, { + user: "ubuntu", + on, + }).trim(); + forkOn(repo); + const touched = bootAndCheckout(); + expect( + "fresh fork, first fetch failed, then the user committed: their copy is kept", + /checkout\.kept \. main is already on the drive/.test(touched) && + head() === usersCommit, + touched.match(/checkout\..*/g)?.join(" | "), + ); + + writeConf(driveIdentity); + + const refuses = (label: string, phase: string) => { + const answer = bootAsUser(on); + const log = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + `${label}: refused before host-service, stamped ${phase}`, + /exit=[1-9]/.test(answer) && + new RegExp(`${phase.replace(".", "\\.")} `).test(log) && + !/host\.exec|checkout\.start/.test(log) && + exec(`mountpoint -q ${ws} && echo bound`, { + check: false, + on, + }).trim() === "", + log.match(/drive\..*/)?.[0] ?? answer.slice(-200), + ); + }; + + // Another workspace's drive. + endSession(); + writeConf({ ...driveIdentity, SUPERSET_SANDBOX_WORKSPACE_ID: randomUUID() }); + refuses("drive foreign", "drive.foreign"); + writeConf(driveIdentity); + + // A drive that mounts empty though the workspace was laid out before. + endSession(); + exec(`umount ${drive} && mount -t tmpfs -o exec,mode=0755 tmpfs ${drive}`, { + on, + }); + writeConf({ ...driveIdentity, SUPERSET_SANDBOX_DRIVE_LAID_OUT: "1" }); + refuses("drive empty", "drive.unlaid"); + + // A session whose drive did not mount, while sandbox.conf still names one. + endSession(); + exec(`umount ${drive}`, { on }); + const refused = bootAsUser(on); + const log3 = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive missing: boot exits non-zero", + /exit=[1-9]/.test(refused), + refused.slice(-200).replaceAll("\n", " | "), + ); + expect( + "drive missing: stamped drive.missing", + /drive\.missing /.test(log3), + log3.match(/drive\.missing.*/)?.[0], + ); + expect( + "drive missing: no host-service, no checkout", + !/host\.exec|checkout\.start|steps\.start/.test(log3) && + exec(`ls ${SANDBOX_PATHS.run}`, { check: false, on }).trim() === "", + log3, + ); + expect( + "drive missing: /workspace is not bound", + exec(`mountpoint -q ${SANDBOX_PATHS.workspace} && echo bound`, { + check: false, + on, + }).trim() === "", + ); +} + +docker(["rm", "-f", NAME, DRIVE_NAME], { check: false }); // Privileged for the /dev/shm remount and the X socket dir, like the VM. No // network policy here; the checkout needs the internet. docker([ @@ -131,6 +706,7 @@ docker([ "--privileged", "--user", "root", + ...CA_MOUNT, IMAGE, "sleep", "infinity", @@ -471,8 +1047,10 @@ try { exec(`ls ${SANDBOX_PATHS.run}`, { check: false }), ), ); + + drivePass(); } finally { - docker(["rm", "-f", NAME], { check: false }); + docker(["rm", "-f", NAME, DRIVE_NAME], { check: false }); } if (failures.length) { console.error( diff --git a/packages/sandbox/src/drive-comparison.test.ts b/packages/sandbox/src/drive-comparison.test.ts new file mode 100644 index 00000000000..815b8e0aab6 --- /dev/null +++ b/packages/sandbox/src/drive-comparison.test.ts @@ -0,0 +1,64 @@ +import { describe, expect, test } from "bun:test"; +import { readSentinels, renderComparison } from "./drive-comparison"; + +describe("renderComparison", () => { + test("one column per box, a dash where a box has no measure, LOST for a lost sentinel", () => { + const table = renderComparison([ + { + label: "plain", + measures: { + createToReadyMs: 31_000, + firstStop: { ms: 40_000, snapshotBytes: 7 * 1024 ** 3 }, + sentinels: { "uncommitted file": true }, + }, + }, + { + label: "drive", + measures: { + forkMs: 500, + firstStop: { ms: 12_000, snapshotBytes: 2.11 * 1024 ** 3 }, + sentinels: { "uncommitted file": false }, + }, + }, + ]).split("\n"); + expect(table[0]).toMatch(/plain\s+drive$/); + expect(table.find((row) => row.startsWith("create to host.ready"))).toMatch( + /31\.0 s\s+-$/, + ); + expect(table.find((row) => row.startsWith("drive fork"))).toMatch( + /-\s+0\.5 s$/, + ); + expect( + table.find((row) => row.startsWith("root snapshot, 1st stop")), + ).toMatch(/7\.00 GB\s+2\.11 GB$/); + expect( + table.find((row) => row.startsWith("kept: uncommitted file")), + ).toMatch(/yes\s+LOST$/); + }); +}); + +describe("readSentinels", () => { + test("a sentinel is kept only when it reads back exactly", async () => { + const answers: Record = { + "superset-sentinel-uncommitted.txt": "t0k", + "rev-parse HEAD": "abc", + ".cache/superset-sentinel": "t0k", + ".config/superset-sentinel": "other", + "image inspect": "", + }; + const run = async (command: string) => { + const key = Object.keys(answers).find((part) => command.includes(part)); + const out = key ? (answers[key] ?? "") : ""; + return { out, detail: out }; + }; + expect( + await readSentinels(run, { token: "t0k", commit: "abc", image: "img" }), + ).toEqual({ + "uncommitted file": true, + "local commit": true, + "file in a bound home subtree": true, + "file on the root disk": false, + "docker image": false, + }); + }); +}); diff --git a/packages/sandbox/src/drive-comparison.ts b/packages/sandbox/src/drive-comparison.ts new file mode 100644 index 00000000000..206fc431efa --- /dev/null +++ b/packages/sandbox/src/drive-comparison.ts @@ -0,0 +1,204 @@ +/** + * The same work on a box with a workspace drive and on one without, measured + * the way a person meets it: create, install, stop, wake, and whether what + * they wrote is still there. The REAL_SANDBOX_DRIVE pass of real-sandbox.ts + * runs it; this module holds the work, the sentinels and the table. + */ + +import { SANDBOX_PATHS } from "@superset/shared/sandbox/contract"; + +export const SENTINELS = [ + "uncommitted file", + "local commit", + "file in a bound home subtree", + "file on the root disk", + "docker image", +] as const; +export type SentinelName = (typeof SENTINELS)[number]; + +export interface StopMeasure { + ms: number; + snapshotBytes: number; +} + +export interface BoxMeasures { + createToReadyMs?: number; + forkMs?: number; + installMs?: number; + writeMs?: number; + firstStop?: StopMeasure; + secondStop?: StopMeasure; + wake?: { + detachMs?: number; + mountMs?: number; + resumeMs?: number; + bootMs?: number; + totalMs: number; + }; + /** From the start of the create or wake until git and node_modules answer in /workspace. */ + usableMs?: number; + sentinels?: Partial>; +} + +/** A command on the box: stdout only, since sudo there warns on stderr. */ +export type BoxRun = ( + command: string, + options?: { root?: boolean }, +) => Promise<{ out: string; detail: string }>; + +const WORKSPACE = SANDBOX_PATHS.workspace; +const DOCKER_UP = + "pgrep -x dockerd > /dev/null || (dockerd > /tmp/dockerd.log 2>&1 &); for i in $(seq 1 60); do docker info > /dev/null 2>&1 && break; sleep 1; done"; + +async function timed(run: () => Promise): Promise { + const started = Date.now(); + await run(); + return Date.now() - started; +} + +export async function waitForCheckout(run: BoxRun): Promise { + const ready = await run( + `for i in $(seq 1 600); do [ -f ${SANDBOX_PATHS.run}/checkout.ready ] && echo ready && exit 0; sleep 1; done; echo missing`, + ); + return ready.out === "ready"; +} + +/** A real install, then `gb` GB of incompressible data under /workspace. */ +export async function doWork( + run: BoxRun, + gb: number, +): Promise<{ installMs: number; writeMs: number; installed: string }> { + let installed = ""; + const installMs = await timed(async () => { + installed = ( + await run( + `cd ${WORKSPACE} && bun install --frozen-lockfile > /tmp/bun-install.log 2>&1 && echo installed || tail -5 /tmp/bun-install.log`, + ) + ).out; + }); + const writeMs = await timed(() => + run( + `head -c ${gb * 1024 * 1024 * 1024} /dev/urandom > ${WORKSPACE}/.superset-work-filler && sync`, + ), + ); + return { installMs, writeMs, installed }; +} + +export interface SentinelValues { + token: string; + commit: string; + image: string; +} + +export async function writeSentinels( + run: BoxRun, + token: string, +): Promise { + const commit = await run( + [ + `echo ${token} > ${WORKSPACE}/superset-sentinel-uncommitted.txt`, + `git -C ${WORKSPACE} -c user.name=check -c user.email=check@example.invalid commit -q --allow-empty -m "sentinel ${token}"`, + "mkdir -p ~/.cache ~/.config", + `echo ${token} > ~/.cache/superset-sentinel`, + `echo ${token} > ~/.config/superset-sentinel`, + `git -C ${WORKSPACE} rev-parse HEAD`, + ].join(" && "), + ); + const image = await run( + `${DOCKER_UP}; mkdir -p /tmp/superset-sentinel && echo ${token} > /tmp/superset-sentinel/sentinel && tar -C /tmp/superset-sentinel -c . | docker import - superset-sentinel:check`, + { root: true }, + ); + return { token, commit: commit.out, image: image.out }; +} + +export async function readSentinels( + run: BoxRun, + values: SentinelValues, +): Promise> { + const read = async (command: string, root = false) => + (await run(command, { root })).out; + return { + "uncommitted file": + (await read(`cat ${WORKSPACE}/superset-sentinel-uncommitted.txt`)) === + values.token, + "local commit": + values.commit !== "" && + (await read(`git -C ${WORKSPACE} rev-parse HEAD`)) === values.commit, + "file in a bound home subtree": + (await read("cat ~/.cache/superset-sentinel")) === values.token, + "file on the root disk": + (await read("cat ~/.config/superset-sentinel")) === values.token, + "docker image": + values.image !== "" && + (await read( + `${DOCKER_UP}; docker image inspect -f '{{.Id}}' superset-sentinel:check`, + true, + )) === values.image, + }; +} + +/** Milliseconds from `since` until /workspace answers git and has node_modules. */ +export async function usableSince( + run: BoxRun, + since: number, +): Promise { + const usable = await run( + `for i in $(seq 1 300); do git -C ${WORKSPACE} status --porcelain > /dev/null 2>&1 && test -d ${WORKSPACE}/node_modules && echo usable && exit 0; sleep 1; done; echo never`, + ); + return usable.out === "usable" ? Date.now() - since : undefined; +} + +const seconds = (ms: number | undefined) => + ms === undefined ? "-" : `${(ms / 1000).toFixed(1)} s`; +const gigabytes = (bytes: number | undefined) => + bytes === undefined ? "-" : `${(bytes / 1024 ** 3).toFixed(2)} GB`; + +/** One row per measure, one column per box. */ +export function renderComparison( + columns: Array<{ label: string; measures: BoxMeasures }>, +): string { + const rows: Array<[string, (m: BoxMeasures) => string]> = [ + ["create to host.ready", (m) => seconds(m.createToReadyMs)], + ["drive fork at create", (m) => seconds(m.forkMs)], + ["bun install", (m) => seconds(m.installMs)], + ["write the filler", (m) => seconds(m.writeMs)], + ["root snapshot, 1st stop", (m) => gigabytes(m.firstStop?.snapshotBytes)], + ["stop time, 1st stop", (m) => seconds(m.firstStop?.ms)], + ["wake: detach wait", (m) => seconds(m.wake?.detachMs)], + ["wake: mount update", (m) => seconds(m.wake?.mountMs)], + ["wake: resume", (m) => seconds(m.wake?.resumeMs)], + ["wake: boot to host.ready", (m) => seconds(m.wake?.bootMs)], + ["wake: total", (m) => seconds(m.wake?.totalMs)], + ["until /workspace is usable", (m) => seconds(m.usableMs)], + ["root snapshot, 2nd stop", (m) => gigabytes(m.secondStop?.snapshotBytes)], + ["stop time, 2nd stop", (m) => seconds(m.secondStop?.ms)], + ...SENTINELS.map((name): [string, (m: BoxMeasures) => string] => [ + `kept: ${name}`, + (m) => { + const kept = m.sentinels?.[name]; + return kept === undefined ? "-" : kept ? "yes" : "LOST"; + }, + ]), + ]; + const table = [ + ["", ...columns.map((column) => column.label)], + ...rows.map(([label, cell]) => [ + label, + ...columns.map((column) => cell(column.measures)), + ]), + ]; + const widths = table[0]?.map((_, i) => + Math.max(...table.map((row) => row[i]?.length ?? 0)), + ); + return table + .map((row) => + row + .map((cell, i) => + i === 0 + ? cell.padEnd(widths?.[i] ?? 0) + : cell.padStart(widths?.[i] ?? 0), + ) + .join(" "), + ) + .join("\n"); +} diff --git a/packages/sandbox/src/drive-sweep.test.ts b/packages/sandbox/src/drive-sweep.test.ts new file mode 100644 index 00000000000..5a58da087c5 --- /dev/null +++ b/packages/sandbox/src/drive-sweep.test.ts @@ -0,0 +1,80 @@ +import { describe, expect, test } from "bun:test"; +import { orphanCandidates, type SweepHost, sweepDrives } from "./drive-sweep"; + +const DAY = 24 * 60 * 60 * 1000; +const NOW = 100 * DAY; +const old = NOW - 10 * DAY; + +describe("orphanCandidates", () => { + test("any row that names a drive keeps it, whatever its status or age", () => { + const candidates = orphanCandidates({ + drives: [ + { name: "drive-ws-archived-long-ago", createdAt: old }, + { name: "drive-ws-unnamed", createdAt: old }, + { name: "drive-ws-young", createdAt: NOW - DAY / 2 }, + { name: "drive-ws-held", createdAt: old, attachedTo: "box" }, + { name: "env-drive-archived-env", createdAt: old }, + { name: "env-drive-gone", createdAt: old }, + { name: "agent-tools-iad1-feedfacecafe", createdAt: old }, + ], + named: new Set(["drive-ws-archived-long-ago", "env-drive-archived-env"]), + now: NOW, + minAgeMs: 2 * DAY, + }); + expect(candidates.map((drive) => drive.name)).toEqual([ + "drive-ws-unnamed", + "env-drive-gone", + ]); + }); +}); + +describe("sweepDrives", () => { + test("a drive an unarchive named after the listing is kept: each delete checks again", async () => { + const deleted: string[] = []; + let named = new Set(); + const host: SweepHost = { + listDrives: async () => [ + { name: "drive-ws-a", createdAt: old }, + { name: "drive-ws-b", createdAt: old }, + ], + namedDrives: async () => { + const listedAt = named; + // The user unarchives workspace a right after the listing. + named = new Set(["drive-ws-a"]); + return listedAt; + }, + stillOrphaned: async (name) => !named.has(name), + remove: async (name) => { + deleted.push(name); + }, + }; + const result = await sweepDrives({ + host, + now: NOW, + minAgeMs: 2 * DAY, + remove: true, + log: () => {}, + }); + expect(deleted).toEqual(["drive-ws-b"]); + expect(result).toEqual({ candidates: 2, deleted: 1, failed: 0 }); + }); + + test("a dry run deletes nothing", async () => { + const deleted: string[] = []; + await sweepDrives({ + host: { + listDrives: async () => [{ name: "drive-ws-a", createdAt: old }], + namedDrives: async () => new Set(), + stillOrphaned: async () => true, + remove: async (name) => { + deleted.push(name); + }, + }, + now: NOW, + minAgeMs: 2 * DAY, + remove: false, + log: () => {}, + }); + expect(deleted).toEqual([]); + }); +}); diff --git a/packages/sandbox/src/drive-sweep.ts b/packages/sandbox/src/drive-sweep.ts new file mode 100644 index 00000000000..a57330a775d --- /dev/null +++ b/packages/sandbox/src/drive-sweep.ts @@ -0,0 +1,199 @@ +/** + * Finds workspace and golden drives that nothing names any more: no + * workspace row and no environment row, whatever its status, no sandbox + * attached, and older than a safe age. Lists them; deletes them only with + * --delete, and checks each again just before its delete. + * + * bun run drive-sweep DATABASE_URL, dry run + * bun run drive-sweep --production the production database + * bun run drive-sweep --production --delete deletes what it lists + * --min-age-days default 2 + * + * Needs VERCEL_SANDBOX_TOKEN, VERCEL_SANDBOX_TEAM_ID and VERCEL_SANDBOX_PROJECT_ID. + */ + +const DAY_MS = 24 * 60 * 60 * 1000; +export const SWEPT_PREFIXES = ["drive-", "env-drive-"] as const; + +export interface SweptDrive { + name: string; + createdAt: number; + attachedTo?: string; +} + +/** + * A drive is a candidate when no row names it and no sandbox holds it, and + * it is older than `minAgeMs`: a release writes its golden drive's name only + * once the golden is built. A row that is really torn down has its names + * cleared in the same write (retireAndTearDown), so any row that still names + * a drive, archived or not, may still be undone onto it. + */ +export function orphanCandidates(args: { + drives: SweptDrive[]; + named: ReadonlySet; + now: number; + minAgeMs: number; +}): SweptDrive[] { + return args.drives.filter( + (drive) => + SWEPT_PREFIXES.some((prefix) => drive.name.startsWith(prefix)) && + !args.named.has(drive.name) && + !drive.attachedTo && + args.now - drive.createdAt > args.minAgeMs, + ); +} + +export interface SweepHost { + listDrives(): Promise; + namedDrives(): Promise>; + /** Read at delete time, not from the list: an unarchive may have named or mounted it since. */ + stillOrphaned(name: string): Promise; + remove(name: string): Promise; +} + +export async function sweepDrives(args: { + host: SweepHost; + now: number; + minAgeMs: number; + remove: boolean; + log: (line: string) => void; +}): Promise<{ candidates: number; deleted: number; failed: number }> { + const drives = await args.host.listDrives(); + const candidates = orphanCandidates({ + drives, + named: await args.host.namedDrives(), + now: args.now, + minAgeMs: args.minAgeMs, + }); + args.log(`${drives.length} drives listed, ${candidates.length} orphaned`); + let deleted = 0; + let failed = 0; + for (const drive of candidates) { + const age = ((args.now - drive.createdAt) / DAY_MS).toFixed(1); + if (!args.remove) { + args.log(`would delete ${drive.name} (${age} days old)`); + continue; + } + try { + if (!(await args.host.stillOrphaned(drive.name))) { + args.log( + `kept ${drive.name}: a row names it or a sandbox holds it now`, + ); + continue; + } + await args.host.remove(drive.name); + deleted++; + args.log(`deleted ${drive.name} (${age} days old)`); + } catch (error) { + failed++; + args.log(`::warning::could not delete ${drive.name}: ${error}`); + } + } + if (!args.remove && candidates.length) { + args.log("dry run: pass --delete to delete"); + } + return { candidates: candidates.length, deleted, failed }; +} + +if (import.meta.main) { + const argv = process.argv.slice(2); + const ageFlag = argv.indexOf("--min-age-days"); + const minAgeDays = ageFlag >= 0 ? Number(argv[ageFlag + 1]) : 2; + const credentials = { + token: process.env.VERCEL_SANDBOX_TOKEN ?? "", + teamId: process.env.VERCEL_SANDBOX_TEAM_ID ?? "", + projectId: process.env.VERCEL_SANDBOX_PROJECT_ID ?? "", + }; + if (!credentials.token || !credentials.teamId || !credentials.projectId) { + console.error( + "VERCEL_SANDBOX_TOKEN, VERCEL_SANDBOX_TEAM_ID and VERCEL_SANDBOX_PROJECT_ID are required", + ); + process.exit(64); + } + if (!Number.isFinite(minAgeDays) || minAgeDays < 1) { + console.error("--min-age-days must be at least 1"); + process.exit(64); + } + if (argv.includes("--production")) { + const { useProductionDatabase } = await import( + "./environments/production-database" + ); + console.log(`database: Neon branch ${await useProductionDatabase()}`); + } else if (!process.env.DATABASE_URL) { + console.error("DATABASE_URL is required (or pass --production)"); + process.exit(64); + } + process.env.SKIP_ENV_VALIDATION ??= "1"; + const { db } = await import("@superset/db/client"); + const { cloudWorkspaces, environments } = await import("@superset/db/schema"); + const { eq, isNotNull } = await import("drizzle-orm"); + const { Drive } = await import("@vercel/sandbox"); + + const named = async (name: string) => + ( + await Promise.all([ + db + .select({ id: cloudWorkspaces.id }) + .from(cloudWorkspaces) + .where(eq(cloudWorkspaces.driveName, name)) + .limit(1), + db + .select({ id: environments.id }) + .from(environments) + .where(eq(environments.goldenDriveName, name)) + .limit(1), + ]) + ).some((rows) => rows.length > 0); + const host: SweepHost = { + async listDrives() { + const drives = new Map(); + for (const prefix of SWEPT_PREFIXES) { + for await (const drive of await Drive.list({ + ...credentials, + namePrefix: prefix, + sortBy: "name", + })) { + drives.set(drive.name, { + name: drive.name, + createdAt: drive.createdAt.getTime(), + attachedTo: drive.currentSandboxName, + }); + } + } + return [...drives.values()]; + }, + async namedDrives() { + const [workspaces, goldens] = await Promise.all([ + db + .select({ name: cloudWorkspaces.driveName }) + .from(cloudWorkspaces) + .where(isNotNull(cloudWorkspaces.driveName)), + db + .select({ name: environments.goldenDriveName }) + .from(environments) + .where(isNotNull(environments.goldenDriveName)), + ]); + return new Set( + [...workspaces, ...goldens].flatMap((row) => + row.name ? [row.name] : [], + ), + ); + }, + async stillOrphaned(name) { + if (await named(name)) return false; + const drive = await Drive.get({ ...credentials, name }); + return !drive.currentSandboxName; + }, + async remove(name) { + await (await Drive.get({ ...credentials, name })).delete(); + }, + }; + const { failed } = await sweepDrives({ + host, + now: Date.now(), + minAgeMs: minAgeDays * DAY_MS, + remove: argv.includes("--delete"), + log: console.log, + }); + process.exit(failed ? 1 : 0); +} diff --git a/packages/sandbox/src/environments/probe.ts b/packages/sandbox/src/environments/probe.ts index e1b198bd79d..c5425d7a937 100644 --- a/packages/sandbox/src/environments/probe.ts +++ b/packages/sandbox/src/environments/probe.ts @@ -175,12 +175,40 @@ export async function probeBox(args: ProbeArgs): Promise { /up/, 90, ); + await until( + "websockify listening", + `ss -ltn | grep -q ':${SANDBOX_PORTS.desktop} ' && echo listening`, + /listening/, + 60, + ); const rfb = await rfbHandshake(host, args.hostSecret); check( "desktop stream answers RFB through host-service", rfb.startsWith("RFB "), JSON.stringify(rfb), ); + if (!rfb.startsWith("RFB ")) { + const plainGet = await status( + `${host}/desktop/websockify?token=${encodeURIComponent(args.hostSecret)}`, + ); + const onBox = await run( + [ + `echo '--- listening'`, + `ss -ltn | grep -E ':(${SANDBOX_PORTS.desktop}|${SANDBOX_PORTS.vnc}) '`, + `echo '--- processes'`, + `pgrep -a -f 'websockify|Xvnc'`, + `echo '--- loopback handshake'`, + `curl -s --max-time 5 -o /dev/null -w '%{http_code}' -H 'Connection: Upgrade' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' http://127.0.0.1:${SANDBOX_PORTS.desktop}/`, + `echo; echo '--- desktop.log'`, + `tail -20 ${SANDBOX_PATHS.logs}/desktop.log`, + `echo '--- host-service.log, desktop lines'`, + `grep -i -E 'desktop|websockify' ${SANDBOX_PATHS.logs}/host-service.log | tail -20`, + ].join("; "), + ); + log( + `RFB diagnosis: GET through host-service answered ${plainGet}\n${onBox}`, + ); + } if (args.expectAnthropicRule) { const models = (key: string) => run( diff --git a/packages/sandbox/src/environments/production-database.ts b/packages/sandbox/src/environments/production-database.ts new file mode 100644 index 00000000000..7815aec8304 --- /dev/null +++ b/packages/sandbox/src/environments/production-database.ts @@ -0,0 +1,37 @@ +/** + * Points DATABASE_URL at the Neon project's default branch, the production + * database (NEON_API_KEY, NEON_PROJECT_ID). Returns the branch it chose. + */ +export async function useProductionDatabase(): Promise { + const key = process.env.NEON_API_KEY; + const project = process.env.NEON_PROJECT_ID; + if (!key || !project) { + throw new Error("--production needs NEON_API_KEY and NEON_PROJECT_ID"); + } + const headers = { authorization: `Bearer ${key}` }; + const branchesResponse = await fetch( + `https://console.neon.tech/api/v2/projects/${project}/branches`, + { headers }, + ); + if (!branchesResponse.ok) { + throw new Error(`Neon branches: ${branchesResponse.status}`); + } + const branches = (await branchesResponse.json()) as { + branches: Array<{ id: string; name: string; default?: boolean }>; + }; + const main = branches.branches.find((b) => b.default); + if (!main) throw new Error("no default branch in the Neon project"); + const uriResponse = await fetch( + `https://console.neon.tech/api/v2/projects/${project}/connection_uri?branch_id=${main.id}&database_name=neondb&role_name=neondb_owner&pooled=false`, + { headers }, + ); + if (!uriResponse.ok) { + throw new Error(`Neon connection_uri: ${uriResponse.status}`); + } + const uri = (await uriResponse.json()) as { uri?: string }; + if (!uri.uri) + throw new Error("could not resolve the production connection string"); + process.env.DATABASE_URL = uri.uri; + process.env.DATABASE_URL_UNPOOLED = uri.uri; + return `${main.name} (${main.id})`; +} diff --git a/packages/sandbox/src/environments/release.ts b/packages/sandbox/src/environments/release.ts index 95dd3faf6c6..21f96e697dc 100644 --- a/packages/sandbox/src/environments/release.ts +++ b/packages/sandbox/src/environments/release.ts @@ -51,6 +51,7 @@ import { SANDBOX_REGION_IDS, } from "@superset/shared/sandbox/regions"; import { Sandbox } from "@vercel/sandbox"; +import { useProductionDatabase } from "./production-database"; // The provisioning code imports the API env schema; an operator running this // should not need every API secret to exist locally. @@ -99,31 +100,10 @@ if (!credentials.token || !credentials.teamId || !credentials.projectId) { if (!process.env.SANDBOX_GATE_SECRET) fail("SANDBOX_GATE_SECRET is required (the host secret derives from it)"); if (PRODUCTION) { - const key = process.env.NEON_API_KEY; - const project = process.env.NEON_PROJECT_ID; - if (!key || !project) - fail("--production needs NEON_API_KEY and NEON_PROJECT_ID"); - const headers = { authorization: `Bearer ${key}` }; - const branchesResponse = await fetch( - `https://console.neon.tech/api/v2/projects/${project}/branches`, - { headers }, + const branch = await useProductionDatabase().catch((error: unknown) => + fail(String(error instanceof Error ? error.message : error)), ); - if (!branchesResponse.ok) fail(`Neon branches: ${branchesResponse.status}`); - const branches = (await branchesResponse.json()) as { - branches: Array<{ id: string; name: string; default?: boolean }>; - }; - const main = branches.branches.find((b) => b.default); - if (!main) fail("no default branch in the Neon project"); - const uriResponse = await fetch( - `https://console.neon.tech/api/v2/projects/${project}/connection_uri?branch_id=${main.id}&database_name=neondb&role_name=neondb_owner&pooled=false`, - { headers }, - ); - if (!uriResponse.ok) fail(`Neon connection_uri: ${uriResponse.status}`); - const uri = (await uriResponse.json()) as { uri?: string }; - if (!uri.uri) fail("could not resolve the production connection string"); - process.env.DATABASE_URL = uri.uri; - process.env.DATABASE_URL_UNPOOLED = uri.uri; - log(`database: Neon branch ${main.name} (${main.id})`); + log(`database: Neon branch ${branch}`); } else if (!process.env.DATABASE_URL) { fail("DATABASE_URL is required (or pass --production)"); } @@ -204,11 +184,22 @@ if (SKIP_IMAGE) { // email clients, whose secrets this job does not have. const { deleteSandbox, + deleteSandboxAndDrive, provisionSandbox, stopAndSnapshot, stripWorkspaceIdentity, wakeSandbox, } = await import("@superset/trpc/lib/sandbox/vercel"); +const { + forkWorkspaceDrive, + goldenDriveNameFor, + vercelDrives, + waitForDriveDetached, + workspaceDriveIdentity, + workspaceDriveMounts, + workspaceDriveNameFor, + workspaceDrivesEnabled, +} = await import("@superset/trpc/lib/sandbox/drives"); const { deriveSandboxCredentials } = await import( "@superset/trpc/lib/sandbox/credentials" ); @@ -223,7 +214,11 @@ const setupScript = readFileSync( ); const setupHook = [setupScript]; -function identityFor(workspaceId: string, sourceRef: string): SandboxIdentity { +function identityFor( + workspaceId: string, + sourceRef: string, + driveName: string | null, +): SandboxIdentity { return { SUPERSET_SANDBOX_CONTRACT: String(SANDBOX_CONTRACT_VERSION) as "1", SUPERSET_BUNDLE_SHA: bundle.sha256, @@ -235,12 +230,23 @@ function identityFor(workspaceId: string, sourceRef: string): SandboxIdentity { ]), SUPERSET_SANDBOX_IMAGE_TAG: sourceRef, SUPERSET_SANDBOX_PROVIDER: "vercel", + ...workspaceDriveIdentity(driveName, { laidOut: false }), }; } const golden = `env-internal-${Date.now().toString(36)}`; const goldenWorkspaceId = randomUUID(); const goldenSecret = await sandboxHostSecretFor(goldenWorkspaceId); +// In drive mode the golden boots with its drive from the first boot, so the +// checkout and the installs never touch its root disk. +const drives = vercelDrives(); +const goldenDrive = (await workspaceDrivesEnabled(ORGANIZATION_ID as string)) + ? goldenDriveNameFor(golden) + : null; +if (goldenDrive) { + await drives.create(goldenDrive, REGION); + log(`golden: drive mode; ${goldenDrive} created in ${REGION}`); +} log(`golden: provisioning ${golden} from ${goldenImage}`); // A freshly pushed image sits in `Preparing` while the registry optimises it // for sandboxes, and create answers 409 until then. @@ -256,13 +262,14 @@ for (;;) { region: REGION, }, claim: { - identity: identityFor(goldenWorkspaceId, goldenImage), + identity: identityFor(goldenWorkspaceId, goldenImage, goldenDrive), hostSecret: goldenSecret, managedEnv: {}, networkPolicy: "allow-all", vcpus: GOLDEN_BOX_VCPUS, agentTools: true, }, + mounts: workspaceDriveMounts(goldenDrive), }); break; } catch (error) { @@ -280,7 +287,7 @@ for (;;) { await wakeSandbox({ providerSandboxId: golden, claim: { - identity: identityFor(goldenWorkspaceId, goldenImage), + identity: identityFor(goldenWorkspaceId, goldenImage, goldenDrive), hostSecret: goldenSecret, managedEnv: {}, networkPolicy: "allow-all", @@ -383,8 +390,9 @@ if (failed) fail(`${failed} check(s) failed; ${golden} left for inspection`); // A fork must not come up as the golden's workspace; and the stop is the // snapshot forks start from. -await stripWorkspaceIdentity(goldenBox); +await stripWorkspaceIdentity(goldenBox, { withDrive: goldenDrive !== null }); await stopAndSnapshot(golden); +if (goldenDrive) await waitForDriveDetached(goldenDrive, { drives }); log(`golden: ${golden} stripped, stopped and snapshotted`); // 4. probe, as a workspace @@ -441,21 +449,36 @@ const { networkPolicy, managedEnv } = await deriveSandboxCredentials({ githubToken: null, gitAuthor: { name: "Superset release", email: "noreply@superset.sh" }, }); +const probeDrive = goldenDrive ? workspaceDriveNameFor(probe) : null; const probeClaim = { - identity: identityFor(probeWorkspaceId, golden), + identity: identityFor(probeWorkspaceId, golden, probeDrive), hostSecret: probeSecret, managedEnv, networkPolicy, vcpus: GOLDEN_BOX_VCPUS, agentTools: true, }; +const probeMounts = workspaceDriveMounts(probeDrive); +if (probeDrive) { + await forkWorkspaceDrive({ + name: probeDrive, + region: REGION, + goldenDriveName: goldenDrive, + drives, + }); +} log(`probe: provisioning ${probe} as a fork of ${golden}`); await provisionSandbox({ name: probe, environment: { sourceKind: "fork", sourceRef: golden, region: REGION }, claim: probeClaim, + mounts: probeMounts, +}); +await wakeSandbox({ + providerSandboxId: probe, + claim: probeClaim, + mounts: probeMounts, }); -await wakeSandbox({ providerSandboxId: probe, claim: probeClaim }); let probeFailed = await probeBox({ name: probe, credentials, @@ -500,14 +523,18 @@ if (ENV_FILE) { { const box = await Sandbox.get({ ...credentials, name: probe }); await box.stop(); - await wakeSandbox({ providerSandboxId: probe, claim: probeClaim }); + await wakeSandbox({ + providerSandboxId: probe, + claim: probeClaim, + mounts: probeMounts, + }); probeFailed += await checkWakeLog({ name: probe, credentials, log }); } if (probeFailed) fail( `${probeFailed} probe check(s) failed; ${golden} and ${probe} left for inspection`, ); -await deleteSandbox(probe); +await deleteSandboxAndDrive(probe, probeDrive, drives); log(`probe: ${probe} deleted`); if (!SKIP_IMAGE) { @@ -560,6 +587,7 @@ const fromGolden = { sourceKind: "fork" as const, sourceRef: golden, region: REGION, + goldenDriveName: goldenDrive, bundleSha: bundle.sha256, primaryRepositoryId: monorepo.id, archivedAt: null, @@ -649,6 +677,12 @@ if ( } else { await deleteSandbox(previous.sourceRef); log(`previous golden ${previous.sourceRef} deleted`); + const { deleteGoldenDriveIfUnreferenced } = await import( + "@superset/trpc/lib/sandbox/golden-drives" + ); + await deleteGoldenDriveIfUnreferenced(previous.goldenDriveName, drives); + if (previous.goldenDriveName) + log(`previous golden drive ${previous.goldenDriveName} deleted`); } } log("release done"); diff --git a/packages/sandbox/src/real-sandbox.ts b/packages/sandbox/src/real-sandbox.ts index 1ce5208b71a..bc5ed2d9851 100644 --- a/packages/sandbox/src/real-sandbox.ts +++ b/packages/sandbox/src/real-sandbox.ts @@ -5,10 +5,26 @@ * built here from a throwaway identity. * * bun run src/real-sandbox.ts (VERCEL_SANDBOX_*, SANDBOX_GATE_SECRET, CDN_R2_*) + * REAL_SANDBOX_DRIVE=1 bun run src/real-sandbox.ts * * Publishes the bundle first so the box can fetch it if the image is older, * and its agent-tools drive, which the box mounts. - * Leaves the sandbox up for inspection on failure; deletes it on success. + * + * With REAL_SANDBOX_DRIVE=1, whatever the first box's result, the same work + * runs on a box in drive mode and on a plain box, side by side: the checkout, + * a real `bun install`, REAL_SANDBOX_WORK_GB (default 4) GB written under + * /workspace, persistence sentinels, a stop, a wake, a second stop. The drive + * box also checks the layout, the stores and Docker on the drive. Then + * it is stopped uncleanly (host-service killed, writers still going) and must + * boot with host.db and the checkout intact, and it is promoted to a + * drive-mode golden that a new workspace is created from on a new branch, the + * production path. One table compares them (drive-comparison.ts). A wake + * that is not served from a warm snapshot cache cannot be forced from here. + * + * Every box and drive it made is deleted at the end, pass or fail: the box + * stopped first, each drive once detached. REAL_SANDBOX_KEEP=1 keeps the + * boxes (stopped) and the box drive for inspection instead. Exits 1 when + * any check fails. */ import { randomUUID } from "node:crypto"; @@ -16,10 +32,24 @@ import { GOLDEN_BOX_VCPUS } from "@superset/shared/billing"; import { SANDBOX_IMAGE_NAME } from "@superset/shared/constants"; import { SANDBOX_CONTRACT_VERSION, + SANDBOX_PATHS, + SANDBOX_USER, type SandboxIdentity, } from "@superset/shared/sandbox/contract"; import { DEFAULT_SANDBOX_REGION } from "@superset/shared/sandbox/regions"; -import { Sandbox } from "@vercel/sandbox"; +import { Sandbox, Snapshot } from "@vercel/sandbox"; +import { + type BoxMeasures, + type BoxRun, + doWork, + readSentinels, + renderComparison, + SENTINELS, + type StopMeasure, + usableSince, + waitForCheckout, + writeSentinels, +} from "./drive-comparison"; process.env.SKIP_ENV_VALIDATION ??= "1"; @@ -71,9 +101,14 @@ if (gaps.length) { process.exit(1); } -const { deleteSandbox, provisionSandbox, wakeSandbox } = await import( - "@superset/trpc/lib/sandbox/vercel" -); +const { + deleteSandbox, + deleteSandboxAndDrive, + promoteSandboxToEnvironment, + provisionSandbox, + stopSandbox, + wakeSandbox, +} = await import("@superset/trpc/lib/sandbox/vercel"); const { sandboxHostSecretFor } = await import( "@superset/trpc/lib/sandbox/access" ); @@ -111,62 +146,616 @@ const claim = { agentTools: true, }; -log( - `provisioning ${name} from ${SANDBOX_IMAGE_NAME} on bundle ${bundle.sha256.slice(0, 12)}`, -); // A freshly pushed image sits in `Preparing` while the registry optimises // it, and create answers 409 until then. -const deadline = Date.now() + 20 * 60_000; -for (;;) { - try { - await provisionSandbox({ - name, - environment: { - sourceKind: "image", - sourceRef: SANDBOX_IMAGE_NAME, - region: DEFAULT_SANDBOX_REGION, - }, - claim, - }); - break; - } catch (error) { - const notReady = - String(error).includes("409") || - String(error).includes("image_not_ready"); - if (!notReady || Date.now() > deadline) throw error; - log("image still preparing in the registry, waiting"); - await new Promise((resolve) => setTimeout(resolve, 15_000)); +async function provisionFromImage( + box: string, + mounts?: Parameters[0]["mounts"], + boxClaim: Parameters[0]["claim"] = claim, +): Promise { + const deadline = Date.now() + 20 * 60_000; + for (;;) { + try { + await provisionSandbox({ + name: box, + environment: { + sourceKind: "image", + sourceRef: SANDBOX_IMAGE_NAME, + region: DEFAULT_SANDBOX_REGION, + }, + claim: boxClaim, + mounts, + }); + return; + } catch (error) { + const notReady = + String(error).includes("409") || + String(error).includes("image_not_ready"); + if (!notReady || Date.now() > deadline) throw error; + log("image still preparing in the registry, waiting"); + await new Promise((resolve) => setTimeout(resolve, 15_000)); + } } } -const woken = await wakeSandbox({ providerSandboxId: name, claim }); -if (!woken) throw new Error(`${name} did not wake`); + log( - `host-service answered at ${woken.hostTarget} (${Date.now() - started} ms since start)`, + `provisioning ${name} from ${SANDBOX_IMAGE_NAME} on bundle ${bundle.sha256.slice(0, 12)}`, ); -let failed = await probeBox({ - name, - credentials, - hostSecret, - bundleSha: bundle.sha256, - branch: workingBranch, - primaryPath: ".", - gate: process.env.SANDBOX_GATE_ORIGIN - ? { workspaceId, userId: randomUUID() } - : null, - log, -}); -{ +const withDrivePass = process.env.REAL_SANDBOX_DRIVE === "1"; +const keep = process.env.REAL_SANDBOX_KEEP === "1"; +let failed = 0; +try { + await provisionFromImage(name); + const woken = await wakeSandbox({ providerSandboxId: name, claim }); + if (!woken) throw new Error(`${name} did not wake`); + log( + `host-service answered at ${woken.hostTarget} (${Date.now() - started} ms since start)`, + ); + failed += await probeBox({ + name, + credentials, + hostSecret, + bundleSha: bundle.sha256, + branch: workingBranch, + primaryPath: ".", + gate: process.env.SANDBOX_GATE_ORIGIN + ? { workspaceId, userId: randomUUID() } + : null, + log, + }); const box = await Sandbox.get({ ...credentials, name }); await box.stop(); const wakeStarted = Date.now(); await wakeSandbox({ providerSandboxId: name, claim }); log(`woken again in ${Date.now() - wakeStarted} ms`); failed += await checkWakeLog({ name, credentials, log }); +} catch (error) { + console.error( + `check of ${name} stopped early: ${String(error).slice(0, 300)}`, + ); + failed++; } -if (failed) { - console.error(`${failed} check(s) failed; ${name} left for inspection`); +if (keep) { + await stopSandbox(name) + .then(() => log(`${name} stopped and kept (REAL_SANDBOX_KEEP=1)`)) + .catch((error) => log(`could not stop ${name}: ${error}`)); +} else { + await stopSandbox(name) + .then(() => deleteSandbox(name)) + .then(() => log(`${name} deleted`)) + .catch((error) => log(`could not delete ${name}: ${error}`)); +} + +const driveFailures = withDrivePass ? await drivePass() : 0; +if (failed || driveFailures) { + console.error( + `${failed} check(s) and ${driveFailures} drive check(s) failed`, + ); process.exit(1); } -await deleteSandbox(name); -log(`${name} deleted; all checks passed`); +log("all checks passed"); process.exit(0); + +async function drivePass(): Promise { + const { + forkWorkspaceDrive, + vercelDrives, + waitForDriveDetached, + workspaceDriveIdentity, + workspaceDriveMounts, + goldenDriveNameFor, + workspaceDriveNameFor, + } = await import("@superset/trpc/lib/sandbox/drives"); + + const drives = vercelDrives(); + const workGb = Number(process.env.REAL_SANDBOX_WORK_GB) || 4; + const suffix = Date.now().toString(36); + const goldenDrive = `env-drive-real-check-${suffix}`; + const box = `ws-real-drive-${suffix}`; + const plainBox = `ws-real-plain-${suffix}`; + const goldenName = `env-real-check-${suffix}`; + const goldenBoxDrive = goldenDriveNameFor(goldenName); + const forkBox = `ws-real-golden-${suffix}`; + const boxDrive = workspaceDriveNameFor(box); + const forkDrive = workspaceDriveNameFor(forkBox); + const mounts = workspaceDriveMounts(boxDrive); + const claimWith = (driveName: string | null, laidOut: boolean) => ({ + ...claim, + identity: { + ...identity, + ...workspaceDriveIdentity(driveName, { laidOut }), + }, + }); + const drive = SANDBOX_PATHS.drive; + const expectations = [ + "sandbox.conf names the drive", + "layout stamped", + "host-service ready", + "/workspace is a bind mount of the drive", + "the checkout landed on the drive", + "docker info reports the drive data-root", + "the checkout is still there after a wake", + "the layout was redone on the wake", + "/workspace is a mount point again", + "bun install on the plain box", + "bun install on the drive box", + ...SENTINELS.map((name) => `plain box kept the ${name}`), + ...SENTINELS.map((name) => `drive box kept the ${name}`), + "the drive box's root snapshot is smaller than the plain box's", + "unclean stop: the box boots again", + "unclean stop: host.db passes integrity_check", + "unclean stop: the checkout is intact", + "a workspace of a drive-mode golden boots on its new branch", + "a workspace of a drive-mode golden has the golden's dependencies", + ]; + const reported = new Set(); + let failures = 0; + const check = (label: string, ok: boolean, detail = "") => { + reported.add(label); + log( + `${ok ? "ok " : "FAIL"} drive: ${label}${detail ? `: ${detail.replaceAll("\n", " | ")}` : ""}`, + ); + if (!ok) failures++; + }; + // Only stdout is compared: sudo on the box warns on stderr that it cannot resolve the hostname. + const runOn = + (name: string): BoxRun => + async (command, options = {}) => { + const sandbox = await Sandbox.get({ ...credentials, name }); + const script = Buffer.from(command).toString("base64"); + const result = await sandbox.runCommand({ + cmd: "bash", + args: options.root + ? ["-lc", command] + : [ + "-c", + `echo ${script} | base64 -d | runuser -u ${SANDBOX_USER} -- bash -l`, + ], + sudo: true, + }); + const out = (await result.stdout()).trim(); + const err = (await result.stderr()).trim(); + return { + out, + detail: [out, err && `stderr: ${err}`, `exit=${result.exitCode}`] + .filter(Boolean) + .join("\n"), + }; + }; + const run = runOn(box); + const root = (command: string) => run(command, { root: true }); + const layouts = async () => + Number( + ( + await root( + `grep -c 'drive\\.layout ms=' ${SANDBOX_PATHS.bootLog} || true`, + ) + ).out, + ); + const diagnose = async (when: string, on: BoxRun = run) => { + const report = await on( + [ + `echo '--- findmnt ${drive} and /workspace'`, + `findmnt -no SOURCE,MAJ:MIN,FSROOT,OPTIONS ${drive} ${SANDBOX_PATHS.workspace}`, + `echo '--- ls ${drive} ${drive}/workspace ${drive}/state'`, + `ls -la ${drive} ${drive}/workspace ${drive}/state 2>&1 | head -60`, + `echo '--- ${SANDBOX_PATHS.conf} drive keys'`, + `grep SUPERSET_SANDBOX_DRIVE ${SANDBOX_PATHS.conf}`, + `echo '--- run dir'`, + `ls ${SANDBOX_PATHS.run}`, + `echo '--- /etc/docker/daemon.json'`, + `cat /etc/docker/daemon.json`, + `echo '--- df ${drive}'`, + `df -h ${drive} ${SANDBOX_PATHS.workspace} 2>&1`, + `echo '--- git status of /workspace'`, + `runuser -u ${SANDBOX_USER} -- git -C ${SANDBOX_PATHS.workspace} status --short --branch 2>&1 | head -20`, + `echo '--- checkout.log'`, + `tail -30 ${SANDBOX_PATHS.logs}/checkout.log`, + `echo '--- boot log'`, + `cat ${SANDBOX_PATHS.bootLog}`, + ].join("; "), + { root: true }, + ).catch((error) => ({ detail: `diagnosis failed: ${error}` })); + log(`drive: diagnosis ${when}:\n${report.detail}`); + }; + + const stopMeasured = async (name: string): Promise => { + const before = (await Sandbox.get({ ...credentials, name, resume: false })) + .currentSnapshotId; + const stopStarted = Date.now(); + await stopSandbox(name); + while (Date.now() - stopStarted < 10 * 60_000) { + const now = await Sandbox.get({ ...credentials, name, resume: false }); + if ( + now.status === "stopped" && + now.currentSnapshotId && + now.currentSnapshotId !== before + ) { + const ms = Date.now() - stopStarted; + const snapshot = await Snapshot.get({ + ...credentials, + snapshotId: now.currentSnapshotId, + }); + return { ms, snapshotBytes: snapshot.sizeBytes }; + } + await new Promise((resolve) => setTimeout(resolve, 1000)); + } + throw new Error(`${name} did not snapshot within 10 minutes of its stop`); + }; + + const measures: Record<"plain" | "drive" | "golden", BoxMeasures> = { + plain: {}, + drive: {}, + golden: {}, + }; + const token = randomUUID().replaceAll("-", ""); + + /** Create, work, stop, wake, read the sentinels back, stop again. */ + const measureBox = async (args: { + name: string; + label: "plain" | "drive"; + provisionClaim: typeof claim; + wakeClaim: typeof claim; + mounts?: typeof mounts; + afterFirstBoot?: () => Promise; + beforeStop?: () => Promise; + afterWake?: () => Promise; + }) => { + const m = measures[args.label]; + const boxRun = runOn(args.name); + const createStarted = Date.now(); + await provisionFromImage(args.name, args.mounts, args.provisionClaim); + await wakeSandbox({ + providerSandboxId: args.name, + claim: args.provisionClaim, + mounts: args.mounts, + }); + m.createToReadyMs = Date.now() - createStarted; + if (!(await waitForCheckout(boxRun))) { + throw new Error(`${args.name}: the checkout never became ready`); + } + await args.afterFirstBoot?.(); + const work = await doWork(boxRun, workGb); + m.installMs = work.installMs; + m.writeMs = work.writeMs; + check( + `bun install on the ${args.label} box`, + work.installed === "installed", + work.installed, + ); + const values = await writeSentinels(boxRun, token); + await args.beforeStop?.(); + m.firstStop = await stopMeasured(args.name); + log(`drive: ${args.label} box stopped once`); + const phases: Partial> = {}; + const wakeStarted = Date.now(); + await wakeSandbox({ + providerSandboxId: args.name, + claim: args.wakeClaim, + mounts: args.mounts, + onPhase: (phase, ms) => { + phases[phase] = (phases[phase] ?? 0) + ms; + }, + }); + m.wake = { + detachMs: phases.detach, + mountMs: phases.mount, + resumeMs: phases.resume, + bootMs: phases.boot, + totalMs: Date.now() - wakeStarted, + }; + m.usableMs = await usableSince(boxRun, wakeStarted); + m.sentinels = await readSentinels(boxRun, values); + for (const name of SENTINELS) { + check(`${args.label} box kept the ${name}`, m.sentinels[name] === true); + } + await args.afterWake?.(); + m.secondStop = await stopMeasured(args.name); + log(`drive: ${args.label} box stopped twice`); + }; + + const stoppedAt: string[] = []; + const plain = measureBox({ + name: plainBox, + label: "plain", + provisionClaim: claim, + wakeClaim: claim, + }).catch((error) => { + stoppedAt.push(`plain box: ${String(error).slice(0, 300)}`); + }); + const driveBox = (async () => { + const forkStarted = Date.now(); + await drives.create(goldenDrive, DEFAULT_SANDBOX_REGION); + await forkWorkspaceDrive({ + name: boxDrive, + region: DEFAULT_SANDBOX_REGION, + goldenDriveName: goldenDrive, + drives, + }); + measures.drive.forkMs = Date.now() - forkStarted; + log(`drive: ${boxDrive} forked from ${goldenDrive}; provisioning ${box}`); + let before = 0; + await measureBox({ + name: box, + label: "drive", + provisionClaim: claimWith(boxDrive, false), + wakeClaim: claimWith(boxDrive, true), + mounts, + afterFirstBoot: async () => { + const conf = await root( + `grep -c "^SUPERSET_SANDBOX_DRIVE='${boxDrive}'$" ${SANDBOX_PATHS.conf}`, + ); + check("sandbox.conf names the drive", conf.out === "1", conf.detail); + const bootLog = (await root(`cat ${SANDBOX_PATHS.bootLog}`)).out; + check( + "layout stamped", + /drive\.layout ms=\d+/.test(bootLog), + bootLog.match(/drive\.(layout|missing|foreign|unlaid).*/)?.[0], + ); + check("host-service ready", /host\.ready/.test(bootLog)); + // A bind mount of a subdirectory: the drive's device, at fsroot /workspace. + const backing = await root( + `findmnt -rno MAJ:MIN ${drive}; findmnt -rno MAJ:MIN,FSROOT ${SANDBOX_PATHS.workspace}`, + ); + const [device, workspaceMount] = backing.out.split("\n"); + check( + "/workspace is a bind mount of the drive", + Boolean(device) && workspaceMount === `${device} /workspace`, + backing.detail, + ); + const checkout = await root( + `test -d ${drive}/workspace/.git && echo on-drive`, + ); + check( + "the checkout landed on the drive", + checkout.out === "on-drive", + checkout.detail, + ); + const dockerRoot = await root( + `(dockerd > /tmp/dockerd.log 2>&1 &); for i in $(seq 1 60); do docker info --format '{{.DockerRootDir}}' 2>/dev/null && exit 0; sleep 1; done; tail -5 /tmp/dockerd.log`, + ); + check( + "docker info reports the drive data-root", + dockerRoot.out === `${drive}/docker`, + dockerRoot.detail, + ); + if (failures) await diagnose("after the first boot"); + }, + beforeStop: async () => { + before = await layouts(); + }, + afterWake: async () => { + const kept = await run( + `git -C ${SANDBOX_PATHS.workspace} rev-parse --is-inside-work-tree`, + ); + check( + "the checkout is still there after a wake", + kept.out === "true", + kept.detail, + ); + const after = await layouts(); + check( + "the layout was redone on the wake", + after > before, + `${before} layout stamps before the wake, ${after} after`, + ); + const mounted = await root( + `mountpoint ${SANDBOX_PATHS.workspace} && echo yes`, + ); + check( + "/workspace is a mount point again", + mounted.out.endsWith("yes"), + mounted.detail, + ); + }, + }); + })().catch((error) => { + stoppedAt.push(`drive box: ${String(error).slice(0, 300)}`); + return "failed" as const; + }); + const [, driveOutcome] = await Promise.all([plain, driveBox]); + + if (driveOutcome !== "failed") { + // An unclean stop: host-service killed, writers into host.db and /workspace still going. + try { + await wakeSandbox({ + providerSandboxId: box, + claim: claimWith(boxDrive, true), + mounts, + }); + const writer = Buffer.from( + [ + 'const { Database } = require("bun:sqlite");', + 'process.on("SIGTERM", () => {});', + "const db = new Database(process.argv[2]);", + 'db.exec("pragma journal_mode=wal; create table if not exists superset_probe (i integer, at integer)");', + 'for (let i = 0; ; i++) db.run("insert into superset_probe values (?, ?)", [i, Date.now()]);', + ].join("\n"), + ).toString("base64"); + await run( + `echo ${writer} | base64 -d > /tmp/superset-probe-writer.js && (setsid nohup bun /tmp/superset-probe-writer.js ${SANDBOX_PATHS.driveState}/host.db > /tmp/superset-probe-writer.log 2>&1 &) && (setsid nohup bash -c "trap '' TERM; while true; do head -c 1048576 /dev/urandom >> ${SANDBOX_PATHS.workspace}/.superset-unclean; done" > /dev/null 2>&1 &)`, + ); + await root("sleep 3; pkill -9 -f host-service.js; true"); + await stopMeasured(box); + await wakeSandbox({ + providerSandboxId: box, + claim: claimWith(boxDrive, true), + mounts, + }); + check("unclean stop: the box boots again", true); + const integrity = await run( + `bun -e 'const { Database } = require("bun:sqlite"); console.log(Object.values(new Database(process.argv[1], { readonly: true }).query("pragma integrity_check").get() ?? {})[0])' ${SANDBOX_PATHS.driveState}/host.db`, + ); + check( + "unclean stop: host.db passes integrity_check", + integrity.out === "ok", + integrity.detail, + ); + const checkout = await run( + `git -C ${SANDBOX_PATHS.workspace} fsck --connectivity-only --no-progress > /dev/null && git -C ${SANDBOX_PATHS.workspace} status --porcelain > /dev/null && echo intact`, + ); + check( + "unclean stop: the checkout is intact", + checkout.out === "intact", + checkout.detail, + ); + } catch (error) { + stoppedAt.push(`unclean stop: ${String(error).slice(0, 300)}`); + } + + // The production path: promote the drive box to a drive-mode golden, then a workspace of it. + try { + const golden = await promoteSandboxToEnvironment({ + sourceSandbox: box, + goldenName, + organizationId: identity.SUPERSET_SANDBOX_ORGANIZATION_ID, + sourceDriveName: boxDrive, + drives, + }); + await stopSandbox(box); + const forkWorkspaceId = randomUUID(); + const forkBranch = `superset/real-check-golden-${forkWorkspaceId.slice(0, 8)}`; + const forkClaim = { + ...claim, + hostSecret: await sandboxHostSecretFor(forkWorkspaceId), + identity: { + ...identity, + SUPERSET_SANDBOX_WORKSPACE_ID: forkWorkspaceId, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { + url: "https://github.com/paradevai/paradev.git", + branch: forkBranch, + baseBranch: "main", + path: ".", + }, + ]), + SUPERSET_SANDBOX_IMAGE_TAG: goldenName, + ...workspaceDriveIdentity(forkDrive, { laidOut: false }), + }, + }; + const forkMounts = workspaceDriveMounts(forkDrive); + const createStarted = Date.now(); + await forkWorkspaceDrive({ + name: forkDrive, + region: golden.region, + goldenDriveName: golden.goldenDriveName, + drives, + }); + measures.golden.forkMs = Date.now() - createStarted; + await provisionSandbox({ + name: forkBox, + environment: { + sourceKind: "fork", + sourceRef: goldenName, + region: golden.region, + }, + claim: forkClaim, + mounts: forkMounts, + }); + await wakeSandbox({ + providerSandboxId: forkBox, + claim: forkClaim, + mounts: forkMounts, + }); + measures.golden.createToReadyMs = Date.now() - createStarted; + const forkRun = runOn(forkBox); + measures.golden.usableMs = await usableSince(forkRun, createStarted); + await waitForCheckout(forkRun); + const branch = await forkRun( + `git -C ${SANDBOX_PATHS.workspace} rev-parse --abbrev-ref HEAD`, + ); + check( + "a workspace of a drive-mode golden boots on its new branch", + branch.out === forkBranch, + branch.detail, + ); + const deps = await forkRun( + `test -d ${SANDBOX_PATHS.workspace}/node_modules && echo present`, + ); + check( + "a workspace of a drive-mode golden has the golden's dependencies", + deps.out === "present", + deps.detail, + ); + if (deps.out !== "present") { + await diagnose("of the workspace of the drive-mode golden", forkRun); + } + } catch (error) { + stoppedAt.push(`golden path: ${String(error).slice(0, 300)}`); + } + } + const plainSnapshot = measures.plain.firstStop?.snapshotBytes; + const driveSnapshot = measures.drive.firstStop?.snapshotBytes; + if (plainSnapshot !== undefined && driveSnapshot !== undefined) { + check( + "the drive box's root snapshot is smaller than the plain box's", + driveSnapshot < plainSnapshot, + `${(driveSnapshot / 1024 ** 3).toFixed(2)} GB against ${(plainSnapshot / 1024 ** 3).toFixed(2)} GB`, + ); + } + for (const label of expectations) { + if (!reported.has(label)) { + check( + label, + false, + `not reached${stoppedAt.length ? `: ${stoppedAt.join("; ")}` : ""}`, + ); + } + } + if (failures) await diagnose("at the end"); + log( + `drive: comparison, ${workGb} GB written under /workspace\n${renderComparison( + [ + { label: "plain", measures: measures.plain }, + { label: "drive", measures: measures.drive }, + { label: "workspace of a drive golden", measures: measures.golden }, + ], + )}`, + ); + + const cleanups: Array<[string, () => Promise]> = keep + ? [ + [plainBox, () => stopSandbox(plainBox)], + [box, () => stopSandbox(box)], + [forkBox, () => stopSandbox(forkBox)], + [goldenName, () => stopSandbox(goldenName)], + ] + : [ + [ + plainBox, + () => stopSandbox(plainBox).then(() => deleteSandbox(plainBox)), + ], + [ + `${box} and ${boxDrive}`, + () => deleteSandboxAndDrive(box, boxDrive, drives), + ], + [ + `${forkBox} and ${forkDrive}`, + () => deleteSandboxAndDrive(forkBox, forkDrive, drives), + ], + [ + `${goldenName} and ${goldenBoxDrive}`, + () => deleteSandboxAndDrive(goldenName, goldenBoxDrive, drives), + ], + ]; + for (const [what, cleanup] of cleanups) { + await cleanup() + .then(() => + log( + `drive: ${what} ${keep ? "stopped and kept (REAL_SANDBOX_KEEP=1)" : "deleted"}`, + ), + ) + .catch((error) => { + failures++; + log(`FAIL drive: could not clean up ${what}: ${error}`); + }); + } + await waitForDriveDetached(goldenDrive, { drives }) + .then(() => drives.remove(goldenDrive)) + .then(() => log(`drive: ${goldenDrive} deleted`)) + .catch((error) => { + failures++; + log(`FAIL drive: could not delete ${goldenDrive}: ${error}`); + }); + return failures; +} diff --git a/packages/shared/src/constants.ts b/packages/shared/src/constants.ts index 76cdb18e648..07eb433703b 100644 --- a/packages/shared/src/constants.ts +++ b/packages/shared/src/constants.ts @@ -206,6 +206,13 @@ export const FEATURE_FLAGS = { SANDBOX_BOX_LOGS: "sandbox-box-logs", /** One answer for every box (asked for the subject `sandbox-fleet`, no group): off stops box logs at each box's next wake. */ SANDBOX_BOX_LOGS_FLEET: "sandbox-box-logs-fleet", + /** + * Lets a golden build or a promote put an organization's environment in + * drive mode (`environments.golden_drive_name`). Evaluated per organization. + * Workspaces follow their environment, never this flag. Unloaded or + * offline means no drive. + */ + CLOUD_WORKSPACE_DRIVES: "cloud-workspace-drives", /** * Moving a workspace and its chats to the cloud or to another computer: * shows the Move items, and gates diff --git a/packages/shared/src/sandbox/contract.ts b/packages/shared/src/sandbox/contract.ts index a383b7579a5..2334b528a99 100644 --- a/packages/shared/src/sandbox/contract.ts +++ b/packages/shared/src/sandbox/contract.ts @@ -37,8 +37,56 @@ export const SANDBOX_PATHS = { backgrounds: "/usr/share/backgrounds/superset", /** A read-only snapshot of the region's agent-tools drive, when one is mounted. */ agents: "/opt/agents", + /** The workspace's own drive, read-write, when its environment is in drive mode. */ + drive: "/mnt/superset-drive", + /** + * In drive mode, what describes the drive's contents lives on the drive, so + * a root disk that rolls back cannot disagree with it: `host.db`, the + * checkout markers and `workspace-id`, the drive's owner. + */ + driveState: "/mnt/superset-drive/state", } as const; +/** + * Home subtrees that live on the workspace drive, relative to the home + * directory. Each is bind-mounted from `/home/`. + */ +export const SANDBOX_DRIVE_HOME_SUBTREES = [ + ".cache", + ".local/share", + ".local/state", + ".npm", + ".cargo/registry", + ".rustup", + ".claude", + ".codex/sessions", + "go/pkg", +] as const; + +/** Package stores inside the `/workspace` bind mount: a hardlink across two mounts fails with EXDEV. */ +export const SANDBOX_DRIVE_STORES = { + bun: `${SANDBOX_PATHS.workspace}/.superset/stores/bun`, + pnpm: `${SANDBOX_PATHS.workspace}/.superset/stores/pnpm`, +} as const; + +/** + * Where a path of the laid-out box lives on the drive while no boot has + * bind-mounted it; null for a path on the root disk. + */ +export function sandboxDrivePathOf(path: string): string | null { + const workspace = SANDBOX_PATHS.workspace; + if (path === workspace || path.startsWith(`${workspace}/`)) { + return `${SANDBOX_PATHS.drive}/workspace${path.slice(workspace.length)}`; + } + for (const subtree of SANDBOX_DRIVE_HOME_SUBTREES) { + const mounted = `${SANDBOX_PATHS.home}/${subtree}`; + if (path === mounted || path.startsWith(`${mounted}/`)) { + return `${SANDBOX_PATHS.drive}/home${path.slice(SANDBOX_PATHS.home.length)}`; + } + } + return null; +} + /** * The agent CLIs of one bundle in one region: written once by the release, * then only ever mounted read-only. A drive only mounts in its own region. @@ -111,6 +159,10 @@ export const sandboxIdentitySchema = z.object({ /** The environment row's source (image name or golden), for telemetry. */ SUPERSET_SANDBOX_IMAGE_TAG: z.string().min(1), SUPERSET_SANDBOX_PROVIDER: z.string().min(1), + /** The workspace drive's name; the box refuses to boot when it is not mounted. */ + SUPERSET_SANDBOX_DRIVE: z.string().min(1).optional(), + /** Set after the provision: the drive must already carry this workspace's id. */ + SUPERSET_SANDBOX_DRIVE_LAID_OUT: z.literal("1").optional(), HOST_SERVICE_SENTRY_DSN: z.string().optional(), HOST_SERVICE_SENTRY_ENVIRONMENT: z.string().optional(), /** Starts `superset-logship`, which sends the platform's box logs to Sentry. Absent is off. */ @@ -239,6 +291,11 @@ export function renderContractShell(): string { SUPERSET_DESKTOP_TEMPLATES: SANDBOX_PATHS.desktopTemplates, SUPERSET_BACKGROUNDS_DIR: SANDBOX_PATHS.backgrounds, SUPERSET_AGENTS_DIR: SANDBOX_PATHS.agents, + SUPERSET_DRIVE_DIR: SANDBOX_PATHS.drive, + SUPERSET_DRIVE_STATE_DIR: SANDBOX_PATHS.driveState, + SUPERSET_DRIVE_HOME_SUBTREES: SANDBOX_DRIVE_HOME_SUBTREES.join(" "), + SUPERSET_BUN_STORE: SANDBOX_DRIVE_STORES.bun, + SUPERSET_PNPM_STORE: SANDBOX_DRIVE_STORES.pnpm, SUPERSET_HOST_SERVICE_PORT: SANDBOX_PORTS.hostService, SUPERSET_DESKTOP_PORT: SANDBOX_PORTS.desktop, SUPERSET_VNC_PORT: SANDBOX_PORTS.vnc, diff --git a/packages/trpc/package.json b/packages/trpc/package.json index c444ba70db4..58e052f41e0 100644 --- a/packages/trpc/package.json +++ b/packages/trpc/package.json @@ -136,6 +136,10 @@ "types": "./src/router/cloud-workspace/reap.ts", "default": "./src/router/cloud-workspace/reap.ts" }, + "./cloud-workspace-teardown": { + "types": "./src/router/cloud-workspace/teardown.ts", + "default": "./src/router/cloud-workspace/teardown.ts" + }, "./automation-dispatch": { "types": "./src/router/automation/dispatch.ts", "default": "./src/router/automation/dispatch.ts" diff --git a/packages/trpc/src/lib/sandbox/claim.test.ts b/packages/trpc/src/lib/sandbox/claim.test.ts new file mode 100644 index 00000000000..e8b57417791 --- /dev/null +++ b/packages/trpc/src/lib/sandbox/claim.test.ts @@ -0,0 +1,112 @@ +import { describe, expect, test } from "bun:test"; +import { stub } from "../../../test/stub"; +import * as environments from "../../router/environment/resolve-environment"; +import * as plan from "../billing/plan"; +import * as featureFlags from "../feature-flags"; +import * as access from "./access"; +import { buildSandboxClaim } from "./claim"; +import * as credentials from "./credentials"; +import * as drives from "./drives"; +import * as plugins from "./plugins"; +import * as repositories from "./repositories"; + +let goldenDriveName: string | null = null; +let drivesOn = true; + +stub(environments, { + resolveEnvironment: async () => ({ + id: "env-1", + provider: "vercel", + sourceKind: "fork", + sourceRef: "env-golden", + region: "iad1", + bundleSha: "a".repeat(64), + primaryRepositoryId: null, + goldenDriveName, + envs: {}, + }), +}); +stub(repositories, { + workspaceRepositories: async () => [], + installationTokenFor: async () => null, +}); +stub(plugins, { creatorPlugins: async () => [] }); +stub(access, { sandboxHostSecretFor: async () => "secret" }); +stub(featureFlags, { + organizationFlagEnabled: async () => false, + boxLogsEnabled: async () => false, +}); +stub(credentials, { + deriveSandboxCredentials: async () => ({ + networkPolicy: "allow-all", + managedEnv: {}, + files: [], + }), +}); +stub(plan, { resolveMachineSize: async () => ({ vcpus: 4 }) }); +stub(drives, { workspaceDrivesEnabled: async () => drivesOn }); + +const row = (driveName: string | null, driveLaidOutAt: Date | null = null) => + ({ + id: "00000000-0000-4000-8000-000000000001", + organizationId: "00000000-0000-4000-8000-000000000002", + environmentId: "00000000-0000-4000-8000-000000000003", + createdByUserId: null, + providerSandboxId: "ws-1", + provider: "vercel", + baseBranch: "main", + branch: "superset/ws-1", + machineSize: null, + driveName, + driveLaidOutAt, + }) as unknown as Parameters[0]["row"]; + +describe("buildSandboxClaim drive", () => { + test("an existing workspace gets no drive on a wake when its environment enters drive mode", async () => { + goldenDriveName = "env-drive-env-golden"; + const { claim } = await buildSandboxClaim({ row: row(null) }); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE).toBeUndefined(); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT).toBeUndefined(); + }); + + test("a provision in drive mode claims a new drive, not yet laid out", async () => { + goldenDriveName = "env-drive-env-golden"; + const { claim } = await buildSandboxClaim({ + row: row(null), + provisioning: true, + }); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE).toBe("drive-ws-1"); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT).toBeUndefined(); + }); + + test("a wake expects the drive laid out only once the row recorded a boot that laid it out", async () => { + goldenDriveName = null; + const laidOut = await buildSandboxClaim({ + row: row("drive-ws-1", new Date()), + }); + expect(laidOut.claim.identity.SUPERSET_SANDBOX_DRIVE).toBe("drive-ws-1"); + expect(laidOut.claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT).toBe("1"); + const never = await buildSandboxClaim({ row: row("drive-ws-1") }); + expect(never.claim.identity.SUPERSET_SANDBOX_DRIVE).toBe("drive-ws-1"); + expect( + never.claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT, + ).toBeUndefined(); + }); + + test("with the organization's flag off a new workspace gets no drive; one that has a drive keeps it", async () => { + goldenDriveName = "env-drive-env-golden"; + drivesOn = false; + const fresh = await buildSandboxClaim({ + row: row(null), + provisioning: true, + }); + expect(fresh.driveName).toBeNull(); + expect(fresh.claim.identity.SUPERSET_SANDBOX_DRIVE).toBeUndefined(); + const unarchived = await buildSandboxClaim({ + row: row("drive-ws-1"), + provisioning: true, + }); + expect(unarchived.driveName).toBe("drive-ws-1"); + drivesOn = true; + }); +}); diff --git a/packages/trpc/src/lib/sandbox/claim.ts b/packages/trpc/src/lib/sandbox/claim.ts index c89ecfe46b6..4de5983b2cc 100644 --- a/packages/trpc/src/lib/sandbox/claim.ts +++ b/packages/trpc/src/lib/sandbox/claim.ts @@ -35,6 +35,7 @@ import { boxLogsEnabled, organizationFlagEnabled } from "../feature-flags"; import { githubUserConnectionFor, githubUserTokenFor } from "../github-user"; import { sandboxHostSecretFor } from "./access"; import { deriveSandboxCredentials, gitAuthorFor } from "./credentials"; +import { claimedWorkspaceDrive, workspaceDrivesEnabled } from "./drives"; import { creatorPlugins } from "./plugins"; import { readRepoHooks } from "./repo-hooks"; import { @@ -87,11 +88,15 @@ export async function buildSandboxClaim(args: { * are fixed once the box exists), and it costs a GitHub request. */ withRepoHooks?: boolean; + /** A create, the only claim that may give the workspace its environment's drive. */ + provisioning?: boolean; }): Promise<{ claim: SandboxClaim; environment: SandboxEnvironment; repositories: SandboxRepository[]; agentCredentialDigest: string; + /** The drive the claim names; what a provision must fork and mount. */ + driveName: string | null; }> { const [environment, signIns, boxLogs] = await Promise.all([ resolveEnvironment(args.row.environmentId, args.row.organizationId), @@ -137,6 +142,18 @@ export async function buildSandboxClaim(args: { : null; const repositories = toSandboxRepositories(checkouts); + const provisioning = args.provisioning ?? false; + const drive = claimedWorkspaceDrive({ + row: args.row, + goldenDriveName: environment.goldenDriveName, + provisioning, + // The kill switch: with the flag off, a new workspace gets no drive. + drivesEnabled: + provisioning && + !args.row.driveName && + Boolean(environment.goldenDriveName) && + (await workspaceDrivesEnabled(args.row.organizationId)), + }); const identity: SandboxIdentity = { SUPERSET_SANDBOX_CONTRACT: String(SANDBOX_CONTRACT_VERSION) as "1", SUPERSET_API_URL: env.NEXT_PUBLIC_API_URL, @@ -147,6 +164,7 @@ export async function buildSandboxClaim(args: { SUPERSET_SANDBOX_PLUGINS: JSON.stringify(plugins), SUPERSET_SANDBOX_IMAGE_TAG: environment.sourceRef, SUPERSET_SANDBOX_PROVIDER: args.row.provider, + ...drive.identity, SUPERSET_BUNDLE_SHA: environment.bundleSha, ...(env.SENTRY_DSN_SANDBOX ? { @@ -194,8 +212,10 @@ export async function buildSandboxClaim(args: { sourceKind: environment.sourceKind, sourceRef: environment.sourceRef, region: environment.region, + goldenDriveName: environment.goldenDriveName, }, repositories, agentCredentialDigest: agentCredentialDigest(signIns), + driveName: drive.driveName, }; } diff --git a/packages/trpc/src/lib/sandbox/drives.test.ts b/packages/trpc/src/lib/sandbox/drives.test.ts new file mode 100644 index 00000000000..5ada5f501b8 --- /dev/null +++ b/packages/trpc/src/lib/sandbox/drives.test.ts @@ -0,0 +1,212 @@ +import { beforeEach, describe, expect, test } from "bun:test"; +import { renderSandboxConf } from "@superset/shared/sandbox/contract"; +import { + type DriveHost, + DriveStillAttachedError, + deleteWorkspaceDrive, + forkWorkspaceDrive, + WORKSPACE_DRIVE_MOUNT_PATH, + waitForDriveDetached, + workspaceDriveIdentity, + workspaceDriveMounts, + workspaceDriveNameToClaim, +} from "./drives"; + +let drives: Map; +let calls: string[]; + +const host: DriveHost = { + find: async (name) => drives.get(name) ?? null, + create: async (name, region) => { + calls.push(`create:${name}@${region}`); + drives.set(name, { region }); + }, + fork: async (source, name) => { + calls.push(`fork:${source}->${name}`); + const parent = drives.get(source); + if (!parent) throw new Error(`no ${source}`); + drives.set(name, { region: parent.region }); + }, + remove: async (name) => { + if (drives.get(name)?.attachedTo) throw new Error(`${name} attached`); + if (drives.delete(name)) calls.push(`delete:${name}`); + }, +}; + +beforeEach(() => { + drives = new Map(); + calls = []; +}); + +describe("workspaceDriveMounts", () => { + test("mounts the drive read-write at one path", () => { + expect(workspaceDriveMounts("drive-ws-1")).toEqual({ + [WORKSPACE_DRIVE_MOUNT_PATH]: { drive: "drive-ws-1", mode: "read-write" }, + }); + }); + + test("mounts nothing for a workspace without a drive", () => { + expect(workspaceDriveMounts(null)).toBeUndefined(); + expect(workspaceDriveMounts(undefined)).toBeUndefined(); + }); +}); + +describe("workspaceDriveIdentity", () => { + const identity = { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "a".repeat(64), + SUPERSET_API_URL: "https://api.example.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "image", + SUPERSET_SANDBOX_PROVIDER: "vercel", + } as const; + + test("a workspace with a drive names it in sandbox.conf, so the box can refuse without it", () => { + expect( + renderSandboxConf({ + ...identity, + ...workspaceDriveIdentity("drive-ws-1", { laidOut: true }), + }), + ).toContain( + "SUPERSET_SANDBOX_DRIVE='drive-ws-1'\nSUPERSET_SANDBOX_DRIVE_LAID_OUT='1'\n", + ); + }); + + test("a workspace without a drive writes no drive key", () => { + expect(workspaceDriveIdentity(null, { laidOut: true })).toEqual({}); + expect( + renderSandboxConf({ + ...identity, + ...workspaceDriveIdentity(undefined, { laidOut: false }), + }), + ).not.toContain("SUPERSET_SANDBOX_DRIVE"); + }); +}); + +describe("workspaceDriveNameToClaim", () => { + test("an environment in drive mode gives a new workspace a drive", () => { + expect( + workspaceDriveNameToClaim({ + existing: null, + goldenDriveName: "env-drive-env-1", + providerSandboxId: "ws-1", + }), + ).toBe("drive-ws-1"); + }); + + test("an environment without a golden drive gives no drive at all", () => { + expect( + workspaceDriveNameToClaim({ + existing: null, + goldenDriveName: null, + providerSandboxId: "ws-1", + }), + ).toBeNull(); + }); + + test("a re-delivered provision keeps the name already on the row", () => { + expect( + workspaceDriveNameToClaim({ + existing: "drive-ws-old", + goldenDriveName: "env-drive-env-1", + providerSandboxId: "ws-new", + }), + ).toBe("drive-ws-old"); + }); +}); + +describe("forkWorkspaceDrive", () => { + test("forks the golden drive", async () => { + drives.set("golden", { region: "iad1" }); + await forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: "golden", + drives: host, + }); + expect(calls).toEqual(["fork:golden->drive-ws-1"]); + }); + + test("finds the drive a previous delivery made", async () => { + drives.set("golden", { region: "iad1" }); + drives.set("drive-ws-1", { region: "iad1" }); + await forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: "golden", + drives: host, + }); + expect(calls).toEqual([]); + }); + + test("refuses a golden in another region", async () => { + drives.set("golden", { region: "iad1" }); + await expect( + forkWorkspaceDrive({ + name: "drive-ws-1", + region: "sfo1", + goldenDriveName: "golden", + drives: host, + }), + ).rejects.toThrow("is in iad1"); + expect(calls).toEqual([]); + }); + + test("never creates an empty drive", async () => { + await expect( + forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: null, + drives: host, + }), + ).rejects.toThrow("no golden drive"); + await expect( + forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: "gone", + drives: host, + }), + ).rejects.toThrow("does not exist"); + expect(calls).toEqual([]); + }); +}); + +describe("waitForDriveDetached", () => { + test("returns once the previous session lets go of the drive", async () => { + let polls = 0; + const flushing: DriveHost = { + ...host, + find: async () => { + polls++; + return { region: "iad1", attachedTo: polls < 3 ? "ws-1" : undefined }; + }, + }; + await waitForDriveDetached("drive-ws-1", { drives: flushing, pollMs: 1 }); + expect(polls).toBe(3); + }); + + test("fails rather than mount a drive another session still writes", async () => { + drives.set("drive-ws-1", { region: "iad1", attachedTo: "ws-1" }); + await expect( + waitForDriveDetached("drive-ws-1", { + drives: host, + timeoutMs: 5, + pollMs: 1, + }), + ).rejects.toBeInstanceOf(DriveStillAttachedError); + }); +}); + +describe("deleteWorkspaceDrive", () => { + test("deletes the drive, and treats one already gone as deleted", async () => { + drives.set("drive-ws-1", { region: "iad1" }); + await deleteWorkspaceDrive("drive-ws-1", host); + await deleteWorkspaceDrive("drive-ws-1", host); + await deleteWorkspaceDrive(null, host); + expect(calls).toEqual(["delete:drive-ws-1"]); + }); +}); diff --git a/packages/trpc/src/lib/sandbox/drives.ts b/packages/trpc/src/lib/sandbox/drives.ts new file mode 100644 index 00000000000..28b23a0dedc --- /dev/null +++ b/packages/trpc/src/lib/sandbox/drives.ts @@ -0,0 +1,250 @@ +import { FEATURE_FLAGS } from "@superset/shared/constants"; +import { + SANDBOX_PATHS, + type SandboxIdentity, +} from "@superset/shared/sandbox/contract"; +import { + APIError, + Drive, + type SandboxMounts, + type SandboxRegion, +} from "@vercel/sandbox"; +import { credentials } from "./vercel-credentials"; + +/** Vercel allows one read-write mount per drive, so the whole drive sits at one path. */ +export const WORKSPACE_DRIVE_MOUNT_PATH = SANDBOX_PATHS.drive; + +export function workspaceDriveNameFor(providerSandboxId: string): string { + return `drive-${providerSandboxId}`; +} + +/** + * Fork or nothing: a workspace has a drive exactly when its environment is + * in drive mode, because the golden's root disk was built without /workspace. + */ +export function workspaceDriveNameToClaim(args: { + existing: string | null; + goldenDriveName: string | null | undefined; + providerSandboxId: string; +}): string | null { + if (args.existing) return args.existing; + if (!args.goldenDriveName) return null; + return workspaceDriveNameFor(args.providerSandboxId); +} + +export function workspaceDriveIdentity( + driveName: string | null | undefined, + { laidOut }: { laidOut: boolean }, +): Pick< + SandboxIdentity, + "SUPERSET_SANDBOX_DRIVE" | "SUPERSET_SANDBOX_DRIVE_LAID_OUT" +> { + if (!driveName) return {}; + return { + SUPERSET_SANDBOX_DRIVE: driveName, + ...(laidOut ? { SUPERSET_SANDBOX_DRIVE_LAID_OUT: "1" as const } : {}), + }; +} + +/** + * The drive a claim names. Only a provision gives a workspace its + * environment's drive, and only while the organization's flag is on; a wake + * mounts the row's own drive or none, so a workspace made before its + * environment entered drive mode boots exactly as it did. A wake expects the + * drive laid out once the row recorded a boot that laid it out. + */ +export function claimedWorkspaceDrive(args: { + row: { + driveName: string | null; + driveLaidOutAt: Date | null; + providerSandboxId: string; + }; + goldenDriveName: string | null | undefined; + provisioning: boolean; + drivesEnabled: boolean; +}): { + driveName: string | null; + identity: ReturnType; +} { + if (!args.provisioning) { + return { + driveName: args.row.driveName, + identity: workspaceDriveIdentity(args.row.driveName, { + laidOut: args.row.driveLaidOutAt !== null, + }), + }; + } + const driveName = workspaceDriveNameToClaim({ + existing: args.row.driveName, + goldenDriveName: args.drivesEnabled ? args.goldenDriveName : null, + providerSandboxId: args.row.providerSandboxId, + }); + return { + driveName, + identity: workspaceDriveIdentity(driveName, { laidOut: false }), + }; +} + +export function goldenDriveNameFor(goldenName: string): string { + return `env-drive-${goldenName}`; +} + +/** Vercel does not carry mounts from one session to the next, so every create and wake passes these. */ +export function workspaceDriveMounts( + driveName: string | null | undefined, +): SandboxMounts | undefined { + if (!driveName) return undefined; + return { + [WORKSPACE_DRIVE_MOUNT_PATH]: { drive: driveName, mode: "read-write" }, + }; +} + +/** + * Whether a golden build or a promote may put an organization's environment + * in drive mode. Workspaces follow their environment's row, never this. + * Fails closed: PostHog unreachable means no drive. + */ +export async function workspaceDrivesEnabled( + organizationId: string, +): Promise { + // Lazily, so the release scripts that import this need no PostHog key. + const { posthog } = await import("../analytics"); + const enabled = await posthog + .isFeatureEnabled(FEATURE_FLAGS.CLOUD_WORKSPACE_DRIVES, organizationId, { + groups: { organization: organizationId }, + sendFeatureFlagEvents: false, + }) + .catch(() => undefined); + return enabled === true; +} + +/** What drive bookkeeping needs from the provider; `vercelDrives` is the real one. */ +export interface DriveHost { + /** Null when no drive has the name. */ + find(name: string): Promise<{ region: string; attachedTo?: string } | null>; + create(name: string, region: string): Promise; + fork(source: string, name: string): Promise; + /** Throws while the drive is attached to a sandbox. */ + remove(name: string): Promise; +} + +function isNotFound(error: unknown): boolean { + return error instanceof APIError && error.response.status === 404; +} + +export function vercelDrives(): DriveHost { + const get = async (name: string) => { + try { + return await Drive.get({ ...credentials(), name }); + } catch (error) { + if (isNotFound(error)) return null; + throw error; + } + }; + return { + async find(name) { + const drive = await get(name); + return drive + ? { region: drive.region, attachedTo: drive.currentSandboxName } + : null; + }, + async create(name, region) { + await Drive.getOrCreate({ + ...credentials(), + name, + region: region as SandboxRegion, + }); + }, + async fork(source, name) { + const parent = await Drive.get({ ...credentials(), name: source }); + await parent.fork({ name }); + }, + async remove(name) { + await (await get(name))?.delete(); + }, + }; +} + +/** + * The workspace's drive, forked from its environment's golden drive. Forks + * are copy-on-write and near instant. Idempotent on the name, so a + * re-delivered provision finds the drive it already made. + */ +export async function forkWorkspaceDrive(args: { + name: string; + region: string; + goldenDriveName: string | null; + drives?: DriveHost; +}): Promise<"existing" | "forked"> { + const drives = args.drives ?? vercelDrives(); + if (await drives.find(args.name)) return "existing"; + if (!args.goldenDriveName) { + throw new Error( + `Drive ${args.name} is missing and its environment has no golden drive to fork`, + ); + } + const golden = await drives.find(args.goldenDriveName); + if (!golden) { + throw new Error(`Golden drive ${args.goldenDriveName} does not exist`); + } + if (golden.region !== args.region) { + throw new Error( + `Golden drive ${args.goldenDriveName} is in ${golden.region}; the sandbox runs in ${args.region}`, + ); + } + await drives.fork(args.goldenDriveName, args.name); + return "forked"; +} + +/** + * A drive already gone is the state we wanted. Throws while the drive is + * still attached, so the caller stops its sandbox first and retries. + */ +export async function deleteWorkspaceDrive( + name: string | null | undefined, + drives: DriveHost = vercelDrives(), +): Promise { + if (!name) return; + await drives.remove(name); +} + +export const DRIVE_DETACH_TIMEOUT_MS = 2 * 60 * 1000; + +export class DriveStillAttachedError extends Error { + constructor( + readonly driveName: string, + readonly sandboxName: string, + ) { + super( + `Drive ${driveName} is still attached to ${sandboxName}; the previous session has not finished writing it`, + ); + this.name = "DriveStillAttachedError"; + } +} + +/** + * A session that stops keeps its drive attached while it flushes, and a mount + * taken meanwhile reads a corrupt or empty filesystem. + */ +export async function waitForDriveDetached( + driveName: string, + options: { + drives?: DriveHost; + timeoutMs?: number; + pollMs?: number; + /** Whether the sandbox holding the drive is the one about to mount it, already running. */ + heldByCaller?: (attachedTo: string) => Promise; + } = {}, +): Promise<"detached" | "held"> { + const drives = options.drives ?? vercelDrives(); + const deadline = Date.now() + (options.timeoutMs ?? DRIVE_DETACH_TIMEOUT_MS); + for (;;) { + const drive = await drives.find(driveName); + if (!drive?.attachedTo) return "detached"; + if (await options.heldByCaller?.(drive.attachedTo)) return "held"; + if (Date.now() >= deadline) { + throw new DriveStillAttachedError(driveName, drive.attachedTo); + } + await new Promise((resolve) => setTimeout(resolve, options.pollMs ?? 2000)); + } +} diff --git a/packages/trpc/src/lib/sandbox/golden-drives.ts b/packages/trpc/src/lib/sandbox/golden-drives.ts new file mode 100644 index 00000000000..ce7a16d8d2c --- /dev/null +++ b/packages/trpc/src/lib/sandbox/golden-drives.ts @@ -0,0 +1,24 @@ +import { db } from "@superset/db/client"; +import { environments } from "@superset/db/schema"; +import { and, eq, isNull } from "drizzle-orm"; +import { type DriveHost, vercelDrives } from "./drives"; + +/** + * An environment's golden drive goes once no live environment names it. Its + * forks are independent drives and stay readable after it is deleted. + */ +export async function deleteGoldenDriveIfUnreferenced( + name: string | null | undefined, + drives: DriveHost = vercelDrives(), +): Promise { + if (!name) return; + const referenced = await db.query.environments.findFirst({ + where: and( + eq(environments.goldenDriveName, name), + isNull(environments.archivedAt), + ), + columns: { id: true }, + }); + if (referenced) return; + await drives.remove(name); +} diff --git a/packages/trpc/src/lib/sandbox/index.ts b/packages/trpc/src/lib/sandbox/index.ts index ec376ff4943..2f9b6da808c 100644 --- a/packages/trpc/src/lib/sandbox/index.ts +++ b/packages/trpc/src/lib/sandbox/index.ts @@ -26,6 +26,18 @@ export { } from "./box-pressure"; export { buildSandboxClaim } from "./claim"; export { deriveSandboxCredentials } from "./credentials"; +export { + DriveStillAttachedError, + deleteWorkspaceDrive, + forkWorkspaceDrive, + goldenDriveNameFor, + WORKSPACE_DRIVE_MOUNT_PATH, + waitForDriveDetached, + workspaceDriveMounts, + workspaceDriveNameToClaim, + workspaceDrivesEnabled, +} from "./drives"; +export { deleteGoldenDriveIfUnreferenced } from "./golden-drives"; export { listRemoteBranches, type RemoteBranch, @@ -56,6 +68,7 @@ export { type ActiveWorkspaceSandbox, type BoxSession, deleteSandbox, + deleteSandboxAndDrive, describeSandbox, findSandboxSession, HOST_SERVICE_PORT, @@ -65,7 +78,11 @@ export { promoteSandboxToEnvironment, provisionSandbox, pushManagedEnv, + type ResumeLock, + SandboxBusyError, type SandboxClaim, + SandboxDriveMissingError, + SandboxDriveRefusedError, type SandboxEnvironment, SandboxNotReadyError, SandboxUnavailableError, diff --git a/packages/trpc/src/lib/sandbox/vercel-credentials.ts b/packages/trpc/src/lib/sandbox/vercel-credentials.ts new file mode 100644 index 00000000000..3c97aa04053 --- /dev/null +++ b/packages/trpc/src/lib/sandbox/vercel-credentials.ts @@ -0,0 +1,9 @@ +import { env } from "../../env"; + +export function credentials() { + return { + token: env.VERCEL_SANDBOX_TOKEN, + teamId: env.VERCEL_SANDBOX_TEAM_ID, + projectId: env.VERCEL_SANDBOX_PROJECT_ID, + }; +} diff --git a/packages/trpc/src/lib/sandbox/vercel.test.ts b/packages/trpc/src/lib/sandbox/vercel.test.ts index 2141b68005c..6a071d53891 100644 --- a/packages/trpc/src/lib/sandbox/vercel.test.ts +++ b/packages/trpc/src/lib/sandbox/vercel.test.ts @@ -1,26 +1,40 @@ import { afterAll, beforeAll, + beforeEach, describe, expect, setSystemTime, test, } from "bun:test"; +import { SANDBOX_PATHS } from "@superset/shared/sandbox/contract"; import { APIError, Drive, Sandbox } from "@vercel/sandbox"; import { stub } from "../../../test/stub"; import { agentToolsMounts, applySandboxPolicy, + composeMounts, + DRIVE_REFUSAL_LIMIT, deleteSandbox, + deleteSandboxAndDrive, describeSandbox, + driveBootsIn, + driveRefusalIn, findSandboxSession, listActiveWorkspaceSandboxes, listSandboxSessions, + persistentDriveRefusals, promoteSandboxToEnvironment, + provisionSandbox, + type ResumeLock, type SandboxClaim, + SandboxDriveMissingError, + SandboxDriveRefusedError, + SandboxNotReadyError, SandboxUnavailableError, SESSION_TIMEOUT_MS, sandboxExists, + settleSandbox, stopSandbox, topUpSandboxSession, wakeSandbox, @@ -313,7 +327,7 @@ describe("promoteSandboxToEnvironment", () => { let createTags: unknown; const golden = { currentSnapshotId: "snap_golden_0", - runCommand: async () => {}, + runCommand: async () => ({ exitCode: 0 }), stop: async () => {}, }; stub(Sandbox, { @@ -342,6 +356,7 @@ describe("promoteSandboxToEnvironment", () => { expect(await promote()).toEqual({ goldenName: "env-1", region: "iad1", + goldenDriveName: null, stopped: { sessionId: "sess_A" }, }); expect(sourceSnapshot).toBe("snap_now"); @@ -420,7 +435,7 @@ describe("wakeSandbox", () => { SUPERSET_SANDBOX_PROVIDER: "vercel", SUPERSET_BUNDLE_SHA: "a".repeat(64), }; - async function wake(): Promise { + async function _wake(): Promise { const authorized: Box[] = []; await wakeSandbox({ providerSandboxId: "ws-box", @@ -435,22 +450,6 @@ describe("wakeSandbox", () => { return authorized; } - test("a box read running that a call resumed runs beforeResume for the new session", async () => { - running = true; - expect(await wake()).toEqual([ - { - memoryMb: 16_384, - vcpus: 8, - resumedSession: { sessionId: "sess_new", expiresAt: box.expiresAt }, - }, - ]); - }); - - test("a box read stopped runs beforeResume once, before its own resume", async () => { - running = false; - expect(await wake()).toEqual([{ memoryMb: 8192, vcpus: 4 }]); - }); - test("a wake only for a running box leaves a box that stopped since the caller looked", async () => { running = false; calls.length = 0; @@ -485,6 +484,868 @@ describe("listActiveWorkspaceSandboxes", () => { }); }); +describe("wakeSandbox mounts", () => { + const RESUMED = new Error("resumed"); + let status: "running" | "stopped"; + let current: Record; + let events: string[]; + let updates: Array>; + const fake = { + get status() { + return status; + }, + get mounts() { + return current; + }, + vcpus: 4, + region: "iad1", + domain: () => "https://box.invalid", + update: async (params: Record) => { + events.push("update"); + updates.push(params); + }, + extendTimeout: async () => {}, + writeFiles: async () => { + throw RESUMED; + }, + currentSession: () => fake, + }; + stub(Sandbox, { get: async () => fake }); + let attachedTo: string | undefined; + stub(Drive, { + get: async ({ name }: { name: string }) => { + events.push(`drive:${name}`); + // The concurrent wake that took the drive has resumed the box. + if (attachedTo) status = "running"; + return { name, currentSandboxName: attachedTo }; + }, + }); + stub(globalThis, { + fetch: async () => { + throw new Error("no network in tests"); + }, + }); + const drive = { drive: "drive-1", mode: "read-write" }; + const mounts = { [SANDBOX_PATHS.drive]: drive }; + const claim = { vcpus: 4, files: [], networkPolicy: {} } as never; + const withAgentTools = { + ...(claim as object), + agentTools: true, + identity: { SUPERSET_BUNDLE_SHA: `feedfacecafe${"0".repeat(52)}` }, + } as never; + const stopAtResume = async () => { + events.push("resume"); + throw RESUMED; + }; + + beforeEach(() => { + current = {}; + events = []; + updates = []; + attachedTo = undefined; + }); + + test("a drive a concurrent wake already resumed this box with is not foreign", async () => { + status = "stopped"; + attachedTo = "sbx"; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts: mounts as never, + beforeResume: stopAtResume, + }), + ).rejects.toThrow(); + expect(events).not.toContain("resume"); + expect(updates.some((update) => "mounts" in update)).toBe(false); + }); + + test("a stopped box waits for its drive, then gets its mounts, then resumes", async () => { + status = "stopped"; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts: mounts as never, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(events).toEqual(["drive:drive-1", "update", "resume"]); + expect(updates).toEqual([{ mounts }]); + }); + + test("the drive and the agent-tools drive share one mounts object", async () => { + status = "stopped"; + const agents = "agent-tools-iad1-feedfacecafe"; + current = { + [SANDBOX_PATHS.agents]: { drive: "agent-tools-old", mode: "snapshot" }, + }; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim: withAgentTools, + mounts: mounts as never, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(updates).toEqual([ + { + mounts: { + [SANDBOX_PATHS.drive]: drive, + [SANDBOX_PATHS.agents]: { drive: agents, mode: "snapshot" }, + }, + }, + ]); + }); + + test("an agent-tools update keeps the drive the box already has", async () => { + status = "stopped"; + current = { [SANDBOX_PATHS.drive]: drive }; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim: withAgentTools, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(updates[0]?.mounts).toMatchObject({ [SANDBOX_PATHS.drive]: drive }); + }); + + test("a box without a drive gets no mount update", async () => { + status = "stopped"; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(updates).toEqual([]); + }); + + test("a running box keeps the mounts its session started with", async () => { + status = "running"; + await expect( + wakeSandbox({ providerSandboxId: "sbx", claim, mounts: mounts as never }), + ).rejects.toThrow(); + expect(updates).toEqual([{ networkPolicy: {} }]); + expect(events).not.toContain("drive:drive-1"); + }); +}); + +describe("resume under one lock", () => { + let status: "running" | "stopped"; + let attachedTo: string | undefined; + let events: string[]; + let gate: Promise; + let bootLog: string; + let refuse: boolean; + const box = () => ({ + get status() { + return status; + }, + mounts: {}, + vcpus: 4, + domain: () => "https://box.invalid", + expiresAt: undefined, + update: async (params: Record) => { + if (params.mounts) { + events.push("mount"); + await gate; + } + }, + writeFiles: async () => { + events.push("write"); + }, + runCommand: async (command: { env?: Record }) => { + events.push("boot"); + status = "running"; + const id = command.env?.SUPERSET_BOOT_ID ?? ""; + if (refuse) { + bootLog = `1 boot.start pid=1 boot=${id}\n2 drive.missing drive-1 is not mounted at /mnt/superset-drive\n`; + } + }, + readFileToBuffer: async () => Buffer.from(bootLog), + stop: async () => { + events.push("stop"); + status = "stopped"; + }, + }); + stub(Sandbox, { + get: async () => { + const fake = box(); + return { ...fake, currentSession: () => fake }; + }, + }); + stub(Drive, { + get: async ({ name }: { name: string }) => { + events.push(`drive:${attachedTo ?? "detached"}`); + const holder = attachedTo; + // The session that stopped finishes saving and lets go. + attachedTo = undefined; + return { name, currentSandboxName: holder }; + }, + }); + stub(globalThis, { + fetch: async () => { + throw new Error("not answering"); + }, + }); + const mounts = { + [SANDBOX_PATHS.drive]: { drive: "drive-1", mode: "read-write" }, + } as never; + const claim = { + identity: { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "0".repeat(64), + SUPERSET_API_URL: "https://api.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "env-g", + SUPERSET_SANDBOX_PROVIDER: "vercel", + SUPERSET_SANDBOX_DRIVE: "drive-1", + }, + hostSecret: "s", + files: [], + vcpus: 4, + networkPolicy: {}, + } as never; + let holder: string | null; + const lock = + (who: string): ResumeLock => + async (step) => { + if (holder) throw new SandboxNotReadyError("sbx"); + holder = who; + events.push(`lock:${who}`); + try { + return await step(); + } finally { + holder = null; + } + }; + + beforeEach(() => { + status = "stopped"; + attachedTo = undefined; + events = []; + gate = Promise.resolve(); + bootLog = ""; + refuse = false; + holder = null; + }); + + test("a redelivered provision whose box stopped waits for the drive before anything resumes it", async () => { + attachedTo = "sbx"; + await provisionSandbox({ + name: "sbx", + environment: { sourceKind: "fork", sourceRef: "env-g", region: "iad1" }, + claim, + mounts, + exclusive: lock("provision"), + }); + expect(events).toEqual([ + "lock:provision", + "drive:sbx", + "drive:detached", + "mount", + "write", + "boot", + ]); + }); + + test("a second wake while the first resumes does not resume the box; it answers still starting", async () => { + let open = () => {}; + gate = new Promise((resolve) => { + open = resolve; + }); + const first = wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts, + exclusive: lock("first"), + }).catch((error: unknown) => error); + await new Promise((resolve) => setTimeout(resolve, 0)); + const second = await wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts, + exclusive: lock("second"), + }).catch((error: unknown) => error); + expect(second).toBeInstanceOf(SandboxNotReadyError); + expect(events).toEqual(["lock:first", "drive:detached", "mount"]); + refuse = true; + open(); + const outcome = await first; + expect(outcome).toBeInstanceOf(SandboxDriveMissingError); + // The refused box is stopped under the lock, so no wake can resume it in between. + expect(events).toEqual([ + "lock:first", + "drive:detached", + "mount", + "write", + "boot", + "lock:first", + "stop", + ]); + }); +}); + +describe("a box found running", () => { + const ended = new Error("session ended"); + const booted = new Error("booted on the new session"); + let calls: string[]; + let status: "running" | "stopped"; + let attachedTo: string | undefined; + const endedSession = { + extendTimeout: async () => { + calls.push("old-session:extend"); + // The extension past the plan's cap is refused, and the session ends. + status = "stopped"; + attachedTo = "sbx"; + throw ended; + }, + update: async () => { + calls.push("old-session:policy"); + throw ended; + }, + writeFiles: async () => { + calls.push("old-session:write"); + throw ended; + }, + runCommand: async () => { + calls.push("old-session:boot"); + throw ended; + }, + }; + const newSession = { + sessionId: "sess-new", + vcpus: 8, + extendTimeout: async () => {}, + update: async () => {}, + writeFiles: async () => { + calls.push("new-session:write"); + }, + runCommand: async () => { + calls.push("new-session:boot"); + throw booted; + }, + }; + // A Sandbox method other than a config update would resume a session that ended. + const resumes = (name: string) => async () => { + calls.push(`sandbox:${name}`); + }; + stub(Sandbox, { + get: async ({ resume }: { resume?: boolean }) => { + if (resume) { + calls.push("resume"); + status = "running"; + } + return { + status, + mounts: {}, + vcpus: 4, + expiresAt: new Date(Date.now() + 1000), + domain: () => "https://box.invalid", + extendTimeout: resumes("extend"), + update: async (params: Record) => { + calls.push(params.mounts ? "mount" : "sandbox:update"); + }, + writeFiles: resumes("write"), + runCommand: resumes("boot"), + currentSession: () => (resume ? newSession : endedSession), + }; + }, + }); + stub(Drive, { + get: async ({ name }: { name: string }) => { + calls.push(`drive:${attachedTo ?? "detached"}`); + const holder = attachedTo; + attachedTo = undefined; + return { name, currentSandboxName: holder }; + }, + }); + stub(globalThis, { + fetch: async () => { + throw new Error("not answering"); + }, + }); + + beforeEach(() => { + calls = []; + status = "running"; + attachedTo = undefined; + }); + + const wake = ( + beforeResume?: Parameters[0]["beforeResume"], + ) => + wakeSandbox({ + providerSandboxId: "sbx", + claim: { + identity: { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "0".repeat(64), + SUPERSET_API_URL: "https://api.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: + "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "env-g", + SUPERSET_SANDBOX_PROVIDER: "vercel", + SUPERSET_SANDBOX_DRIVE: "drive-1", + }, + files: [{ path: "/x", content: "y" }], + networkPolicy: {}, + vcpus: 4, + } as never, + mounts: { + [SANDBOX_PATHS.drive]: { drive: "drive-1", mode: "read-write" }, + } as never, + beforeResume, + }); + + test("whose session ends during the wake is resumed only after the drive check", async () => { + await expect(wake()).rejects.toBe(booted); + expect(calls.filter((call) => call.startsWith("sandbox:"))).toEqual([]); + const resumedAt = calls.indexOf("resume"); + expect(calls.slice(0, resumedAt)).toContain("drive:sbx"); + expect(calls.slice(0, resumedAt)).toContain("drive:detached"); + expect(calls.slice(0, resumedAt)).toContain("mount"); + expect(calls.slice(resumedAt)).toEqual([ + "resume", + "new-session:write", + "new-session:write", + "new-session:boot", + ]); + }); + + test("whose session ends during the wake meters the session resumed here before it boots", async () => { + const metered: unknown[] = []; + await expect( + wake(async (box) => { + metered.push(box); + calls.push("metered"); + }), + ).rejects.toBe(booted); + expect(metered).toEqual([ + { + memoryMb: 16_384, + vcpus: 8, + resumedSession: { sessionId: "sess-new", expiresAt: expect.any(Date) }, + }, + ]); + expect(calls.indexOf("metered")).toBeLessThan( + calls.indexOf("new-session:boot"), + ); + }); +}); + +describe("a new box on a drive", () => { + let events: string[]; + let attachedTo: string | undefined; + const session = { + writeFiles: async () => { + events.push("write"); + }, + runCommand: async () => { + events.push("boot"); + }, + }; + stub(Sandbox, { + get: async () => { + throw new APIError(new Response(null, { status: 404 })); + }, + fork: async () => { + events.push("create"); + return { + domain: () => "https://box.invalid", + expiresAt: undefined, + currentSession: () => session, + }; + }, + }); + stub(Drive, { + get: async ({ name }: { name: string }) => { + events.push(`drive:${attachedTo ?? "detached"}`); + const holder = attachedTo; + attachedTo = undefined; + return { name, currentSandboxName: holder }; + }, + }); + + beforeEach(() => { + events = []; + attachedTo = undefined; + }); + + test("is created only once a box of another name has let go of the drive", async () => { + // An undo's second provision: the first job's box still holds the drive. + attachedTo = "ws-box-1"; + await provisionSandbox({ + name: "ws-box-2", + environment: { sourceKind: "fork", sourceRef: "env-g", region: "iad1" }, + claim: { + identity: { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "0".repeat(64), + SUPERSET_API_URL: "https://api.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: + "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "env-g", + SUPERSET_SANDBOX_PROVIDER: "vercel", + }, + hostSecret: "s", + files: [], + vcpus: 4, + networkPolicy: {}, + } as never, + mounts: { + [SANDBOX_PATHS.drive]: { drive: "drive-1", mode: "read-write" }, + } as never, + }); + expect(events).toEqual([ + "drive:ws-box-1", + "drive:detached", + "create", + "write", + "boot", + ]); + }); +}); + +describe("composeMounts", () => { + test("merges parts, a later part winning a shared path, and drops empty", () => { + const a = { "/a": { drive: "a", mode: "read-write" as const } }; + const b = { "/a": { drive: "b", mode: "snapshot" as const } }; + expect(composeMounts(a, undefined, b)).toEqual(b); + expect(composeMounts(undefined, {})).toBeUndefined(); + }); +}); + +describe("drive refusals in the boot log", () => { + const boot = (at: number, id: string, ...lines: string[]) => + [`${at} boot.start pid=1 bundle=abc boot=${id}`, ...lines].join("\n"); + + test("only this boot's lines count, so a previous boot's refusal never stops a healthy box", () => { + const log = [ + boot( + 1, + "old", + "2 drive.missing drive-1 is not mounted at /mnt/superset-drive", + ), + boot(3, "new", "4 run.cleared"), + ].join("\n"); + expect(driveRefusalIn(log, "old")).toBe( + "drive.missing drive-1 is not mounted at /mnt/superset-drive", + ); + expect(driveRefusalIn(log, "new")).toBeNull(); + expect(driveRefusalIn(log, "not-started-yet")).toBeNull(); + }); + + test("a reexec's second boot.start joins its boot", () => { + const log = boot( + 1, + "b", + "2 boot.reexec into the installed bundle's script", + "3 boot.start pid=2 bundle=abc boot=b", + "4 drive.foreign drive-1 belongs to workspace w", + ); + expect(driveBootsIn(log)).toHaveLength(1); + expect(driveRefusalIn(log, "b")).toBe( + "drive.foreign drive-1 belongs to workspace w", + ); + }); + + test("counts the persistent refusals in a row; a missing drive or a laid-out boot ends the run", () => { + const foreign = (at: number, id: string) => + boot(at, id, `${at + 1} drive.foreign d belongs to workspace w`); + const log = [ + boot(1, "a", "2 drive.layout ms=3", "3 host.exec 1.0"), + foreign(10, "b"), + boot(20, "c", "21 host.exec 1.0"), + foreign(30, "d"), + ].join("\n"); + expect(persistentDriveRefusals(log)).toBe(3); + expect( + persistentDriveRefusals( + `${log}\n${boot(40, "e", "41 drive.missing d is not mounted")}`, + ), + ).toBe(0); + }); +}); + +describe("settleSandbox with a drive", () => { + let health: unknown = null; + let bootLog = ""; + stub(globalThis, { + fetch: async (url: string) => { + if (String(url).endsWith("/trpc/health.check") && health !== null) { + return new Response(JSON.stringify(health)); + } + if (String(url).endsWith("/trpc/sandbox.setEnvironment")) { + return new Response("{}"); + } + throw new Error("not answering"); + }, + }); + const box = { + status: "running", + readFileToBuffer: async () => Buffer.from(bootLog), + } as never; + const claim = { + identity: { SUPERSET_SANDBOX_DRIVE: "drive-1" }, + hostSecret: "secret", + managedEnv: {}, + } as never; + const answering = (phases: string[]) => ({ + result: { + data: { + json: { + sandboxBoot: { + stamps: phases.map((phase, at) => ({ phase, at })), + }, + }, + }, + }, + }); + + test("a refused boot is a retryable not-ready, found without the full host wait", async () => { + health = null; + bootLog = + "1 boot.start pid=1 boot=this\n2 drive.missing drive-1 is not mounted at /mnt/superset-drive\n"; + const started = Date.now(); + const error = await settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + bootId: "this", + getSession: async () => box, + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SandboxDriveMissingError); + expect(error).toBeInstanceOf(SandboxNotReadyError); + expect(Date.now() - started).toBeLessThan(1_000); + }); + + test("a refusal no remount can change ends the retries after several boots in a row", async () => { + health = null; + bootLog = Array.from( + { length: DRIVE_REFUSAL_LIMIT }, + (_, i) => + `${i * 10} boot.start pid=1 boot=b${i}\n${i * 10 + 1} drive.unlaid drive-1 carries no workspace id`, + ).join("\n"); + const error = await settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + bootId: `b${DRIVE_REFUSAL_LIMIT - 1}`, + getSession: async () => box, + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SandboxDriveRefusedError); + expect(error).toBeInstanceOf(SandboxUnavailableError); + expect(String(error)).toContain(`in ${DRIVE_REFUSAL_LIMIT} boots in a row`); + }); + + test("a boot that answers without laying the drive out fails the settle", async () => { + health = answering(["boot.start", "host.ready"]); + bootLog = ""; + await expect( + settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + getSession: async () => box, + }), + ).rejects.toBeInstanceOf(SandboxDriveMissingError); + }); + + test("a boot that laid the drive out settles", async () => { + health = answering(["boot.start", "drive.layout", "host.ready"]); + bootLog = ""; + const answer = await settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + getSession: async () => box, + }); + expect(answer.stamps.map((stamp) => stamp.phase)).toContain("drive.layout"); + }); +}); + +describe("promoteSandboxToEnvironment", () => { + let events: string[]; + let created: Record | null; + let goldenStatus: string; + let goldenSnapshot: string; + let stripExit: number; + const source = { + status: "running", + region: "iad1", + vcpus: 4, + currentSession: () => ({ + sessionId: "sess-source", + snapshot: async () => { + events.push("snapshot source"); + return { snapshotId: "snap-1" }; + }, + }), + }; + const golden = { + get status() { + return goldenStatus; + }, + get currentSnapshotId() { + return goldenSnapshot; + }, + runCommand: async (command: { args: string[] }) => { + events.push(`strip ${command.args.join(" ")}`); + return { exitCode: stripExit }; + }, + stop: async () => { + goldenStatus = "stopped"; + goldenSnapshot = "snap-golden"; + }, + }; + stub(Sandbox, { + get: async ({ name }: { name: string }) => + name === "ws-source" ? source : golden, + create: async (params: Record) => { + created = params; + return golden; + }, + }); + const drives = { + find: async (name: string) => { + events.push(`find ${name}`); + return { region: "iad1" }; + }, + create: async () => {}, + fork: async (from: string, to: string) => { + events.push(`fork ${from} ${to}`); + }, + remove: async () => {}, + }; + beforeEach(() => { + events = []; + created = null; + goldenStatus = "running"; + goldenSnapshot = "snap-0"; + stripExit = 0; + }); + + test("a golden whose drive did not mount is not stripped and fails the promote", async () => { + stripExit = 3; + await expect( + promoteSandboxToEnvironment({ + sourceSandbox: "ws-source", + goldenName: "env-g", + organizationId: "org_1", + sourceDriveName: "drive-ws-source", + drives, + }), + ).rejects.toThrow(`${SANDBOX_PATHS.drive} is not mounted`); + const strip = events.find((event) => event.startsWith("strip")) ?? ""; + expect(strip).toContain(`mountpoint -q ${SANDBOX_PATHS.drive} || exit 3;`); + }); + + test("forks the source's drive once it is released, strips identity on the drive, and leaves the source's restart to the caller", async () => { + const result = await promoteSandboxToEnvironment({ + sourceSandbox: "ws-source", + goldenName: "env-g", + organizationId: "org_1", + sourceDriveName: "drive-ws-source", + drives, + }); + expect(result.goldenDriveName).toBe("env-drive-env-g"); + expect(created?.mounts).toEqual({ + [SANDBOX_PATHS.drive]: { drive: "env-drive-env-g", mode: "read-write" }, + }); + const strip = events.find((event) => event.startsWith("strip")) ?? ""; + expect(strip).toContain(`${SANDBOX_PATHS.drive}/workspace/.env`); + expect(strip).toContain(`${SANDBOX_PATHS.drive}/home/.codex/sessions`); + expect(strip).toContain(SANDBOX_PATHS.driveState); + expect(events).toEqual([ + "snapshot source", + "find drive-ws-source", + "fork drive-ws-source env-drive-env-g", + strip, + "find env-drive-env-g", + ]); + expect(result.stopped).toEqual({ sessionId: "sess-source" }); + }); + + test("a source without a drive leaves the environment out of drive mode", async () => { + const result = await promoteSandboxToEnvironment({ + sourceSandbox: "ws-source", + goldenName: "env-g", + organizationId: "org_1", + drives, + }); + expect(result.goldenDriveName).toBeNull(); + expect(created).not.toHaveProperty("mounts"); + expect(events.some((event) => event.startsWith("fork"))).toBe(false); + expect(events.join("\n")).not.toContain(SANDBOX_PATHS.drive); + }); +}); + +describe("deleteSandboxAndDrive", () => { + let events: string[]; + let status: string; + let expired = false; + stub(Sandbox, { + get: async () => { + if (expired) throw new APIError(new Response(null, { status: 410 })); + return box(); + }, + }); + const box = () => ({ + get status() { + return status; + }, + currentSession: () => ({ sessionId: "s" }), + stop: async () => { + events.push("stop"); + status = "stopped"; + }, + delete: async () => { + events.push("delete sandbox"); + }, + }); + const drives = { + find: async () => { + events.push("find"); + return { region: "iad1" }; + }, + create: async () => {}, + fork: async () => {}, + remove: async (name: string) => { + events.push(`delete ${name}`); + }, + }; + + test("stops the box and waits for the drive before deleting either", async () => { + events = []; + status = "running"; + await deleteSandboxAndDrive("ws-1", "drive-ws-1", drives); + expect(events).toEqual([ + "stop", + "find", + "delete sandbox", + "delete drive-ws-1", + ]); + }); + + test("a box whose snapshot expired, which cannot be fetched, still lets its drive go", async () => { + events = []; + expired = true; + await deleteSandboxAndDrive("ws-1", "drive-ws-1", drives); + expired = false; + expect(events).toEqual(["find", "delete drive-ws-1"]); + }); +}); + describe("topUpSandboxSession", () => { let expiresAt: Date; const extended: number[] = []; diff --git a/packages/trpc/src/lib/sandbox/vercel.ts b/packages/trpc/src/lib/sandbox/vercel.ts index b79ff3809e9..205781d70d3 100644 --- a/packages/trpc/src/lib/sandbox/vercel.ts +++ b/packages/trpc/src/lib/sandbox/vercel.ts @@ -10,6 +10,7 @@ * ticket-checking gate in front of every port (`access.ts`). */ +import { randomUUID } from "node:crypto"; import * as Sentry from "@sentry/core"; import { renderSandboxConf, @@ -18,6 +19,7 @@ import { SANDBOX_PUBLISHED_PORTS, type SandboxIdentity, sandboxAgentToolsDrive, + sandboxDrivePathOf, } from "@superset/shared/sandbox/contract"; import { APIError, @@ -26,14 +28,23 @@ import { Sandbox, type SandboxMounts, type SandboxRegion, + type Session, } from "@vercel/sandbox"; -import { env } from "../../env"; import { bootStampsFrom, type HostAnswer } from "./boot-trace"; import { CODEX_AUTH_PATH, CODEX_SYSTEM_CONFIG_PATH, type SandboxFile, } from "./credentials"; +import { + type DriveHost, + deleteWorkspaceDrive, + goldenDriveNameFor, + vercelDrives, + waitForDriveDetached, + workspaceDriveMounts, +} from "./drives"; +import { credentials } from "./vercel-credentials"; export const HOST_SERVICE_PORT = SANDBOX_PORTS.hostService; /** @@ -53,14 +64,6 @@ const GOLDEN_SESSION_TIMEOUT_MS = 30 * 60 * 1000; const BOOT_COMMAND = "/usr/local/bin/superset-boot"; const HOST_PROBE_TIMEOUT_MS = 1_500; -function credentials() { - return { - token: env.VERCEL_SANDBOX_TOKEN, - teamId: env.VERCEL_SANDBOX_TEAM_ID, - projectId: env.VERCEL_SANDBOX_PROJECT_ID, - }; -} - function isNotFound(error: unknown): boolean { return error instanceof APIError && error.response.status === 404; } @@ -79,10 +82,14 @@ export class SandboxUnavailableError extends Error { } get snapshotExpired(): boolean { - return this.cause instanceof APIError && this.cause.response.status === 410; + return isSnapshotExpired(this.cause); } } +function isSnapshotExpired(error: unknown): boolean { + return error instanceof APIError && error.response.status === 410; +} + function isUnavailable(error: unknown): boolean { return ( error instanceof APIError && @@ -158,6 +165,8 @@ export interface SandboxEnvironment { sourceKind: "image" | "fork"; sourceRef: string; region: string; + /** Set when the environment is in drive mode; see `drives.ts`. */ + goldenDriveName?: string | null; } /** Everything the box needs to become one workspace; nothing of it is a create-time env. */ @@ -242,6 +251,21 @@ export function agentToolsDriveName( : null; } +/** One `mounts` object for a session; a later part wins a path both name. */ +export function composeMounts( + ...parts: Array +): SandboxMounts | undefined { + const mounts = Object.assign({}, ...parts.filter(Boolean)) as SandboxMounts; + return Object.keys(mounts).length > 0 ? mounts : undefined; +} + +function readWriteDriveAt(mounts: SandboxMounts, path: string): string | null { + const mount = mounts[path]; + if (!mount) return null; + if (mount instanceof Drive) return mount.name; + return mount.mode === "read-write" ? mount.drive : null; +} + function publishedPorts(extra: readonly number[] = []): number[] { return [...new Set([...SANDBOX_PUBLISHED_PORTS, ...extra])]; } @@ -253,7 +277,7 @@ function publishedPorts(extra: readonly number[] = []): number[] { * without a size limit and so the runner's inputs never change shape. */ async function writeIdentity( - sandbox: Sandbox, + sandbox: Pick, identity: SandboxIdentity, ): Promise { await sandbox.writeFiles([ @@ -271,7 +295,7 @@ async function writeIdentity( * started there. They hold placeholders only. */ async function writeSignInFiles( - sandbox: Sandbox, + sandbox: Pick, files: SandboxFile[] = [], ): Promise { if (files.length === 0) return; @@ -285,13 +309,19 @@ async function writeSignInFiles( * argv. The runner refuses to stack a second host-service on a live one, so * a wake that races a wake is harmless. */ -async function runBoot(sandbox: Sandbox, hostSecret: string): Promise { +async function runBoot( + sandbox: Pick, + hostSecret: string, +): Promise { + // Stamped on boot.start, so a wait reads only this boot's lines of the log. + const bootId = randomUUID(); await sandbox.runCommand({ cmd: "sudo", - args: ["--preserve-env=HOST_SERVICE_SECRET", BOOT_COMMAND], + args: ["--preserve-env=HOST_SERVICE_SECRET,SUPERSET_BOOT_ID", BOOT_COMMAND], detached: true, - env: { HOST_SERVICE_SECRET: hostSecret }, + env: { HOST_SERVICE_SECRET: hostSecret, SUPERSET_BOOT_ID: bootId }, }); + return bootId; } /** @@ -306,12 +336,16 @@ export async function provisionSandbox(args: { claim: SandboxClaim; /** A golden under construction keeps its snapshots until its row goes. */ kind?: "workspace" | "environment"; + mounts?: SandboxMounts; + exclusive?: ResumeLock; }): Promise<{ providerSandboxId: string; sandboxUrl: string; hostTarget: string; expiresAt: Date | null; sessionId: string | null; + /** Ties a wait for host-service to this boot's lines of the boot log. */ + bootId: string; }> { const kind = args.kind ?? "workspace"; const config = { @@ -327,53 +361,89 @@ export async function provisionSandbox(args: { keepLastSnapshots: { count: 1 }, tags: sandboxTags(kind, args.claim.identity), }; - const mounts = await agentToolsMounts( - agentToolsDriveName(args.claim, args.environment.region), + const mounts = composeMounts( + args.mounts, + await agentToolsMounts( + agentToolsDriveName(args.claim, args.environment.region), + ), ); - const sandbox = - (await getSandboxToWake(args.name)) ?? - (await Sentry.startSpan( - { - name: - args.environment.sourceKind === "fork" ? "fork" : "create from image", - op: "sandbox.provider", - attributes: { - "sandbox.name": args.name, - "sandbox.source": args.environment.sourceRef, - "sandbox.region": args.environment.region, - "sandbox.vcpus": args.claim.vcpus, + return (args.exclusive ?? unlocked)(async () => { + const existing = await getSandboxToWake(args.name); + const drive = mounts ? readWriteDriveAt(mounts, SANDBOX_PATHS.drive) : null; + let live: { sandbox: Sandbox; session: Session }; + if (existing?.status === "running") { + live = { sandbox: existing, session: existing.currentSession() }; + } else if (existing) { + // A redelivered provision whose box has stopped: its drive may still be flushing. + const outcome = drive + ? await waitForDriveDetached(drive, { + heldByCaller: heldByThisRunningBox(args.name), + }) + : "detached"; + if (outcome === "detached" && mounts) await existing.update({ mounts }); + live = await resumeSession(args.name); + } else { + // A drive has one writer whatever the box is called: a box of another name may still hold it. + if (drive) await waitForDriveDetached(drive); + const created = await Sentry.startSpan( + { + name: + args.environment.sourceKind === "fork" + ? "fork" + : "create from image", + op: "sandbox.provider", + attributes: { + "sandbox.name": args.name, + "sandbox.source": args.environment.sourceRef, + "sandbox.region": args.environment.region, + "sandbox.vcpus": args.claim.vcpus, + }, }, - }, + () => + args.environment.sourceKind === "fork" + ? // A fork copies the golden's region; a snapshot only exists where + // it was taken. + Sandbox.fork({ + ...config, + ...(mounts ? { mounts } : {}), + sourceSandbox: args.environment.sourceRef, + resources: { vcpus: args.claim.vcpus }, + }) + : Sandbox.create({ + ...config, + ...(mounts ? { mounts } : {}), + image: args.environment.sourceRef, + region: args.environment.region as SandboxRegion, + resources: { vcpus: args.claim.vcpus }, + }), + ); + live = { sandbox: created, session: created.currentSession() }; + } + const booted = await Sentry.startSpan( + { name: "start boot", op: "sandbox" }, () => - args.environment.sourceKind === "fork" - ? // A fork copies the golden's region; a snapshot only exists where - // it was taken. - Sandbox.fork({ - ...config, - ...(mounts ? { mounts } : {}), - sourceSandbox: args.environment.sourceRef, - resources: { vcpus: args.claim.vcpus }, - }) - : Sandbox.create({ - ...config, - ...(mounts ? { mounts } : {}), - image: args.environment.sourceRef, - region: args.environment.region as SandboxRegion, - resources: { vcpus: args.claim.vcpus }, - }), - )); - await Sentry.startSpan({ name: "start boot", op: "sandbox" }, async () => { - await writeIdentity(sandbox, args.claim.identity); - await writeSignInFiles(sandbox, args.claim.files); - await runBoot(sandbox, args.claim.hostSecret); + withLiveSession({ + providerSandboxId: args.name, + mounts, + live, + use: async (current) => { + await writeIdentity(current.session, args.claim.identity); + await writeSignInFiles(current.session, args.claim.files); + return runBoot(current.session, args.claim.hostSecret); + }, + }), + ); + const bootId = booted.result; + const { sandbox } = booted.live; + return { + providerSandboxId: args.name, + sandboxUrl: sandbox.domain(HOST_SERVICE_PORT), + hostTarget: sandbox.domain(HOST_SERVICE_PORT), + expiresAt: sandbox.expiresAt ?? null, + sessionId: sessionIdOf(sandbox), + bootId, + }; }); - return { - providerSandboxId: args.name, - sandboxUrl: sandbox.domain(HOST_SERVICE_PORT), - hostTarget: sandbox.domain(HOST_SERVICE_PORT), - expiresAt: sandbox.expiresAt ?? null, - sessionId: sessionIdOf(sandbox), - }; } const HOST_READY_TIMEOUT_MS = 60_000; @@ -386,12 +456,134 @@ export class SandboxNotReadyError extends Error { } } +/** Another caller holds the box's resume lock: retry later, nothing is wrong with the box. */ +export class SandboxBusyError extends SandboxNotReadyError { + constructor(key: string) { + super(key); + this.name = "SandboxBusyError"; + this.message = `Another caller is resuming or creating the box on ${key}`; + } +} + +/** + * The box did not come up on the drive its identity names: the drive did not + * mount, belongs to another workspace, or the boot never laid it out. The + * data is intact, so it is a not-ready the next wake retries, never a failure. + */ +export class SandboxDriveMissingError extends SandboxNotReadyError { + constructor( + providerSandboxId: string, + readonly driveName: string, + readonly reason: string, + ) { + super(providerSandboxId); + this.name = "SandboxDriveMissingError"; + this.message = `Sandbox ${providerSandboxId} did not come up on its drive ${driveName}: ${reason}`; + } +} + +/** + * The drive refused for good: the same refusal that no remount can change, in + * several boots in a row. The workspace is marked failed with this message; + * its drive and box are kept. + */ +export class SandboxDriveRefusedError extends SandboxUnavailableError { + constructor( + providerSandboxId: string, + readonly driveName: string, + readonly reason: string, + readonly boots: number, + ) { + super(providerSandboxId, new Error(reason)); + this.name = "SandboxDriveRefusedError"; + this.message = `Sandbox ${providerSandboxId} refused its drive ${driveName} in ${boots} boots in a row: ${reason}`; + } +} + +/** Boots in a row that a refusal no remount can change ends the retries. */ +export const DRIVE_REFUSAL_LIMIT = 3; + +const STAMP = /^(\d+) ([a-z][a-z0-9-]*(?:\.[a-z0-9-]+)+)(?: (.*))?$/; + +interface DriveBoot { + id: string | null; + refusal: string | null; + laidOut: boolean; + hostStarted: boolean; +} + +/** + * The boot log is append-only across sessions. One entry per boot; a + * reexec's second `boot.start` carries the same boot id and joins its boot. + */ +export function driveBootsIn(bootLog: string): DriveBoot[] { + const boots: DriveBoot[] = []; + for (const line of bootLog.split("\n")) { + const match = STAMP.exec(line.trim()); + if (!match) continue; + const [, , phase, detail = ""] = match; + if (phase === "boot.start") { + const id = /(?:^| )boot=(\S+)/.exec(detail)?.[1] ?? null; + const last = boots.at(-1); + if (!(last && id && last.id === id)) { + boots.push({ id, refusal: null, laidOut: false, hostStarted: false }); + } + continue; + } + const boot = boots.at(-1); + if (!boot) continue; + if (/^drive\.(missing|foreign|unlaid|downgrade)$/.test(phase ?? "")) { + boot.refusal ??= `${phase} ${detail}`.trim(); + } else if (phase === "drive.layout") { + boot.laidOut = true; + } else if (phase === "host.exec") { + boot.hostStarted = true; + } + } + return boots; +} + +/** This boot's refusal, if it refused; another boot's line never counts. */ +export function driveRefusalIn(bootLog: string, bootId: string): string | null { + const boot = driveBootsIn(bootLog) + .reverse() + .find((entry) => entry.id === bootId); + return boot?.refusal ?? null; +} + +/** + * How many of the last boots in a row failed in a way a remount cannot fix: + * a drive of another workspace, a drive with no owner, a boot script that + * predates the drive, or a boot that started host-service without a layout. + */ +export function persistentDriveRefusals(bootLog: string): number { + let count = 0; + for (const boot of driveBootsIn(bootLog).reverse()) { + const persistent = boot.refusal + ? !boot.refusal.startsWith("drive.missing") + : boot.hostStarted && !boot.laidOut; + if (!persistent) break; + count++; + } + return count; +} + +/** How often a wait for host-service also reads the boot log for a drive refusal. */ +const DRIVE_REFUSAL_POLL_MS = 2_000; + async function waitForHostService( target: string, providerSandboxId: string, + refusal?: () => Promise, ): Promise { const deadline = Date.now() + HOST_READY_TIMEOUT_MS; + let nextRefusalCheck = 0; while (Date.now() < deadline) { + if (refusal && Date.now() >= nextRefusalCheck) { + const refused = await refusal(); + if (refused) throw refused; + nextRefusalCheck = Date.now() + DRIVE_REFUSAL_POLL_MS; + } const body: unknown = await fetch(`${target}/trpc/health.check`, { signal: AbortSignal.timeout(HOST_READY_POLL_MS * 6), }) @@ -416,11 +608,71 @@ export async function settleSandbox(args: { providerSandboxId: string; hostTarget: string; claim: SandboxClaim; + /** The boot this settle waits for; without one, no refusal is read from the log. */ + bootId?: string; + /** The running session to read the boot log from, fetched fresh for each read; null when the box is not running. */ + getSession?: () => Promise | null>; }): Promise { + const driveName = args.claim.identity.SUPERSET_SANDBOX_DRIVE; + const getSession = + args.getSession ?? + (async () => { + const box = await getSandbox(args.providerSandboxId); + return box?.status === "running" ? box.currentSession() : null; + }); + const bootLog = async () => { + // Through the session: a read from a box another caller stopped never resumes it. + const session = await getSession(); + if (!session) return ""; + const log = await session + .readFileToBuffer({ path: SANDBOX_PATHS.bootLog }) + .catch(() => null); + return log?.toString("utf8") ?? ""; + }; + const driveFailure = (driveName: string, reason: string, log: string) => { + const boots = persistentDriveRefusals(log); + if (boots < DRIVE_REFUSAL_LIMIT) { + console.warn(`[sandbox] ${args.providerSandboxId}: ${reason}`); + return new SandboxDriveMissingError( + args.providerSandboxId, + driveName, + reason, + ); + } + const refused = new SandboxDriveRefusedError( + args.providerSandboxId, + driveName, + reason, + boots, + ); + Sentry.captureException(refused); + return refused; + }; + const bootId = args.bootId; + const refusal = + driveName && bootId + ? async () => { + const log = await bootLog(); + const refused = driveRefusalIn(log, bootId); + return refused ? driveFailure(driveName, refused, log) : null; + } + : undefined; const answer = await waitForHostService( args.hostTarget, args.providerSandboxId, + refusal, ); + // A pinned bundle whose boot predates the drive layout serves the root disk's empty /workspace. + if ( + driveName && + !answer.stamps.some((stamp) => stamp.phase === "drive.layout") + ) { + throw driveFailure( + driveName, + "the boot answered without laying the drive out", + await bootLog(), + ); + } await pushManagedEnv( args.hostTarget, args.claim.hostSecret, @@ -548,21 +800,96 @@ async function hostServiceAnswers(hostTarget: string): Promise { * moved), the credential rules are re-applied (a token may have aged), and * the managed environment is pushed again (a fresh session holds none). */ +/** + * Runs `step` alone among everything that may resume this sandbox, and throws + * SandboxNotReadyError when another caller holds it. A stopped box's drive + * check and the call that resumes it are one step under it, so no two + * sessions ever hold the drive. + */ +export type ResumeLock = (step: () => Promise) => Promise; +const unlocked: ResumeLock = (step) => step(); + +/** + * The one call in this module that resumes a stopped box; it runs under the + * resume lock, after the drive check. Every other call goes through the + * returned Session: a Sandbox method resumes a session that ended, a Session + * method fails instead. + */ +async function resumeSession( + providerSandboxId: string, +): Promise<{ sandbox: Sandbox; session: Session }> { + const sandbox = await Sandbox.get({ + ...credentials(), + name: providerSandboxId, + resume: true, + }); + return { sandbox, session: sandbox.currentSession() }; +} + +/** + * Runs `use` on the box's session as it was read. If that session ended in + * between, a Session call fails rather than resume it; the box then takes + * the stopped path, under the caller's lock: wait for the drive, remount, + * resume, and `use` runs once more on the new session. + */ +async function withLiveSession(args: { + providerSandboxId: string; + mounts: SandboxMounts | undefined; + live: { sandbox: Sandbox; session: Session }; + use: (live: { sandbox: Sandbox; session: Session }) => Promise; + onResumed?: (live: { sandbox: Sandbox; session: Session }) => Promise; +}): Promise<{ result: T; live: { sandbox: Sandbox; session: Session } }> { + try { + return { result: await args.use(args.live), live: args.live }; + } catch (error) { + const now = await getSandbox(args.providerSandboxId); + if (!now || now.status === "running") throw error; + const drive = args.mounts + ? readWriteDriveAt(args.mounts, SANDBOX_PATHS.drive) + : null; + const outcome = drive + ? await waitForDriveDetached(drive, { + heldByCaller: heldByThisRunningBox(args.providerSandboxId), + }) + : "detached"; + if (outcome === "detached" && args.mounts) { + await now.update({ mounts: args.mounts }); + } + const live = await resumeSession(args.providerSandboxId); + await args.onResumed?.(live); + return { result: await args.use(live), live }; + } +} + +/** A drive attached to this very sandbox, running again, is held by the session another caller resumed. */ +function heldByThisRunningBox(providerSandboxId: string) { + return async (attachedTo: string) => + attachedTo === providerSandboxId && + (await getSandbox(providerSandboxId))?.status === "running"; +} + +export type WakePhase = "detach" | "mount" | "resume" | "boot"; + export async function wakeSandbox(args: { providerSandboxId: string; claim: SandboxClaim; + /** Replaces the box's mounts before a stopped session resumes; a running one keeps its own. */ + mounts?: SandboxMounts; /** A stopped box is left stopped, and the wake answers null. */ onlyIfRunning?: boolean; /** * Runs once the box is known to be stopped, before anything resumes it; or, - * for a box read running that a call here resumed (`resumedSession`), once - * those calls end. + * for a box read running whose session ended and was resumed here + * (`resumedSession`), once that resume is done. */ beforeResume?: (box: { memoryMb: number; vcpus: number; resumedSession?: { sessionId: string | null; expiresAt: Date | null }; }) => Promise; + /** How long each part of the wake took, for a check that measures it. */ + onPhase?: (phase: WakePhase, ms: number) => void; + exclusive?: ResumeLock; }): Promise<{ hostTarget: string; /** The session was stopped, so every process on the box started from this claim. */ @@ -573,111 +900,182 @@ export async function wakeSandbox(args: { expiresAt: Date | null; sessionId: string | null; } | null> { - try { - let wasRunning = false; - let resumed = null as { vcpus: number; sessionId: string | null } | null; - const sandbox = await getSandboxToWake( - args.providerSandboxId, - async (box) => { - if (!wasRunning) return; - const session = box.currentSession(); - resumed = { vcpus: session.vcpus, sessionId: session.sessionId }; - }, - ); - if (!sandbox) throw new SandboxUnavailableError(args.providerSandboxId); - const hostTarget = sandbox.domain(HOST_SERVICE_PORT); - wasRunning = sandbox.status === "running"; - if (args.onlyIfRunning && !wasRunning) return null; - let vcpus = sandbox.vcpus ?? args.claim.vcpus; - if (!wasRunning && vcpus !== args.claim.vcpus) { - await Sentry.startSpan({ name: "resize", op: "sandbox.provider" }, () => - sandbox - .update({ resources: { vcpus: args.claim.vcpus } }) - .then(() => { - vcpus = args.claim.vcpus; - }) - .catch((error) => - console.warn( - `[sandbox] resize to ${args.claim.vcpus} vCPUs failed for ${args.providerSandboxId}`, - error, - ), - ), - ); + const timed = async (phase: WakePhase, run: () => Promise) => { + const started = Date.now(); + try { + return await run(); + } finally { + args.onPhase?.(phase, Date.now() - started); } - if (!wasRunning) { - const { [SANDBOX_PATHS.agents]: previous, ...others } = + }; + const exclusive = args.exclusive ?? unlocked; + try { + const resumed = await exclusive(async () => { + const sandbox = await getSandboxToWake(args.providerSandboxId); + if (!sandbox) throw new SandboxUnavailableError(args.providerSandboxId); + const hostTarget = sandbox.domain(HOST_SERVICE_PORT); + const wasRunning = sandbox.status === "running"; + if (args.onlyIfRunning && !wasRunning) return null; + let vcpus = sandbox.vcpus ?? args.claim.vcpus; + let resumedByAnotherWake = false; + const { [SANDBOX_PATHS.agents]: previousAgents, ...kept } = sandbox.mounts ?? {}; const agents = await agentToolsMounts( agentToolsDriveName(args.claim, sandbox.region), ); - if (agents || previous) { - await sandbox - .update({ mounts: { ...others, ...agents } }) - .catch((error) => - console.warn( - `[sandbox] mount update failed for ${args.providerSandboxId}`, - error, - ), + const mounts = composeMounts(kept, args.mounts, agents) ?? {}; + if (!wasRunning) { + const drive = readWriteDriveAt(mounts, SANDBOX_PATHS.drive); + if (drive) { + const outcome = await Sentry.startSpan( + { name: "drive detach", op: "sandbox.provider" }, + () => + timed("detach", () => + waitForDriveDetached(drive, { + heldByCaller: heldByThisRunningBox(args.providerSandboxId), + }), + ), ); - } - } - if (!wasRunning && args.beforeResume) { - await args.beforeResume({ memoryMb: vcpus * MEMORY_MB_PER_VCPU, vcpus }); - } - if (wasRunning) { - const remaining = (sandbox.expiresAt?.getTime() ?? 0) - Date.now(); - if (remaining < EXTEND_BELOW_MS) { - // Past the plan's per-session cap the extension is refused; the - // session then ends and the next open resumes it. - await sandbox.extendTimeout(SESSION_TIMEOUT_MS).catch(() => {}); - } - } - // A box that is already serving needs neither a fresh identity file nor - // another boot. The policy still goes out: a credential may have aged. - const serving = wasRunning && (await hostServiceAnswers(hostTarget)); - await Sentry.startSpan( - { name: "apply claim", op: "sandbox.provider" }, - () => - Promise.all([ - sandbox - .update({ networkPolicy: args.claim.networkPolicy }) - .catch((error) => + // A concurrent wake resumed this box with its drive meanwhile. + resumedByAnotherWake = outcome === "held"; + } + if (!resumedByAnotherWake && vcpus !== args.claim.vcpus) { + await Sentry.startSpan( + { name: "resize", op: "sandbox.provider" }, + () => + sandbox + .update({ resources: { vcpus: args.claim.vcpus } }) + .then(() => { + vcpus = args.claim.vcpus; + }) + .catch((error) => + console.warn( + `[sandbox] resize to ${args.claim.vcpus} vCPUs failed for ${args.providerSandboxId}`, + error, + ), + ), + ); + } + if ( + !resumedByAnotherWake && + (agents || previousAgents || args.mounts) + ) { + const update = Sentry.startSpan( + { name: "mount", op: "sandbox.provider" }, + () => timed("mount", () => sandbox.update({ mounts })), + ); + // Without its drive the workspace would boot on an empty /workspace. + if (args.mounts) await update; + else + await update.catch((error) => console.warn( - `[sandbox] policy update failed for ${args.providerSandboxId}`, + `[sandbox] mount update failed for ${args.providerSandboxId}`, error, ), - ), - serving ? null : writeIdentity(sandbox, args.claim.identity), - writeSignInFiles(sandbox, args.claim.files), - ]), - ); - if (!serving) { - await Sentry.startSpan({ name: "start boot", op: "sandbox" }, () => - runBoot(sandbox, args.claim.hostSecret), - ); - } - if (resumed && args.beforeResume) { - await args.beforeResume({ - memoryMb: resumed.vcpus * MEMORY_MB_PER_VCPU, - vcpus: resumed.vcpus, - resumedSession: { - sessionId: resumed.sessionId, - expiresAt: sandbox.expiresAt ?? null, + ); + } + } + const running = wasRunning || resumedByAnotherWake; + if (!running && args.beforeResume) { + await args.beforeResume({ + memoryMb: vcpus * MEMORY_MB_PER_VCPU, + vcpus, + }); + } + const resumeStarted = Date.now(); + const first = wasRunning + ? { sandbox, session: sandbox.currentSession() } + : await resumeSession(args.providerSandboxId); + const { result, live } = await withLiveSession({ + providerSandboxId: args.providerSandboxId, + mounts, + live: first, + onResumed: async (resumedLive) => { + await args.beforeResume?.({ + memoryMb: resumedLive.session.vcpus * MEMORY_MB_PER_VCPU, + vcpus: resumedLive.session.vcpus, + resumedSession: { + sessionId: resumedLive.session.sessionId, + expiresAt: resumedLive.sandbox.expiresAt ?? null, + }, + }); + }, + use: async (current) => { + if (current === first && wasRunning) { + const remaining = (sandbox.expiresAt?.getTime() ?? 0) - Date.now(); + if (remaining < EXTEND_BELOW_MS) { + // Past the plan's per-session cap the extension is refused; the + // session then ends and the next open resumes it. + await current.session + .extendTimeout(SESSION_TIMEOUT_MS) + .catch(() => {}); + } + } + // A box that is already serving needs neither a fresh identity file nor + // another boot. The policy still goes out: a credential may have aged. + const serving = + current === first && + running && + (await hostServiceAnswers(hostTarget)); + await Sentry.startSpan( + { name: "apply claim", op: "sandbox.provider" }, + () => + Promise.all([ + current.session + .update({ networkPolicy: args.claim.networkPolicy }) + .catch((error) => + console.warn( + `[sandbox] policy update failed for ${args.providerSandboxId}`, + error, + ), + ), + serving + ? null + : writeIdentity(current.session, args.claim.identity), + writeSignInFiles(current.session, args.claim.files), + ]), + ); + const bootId = serving + ? undefined + : await Sentry.startSpan( + { name: "start boot", op: "sandbox" }, + () => runBoot(current.session, args.claim.hostSecret), + ); + return { serving, bootId, resumedHere: current !== first }; }, }); - } + const { serving, bootId } = result; + const booted = !running || result.resumedHere; + args.onPhase?.("resume", Date.now() - resumeStarted); + return { ...live, hostTarget, booted, serving, bootId }; + }); + if (!resumed) return null; + const { sandbox, session, hostTarget, booted, serving, bootId } = resumed; const answer = await Sentry.startSpan( { name: "settle", op: "sandbox" }, () => - settleSandbox({ - providerSandboxId: args.providerSandboxId, - hostTarget, - claim: args.claim, - }), - ); + timed("boot", () => + settleSandbox({ + providerSandboxId: args.providerSandboxId, + hostTarget, + claim: args.claim, + bootId, + }), + ), + ).catch(async (error) => { + // Stopped: the next wake mounts the drive afresh, and a refused box bills nothing. + if ( + error instanceof SandboxDriveMissingError || + error instanceof SandboxDriveRefusedError + ) { + // Under the lock, so no caller resumes the box between its check and this stop. + await exclusive(() => session.stop()).catch(() => {}); + } + throw error; + }); return { hostTarget, - booted: !wasRunning, + booted, rebooted: !serving, answer, expiresAt: sandbox.expiresAt ?? null, @@ -713,13 +1111,47 @@ const INHERITED_IDENTITY = [ CODEX_SYSTEM_CONFIG_PATH, ]; +/** Agent sessions are the person's; on a drive they would reach every fork. */ +const DRIVE_IDENTITY = [ + `${SANDBOX_PATHS.home}/.claude/projects`, + `${SANDBOX_PATHS.home}/.codex/sessions`, +]; + +/** + * The paths to remove so a fork starts clean. With a drive, also where those + * paths sit on the drive itself, for a box no boot has laid out. + */ +export function inheritedIdentityPaths(withDrive: boolean): string[] { + if (!withDrive) return INHERITED_IDENTITY; + const paths = [...INHERITED_IDENTITY, ...DRIVE_IDENTITY]; + const onDrive = paths.flatMap((path) => sandboxDrivePathOf(path) ?? []); + return [...paths, ...onDrive, SANDBOX_PATHS.driveState]; +} + /** Removes what a box wrote for the workspace it was, so a fork starts clean. */ -export async function stripWorkspaceIdentity(sandbox: Sandbox): Promise { - await sandbox.runCommand({ - cmd: "rm", - args: ["-rf", ...INHERITED_IDENTITY], +export async function stripWorkspaceIdentity( + sandbox: Pick, + options: { withDrive?: boolean } = {}, +): Promise { + const withDrive = options.withDrive ?? false; + // Unmounted, the rm would miss the drive, and every fork would refuse it as another workspace's. + const result = await sandbox.runCommand({ + cmd: "bash", + args: [ + "-c", + `${withDrive ? `mountpoint -q ${SANDBOX_PATHS.drive} || exit 3; ` : ""}rm -rf -- "$@"`, + "strip", + ...inheritedIdentityPaths(withDrive), + ], sudo: true, }); + if (result.exitCode !== 0) { + throw new Error( + result.exitCode === 3 + ? `${SANDBOX_PATHS.drive} is not mounted; the golden's drive keeps the source's identity` + : `stripping the workspace identity exited ${result.exitCode}`, + ); + } } /** @@ -745,25 +1177,39 @@ export async function stopAndSnapshot( * would start from the source's last stop) and created with an empty env. * Taking that snapshot ends the source's session. The source stays stopped: * the caller starts it again through a wake, which meters the new session. + * + * A source with a drive gives the golden a fork of it, taken once the source + * has released it; the golden drive is stripped and released the same way. */ export async function promoteSandboxToEnvironment(args: { sourceSandbox: string; goldenName: string; organizationId: string; + sourceDriveName?: string | null; + drives?: DriveHost; }): Promise<{ goldenName: string; region: string; + goldenDriveName: string | null; /** The session the snapshot ended; null when the source was not running. */ stopped: { sessionId: string | null } | null; }> { + const drives = args.drives ?? vercelDrives(); const source = await getSandbox(args.sourceSandbox); if (!source) throw new SandboxUnavailableError(args.sourceSandbox); const stopped = source.status === "running" ? { sessionId: sessionIdOf(source) } : null; + const snapshotId = await snapshotNow(source); + let goldenDriveName: string | null = null; + if (args.sourceDriveName) { + goldenDriveName = goldenDriveNameFor(args.goldenName); + await waitForDriveDetached(args.sourceDriveName, { drives }); + await drives.fork(args.sourceDriveName, goldenDriveName); + } const golden = await Sandbox.create({ ...credentials(), name: args.goldenName, - source: { type: "snapshot", snapshotId: await snapshotNow(source) }, + source: { type: "snapshot", snapshotId }, ports: publishedPorts(), timeout: GOLDEN_SESSION_TIMEOUT_MS, region: source.region as SandboxRegion, @@ -773,12 +1219,21 @@ export async function promoteSandboxToEnvironment(args: { snapshotExpiration: 0, keepLastSnapshots: { count: 1 }, tags: environmentTags(args.organizationId), + ...(goldenDriveName + ? { mounts: workspaceDriveMounts(goldenDriveName) } + : {}), }); - await stripWorkspaceIdentity(golden); + await stripWorkspaceIdentity(golden, { withDrive: goldenDriveName !== null }); const created = golden.currentSnapshotId; await golden.stop(); await waitForStopSnapshot(args.goldenName, created); - return { goldenName: args.goldenName, region: source.region, stopped }; + if (goldenDriveName) await waitForDriveDetached(goldenDriveName, { drives }); + return { + goldenName: args.goldenName, + region: source.region, + goldenDriveName, + stopped, + }; } /** @@ -837,7 +1292,10 @@ export async function sandboxExists( export async function stopSandbox( providerSandboxId: string, ): Promise<{ sessionId: string | null }> { - const sandbox = await getSandbox(providerSandboxId); + const sandbox = await getSandbox(providerSandboxId).catch((error) => { + if (isSnapshotExpired(error)) return null; + throw error; + }); if (!sandbox) return { sessionId: null }; const sessionId = sessionIdOf(sandbox); if (sandbox.status === "running") await sandbox.stop(); @@ -989,10 +1447,34 @@ export async function listActiveWorkspaceSandboxes( return found; } -/** Best-effort: a sandbox already gone is the state we wanted. */ +/** + * Best-effort: a sandbox already gone is the state we wanted. One whose + * snapshot expired cannot be fetched to delete; it holds no session and no + * snapshot, and its name is retired. + */ export async function deleteSandbox(providerSandboxId: string): Promise { - const sandbox = await getSandbox(providerSandboxId); + const sandbox = await getSandbox(providerSandboxId).catch((error) => { + if (isSnapshotExpired(error)) return null; + throw error; + }); if (!sandbox) return; // Snapshots outlive a sandbox by default and keep billing storage. await sandbox.delete({ deleteOrphanSnapshots: true }); } + +/** + * A sandbox deleted while it holds its drive read-write leaves the drive + * attached for minutes, so the box is stopped and the drive released first. + * Throws while the drive is still attached; the caller retries. + */ +export async function deleteSandboxAndDrive( + providerSandboxId: string, + driveName: string | null | undefined, + drives: DriveHost = vercelDrives(), +): Promise { + if (!driveName) return deleteSandbox(providerSandboxId); + await stopSandbox(providerSandboxId); + await waitForDriveDetached(driveName, { drives }); + await deleteSandbox(providerSandboxId); + await deleteWorkspaceDrive(driveName, drives); +} diff --git a/packages/trpc/src/router/cloud-workspace/client-status.test.ts b/packages/trpc/src/router/cloud-workspace/client-status.test.ts index 7f358b44713..0c3c5918d2e 100644 --- a/packages/trpc/src/router/cloud-workspace/client-status.test.ts +++ b/packages/trpc/src/router/cloud-workspace/client-status.test.ts @@ -1,5 +1,5 @@ import { describe, expect, test } from "bun:test"; -import { cloudWorkspaceStatusFor } from "./client-status"; +import { cloudWorkspaceStatusFor, withClientStatus } from "./client-status"; const desktop = (version: string) => ({ product: "desktop" as const, version }); @@ -30,3 +30,15 @@ describe("cloudWorkspaceStatusFor", () => { ).toBe("asleep"); }); }); + +describe("withClientStatus output", () => { + test("never carries the drive columns, which released clients never saw", () => { + const out = withClientStatus(null, { + id: "ws-1", + status: "awake" as const, + driveName: "drive-ws-1", + driveLaidOutAt: new Date(), + }); + expect(out).toEqual({ id: "ws-1", status: "awake" }); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/client-status.ts b/packages/trpc/src/router/cloud-workspace/client-status.ts index 1aaa16165ef..1ed6fa4c930 100644 --- a/packages/trpc/src/router/cloud-workspace/client-status.ts +++ b/packages/trpc/src/router/cloud-workspace/client-status.ts @@ -38,9 +38,17 @@ export function cloudWorkspaceStatusFor( return status; } +/** Only the control plane reads these; released clients never saw them. */ +type DriveColumns = "driveName" | "driveLaidOutAt"; + export function withClientStatus( client: ApiClientInfo | null, row: Row, -): Row { - return { ...row, status: cloudWorkspaceStatusFor(client, row.status) }; +): Omit { + const { + driveName: _driveName, + driveLaidOutAt: _driveLaidOutAt, + ...rest + } = row as Row & Partial>; + return { ...rest, status: cloudWorkspaceStatusFor(client, row.status) }; } diff --git a/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts b/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts index c9bd47c194d..3cc9b9b1a28 100644 --- a/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts +++ b/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts @@ -37,7 +37,6 @@ import { logControl } from "../../lib/control-log"; import { callMoves } from "../../lib/moves"; import { nudge, readWorkspaceForwards } from "../../lib/realtime"; import { - deleteSandbox, describeSandbox, HOST_SERVICE_PORT, listRemoteBranches, @@ -90,6 +89,7 @@ import { queueProvision, startCloudWorkspace, } from "./start"; +import { retireAndTearDown } from "./teardown"; import { transitionCloudWorkspace } from "./transition"; import { markSandboxUnavailable, @@ -916,7 +916,13 @@ export const cloudWorkspaceRouter = { meterError, ), ); - await deleteSandbox(row.providerSandboxId); + await retireAndTearDown( + row, + and( + eq(cloudWorkspaces.status, "archived"), + eq(cloudWorkspaces.deletedAt, archivedAt), + ), + ); await reapWorkspaceFiles(row.id); }); } diff --git a/packages/trpc/src/router/cloud-workspace/drive-laid-out.ts b/packages/trpc/src/router/cloud-workspace/drive-laid-out.ts new file mode 100644 index 00000000000..ddb4eaf700f --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/drive-laid-out.ts @@ -0,0 +1,18 @@ +import { db } from "@superset/db/client"; +import { cloudWorkspaces } from "@superset/db/schema"; +import { and, eq, isNull, sql } from "drizzle-orm"; + +/** Once a boot answered with the drive laid out, every later wake expects the drive to name this workspace. */ +export async function recordDriveLaidOut( + cloudWorkspaceId: string, +): Promise { + await db + .update(cloudWorkspaces) + .set({ driveLaidOutAt: sql`now()` }) + .where( + and( + eq(cloudWorkspaces.id, cloudWorkspaceId), + isNull(cloudWorkspaces.driveLaidOutAt), + ), + ); +} diff --git a/packages/trpc/src/router/cloud-workspace/provision-drive.test.ts b/packages/trpc/src/router/cloud-workspace/provision-drive.test.ts new file mode 100644 index 00000000000..1b432fc1366 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/provision-drive.test.ts @@ -0,0 +1,288 @@ +import { beforeEach, describe, expect, test } from "bun:test"; +import { db } from "@superset/db/client"; +import { stub } from "../../../test/stub"; +import * as realtime from "../../lib/realtime"; +import * as sandbox from "../../lib/sandbox"; +import * as chatMoves from "../chat-move/utils/queueChatMove"; +import * as laidOut from "./drive-laid-out"; +import * as jobs from "./jobs"; +import * as meter from "./meter"; +import { provisionCloudWorkspace } from "./provision"; +import * as resumeLock from "./resume-lock"; +import * as expiry from "./sandbox-expiry"; +import * as sleep from "./sandbox-sleep"; +import * as teardown from "./teardown"; +import * as transition from "./transition"; + +const ROW = { + id: "00000000-0000-4000-8000-000000000001", + organizationId: "00000000-0000-4000-8000-000000000002", + createdByUserId: null, + status: "provisioning", + providerSandboxId: "ws-box", + driveName: null as string | null, + driveLaidOutAt: null as Date | null, +}; + +let row: typeof ROW; +let events: string[]; +let settle: () => Promise; +let create: () => Promise; +let fork: "existing" | "forked"; +let wins: (to: string) => boolean; +let current: { status: string; providerSandboxId?: string } | undefined; +let lockKeys: string[]; +let rowBox: string; +// The row's status as the transitions see it: another delivery or a wake may have moved it. +let rowStatus: string; +let authorize: () => Promise<{ bootId: string | null }>; +let requeued: unknown[]; + +let reads = 0; +stub(db.query.cloudWorkspaces, { + findFirst: async () => (reads++ === 0 ? row : current), +}); +stub(db, { + update: () => ({ + set: (values: Record) => ({ + where: async () => { + events.push(`update ${Object.keys(values).join(",")}`); + }, + }), + }), +}); +stub(sandbox, { + buildSandboxClaim: async () => ({ + claim: { vcpus: 4 }, + environment: { region: "iad1", goldenDriveName: "env-drive-g" }, + agentCredentialDigest: "d", + driveName: "drive-ws-box", + }), + forkWorkspaceDrive: async () => fork, + provisionSandbox: async () => { + await create(); + return { + providerSandboxId: "ws-box", + sandboxUrl: "https://box.invalid", + hostTarget: "https://box.invalid", + expiresAt: null, + sessionId: "session-1", + bootId: "boot-1", + }; + }, + settleSandbox: async () => settle(), + stopSandbox: async () => { + events.push("stop"); + return { sessionId: "session-1" }; + }, + recordBootSpans: () => {}, +}); +stub(meter, { + authorizeBoot: async () => authorize(), + attachBootSession: async () => {}, + releaseBoot: async () => {}, +}); +type Move = { + from: string[]; + to: string; + set?: Record; + onBox?: string; +}; +function move(args: Move): boolean { + if (!wins(args.to)) return false; + if (!args.from.includes(rowStatus)) return false; + if (args.onBox !== undefined && args.onBox !== rowBox) return false; + events.push( + `to ${args.to}${args.set && "providerSandboxId" in args.set ? ` box=${args.set.providerSandboxId === "ws-box" ? "same" : "new"}` : ""}`, + ); + rowStatus = args.to; + return true; +} +stub(transition, { + transitionCloudWorkspace: async (args: Move) => move(args), + failCloudWorkspace: async (args: Omit) => + move({ ...args, to: "failed" }), +}); +stub(sleep, { + recordSandboxAsleep: async () => { + events.push("asleep"); + }, +}); +stub(expiry, { recordSandboxAwake: async () => {} }); +stub(laidOut, { + recordDriveLaidOut: async () => { + events.push("laid out"); + }, +}); +stub(teardown, { + deleteSandboxAndDriveLater: async (args: { + driveName: string | null; + keepDrive?: boolean; + }) => { + events.push( + `teardown drive=${args.driveName}${args.keepDrive ? " kept" : ""}`, + ); + }, +}); +stub(chatMoves, { + queueChatMovesTo: async () => {}, + failChatMovesTo: async () => {}, +}); +stub(realtime, { nudge: () => {} }); +stub(jobs, { + publishCloudWorkspaceJob: async (job: { body: unknown }) => { + requeued.push(job.body); + }, +}); +stub(resumeLock, { + sandboxResumeLock: (key: string) => { + lockKeys.push(key); + return (step: () => Promise) => step(); + }, +}); + +beforeEach(() => { + row = { ...ROW }; + events = []; + settle = async () => ({ stamps: [], answeredAt: 0 }); + create = async () => {}; + fork = "forked"; + wins = () => true; + current = undefined; + reads = 0; + lockKeys = []; + rowBox = "ws-box"; + rowStatus = "provisioning"; + authorize = async () => ({ bootId: null }); + requeued = []; +}); + +describe("provisionCloudWorkspace with a drive", () => { + test("records the drive laid out once the boot answered", async () => { + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "provisioned", + ); + expect(events).toContain("laid out"); + }); + + test("a drive refusal at create leaves the workspace asleep for the next wake, never failed", async () => { + settle = async () => { + throw new sandbox.SandboxDriveMissingError( + "ws-box", + "drive-ws-box", + "drive.missing", + ); + }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "provisioned", + ); + expect(events).toContain("stop"); + expect(events).toContain("asleep"); + expect(events).not.toContain("to failed"); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + }); + + test("a second delivery on the same box that fails after the first brought the row up leaves the live workspace alone", async () => { + // The first delivery made the row awake and the user works; this one is refused by the spend cap. + rowStatus = "awake"; + authorize = async () => { + throw new Error("spend cap reached"); + }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "failed", + ); + expect(rowStatus).toBe("awake"); + expect(events).not.toContain("stop"); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + expect(events.some((event) => event.startsWith("to "))).toBe(false); + }); + + test("a lock held by another caller requeues the provision; nothing is stopped or failed", async () => { + create = async () => { + throw new sandbox.SandboxBusyError("drive-ws-box"); + }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "retrying", + ); + expect(requeued).toEqual([{ cloudWorkspaceId: ROW.id, lockRetries: 1 }]); + expect(events).not.toContain("stop"); + expect(events.some((event) => event.startsWith("to "))).toBe(false); + }); + + test("a lock still held after the last retry fails only a row still provisioning", async () => { + create = async () => { + throw new sandbox.SandboxBusyError("drive-ws-box"); + }; + await provisionCloudWorkspace({ + cloudWorkspaceId: ROW.id, + lockRetries: 10, + }); + expect(requeued).toEqual([]); + expect(events).toContain("to failed box=same"); + }); + + test("a provision overtaken by an undo's second provision gives up its box, keeps the drive, and never marks the row", async () => { + // Deleted between claim and create, then undone: the row names another box now. + rowBox = "ws-box-2"; + current = { status: "provisioning", providerSandboxId: "ws-box-2" }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "skipped", + ); + expect(events).toContain("teardown drive=drive-ws-box kept"); + expect(events.some((event) => event.startsWith("to "))).toBe(false); + expect(events).not.toContain("laid out"); + }); + + test("the create holds the lock of the drive, whatever the box is called", async () => { + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(lockKeys).toEqual(["drive-ws-box"]); + }); + + test("a provision that lost its row to an archive leaves the box to that archive's reap", async () => { + wins = (to) => to !== "awake"; + current = { status: "archived", providerSandboxId: "ws-box" }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "skipped", + ); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + expect(events).not.toContain("stop"); + }); + + test("a failed create retires the box's name in the same write that fails the row, then tears it down", async () => { + create = async () => { + throw new Error("create refused"); + }; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events).toEqual([ + "update driveName", + "to failed box=new", + "teardown drive=drive-ws-box", + ]); + }); + + test("a failed create whose row was archived meanwhile tears nothing down", async () => { + create = async () => { + throw new Error("create refused"); + }; + wins = (to) => to !== "failed"; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + }); + + test("a failed create deletes a drive it forked, never one it found", async () => { + create = async () => { + throw new Error("create refused"); + }; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events).toContain("teardown drive=drive-ws-box"); + + events = []; + reads = 0; + rowStatus = "provisioning"; + row = { ...ROW, driveName: "drive-ws-box" }; + fork = "existing"; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events).toContain("teardown drive=null"); + expect(events).toContain("to failed box=new"); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/provision.ts b/packages/trpc/src/router/cloud-workspace/provision.ts index 8bc3b4c365c..9bea9312dc6 100644 --- a/packages/trpc/src/router/cloud-workspace/provision.ts +++ b/packages/trpc/src/router/cloud-workspace/provision.ts @@ -1,4 +1,3 @@ -import { randomBytes } from "node:crypto"; import * as Sentry from "@sentry/core"; import { db } from "@superset/db/client"; import type { CloudWorkspaceFailureReason } from "@superset/db/enums"; @@ -7,25 +6,35 @@ import { cloudUsageRefusalKey } from "@superset/shared/billing"; import type { CloudAgentLaunch } from "@superset/shared/cloud-agent-launch"; import { CLOUD_WORKSPACE_PROVISION_TRANSACTION } from "@superset/shared/constants"; import { eq } from "drizzle-orm"; -import { controlTags } from "../../lib/control-log"; +import { type ControlScope, controlTags } from "../../lib/control-log"; import { nudge } from "../../lib/realtime"; import { bootPhases, buildSandboxClaim, - deleteSandbox, + forkWorkspaceDrive, provisionSandbox, + SandboxBusyError, + SandboxDriveMissingError, + SandboxDriveRefusedError, + type SandboxEnvironment, SandboxNotReadyError, settleSandbox, stopSandbox, + workspaceDriveMounts, } from "../../lib/sandbox"; import { failChatMovesTo, queueChatMovesTo, } from "../chat-move/utils/queueChatMove"; +import { recordDriveLaidOut } from "./drive-laid-out"; import { generateCloudWorkspaceName } from "./generate-name"; +import { publishCloudWorkspaceJob } from "./jobs"; import { attachBootSession, authorizeBoot, boxFor, releaseBoot } from "./meter"; +import { sandboxResumeLock } from "./resume-lock"; import { recordSandboxAwake } from "./sandbox-expiry"; -import { recordBootTrace } from "./sandbox-sleep"; +import { nextSandboxNameFor } from "./sandbox-name"; +import { recordBootTrace, recordSandboxAsleep } from "./sandbox-sleep"; +import { deleteSandboxAndDriveLater } from "./teardown"; import { failCloudWorkspace, transitionCloudWorkspace } from "./transition"; export const FALLBACK_NAME = "Cloud workspace"; @@ -35,14 +44,7 @@ export function sandboxNameFor(cloudWorkspaceId: string): string { return `ws-${cloudWorkspaceId.replaceAll("-", "").slice(0, 24)}`; } -/** - * A later box for the same row. Vercel refuses a deleted sandbox's name - * with `image_not_ready` for a few minutes after the delete. - */ -export function nextSandboxNameFor(cloudWorkspaceId: string): string { - const prefix = cloudWorkspaceId.replaceAll("-", "").slice(0, 16); - return `ws-${prefix}${randomBytes(4).toString("hex")}`; -} +export { nextSandboxNameFor } from "./sandbox-name"; export interface ProvisionCloudWorkspaceInput { cloudWorkspaceId: string; @@ -53,8 +55,16 @@ export interface ProvisionCloudWorkspaceInput { namingPrompt?: string; /** A built-in agent to run once the sandbox is up; see cloud-agent-launch. */ launch?: CloudAgentLaunch; + /** How many times this provision found the box's resume lock held and requeued itself. */ + lockRetries?: number; } +/** A lock held this long by others means something is stuck: the row is failed as for any error. */ +const PROVISION_LOCK_RETRIES = 10; +const PROVISION_LOCK_RETRY_SECONDS = 15; + +type CloudWorkspaceRow = typeof cloudWorkspaces.$inferSelect; + function provisionFailureReason(error: unknown): CloudWorkspaceFailureReason { switch (cloudUsageRefusalKey(error)) { case "serverError.cloudWorkspace.usageLimitReached": @@ -62,14 +72,48 @@ function provisionFailureReason(error: unknown): CloudWorkspaceFailureReason { case "serverError.cloudWorkspace.cloudNeedsPaidPlan": return "needs_paid_plan"; } - return error instanceof SandboxNotReadyError + return error instanceof SandboxNotReadyError || + error instanceof SandboxDriveRefusedError ? "host_not_ready" : "create_failed"; } +/** + * The name is on the row before the drive exists, so a delete that races + * this provision still finds it. An unarchived workspace keeps its drive when + * it still exists; a drive forked afresh has not been laid out yet. + */ +async function claimWorkspaceDrive( + row: CloudWorkspaceRow, + driveName: string, + environment: SandboxEnvironment, +): Promise<"existing" | "forked"> { + if (!row.driveName) { + await db + .update(cloudWorkspaces) + .set({ driveName }) + .where(eq(cloudWorkspaces.id, row.id)); + } + const outcome = await Sentry.startSpan({ name: "drive", op: "sandbox" }, () => + forkWorkspaceDrive({ + name: driveName, + region: environment.region, + goldenDriveName: environment.goldenDriveName ?? null, + }), + ); + if (outcome === "forked" && row.driveLaidOutAt) { + await db + .update(cloudWorkspaces) + .set({ driveLaidOutAt: null }) + .where(eq(cloudWorkspaces.id, row.id)); + } + return outcome; +} + export type ProvisionCloudWorkspaceOutcome = | "provisioned" | "skipped" + | "retrying" | "failed"; /** @@ -111,6 +155,10 @@ async function provision( where: eq(cloudWorkspaces.id, input.cloudWorkspaceId), }); if (!row) return "skipped"; + const scope: ControlScope = { + cloudWorkspaceId: row.id, + organizationId: row.organizationId, + }; if (row.status !== "provisioning") return "skipped"; const providerSandboxId = row.providerSandboxId; @@ -128,11 +176,25 @@ async function provision( }, ); let bootId: string | null = null; + let driveName = row.driveName; + // Only a drive this provision forked may go with a failed box; an unarchived workspace's is its work. + let forkedDrive = false; + // Another delivery on the same box name, or a wake, may have brought the row up: only this job's own row is its to fail. + let madeAwake = false; try { - const { claim, environment, agentCredentialDigest } = - await Sentry.startSpan({ name: "claim", op: "sandbox" }, () => - buildSandboxClaim({ row, launch: input.launch, withRepoHooks: true }), - ); + const { + claim, + environment, + agentCredentialDigest, + driveName: claimedDrive, + } = await Sentry.startSpan({ name: "claim", op: "sandbox" }, () => + buildSandboxClaim({ + row, + launch: input.launch, + withRepoHooks: true, + provisioning: true, + }), + ); ({ bootId } = await Sentry.startSpan( { name: "authorize boot", op: "billing" }, () => @@ -145,31 +207,61 @@ async function provision( traceId: span.spanContext().traceId, }), )); - if (bootId) span.setAttribute("boot.id", bootId); + if (bootId) { + scope.bootId = bootId; + span.setAttribute("boot.id", bootId); + } + driveName = claimedDrive; + if (driveName) { + forkedDrive = + (await claimWorkspaceDrive(row, driveName, environment)) === "forked"; + } const sandbox = await Sentry.startSpan( { name: "create", op: "sandbox" }, - () => provisionSandbox({ name: providerSandboxId, environment, claim }), + () => + provisionSandbox({ + name: providerSandboxId, + environment, + claim, + mounts: workspaceDriveMounts(driveName), + exclusive: sandboxResumeLock(driveName ?? providerSandboxId, { + wait: true, + }), + }), ); const ready = await transitionCloudWorkspace({ id: row.id, from: ["provisioning"], to: "awake", + onBox: providerSandboxId, set: { providerSandboxId: sandbox.providerSandboxId, sandboxUrl: sandbox.sandboxUrl, bootAgentCredentialDigest: agentCredentialDigest, }, }); + madeAwake = ready; if (!ready) { - // Deleted while the box was being made: the delete won the row, so - // the box it never knew about goes with it. A duplicate delivery of - // this job loses the row to the first one and must leave the box be: - // the name is per workspace, so it is the same box. + // Deleted while the box was being made. The box is the row's, and the + // archive's own reap stops and deletes it after that archive's grace + // period: an undo, more work and a later archive must not lose it to + // this job. A duplicate delivery loses the row to the first one. const current = await db.query.cloudWorkspaces.findFirst({ where: eq(cloudWorkspaces.id, row.id), - columns: { status: true }, + columns: { status: true, providerSandboxId: true }, }); - if (current?.status === "archived") { + // An undo gave the row another box and a second provision: this box is no row's, and must not keep the drive. + const overtaken = + current !== undefined && + current.providerSandboxId !== providerSandboxId; + if (overtaken) { + if (bootId) await releaseBoot(bootId, "provision_abandoned"); + await deleteSandboxAndDriveLater({ + providerSandboxId, + driveName, + keepDrive: true, + }); + } else if (current?.status === "archived") { await failChatMovesTo(row.id, "The cloud workspace was deleted").catch( (error) => console.error( @@ -177,7 +269,6 @@ async function provision( error, ), ); - await deleteSandbox(providerSandboxId); if (bootId) await releaseBoot(bootId, "provision_abandoned"); } await naming.catch(() => {}); @@ -192,13 +283,7 @@ async function provision( } if (bootId) { await attachBootSession({ bootId, providerSandboxId }).catch((error) => - Sentry.captureException(error, { - tags: controlTags({ - cloudWorkspaceId: row.id, - organizationId: row.organizationId, - bootId, - }), - }), + Sentry.captureException(error, { tags: controlTags(scope) }), ); } // The box is booting; the environment it needs arrives once host-service @@ -211,25 +296,71 @@ async function provision( providerSandboxId, hostTarget: sandbox.hostTarget, claim, + bootId: sandbox.bootId, }), ); + if (driveName) await recordDriveLaidOut(row.id); await queueChatMovesTo(row.id).catch((error) => console.error( `[cloud-workspace] could not start the chat moving into ${row.id}`, error, ), ); - await recordBootTrace( - { cloudWorkspaceId: row.id, organizationId: row.organizationId, bootId }, - span, - bootPhases(answer), - ); + await recordBootTrace(scope, span, bootPhases(answer)); await naming.catch((error) => console.error(`[cloud-workspace] naming failed for ${row.id}`, error), ); return "provisioned"; } catch (error) { await naming.catch(() => {}); + // Another caller holds the box's lock: try again later, and touch nothing meanwhile. + const lockRetries = input.lockRetries ?? 0; + if ( + error instanceof SandboxBusyError && + lockRetries < PROVISION_LOCK_RETRIES + ) { + if (bootId) { + await releaseBoot(bootId, "provision_abandoned").catch((releaseError) => + console.error( + `[cloud-workspace] could not release boot ${bootId}`, + releaseError, + ), + ); + } + await publishCloudWorkspaceJob({ + path: "/api/cloud-workspaces/provision", + body: { ...input, lockRetries: lockRetries + 1 }, + delaySeconds: PROVISION_LOCK_RETRY_SECONDS, + runLocally: provisionCloudWorkspace, + }); + return "retrying"; + } + // The drive did not come up but its data is intact: the box sleeps, and the next wake mounts it afresh. + if (error instanceof SandboxDriveMissingError && madeAwake) { + const stopped = await stopSandbox(providerSandboxId).catch(() => ({ + sessionId: null, + })); + await recordSandboxAsleep(row.id, stopped); + nudge(row.organizationId, "cloud_workspaces"); + if (bootId) { + await releaseBoot(bootId, "provision_failed").catch((releaseError) => + console.error( + `[cloud-workspace] could not release boot ${bootId}`, + releaseError, + ), + ); + } + await queueChatMovesTo(row.id).catch((moveError) => + console.error( + `[cloud-workspace] could not start the chat moving into ${row.id}`, + moveError, + ), + ); + console.warn( + `[cloud-workspace] ${row.id} left asleep: its drive did not come up at create`, + ); + return "provisioned"; + } if (bootId) { await releaseBoot(bootId, "provision_failed").catch((releaseError) => console.error( @@ -243,33 +374,48 @@ async function provision( // away, it costs storage rather than compute meanwhile, and a delete // from the sidebar still removes it. Any other failure must not leak // a box, since billing started at provision. - const keepForDiagnosis = error instanceof SandboxNotReadyError; - await (keepForDiagnosis ? stopSandbox : deleteSandbox)( - providerSandboxId, - ).catch((teardownError) => { + const keepForDiagnosis = + error instanceof SandboxNotReadyError || + error instanceof SandboxDriveRefusedError; + // A box that goes is retired with the transition, so nothing can wake it while it is deleted. + const changed = await failCloudWorkspace({ + scope, + from: madeAwake ? ["provisioning", "awake", "asleep"] : ["provisioning"], + onBox: providerSandboxId, + set: keepForDiagnosis + ? { providerSandboxId } + : { + providerSandboxId: nextSandboxNameFor(row.id), + ...(forkedDrive ? { driveName: null, driveLaidOutAt: null } : {}), + }, + reason: provisionFailureReason(error), + error, + }); + const teardownFailed = (teardownError: unknown) => console.error( `[cloud-workspace] ${keepForDiagnosis ? "could not stop" : "leaked"} sandbox ${providerSandboxId}`, teardownError, ); - }); - await failCloudWorkspace({ - scope: { - cloudWorkspaceId: row.id, - organizationId: row.organizationId, - bootId, - }, - from: ["provisioning", "awake", "asleep"], - set: keepForDiagnosis ? { providerSandboxId } : {}, - reason: provisionFailureReason(error), - error, - }); - await failChatMovesTo(row.id, "The cloud workspace could not start").catch( - (moveError) => + if (changed && keepForDiagnosis) { + await stopSandbox(providerSandboxId).catch(teardownFailed); + } + if (changed && !keepForDiagnosis) { + await deleteSandboxAndDriveLater({ + providerSandboxId, + driveName: forkedDrive ? driveName : null, + }).catch(teardownFailed); + } + if (changed) { + await failChatMovesTo( + row.id, + "The cloud workspace could not start", + ).catch((moveError) => console.error( `[cloud-workspace] could not fail the chat moves into ${row.id}`, moveError, ), - ); + ); + } console.error(`[cloud-workspace] provisioning failed for ${row.id}`, error); return "failed"; } diff --git a/packages/trpc/src/router/cloud-workspace/reap.test.ts b/packages/trpc/src/router/cloud-workspace/reap.test.ts index b200c936b57..3d4c903e7f8 100644 --- a/packages/trpc/src/router/cloud-workspace/reap.test.ts +++ b/packages/trpc/src/router/cloud-workspace/reap.test.ts @@ -28,6 +28,22 @@ let onArchiveHook: () => void = () => {}; let reapedFiles: string[] = []; stub(db.query.cloudWorkspaces, { findFirst: () => Promise.resolve(row) }); +// Whether the retiring write still finds the row as the reap read it. +let retires = true; +let retiredTo: Record | null = null; +stub(db, { + update: () => ({ + set: (values: Record) => ({ + where: () => ({ + returning: async () => { + if (!retires) return []; + retiredTo = values; + return [{ id: "row" }]; + }, + }), + }), + }), +}); let recordedAsleep: Array<[string, { sessionId: string | null }]> = []; let endedBoots: unknown[] = []; stub(sandboxSleep, { @@ -187,6 +203,8 @@ describe("reapArchivedCloudWorkspace", () => { reapedFiles = []; recordedAsleep = []; endedBoots = []; + retires = true; + retiredTo = null; }); test("the stop stage ends the stopped session's boot as archived", async () => { @@ -260,6 +278,36 @@ describe("reapArchivedCloudWorkspace", () => { expect(sandboxCalls).toEqual(["delete:ws-box"]); }); + test("the delete stage retires the box's names before it deletes them, so an undo gets a new box and drive", async () => { + row = { ...(row as Row), driveName: "drive-ws-box" } as Row; + expect( + await reapArchivedCloudWorkspace({ ...input, stage: "delete" }), + ).toBe("reaped"); + expect(retiredTo).toMatchObject({ driveName: null, driveLaidOutAt: null }); + expect(retiredTo?.providerSandboxId).not.toBe("ws-box"); + expect(queued).toContainEqual( + expect.objectContaining({ + path: "/api/cloud-workspaces/teardown", + body: { + providerSandboxId: "ws-box", + driveName: "drive-ws-box", + attempt: 0, + }, + }), + ); + }); + + test("an undo that wins the row before the retire leaves the box and drive alone", async () => { + row = { ...(row as Row), driveName: "drive-ws-box" } as Row; + retires = false; + expect( + await reapArchivedCloudWorkspace({ ...input, stage: "delete" }), + ).toBe("skipped"); + expect(sandboxCalls).toEqual([]); + expect(queued).toEqual([]); + expect(reapedFiles).toEqual([]); + }); + test("the delete stage removes the files sent to the workspace", async () => { await reapArchivedCloudWorkspace({ ...input, stage: "delete" }); expect(reapedFiles).toEqual([`file-of-${input.cloudWorkspaceId}`]); diff --git a/packages/trpc/src/router/cloud-workspace/reap.ts b/packages/trpc/src/router/cloud-workspace/reap.ts index 1f815c4a577..271bb126be2 100644 --- a/packages/trpc/src/router/cloud-workspace/reap.ts +++ b/packages/trpc/src/router/cloud-workspace/reap.ts @@ -1,10 +1,9 @@ import { db } from "@superset/db/client"; import { cloudWorkspaces } from "@superset/db/schema"; -import { eq } from "drizzle-orm"; +import { and, eq } from "drizzle-orm"; import { logControl } from "../../lib/control-log"; import { detachAll, reapOrphanFiles } from "../../lib/files"; import { - deleteSandbox, describeSandbox, rerunBoxStartHook, runBoxArchiveHook, @@ -14,6 +13,7 @@ import { import { publishCloudWorkspaceJob } from "./jobs"; import { queueSettleWorkspace, settleWorkspaceNow } from "./meter"; import { recordBootEnd, recordBoxStopped } from "./sandbox-sleep"; +import { retireAndTearDown } from "./teardown"; /** How long an archived workspace keeps its box running, so an undo finds it live. */ export const ARCHIVE_STOP_DELAY_SECONDS = 60; @@ -96,7 +96,15 @@ export async function reapArchivedCloudWorkspace( await settleWorkspaceNow(row.id).catch((error) => console.error(`[cloud-workspace] ${row.id} final settle failed`, error), ); - await deleteSandbox(row.providerSandboxId); + // The names go with this archive's deletion; an undo after it gets a new box and drive. + const retired = await retireAndTearDown( + row, + and( + eq(cloudWorkspaces.status, "archived"), + eq(cloudWorkspaces.deletedAt, new Date(input.archivedAt)), + ), + ); + if (!retired) return "skipped"; await reapWorkspaceFiles(row.id); logControl("archive.deleted", { cloudWorkspaceId: row.id, diff --git a/packages/trpc/src/router/cloud-workspace/resume-lock.ts b/packages/trpc/src/router/cloud-workspace/resume-lock.ts new file mode 100644 index 00000000000..cc50824927c --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/resume-lock.ts @@ -0,0 +1,41 @@ +import { dbWs } from "@superset/db/client"; +import { sql } from "drizzle-orm"; +import { type ResumeLock, SandboxBusyError } from "../../lib/sandbox"; + +/** How long a provision waits for another holder before it requeues itself. */ +const PROVISION_LOCK_WAIT = "20s"; + +/** + * One caller at a time resumes or creates a box on this key: the drive's name + * when there is one, since a drive has one writer whatever the box is + * called, else the box's. A transaction-scoped advisory lock, held while the + * step runs. Another holder throws SandboxBusyError: a wake answers "still + * starting" at once, and its retry sees the box as the first left it; a + * provision waits a moment, then requeues itself rather than fail its row. + */ +export function sandboxResumeLock( + key: string, + { wait = false }: { wait?: boolean } = {}, +): ResumeLock { + const lockKey = `sandbox-resume:${key}`; + return (step) => + dbWs.transaction(async (tx) => { + if (wait) { + await tx.execute( + sql`select set_config('lock_timeout', ${PROVISION_LOCK_WAIT}, true)`, + ); + await tx + .execute(sql`select pg_advisory_xact_lock(hashtext(${lockKey}))`) + .catch(() => { + throw new SandboxBusyError(key); + }); + return step(); + } + const result = await tx.execute( + sql`select pg_try_advisory_xact_lock(hashtext(${lockKey})) as locked`, + ); + const [row] = result.rows as Array<{ locked: boolean }>; + if (!row?.locked) throw new SandboxBusyError(key); + return step(); + }); +} diff --git a/packages/trpc/src/router/cloud-workspace/sandbox-name.ts b/packages/trpc/src/router/cloud-workspace/sandbox-name.ts new file mode 100644 index 00000000000..4c306b21e32 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/sandbox-name.ts @@ -0,0 +1,10 @@ +import { randomBytes } from "node:crypto"; + +/** + * A later box for the same row. Vercel refuses a deleted sandbox's name + * with `image_not_ready` for a few minutes after the delete. + */ +export function nextSandboxNameFor(cloudWorkspaceId: string): string { + const prefix = cloudWorkspaceId.replaceAll("-", "").slice(0, 16); + return `ws-${prefix}${randomBytes(4).toString("hex")}`; +} diff --git a/packages/trpc/src/router/cloud-workspace/teardown-guard.ts b/packages/trpc/src/router/cloud-workspace/teardown-guard.ts new file mode 100644 index 00000000000..cd3987b2743 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/teardown-guard.ts @@ -0,0 +1,57 @@ +import { db } from "@superset/db/client"; +import { cloudWorkspaces, environments } from "@superset/db/schema"; +import { and, eq, isNull } from "drizzle-orm"; + +/** + * Whether a workspace, in any status, or a live environment still names the + * sandbox or the drive. Every real teardown clears the row's names in the + * write that commits to it (retireAndTearDown), so a name still on a row is + * that row's: an archived row may be undone onto it, and a failed one keeps + * its box and drive for diagnosis with the work in them. + */ +export async function liveReferences(args: { + providerSandboxId: string; + driveName: string | null; +}): Promise<{ sandbox: boolean; drive: boolean }> { + const [workspaceBox, goldenBox, workspaceDrive, goldenDrive] = + await Promise.all([ + db + .select({ id: cloudWorkspaces.id }) + .from(cloudWorkspaces) + .where(eq(cloudWorkspaces.providerSandboxId, args.providerSandboxId)) + .limit(1), + db + .select({ id: environments.id }) + .from(environments) + .where( + and( + eq(environments.sourceRef, args.providerSandboxId), + isNull(environments.archivedAt), + ), + ) + .limit(1), + args.driveName + ? db + .select({ id: cloudWorkspaces.id }) + .from(cloudWorkspaces) + .where(eq(cloudWorkspaces.driveName, args.driveName)) + .limit(1) + : [], + args.driveName + ? db + .select({ id: environments.id }) + .from(environments) + .where( + and( + eq(environments.goldenDriveName, args.driveName), + isNull(environments.archivedAt), + ), + ) + .limit(1) + : [], + ]); + return { + sandbox: workspaceBox.length > 0 || goldenBox.length > 0, + drive: workspaceDrive.length > 0 || goldenDrive.length > 0, + }; +} diff --git a/packages/trpc/src/router/cloud-workspace/teardown.test.ts b/packages/trpc/src/router/cloud-workspace/teardown.test.ts new file mode 100644 index 00000000000..5cc817575f2 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/teardown.test.ts @@ -0,0 +1,229 @@ +import { beforeEach, describe, expect, test } from "bun:test"; +import { stub } from "../../../test/stub"; +import * as sandbox from "../../lib/sandbox"; +import * as jobs from "./jobs"; +import { + deleteSandboxAndDriveLater, + TEARDOWN_ATTEMPTS, + teardownSandbox, +} from "./teardown"; +import * as guard from "./teardown-guard"; + +let calls: string[]; +let attached: boolean; +let queued: Array<{ body: unknown; delaySeconds?: number }>; +let live: { sandbox: boolean; drive: boolean }; +let stopFails: boolean; +let holder: string; +let driveDeleteFails: boolean; + +stub(sandbox, { + stopSandbox: async (id: string) => { + if (stopFails) throw new Error("provider down"); + calls.push(`stop:${id}`); + return { sessionId: null }; + }, + waitForDriveDetached: async ( + name: string, + options: { heldByCaller?: (attachedTo: string) => Promise } = {}, + ) => { + calls.push(`wait:${name}`); + if (!attached) return "detached"; + if (await options.heldByCaller?.(holder)) return "held"; + throw new sandbox.DriveStillAttachedError(name, holder); + }, + deleteSandbox: async (id: string) => { + calls.push(`delete:${id}`); + }, + deleteWorkspaceDrive: async (name: string | null) => { + if (driveDeleteFails) throw new Error("drive delete refused"); + if (name) calls.push(`delete-drive:${name}`); + }, +}); +stub(guard, { liveReferences: async () => live }); +stub(jobs, { + publishCloudWorkspaceJob: async (args: { + body: unknown; + delaySeconds?: number; + }) => { + queued.push({ body: args.body, delaySeconds: args.delaySeconds }); + }, +}); + +beforeEach(() => { + calls = []; + attached = false; + queued = []; + live = { sandbox: false, drive: false }; + stopFails = false; + holder = "box"; + driveDeleteFails = false; +}); + +describe("teardownSandbox", () => { + test("stops the box, waits for its drive, then deletes the box and the drive", async () => { + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("deleted"); + expect(calls).toEqual([ + "stop:box", + "wait:drive-box", + "delete:box", + "delete-drive:drive-box", + ]); + }); + + test("a drive still attached requeues the job and deletes nothing", async () => { + attached = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 3, + }), + ).toBe("retrying"); + expect(calls).toEqual(["stop:box", "wait:drive-box"]); + expect(queued).toEqual([ + { + body: { providerSandboxId: "box", driveName: "drive-box", attempt: 4 }, + delaySeconds: 30, + }, + ]); + }); + + test("a box a revived workspace runs on is left alone", async () => { + live = { sandbox: true, drive: true }; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("kept"); + expect(calls).toEqual([]); + }); + + test("an unarchived workspace's new box keeps the drive; only the old box goes", async () => { + live = { sandbox: false, drive: true }; + expect( + await teardownSandbox({ + providerSandboxId: "old-box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("kept"); + expect(calls).toEqual(["stop:old-box", "wait:drive-box", "delete:old-box"]); + }); + + test("a box of the row's lost provision goes; the drive the row runs on is never deleted, failed or not", async () => { + // The row has since gone failed, and still names the drive with the user's work on it. + live = { sandbox: false, drive: false }; + expect( + await teardownSandbox({ + providerSandboxId: "lost-box", + driveName: "drive-box", + attempt: 0, + keepDrive: true, + }), + ).toBe("deleted"); + expect(calls).toEqual([ + "stop:lost-box", + "wait:drive-box", + "delete:lost-box", + ]); + }); + + test("a lost box goes even while the row's live box holds the kept drive", async () => { + attached = true; + holder = "live-box"; + expect( + await teardownSandbox({ + providerSandboxId: "lost-box", + driveName: "drive-box", + attempt: 0, + keepDrive: true, + }), + ).toBe("deleted"); + expect(calls).toEqual([ + "stop:lost-box", + "wait:drive-box", + "delete:lost-box", + ]); + expect(queued).toEqual([]); + }); + + test("any other failure is retried, not dropped", async () => { + stopFails = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("retrying"); + expect(queued).toHaveLength(1); + }); + + test("a drive a golden environment or another row names is never deleted", async () => { + live = { sandbox: false, drive: true }; + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-golden", + attempt: 0, + }); + expect(calls).not.toContain("delete-drive:drive-golden"); + }); + + test("a drive delete that fails is retried", async () => { + driveDeleteFails = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("retrying"); + expect(queued).toEqual([ + { + body: { providerSandboxId: "box", driveName: "drive-box", attempt: 1 }, + delaySeconds: 30, + }, + ]); + }); + + test("gives up after the last attempt instead of requeuing forever", async () => { + attached = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: TEARDOWN_ATTEMPTS - 1, + }), + ).toBe("abandoned"); + expect(queued).toEqual([]); + }); +}); + +describe("deleteSandboxAndDriveLater", () => { + test("a box without a drive is deleted now; one with a drive goes to the job", async () => { + await deleteSandboxAndDriveLater({ + providerSandboxId: "plain", + driveName: null, + }); + await deleteSandboxAndDriveLater({ + providerSandboxId: "box", + driveName: "drive-box", + }); + expect(calls).toEqual(["delete:plain"]); + expect(queued).toEqual([ + { + body: { providerSandboxId: "box", driveName: "drive-box", attempt: 0 }, + delaySeconds: undefined, + }, + ]); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/teardown.ts b/packages/trpc/src/router/cloud-workspace/teardown.ts new file mode 100644 index 00000000000..1b4322058ee --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/teardown.ts @@ -0,0 +1,167 @@ +import * as Sentry from "@sentry/core"; +import { db } from "@superset/db/client"; +import { cloudWorkspaces } from "@superset/db/schema"; +import { and, eq, type SQL } from "drizzle-orm"; +import { + DriveStillAttachedError, + deleteSandbox, + deleteWorkspaceDrive, + stopSandbox, + waitForDriveDetached, +} from "../../lib/sandbox"; +import { publishCloudWorkspaceJob } from "./jobs"; +import { nextSandboxNameFor } from "./sandbox-name"; +import { liveReferences } from "./teardown-guard"; + +export interface TeardownSandboxInput { + providerSandboxId: string; + driveName: string | null; + attempt: number; + /** The drive is only waited for, never deleted: another box of the row runs on it. */ + keepDrive?: boolean; +} + +/** One attempt fits a short request; a drive still flushing is tried again later. */ +export const TEARDOWN_DETACH_WAIT_MS = 20_000; +export const TEARDOWN_RETRY_SECONDS = 30; +export const TEARDOWN_ATTEMPTS = 20; + +export type TeardownOutcome = "deleted" | "kept" | "retrying" | "abandoned"; + +/** + * Stops the box, waits briefly for its drive to detach, then deletes the box + * and the drive. A drive still attached, or any other failure, requeues the + * job rather than wait in the caller. Before each destructive call it reads + * the rows again: what a live workspace or environment names is kept. + */ +export async function teardownSandbox( + input: TeardownSandboxInput, +): Promise { + try { + return await teardownOnce(input); + } catch (error) { + Sentry.captureException(error, { extra: { ...input } }); + return retryOrAbandon(input, error); + } +} + +async function teardownOnce( + input: TeardownSandboxInput, +): Promise { + if ((await liveReferences(input)).sandbox) return kept(input, "sandbox"); + await stopSandbox(input.providerSandboxId); + if (input.driveName) { + try { + await waitForDriveDetached(input.driveName, { + timeoutMs: TEARDOWN_DETACH_WAIT_MS, + // A kept drive only has to leave this box; the row's other box may hold it for good. + ...(input.keepDrive + ? { + heldByCaller: async (attachedTo: string) => + attachedTo !== input.providerSandboxId, + } + : {}), + }); + } catch (error) { + if (!(error instanceof DriveStillAttachedError)) throw error; + return retryOrAbandon(input, error); + } + } + const live = await liveReferences(input); + if (live.sandbox) return kept(input, "sandbox"); + await deleteSandbox(input.providerSandboxId); + if (input.keepDrive) return "deleted"; + if (live.drive) return kept(input, "drive"); + if ((await liveReferences(input)).drive) return kept(input, "drive"); + await deleteWorkspaceDrive(input.driveName); + return "deleted"; +} + +function kept( + input: TeardownSandboxInput, + what: "sandbox" | "drive", +): TeardownOutcome { + console.log( + `[cloud-workspace] teardown kept the ${what} of ${input.providerSandboxId}: a live row names it`, + ); + return "kept"; +} + +async function retryOrAbandon( + input: TeardownSandboxInput, + error: unknown, +): Promise { + if (input.attempt + 1 >= TEARDOWN_ATTEMPTS) { + Sentry.captureMessage( + `cloud workspace teardown gave up on ${input.providerSandboxId} and its drive ${input.driveName}`, + { level: "error", extra: { ...input, error: String(error) } }, + ); + return "abandoned"; + } + await queueTeardown( + { ...input, attempt: input.attempt + 1 }, + TEARDOWN_RETRY_SECONDS, + ); + return "retrying"; +} + +function queueTeardown( + input: TeardownSandboxInput, + delaySeconds?: number, +): Promise { + return publishCloudWorkspaceJob({ + path: "/api/cloud-workspaces/teardown", + body: input, + delaySeconds, + runLocally: teardownSandbox, + }); +} + +/** A box without a drive is deleted now, as before; one with a drive through the retried job. */ +export async function deleteSandboxAndDriveLater(args: { + providerSandboxId: string; + driveName: string | null | undefined; + keepDrive?: boolean; +}): Promise { + if (!args.driveName) return deleteSandbox(args.providerSandboxId); + await queueTeardown({ + providerSandboxId: args.providerSandboxId, + driveName: args.driveName, + attempt: 0, + ...(args.keepDrive ? { keepDrive: true } : {}), + }); +} + +/** + * Takes the row's box and drive names out of use in the same write that + * commits to deleting them, then queues their teardown. An unarchive after it + * finds a name that does not exist and gets a new box and a new drive, never + * the ones being deleted. Returns false when the row no longer matches + * `condition` or names another box: whatever changed it owns the names now. + */ +export async function retireAndTearDown( + row: { id: string; providerSandboxId: string; driveName: string | null }, + condition: SQL | undefined, +): Promise { + const [retired] = await db + .update(cloudWorkspaces) + .set({ + providerSandboxId: nextSandboxNameFor(row.id), + driveName: null, + driveLaidOutAt: null, + }) + .where( + and( + eq(cloudWorkspaces.id, row.id), + eq(cloudWorkspaces.providerSandboxId, row.providerSandboxId), + condition, + ), + ) + .returning({ id: cloudWorkspaces.id }); + if (!retired) return false; + await deleteSandboxAndDriveLater({ + providerSandboxId: row.providerSandboxId, + driveName: row.driveName, + }); + return true; +} diff --git a/packages/trpc/src/router/cloud-workspace/transition.ts b/packages/trpc/src/router/cloud-workspace/transition.ts index 7c26d5cfbe4..f1a94d8fc8f 100644 --- a/packages/trpc/src/router/cloud-workspace/transition.ts +++ b/packages/trpc/src/router/cloud-workspace/transition.ts @@ -16,6 +16,8 @@ type Move = { id: string; from: readonly CloudWorkspaceStatus[]; set?: Partial; + /** Only while the row still names this box: an unarchive may have given it another. */ + onBox?: string; }; async function moveCloudWorkspace( @@ -28,6 +30,9 @@ async function moveCloudWorkspace( and( eq(cloudWorkspaces.id, args.id), inArray(cloudWorkspaces.status, [...args.from]), + args.onBox === undefined + ? undefined + : eq(cloudWorkspaces.providerSandboxId, args.onBox), ), ) .returning({ id: cloudWorkspaces.id }); @@ -60,16 +65,18 @@ export async function failCloudWorkspace(args: { set?: Partial; reason: CloudWorkspaceFailureReason; error: unknown; -}): Promise { + onBox?: string; +}): Promise { const { scope } = args; const changed = await moveCloudWorkspace({ id: scope.cloudWorkspaceId, from: args.from, to: "failed", + onBox: args.onBox, set: { ...args.set, failureReason: args.reason, failedAt: new Date() }, }); nudge(scope.organizationId, "cloud_workspaces"); - if (!changed) return; + if (!changed) return false; const refused = isCloudUsageRefusal(args.error); logControl( "workspace.failed", @@ -80,4 +87,5 @@ export async function failCloudWorkspace(args: { if (!refused) { Sentry.captureException(args.error, { tags: controlTags(scope) }); } + return true; } diff --git a/packages/trpc/src/router/cloud-workspace/wake.test.ts b/packages/trpc/src/router/cloud-workspace/wake.test.ts index b40ad509d43..30fcde46bb2 100644 --- a/packages/trpc/src/router/cloud-workspace/wake.test.ts +++ b/packages/trpc/src/router/cloud-workspace/wake.test.ts @@ -9,6 +9,7 @@ import * as sandbox from "../../lib/sandbox"; import * as meter from "./meter"; import * as sandboxExpiry from "./sandbox-expiry"; import * as sandboxSleep from "./sandbox-sleep"; +import * as teardown from "./teardown"; import { markSandboxUnavailable, restartCloudWorkspace, @@ -440,3 +441,83 @@ describe("markSandboxUnavailable", () => { }); }); }); + +describe("a workspace whose snapshot expired", () => { + const retired: Array<{ + providerSandboxId: string; + driveName: string | null; + }> = []; + stub(realtime, { nudge: () => {} }); + stub(db, rowUpdate(true)); + stub(Sentry, { captureException: () => "" }); + stub(teardown, { + retireAndTearDown: async (row: { + providerSandboxId: string; + driveName: string | null; + }) => { + retired.push({ + providerSandboxId: row.providerSandboxId, + driveName: row.driveName, + }); + return true; + }, + }); + beforeEach(() => { + retired.length = 0; + }); + const row = { ...ROW, driveName: "drive-ws-box" } as Row; + const unavailable = (status: number) => + new sandbox.SandboxUnavailableError( + "ws-box", + new APIError(new Response(null, { status })), + ); + + test("retires its names and queues the teardown of its box and drive", async () => { + await markSandboxUnavailable(row, unavailable(410)); + expect(retired).toEqual([ + { providerSandboxId: "ws-box", driveName: "drive-ws-box" }, + ]); + }); + + test("a box deleted out from under the row keeps the drive", async () => { + await markSandboxUnavailable(row, unavailable(404)); + expect(retired).toEqual([]); + }); + + test("a drive refused for good is kept for diagnosis", async () => { + await markSandboxUnavailable( + row, + new sandbox.SandboxDriveRefusedError( + "ws-box", + "drive-ws-box", + "drive.foreign", + 3, + ), + ); + expect(retired).toEqual([]); + }); + + test("a workspace without a drive is left as it was", async () => { + await markSandboxUnavailable({ ...row, driveName: null }, unavailable(410)); + expect(retired).toEqual([]); + }); +}); + +describe("a wake of a workspace without a drive", () => { + const wakes: Array<{ mounts?: unknown }> = []; + stub(sandbox, { + buildSandboxClaim: async () => ({ claim: {}, agentCredentialDigest: "" }), + wakeSandbox: async (args: { mounts?: unknown }) => { + wakes.push(args); + throw new Error("wake not under test"); + }, + }); + + test("mounts nothing, whatever its environment", async () => { + await expect( + wakeCloudWorkspace({ ...ROW, driveName: null } as Row), + ).rejects.toThrow("wake not under test"); + expect(wakes).toHaveLength(1); + expect(wakes[0]?.mounts).toBeUndefined(); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/wake.ts b/packages/trpc/src/router/cloud-workspace/wake.ts index 2d136ff6a1f..2e842e0b2a2 100644 --- a/packages/trpc/src/router/cloud-workspace/wake.ts +++ b/packages/trpc/src/router/cloud-workspace/wake.ts @@ -7,7 +7,7 @@ import { CLOUD_WORKSPACE_WAKE_TRANSACTION, } from "@superset/shared/constants"; import { waitUntil } from "@vercel/functions"; -import { eq } from "drizzle-orm"; +import { and, eq } from "drizzle-orm"; import { type ControlScope, controlTags, @@ -22,18 +22,22 @@ import { stopAndSnapshot, stopSandbox, wakeSandbox, + workspaceDriveMounts, } from "../../lib/sandbox"; +import { recordDriveLaidOut } from "./drive-laid-out"; import { attachBootSession, authorizeBoot, queueSettleWorkspace, } from "./meter"; +import { sandboxResumeLock } from "./resume-lock"; import { recordSandboxAwake } from "./sandbox-expiry"; import { recordBootTrace, recordBoxStopped, recordSandboxAsleep, } from "./sandbox-sleep"; +import { retireAndTearDown } from "./teardown"; import { failCloudWorkspace } from "./transition"; type CloudWorkspaceRow = typeof cloudWorkspaces.$inferSelect; @@ -108,6 +112,8 @@ async function wake( providerSandboxId: row.providerSandboxId, claim, onlyIfRunning: adoptSignIns, + mounts: workspaceDriveMounts(row.driveName), + exclusive: sandboxResumeLock(row.driveName ?? row.providerSandboxId), beforeResume: async ({ resumedSession, ...box }) => { resumed = resumedSession; ({ bootId } = await Sentry.startSpan( @@ -193,6 +199,8 @@ async function wake( .where(eq(cloudWorkspaces.id, row.id)); } if (expiresAt) await recordSandboxAwake(row.id, { expiresAt, sessionId }); + // settleSandbox only answers for a drive whose boot laid it out. + if (row.driveName && !row.driveLaidOutAt) await recordDriveLaidOut(row.id); if (rebooted) { logControl("wake.completed", scope, { ms: Date.now() - wakeStartedAt, @@ -253,4 +261,18 @@ export async function markSandboxUnavailable( reason: error.snapshotExpired ? "snapshot_expired" : "sandbox_gone", error, }); + if (row.driveName && error.snapshotExpired) { + await retireAndTearDown( + row, + and( + eq(cloudWorkspaces.status, "failed"), + eq(cloudWorkspaces.driveName, row.driveName), + ), + ).catch((teardownError) => + console.error( + `[cloud-workspace] ${row.id} teardown after snapshot expiry`, + teardownError, + ), + ); + } } diff --git a/packages/trpc/src/router/environment/environment.test.ts b/packages/trpc/src/router/environment/environment.test.ts index 76404214510..7381fc238a5 100644 --- a/packages/trpc/src/router/environment/environment.test.ts +++ b/packages/trpc/src/router/environment/environment.test.ts @@ -6,6 +6,7 @@ import * as guards from "../../lib/cloud-guards"; import * as sandbox from "../../lib/sandbox"; import { userError } from "../../trpc"; import * as access from "../cloud-workspace/access"; +import * as teardown from "../cloud-workspace/teardown"; import * as wake from "../cloud-workspace/wake"; import * as integration from "../integration/utils"; import * as currentBundle from "./current-bundle"; @@ -56,9 +57,26 @@ stub(access, { status: "awake", baseBranch: "main", branch: "main", + providerSandboxId: "ws-box", + driveName: "drive-ws-box", }), }); -stub(sandbox, { workspaceRepositories: async () => [] }); +let drivesOn = true; +let promoted = 0; +let abandoned: Array<{ providerSandboxId: string; driveName?: string | null }> = + []; +stub(sandbox, { + workspaceRepositories: async () => [], + workspaceDrivesEnabled: async () => drivesOn, +}); +stub(teardown, { + deleteSandboxAndDriveLater: async (args: { + providerSandboxId: string; + driveName?: string | null; + }) => { + abandoned.push(args); + }, +}); stub(plan, { resolveMachineSize: async (_: string, name: string | null | undefined) => name === "large" || name == null @@ -95,6 +113,9 @@ beforeEach(() => { }; isAdmin = false; written = []; + abandoned = []; + drivesOn = true; + promoted = 0; }); describe("environment.update machineSize", () => { @@ -111,6 +132,17 @@ describe("environment.update machineSize", () => { expect(written).toEqual([{ machineSize: null }]); }); + test("the output never carries the golden drive, which released clients never saw", async () => { + isAdmin = true; + environment.goldenDriveName = "env-drive-env-g"; + const updated = await caller.environment.update({ + id: ENVIRONMENT_ID, + machineSize: null, + }); + expect(updated).not.toHaveProperty("goldenDriveName"); + expect(updated).toMatchObject({ id: ENVIRONMENT_ID }); + }); + test("refuses a size the plan does not offer", async () => { environment.createdByUserId = "user-1"; await expect( @@ -187,3 +219,44 @@ describe("environment.promote of a running workspace", () => { expect(order).toEqual(["save", "restart"]); }); }); + +const failingPromote = { + promoteSandboxToEnvironment: async () => { + promoted++; + throw new Error("snapshot failed after the fork"); + }, +}; + +describe("environment.promote that fails", () => { + stub(sandbox, failingPromote); + test("hands the golden and its forked drive to the retried teardown", async () => { + environment.createdByUserId = "user-1"; + await expect( + caller.environment.promote({ + cloudWorkspaceId: WORKSPACE_ID, + environmentId: ENVIRONMENT_ID, + }), + ).rejects.toThrow("snapshot failed after the fork"); + expect(abandoned).toHaveLength(1); + const [golden] = abandoned; + expect(golden?.providerSandboxId).toMatch(/^env-/); + expect(golden?.driveName).toBe(`env-drive-${golden?.providerSandboxId}`); + expect(written).toEqual([]); + }); +}); + +describe("environment.promote with drives off", () => { + stub(sandbox, failingPromote); + test("a workspace whose files are on a drive does not spread drive mode to a new golden", async () => { + environment.createdByUserId = "user-1"; + drivesOn = false; + await expect( + caller.environment.promote({ + cloudWorkspaceId: WORKSPACE_ID, + environmentId: ENVIRONMENT_ID, + }), + ).rejects.toMatchObject({ code: "PRECONDITION_FAILED" }); + expect(promoted).toBe(0); + expect(abandoned).toEqual([]); + }); +}); diff --git a/packages/trpc/src/router/environment/environment.ts b/packages/trpc/src/router/environment/environment.ts index 060826df00e..fb361d239cd 100644 --- a/packages/trpc/src/router/environment/environment.ts +++ b/packages/trpc/src/router/environment/environment.ts @@ -22,18 +22,22 @@ import { } from "../../lib/cloud-guards"; import { controlTags } from "../../lib/control-log"; import { + deleteGoldenDriveIfUnreferenced, deleteSandbox, + goldenDriveNameFor, loadRepositories, primaryRepository, promoteSandboxToEnvironment, RepositoryError, sortRepositories, + workspaceDrivesEnabled, workspaceRepositories, } from "../../lib/sandbox"; import { personProcedure, userError } from "../../trpc"; import { loadVisibleWorkspace } from "../cloud-workspace/access"; import { isLiveStatus } from "../cloud-workspace/live"; import { assertMachineSizeOffered } from "../cloud-workspace/start"; +import { deleteSandboxAndDriveLater } from "../cloud-workspace/teardown"; import { wakeAfterStop } from "../cloud-workspace/wake"; import { verifyOrgAdmin } from "../integration/utils"; import { currentBundleSha } from "./current-bundle"; @@ -251,6 +255,29 @@ async function loadPromotable( return { workspace, source, checkouts }; } +/** Only the control plane reads the golden drive; released clients never saw it. */ +function publicEnvironment( + row: Row, +): Omit { + const { goldenDriveName: _goldenDriveName, ...rest } = row; + return rest; +} + +/** A golden nothing will point at, and its drive: deleted through the retried job. */ +async function abandonGolden( + goldenName: string, + goldenDriveName: string | null, +): Promise { + await deleteSandboxAndDriveLater({ + providerSandboxId: goldenName, + driveName: goldenDriveName, + }).catch((cleanup: unknown) => + console.error( + `[environment/promote] could not delete unused golden ${goldenName}`, + cleanup, + ), + ); +} export const environmentRouter = { secrets: secretsRouter, @@ -284,7 +311,7 @@ export const environmentRouter = { projectIdsByEnvironment(ids), ]); return rows.map((row) => ({ - ...row, + ...publicEnvironment(row), repositories: repos.get(row.id) ?? [], projectIds: projects.get(row.id) ?? [], })); @@ -300,7 +327,7 @@ export const environmentRouter = { projectIdsByEnvironment([row.id]), ]); return { - ...row, + ...publicEnvironment(row), repositories: repos.get(row.id) ?? [], projectIds: projects.get(row.id) ?? [], }; @@ -374,7 +401,10 @@ export const environmentRouter = { await db.delete(environments).where(eq(environments.id, row.id)); throw error; } - return { ...row, primaryRepositoryId: input.primaryRepositoryId }; + return { + ...publicEnvironment(row), + primaryRepositoryId: input.primaryRepositoryId, + }; }), linkProject: personProcedure @@ -500,10 +530,29 @@ export const environmentRouter = { const limit = target ? null : await assertEnvironmentCapacity(workspace.organizationId); + // The kill switch: with drives off, no promote spreads drive mode to a new golden. + if ( + workspace.driveName && + !(await workspaceDrivesEnabled(workspace.organizationId)) + ) { + throw userError({ + code: "PRECONDITION_FAILED", + message: + "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment.", + i18nKey: "serverError.environment.drivesOff", + }); + } const golden = await promoteSandboxToEnvironment({ sourceSandbox: workspace.providerSandboxId, goldenName, organizationId: workspace.organizationId, + sourceDriveName: workspace.driveName, + }).catch(async (error: unknown) => { + await abandonGolden( + goldenName, + workspace.driveName ? goldenDriveNameFor(goldenName) : null, + ); + throw error; }); const { stopped } = golden; const restart = async () => { @@ -528,6 +577,7 @@ export const environmentRouter = { sourceKind: "fork" as const, sourceRef: goldenName, region: golden.region, + goldenDriveName: golden.goldenDriveName, bundleSha: await currentBundleSha(), primaryRepositoryId, }; @@ -616,25 +666,22 @@ export const environmentRouter = { return saved; }) .catch(async (error: unknown) => { - await deleteSandbox(goldenName).catch((cleanup: unknown) => - console.error( - `[environment/promote] could not delete unused golden ${goldenName}`, - cleanup, - ), - ); + await abandonGolden(goldenName, golden.goldenDriveName); await restart(); throw error; }); if (target?.sourceKind === "fork") { - await deleteSandbox(target.sourceRef).catch((error: unknown) => - console.error( - `[environment/promote] ${target.id} replaced; could not delete its previous golden ${target.sourceRef}`, - error, - ), - ); + await deleteSandbox(target.sourceRef) + .then(() => deleteGoldenDriveIfUnreferenced(target.goldenDriveName)) + .catch((error: unknown) => + console.error( + `[environment/promote] ${target.id} replaced; could not delete its previous golden ${target.sourceRef}`, + error, + ), + ); } await restart(); - return row; + return publicEnvironment(row); }), update: personProcedure @@ -707,14 +754,14 @@ export const environmentRouter = { : {}), }; if (Object.keys(patch).length === 0) { - return loadEnvironment(input.id, ctx); + return publicEnvironment(await loadEnvironment(input.id, ctx)); } const [row] = await db .update(environments) .set(patch) .where(eq(environments.id, input.id)) .returning(); - return row; + return row && publicEnvironment(row); }), archive: personProcedure @@ -729,11 +776,13 @@ export const environmentRouter = { .returning({ sourceKind: environments.sourceKind, sourceRef: environments.sourceRef, + goldenDriveName: environments.goldenDriveName, }); // A golden is one environment's alone, and an archived environment // never forks from it again; without this it bills storage forever. if (archived?.sourceKind === "fork") { await deleteSandbox(archived.sourceRef); + await deleteGoldenDriveIfUnreferenced(archived.goldenDriveName); } return { archived: true }; }), diff --git a/packages/trpc/src/router/environment/resolve-environment.ts b/packages/trpc/src/router/environment/resolve-environment.ts index cc869a25f06..8afb86dc2ba 100644 --- a/packages/trpc/src/router/environment/resolve-environment.ts +++ b/packages/trpc/src/router/environment/resolve-environment.ts @@ -12,6 +12,8 @@ export interface ResolvedEnvironment { region: string; bundleSha: string; primaryRepositoryId: string | null; + /** Set when the environment is in drive mode: every workspace's drive forks from it. */ + goldenDriveName: string | null; envs: Record; } @@ -60,6 +62,7 @@ export async function resolveEnvironment( region: row.region, bundleSha: row.bundleSha, primaryRepositoryId: row.primaryRepositoryId, + goldenDriveName: row.goldenDriveName, envs, }; } diff --git a/plans/20261116-workspace-drive.md b/plans/20261116-workspace-drive.md new file mode 100644 index 00000000000..7d780638dcc --- /dev/null +++ b/plans/20261116-workspace-drive.md @@ -0,0 +1,590 @@ +# Put a cloud workspace's repository and state on its own Vercel drive + +This ExecPlan is a living document. The sections `Progress`, `Surprises & Discoveries`, +`Decision Log`, and `Outcomes & Retrospective` must be kept up to date as work proceeds. + +Reference: this plan follows the conventions in `AGENTS.md`, `packages/trpc/AGENTS.md`, +`packages/trpc/src/lib/sandbox/AGENTS.md` and `.agents/commands/create-plan.md`. + +Status (2026-11-17): coded end to end behind the environment's drive mode and the organization +flag `cloud-workspace-drives`. Every Vercel fact this plan relies on was measured on real Vercel. +The code itself has not yet run against real Vercel: the `REAL_SANDBOX_DRIVE=1` pass of the +Sandbox workflow's `real-sandbox` dispatch is the check for that. + + +## Purpose / Big Picture + +A cloud workspace is a Vercel sandbox (a "box"). When a box stops, Vercel takes a snapshot of its +whole disk, and the next wake restores that snapshot before anything runs. The snapshot is a +block-level, full copy of the disk. It is not a delta over the golden it came from, and blocks +that a deleted file freed stay in it, because the disk has no discard. A typical box snapshot is +about 7 GB, and some are 14 to 27 GB. A wake restores at about 1 GB/s, and the restore is almost +never cached, because every workspace's snapshot is unique. + +About 7 GB of a box is `/workspace` (node_modules is about 5 GB and 213k files; `.git`; build +output) plus state under the home directory. A Vercel *drive* is a persistent block device that +is attached to a sandbox at a mount path and is not part of the sandbox's snapshot. With the +repository and the large home state on a drive, the root snapshot holds only the system: measured +2.11 GB, the same as a bare box from the image. + +The drive is *alongside* the persistent box. The root disk keeps its snapshot exactly as today, +so everything a person does on the box still persists, wherever they write it. There is no rule +for users to learn; the drive only moves the large, known directories off the root disk. + +A person sees it as a faster wake (the provider "resume" span in the `cloud_workspace.wake` +Sentry transaction gets shorter) and `git status` in `/workspace` still works with every file +present. + + +## Terms + +A *golden* is a stopped sandbox whose snapshot new workspaces of one environment fork from. It is +the `environments.source_ref` of an environment with `source_kind = 'fork'`. An *environment* is +the row in `environments` (`packages/db/src/schema/schema.ts`). + +A *drive* is Vercel's `Drive` (`@vercel/sandbox` 4.0.0, `dist/drive.d.ts`): a name unique in the +project, a region, `Drive.get`, `Drive.getOrCreate`, `drive.fork({ name })`, `drive.delete()` and +`currentSandboxName` (the sandbox that holds it now). A sandbox attaches drives through `mounts`, +a record from an absolute path to `{ drive: name, mode: "read-write" | "snapshot" }`. + +*Drive mode*: an environment is in drive mode when `environments.golden_drive_name` is set. Its +golden was built with that drive mounted from its first boot, and every workspace of it gets a +fork of that drive. + +A *bind mount* is the Linux `mount --bind` that `superset-boot` does as root to make a directory +of the drive appear at `/workspace` or under `/home/ubuntu`. + + +## Facts this plan relies on + +Measured on Vercel on 2026-11-16 (default image, `iad1`): + +- A drive is a real ext4 block device. inotify and hardlinks within one mount work on it. +- At stop, processes get SIGTERM and at least 10 s before the snapshot is taken. +- A snapshot is a block-level, full copy of the disk. Deleting a file does not shrink a later + snapshot. Only never writing the bytes, or overwriting freed space with zeros, shrinks it. + +Measured on Vercel on 2026-11-17, on our released image: + +- `drive.fork` is copy-on-write and near instant: 0.5 s for a 12 GB drive of 213k files, a fork + of a fork 0.4 s. Forks are independent: a parent can be deleted while forks exist, and the + forks stay readable. +- The full layout works on a box from our image: the drive mounted at `/mnt/superset-drive`; + `mount --bind` of `/workspace` and the home subtrees took 11 ms (29 ms after a wake). Bind + mounts do not survive a session. +- `bun install` with `BUN_INSTALL_CACHE_DIR=/workspace/.superset/stores/bun` hardlinked all + 212,973 files. A hardlink from `/workspace` into another bind mount of the same drive fails with + `EXDEV`. +- `dockerd --data-root /mnt/superset-drive/docker` (storage driver `fuse-overlayfs`) pulled and + ran an image, also after a wake. +- Stop 12 s, wake 1.9 s. Afterwards the root snapshot was 2.11 GB, identical to a bare box. +- A drive must be stopped and detached before anything else mounts it: a mount taken while a + writer still held it read-write saw a corrupt or empty filesystem. After a stop, poll + `Drive.get(...).currentSandboxName` until it is empty. +- Never delete a running sandbox that holds a drive read-write: a writer deleted while running + left the drive attached for minutes. Stop it first. +- A drive has no discard. Stop time grows with what the session wrote to the drive: 44 s after + 12 GB, 6 s after 1 GB. +- `Drive.list` rejects `namePrefix` unless `sortBy: "name"`. + +Documented by Vercel or the SDK, not measured: + +- One read-write mount per drive at a time. Read-only snapshot mounts can be shared. +- A drive's region must equal the sandbox's region. A sandbox has at most 4 mounts. +- Prices: drive storage $0.05 per GB-month, reads $0.0015 per GB, writes $0.004 per GB. Snapshot + storage $0.08 per GB-month. + + +## Progress + +- [x] (2026-11-16) Milestone 1: columns, drives module, mounts on create and wake, drive + deletion, unit tests. +- [x] (2026-11-17) Milestone 2: the open questions about Vercel, measured by hand (above). +- [x] (2026-11-17) Rebased on main; the drive mount and the agent-tools mount (PR #9844) are one + `mounts` object (`composeMounts` in `vercel.ts`). +- [x] (2026-11-17) Milestone 3: `superset-boot` lays out the drive, exports the store variables, + points Docker at the drive. boot-twice covers it with a tmpfs at + the mount path. +- [x] (2026-11-17) Milestone 4: the release builds the internal golden in drive mode when the + organization's flag is on; promote carries a source's drive to the golden. +- [x] (2026-11-17) Milestone 5: a wake waits for the drive to detach (2 minutes, then fails); + every delete stops the box and waits before it deletes the sandbox and the drive; a golden + drive is deleted once no live environment names it. +- [x] (2026-11-17) Rebased on main (#9861, #9862). The boot refuses a missing drive. The + `REAL_SANDBOX_DRIVE=1` pass runs whatever the first box's result, names every drive + expectation, and deletes what it made unless a drive expectation failed. +- [ ] Milestone 6: run the Sandbox workflow's `real-sandbox` dispatch with `real_sandbox_drive`; + then a release with the flag on for the internal organization; compare wake spans and snapshot + sizes for a week; then other organizations. + + +## Surprises & Discoveries + +- Observation: the first real-Vercel drive run (run 44151145824, 2026-11-17) failed 7 of 10 drive + expectations. All 7 were bugs in the check, not the product. `sudo` on the box writes `unable to + resolve host ` to stderr, and the check compared stdout and stderr together. Also, + findmnt reports a bind mount of a subdirectory as `/dev/vdc[/workspace]`, so its SOURCE never + equals the drive's. The log showed the product doing the right thing: `drive.layout ms=50`, + `/workspace` from `/dev/vdc[/workspace]` and docker's root at + `/mnt/superset-drive/docker`. The check now compares stdout only, and compares MAJ:MIN plus + FSROOT for the bind. + +- Observation: hardlinks do not work across two mount points, even on the same filesystem. + `link(2)` returns `EXDEV` when the two paths are on different `vfsmount`s, and a bind mount is a + separate `vfsmount`. Measured on 2026-11-17. So the bun and pnpm stores live inside the + `/workspace` bind mount. +- Observation: `superset-boot` runs `chown -R` on `/workspace` when its owner is not `ubuntu`. On + a drive, that walks 213k files. The layout creates every drive directory as `ubuntu`, and each + missing component of a home subtree on the root disk too (`install -d -o` only owns the last + one). +- Observation: a repository may track a `.superset/` of its own (the config directory before the + rename to `.paradev/`). So the checkout's exclude names only `/.superset/stores/`, and a reclone + merges a repository's `.superset/` into the one the stores live in instead of failing the move. +- Observation: the image's `superset-boot` runs first, and an older one has no layout. The layout + is idempotent and runs again after the reexec into the bundle's runner, so a box on an older + image is laid out on its first boot of a bundle that has it, before the steps and host-service. + + +## Decision Log + +- Decision: the drive is *alongside* the persistent box, never a replacement for its snapshot. + Rationale: everything a user does still persists; there is no rule to learn about where files + must go. Date/Author: 2026-11-17, product owner. +- Decision: drive mode is per *environment*, set by `environments.golden_drive_name`, not per + user. Rationale: the golden box and the golden drive must agree. If the golden's root disk still + held `/workspace`, every fork would keep carrying those 5 GB in its snapshot under the bind + mount, and the drive would buy nothing. So in drive mode the golden is built with the drive + mounted from the first boot. Date/Author: 2026-11-17, product owner. +- Decision: one flag, per organization, `cloud-workspace-drives` (PostHog group `organization`, + same pattern as `agent-tools.ts`), read only by the release when it builds a golden. It never + decides anything for a workspace. Rationale: a workspace follows its environment's row; a flag + that changes must not strand a workspace without its files. Date/Author: 2026-11-17, product + owner. +- Decision: fork or nothing. A workspace of an environment in drive mode gets + `goldenDrive.fork(...)`; a workspace of any other environment gets no drive at all. There is no + empty drive. Rationale: an empty drive under a golden whose root disk has the repository hides + the repository; under a golden whose root disk has none, it re-clones and re-installs. + Date/Author: 2026-11-17, product owner. +- Decision: promote carries a source's drive whatever the flag says. Rationale: a source in drive + mode has no `/workspace` on its root disk, so a golden without a fork of its drive would be + empty. The flag still gates the only place that starts drive mode, the release. A human may + prefer promote to refuse when the flag is off. Date/Author: 2026-11-17, agent; kept as coded + by the coordinator on 2026-11-17. +- Decision: the identity file names the workspace's drive (`SUPERSET_SANDBOX_DRIVE`), and + `superset-boot` refuses with `drive.missing` when it is set and the drive is not a mount point. + Rationale: a failed mount must never present the root disk's empty `/workspace` as the user's + work. A wake that finds the refusal in the boot log fails the workspace instead of answering + "still starting". Date/Author: 2026-11-17, coordinator. +- Decision: the drive is mounted once, at `/mnt/superset-drive` (`SANDBOX_PATHS.drive`), and + `superset-boot` bind-mounts subdirectories of it. Rationale: one read-write mount per drive, at + most 4 mounts per sandbox, and `/workspace` itself may be the primary checkout. Date/Author: + 2026-11-16, agent. +- Decision: a workspace drive is `drive-`; a golden drive is + `env-drive-`. The workspace's name is written to `cloud_workspaces.drive_name` + before the fork, so a delete that races the provision finds it. Date/Author: 2026-11-16, agent. +- Decision: `golden_drive_name` is read with the rest of the environment in `resolveEnvironment`, + the same read that picks the golden, so a box and its drive come from the same golden even when + a promote replaces it meanwhile. Date/Author: 2026-11-17, agent. +- Decision: a wake of a stopped box waits for its drive to be detached, at most 2 minutes + (`DRIVE_DETACH_TIMEOUT_MS`), then fails with `DriveStillAttachedError` rather than mount. A + failed drive mount update throws; a failed agent-tools mount update only warns. Date/Author: + 2026-11-17, agent. +- Decision: no swap on the drive. Since #9924 cloud boxes have no swap at all, and memguard reaps + early enough without it; the drive has no swap directory. memguard's cleanup of old + `/var/swap` files is main's and covers every box that could have one: no drive box ran the + earlier drive swap, since no environment has been in drive mode. Date/Author: 2026-11-18, + coordinator. +- Decision: Docker's data-root moves to the drive by `superset-boot` rewriting + `/etc/docker/daemon.json` after `apply-rootfs`, merged from the bundle's own file. Rationale: + `apply-rootfs` compares only the hash sidecar, so the rewrite stays until a bundle changes + `daemon.json`, and then the next boot merges again. Date/Author: 2026-11-17, agent. +- Decision: in drive mode `host.db` moves to the drive with the checkout markers + (`/mnt/superset-drive/state`). Rationale: its workspace and project rows name the worktrees on + the drive, and `agent-launched` beside it says whether the agent already ran in them. On the + root disk, a rollback that the drive does not share would make host-service forget a worktree or + launch the agent twice. Terminal sessions in it describe processes, which no snapshot keeps + anyway. Date/Author: 2026-11-17, agent, at the coordinator's request. +- Decision: the drive names its owner (`state/workspace-id`, synced), and the row records the + first boot that laid the drive out (`cloud_workspaces.drive_laid_out_at`, migration 0197). A + wake's claim says the drive was laid out before (`SUPERSET_SANDBOX_DRIVE_LAID_OUT`) only once + that column is set. Rationale: the earlier rule (every wake claims it) made a workspace whose + first session died before its layout refuse every wake as `drive.unlaid`, and a create that met a + transient mount failure could only fail. Replaces the rule of 2026-11-17 morning. + Date/Author: 2026-11-17, agent, at the coordinator's request. +- Decision: a drive refusal is retried, not failed: the box is stopped and the next wake or the + user's first open mounts the drive afresh. A refusal no remount can change (`drive.foreign`, + `drive.unlaid`, `drive.downgrade`, or host-service up without a layout) ends after 3 boots in a + row, counted from the boot log, with the workspace `failed` and a Sentry report. + Date/Author: 2026-11-17, agent. +- Decision: an unarchived workspace reuses its box and drive while their names are not retired, + that is inside the 7-day grace period. The reap's delete stage retires both names in the same + write that commits to deleting them, so an undo after it gets a new box and a fresh drive; the + user's deletion had completed. The teardown job still reads the rows before every destructive + call and keeps what a live row names. A drive forked afresh clears `drive_laid_out_at`. + Rationale: the drive lifecycle model (PR #9914) found a teardown keyed on names deleting a box + an undo had just brought back. Date/Author: 2026-11-17, agent. +- Decision: a provision that lost its row to an archive no longer tears down. Rationale: it cannot + tell the archive it lost to from a later one after an undo; that archive's reap can, by its + `archivedAt`. The cost is that a box the user never saw lives until that reap's stop and + delete stages. Date/Author: 2026-11-17, agent. +- Decision: "fresh fork" is a file on the drive (`state/fresh`), written and synced with the + owner at the first layout, and removed once the first checkout has run, whether it succeeded or + failed. Rationale: an environment variable lived for one boot only, so a first session that + ended before its checkout left the next boot treating the golden's copy as the user's work (the + model's FixFreshDurable). The file goes after a failed checkout too, unlike the model's + proposal: once that checkout has run the box was usable, and a later fresh-path reclone or + `checkout -B` could replace what the user did meanwhile. Date/Author: 2026-11-18, agent. +- Decision: a fresh fork of a golden never recloned. Rationale: the real check lost a golden's + dependencies once in six runs on a fork whose fetch or `checkout -B` failed, because the fresh + fork fell back to main's reclone, which empties the checkout. A fresh fork has no user work, and + the later-boot refetch of an untouched copy covers it, so it retries three times and then keeps + the golden's checkout. Date/Author: 2026-11-18, coordinator. +- Decision: a workspace's drive is deleted when its snapshot expires (30 days after last use). + Rationale: the box can never resume, and the drive would bill storage for good. Vercel expires + the snapshot itself; the control plane learns of it only from the 410 a wake, an agent turn or an + automation dispatch gets, so that is where the drive goes, through the retried teardown. A + workspace nobody opens again keeps its drive until the drive sweep runs. Date/Author: 2026-11-18, + Satya. +- Decision: a checkout with no marker on a drive is fetched only while it is the golden's untouched + copy (`state/golden-head`), else kept (the model's FixFetchIfUntouched). Rationale: removing + `state/fresh` after the first attempt keeps a failed first checkout from being refetched over the + user's work, but alone it left stale code after a failed first fetch; HEAD, a clean tree and the + branch together prove nobody has worked in it. Date/Author: 2026-11-18, model's author and agent. +- Decision: a provision that finds the lock held requeues itself (15 s, at most 10 times) and + touches nothing; after the last try it fails only a row still `provisioning`. A provision fails + or stops a row only if the row is still `provisioning` or this job made it `awake` (the model's + FixProvLockRetry and FixFailOwnOnly). Date/Author: 2026-11-18, model's author and agent. +- Decision: a fresh fork is recognised by its drive having no owner at the first layout, not by + the golden strip removing local branches (the model's FixFresh). Rationale: promote and the + release both strip the drive's state, owner included, so every fork's first boot is fresh and + takes main's fetch and `checkout -B` whatever branches the golden has; removing branches would + add nothing. Date/Author: 2026-11-17, agent. +- Decision: the control plane does not refuse to pin a pre-drive bundle to a workspace with a + drive. Rationale: nothing records which bundle shas know the drive, and both sides already + refuse: the boot will not reexec into a script without the drive-contract line, and a boot that + answers without `drive.layout` is never treated as ready. Date/Author: 2026-11-17, agent. +- Decision: the kill switch is the organization flag, read at provision and promote. It cannot + stop forks inside an environment already in drive mode without leaving their workspaces empty, + so a provision with the flag off gives the workspace no drive at all, and its box clones and + installs onto the root disk. Date/Author: 2026-11-17, agent. +- Decision: do not migrate existing workspaces. See Migration. Date/Author: 2026-11-16, agent; + confirmed out of scope 2026-11-17. +- Decision (rejected): wake by forking the golden. A wake would fork the environment's golden + afresh and mount the workspace's drive, instead of resuming the workspace's own snapshot. + Rejected on 2026-11-17 by the product owner in favour of a drive alongside the persistent box. + Without the workspace's own snapshot, whatever a person wrote outside the drive would be gone, + and that is a rule to learn. It is not a goal of this plan. + + +## Outcomes & Retrospective + +As of 2026-11-17: the code compiles, the unit tests pass, the model check still passes, and +boot-twice covers the layout in a container. Nothing in this branch has run against real Vercel. +No environment is in drive mode, so no workspace changes until a release runs with the flag on. + + +## Context and Orientation + +Affected: the API (`apps/api`, which serves the tRPC routers), the packages `db`, `trpc`, +`shared` and `sandbox`, and the Sandbox workflow. No tRPC input or output changes. + +- Create: `provisionCloudWorkspace` (`packages/trpc/src/router/cloud-workspace/provision.ts`) + builds the claim, which now carries the environment's `goldenDriveName`; if set, it writes the + workspace's drive name to the row, forks the golden drive (`forkWorkspaceDrive`), and passes the + drive's mount to `provisionSandbox`. +- Boot: `superset-boot` (`packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot`). +- Wake: `wakeCloudWorkspace` (`wake.ts`) passes `workspaceDriveMounts(row.driveName)` to + `wakeSandbox` (`packages/trpc/src/lib/sandbox/vercel.ts`). +- Delete: the reap's delete stage (`reap.ts`), the `cloudWorkspace.delete` fallback and the + provision teardowns call `deleteSandboxAndDrive`. +- Promote, replace and archive: `environment.promote` and `environment.archive` in + `packages/trpc/src/router/environment/environment.ts`; `promoteSandboxToEnvironment` in + `vercel.ts`. +- Release: `packages/sandbox/src/environments/release.ts`. + + +## The layout on the box (coded) + +The drive is mounted by Vercel at `/mnt/superset-drive`. It holds: + + /mnt/superset-drive/ + workspace/ bind-mounted on /workspace + .superset/stores/ bun, pnpm stores (inside the /workspace mount, for hardlinks) + home/ subtrees bind-mounted under /home/ubuntu + .cache .local/share .local/state .npm .cargo/registry .rustup + .claude .codex/sessions go/pkg + docker/ dockerd's data-root + +The home subtrees are `SANDBOX_DRIVE_HOME_SUBTREES` in `packages/shared/src/sandbox/contract.ts`, +rendered into `contract.sh` as `SUPERSET_DRIVE_HOME_SUBTREES`; `sandboxDrivePathOf` maps a box +path to its place on the drive. A subtree the drive does not have yet is seeded once from what the +image left at that path. + +`superset-boot` does the layout as root right after it clears `/run/superset`, before the bundle +passes, host-service and the checkout. Each bind is `mountpoint -q || mount --bind ...`, and +a failed bind refuses the boot. It stamps `drive.layout ms=`. If `/mnt/superset-drive` is not a +mount point, nothing changes: that is every box outside drive mode. + +In drive mode the boot also: + +- writes `BUN_INSTALL_CACHE_DIR=/workspace/.superset/stores/bun` and + `npm_config_store_dir=/workspace/.superset/stores/pnpm` to `/run/superset/drive.env` (sourced by + `/etc/profile.d/superset.sh`) and into host-service's env; +- adds `/.superset/stores/` to `.git/info/exclude` of a root checkout; +- rewrites `/etc/docker/daemon.json` with `"data-root": "/mnt/superset-drive/docker"`. + +What stays on the root disk: `/etc/superset`, `/opt/superset`, `/usr`, `/var/lib/superset` +(host.db, checkout markers), `/var/log/superset`, `/home/ubuntu` itself with its dotfiles, +`~/.config`, `~/.gitconfig`, `~/.paradev`, and the sign-in files the control plane writes on every +wake (`~/.codex/auth.json`, `/etc/codex/config.toml`). + + +## Golden drive per environment (coded) + +The release (`release.ts`), when `cloud-workspace-drives` is on for the internal organization: + +1. creates `env-drive-` in the region; +2. provisions the golden with it mounted read-write, so the boot lays it out and the checkout and + the setup hook's installs land on the drive; +3. strips identity (with the drive paths), stops the golden, waits for the drive to detach; +4. probes a workspace on a fork of the golden drive, stop and wake included, then deletes the + probe and its drive; +5. writes `golden_drive_name` with `source_ref` in the same row update; +6. deletes the previous golden and, once no live environment names it, its golden drive. + +When the flag is off, the release runs as before and writes `golden_drive_name = null`. + +Promote (`promoteSandboxToEnvironment`), when the source workspace has a drive: + +1. snapshots the source, which ends its session; waits for the source drive to detach; +2. forks it to `env-drive-`; +3. creates the golden from the snapshot with the fork mounted read-write; +4. strips `INHERITED_IDENTITY` and the agent session directories (`~/.claude/projects`, + `~/.codex/sessions`), both at their box paths and at their drive paths, because no boot has + laid out the golden; +5. stops the golden and waits for its drive to detach; +6. restarts a source that was running, with its own drive in its mounts. + +The environment row gets `golden_drive_name` with `source_ref`. A source without a drive promotes +as before, and the environment leaves drive mode. A replaced or archived environment's golden +drive is deleted after its golden sandbox, once no live environment names it +(`deleteGoldenDriveIfUnreferenced`). Forks stay readable after their parent is deleted. + + +## Mounts on create and wake (coded) + +`composeMounts` merges the workspace drive's mount and the agent-tools mount into the one +`mounts` object a create or a wake passes. On a wake of a stopped box, `wakeSandbox` starts from +the box's current mounts without the agent-tools entry, adds the workspace drive and the current +agent-tools drive, waits for the drive at `/mnt/superset-drive` to be detached, and calls +`update({ mounts })` once. A running box is not touched. + + +## Single writer and the sleep state machine (coded) + +A stop leaves the drive attached while the session flushes. `wakeSandbox` polls +`currentSandboxName` every 2 s and fails after 2 minutes with `DriveStillAttachedError`. The wake +then throws before anything resumes, and the row stays as it was. + +`Asleep.tla` does not need a new state: the wait writes no status, expiry or session id; it +delays `WakeResume` or fails the wake before it. The model still passes unchanged (1,624,931 +distinct states) and the broken config still fails. The single-writer property itself (at most +one session holds the drive) is outside the model; the unit tests in `drives.test.ts` and +`vercel.test.ts` cover the wait. A drive variable in the model is a possible follow-up. + + +## Delete and archive (coded) + +`deleteSandboxAndDrive` stops the box, waits for the drive to detach, deletes the sandbox, then +the drive. It is used by the reap's delete stage, the `cloudWorkspace.delete` fallback and both +provision teardowns. A drive that is still attached after 2 minutes throws; the reap's delivery +retries. Archive keeps the drive for the 7-day grace period, with the stopped sandbox. + + +## Migration of existing workspaces + +Never, in this plan. Existing workspaces and environments not in drive mode are untouched. Freed +blocks stay in a snapshot, so copying `/workspace` to a drive and removing it would not shrink +the snapshot; only zero-filling free space would, which is not acceptable on a wake. + + +## Billing + +Per typical workspace with 7 GB of `/workspace` and home state, per month: + + today: 7 GB snapshot x $0.08 = $0.56 + drive: up to 7 GB drive x $0.05 = $0.35 (forks are copy-on-write; shared blocks + likely bill once, not measured) + + root snapshot 2.11 GB x $0.08 = $0.17 + = at most about $0.52 + +Per event: a bun install that writes node_modules, 5 GB x $0.004 = $0.02. The main gain is the wake: about 5 GB less to restore (the root +snapshot went from ~7 GB to 2.11 GB), roughly 5 s at the measured rate. + + +## Regions + +A drive must be in the sandbox's region, so a golden, its golden drive and every workspace drive +of an environment are in `environments.region`. `forkWorkspaceDrive` refuses a golden drive in +another region. + + +## Failure modes + +- Fork fails at provision: the provision fails as any other create failure does; the row goes to + `failed`, and the sandbox and the drive go to the teardown job. +- The drive is still attached on a wake: the wake waits 2 minutes, then fails with a clear error. + A drive held by this very sandbox, running again, means a concurrent wake resumed it; the wake + goes on as for a running box. +- Mount update fails on a wake: the wake throws. +- A bind or a home seed fails at boot: the boot refuses, so nothing starts on the root disk's empty + `/workspace`. +- The drive does not mount, belongs to another workspace, or mounts without an owner though the + wake says it was laid out: `superset-boot` stamps `drive.missing`, `drive.foreign` or + `drive.unlaid` and exits non-zero before the bundle, host-service and the checkout. The control + plane reads the boot log every 2 s while it waits for host-service and throws + `SandboxDriveMissingError`, a `SandboxNotReadyError`: the wake stops the box and answers "still + starting", and the next wake mounts the drive afresh; a create records the box asleep instead + of failing it. Only this boot's lines count (the boot id on `boot.start`). A refusal no remount + can change ends after 3 boots in a row: the workspace goes `failed` with the reason, Sentry gets + it, and the box and drive are kept. +- A pinned bundle whose boot script predates the drive: a drive-mode boot refuses + (`drive.downgrade`) before it would reexec into it. +- An unclean stop (host-service killed, writers still running): the real check's probe requires + the box to boot with `host.db` passing `integrity_check` and the checkout passing `git fsck`. + A cold wake, one the snapshot cache did not serve, cannot be forced from the check; its wake + times are warm. +- host-service answers without a `drive.layout` stamp for a workspace with a drive (a pinned + bundle older than the layout): the same retryable error, so its empty `/workspace` is never + shown as ready. +- The root disk rolls back (a stop snapshot that did not land) and the drive does not: the + checkout markers and `host.db` are on the drive, so they still describe it. A checkout that + already has the workspace's branch is kept; a failed fetch keeps the checkout; a clone goes only + into an empty `/workspace`. +- Drive delete fails: the teardown job retries every 30 s, 20 times, then logs and gives up. + + +## Invariants + +The owner's two requirements: nothing a user does is ever lost, and an empty `/workspace` is +never shown as their work. + +| Invariant | Enforced in | Test | +| --- | --- | --- | +| A wake never gives a workspace a drive its row does not name; only a provision consults the environment, and only with the organization's flag on. | `claimedWorkspaceDrive` (`drives.ts`) via `buildSandboxClaim({ provisioning })` (`claim.ts`); `wake()` mounts `workspaceDriveMounts(row.driveName)` (`wake.ts`) | `claim.test.ts` "an existing workspace gets no drive on a wake…", "with the organization's flag off a new workspace gets no drive…"; `wake.test.ts` "a workspace without a drive is woken with no mounts" | +| A box whose identity names a drive starts nothing that reads `/workspace` unless that drive is mounted. | `layout_drive` in `superset-boot` (`drive.missing`) | boot-twice "drive missing: …" (4 checks) | +| The mounted drive is this workspace's. | `layout_drive` (`drive.foreign`; `drive.unlaid` when `SUPERSET_SANDBOX_DRIVE_LAID_OUT` is set); the owner is written and synced at the first layout; a wake's claim sets the key only once `cloud_workspaces.drive_laid_out_at` is set | boot-twice "drive foreign", "drive empty", "the drive names its workspace"; `claim.test.ts` "a wake expects the drive laid out only once the row recorded a boot that laid it out" | +| A laid-out drive is recorded durably. | `recordDriveLaidOut` (`drive-laid-out.ts`) after a settle that saw `drive.layout`, in `provision.ts` and `wake.ts`; a fresh fork at provision clears it | `provision.test.ts` "records the drive laid out once the boot answered" | +| A drive-mode boot never runs a boot script that predates the drive. | the `drive-contract` line and the `drive.downgrade` refusal before the reexec in `superset-boot` | boot-twice "drive, older boot script: refused before the reexec, nothing checked out" | +| The control plane never treats a box with a drive as ready unless its boot laid the drive out. | `settleSandbox` (`vercel.ts`) requires `drive.layout` | `vercel.test.ts` "a boot that answers without laying the drive out fails the settle" | +| A drive refusal is read from this boot only, found within about 2 s, and retried by the next wake or create; one no remount can change ends after 3 boots in a row with a clear `failed` and a Sentry report. | `runBoot` boot id, `driveRefusalIn`, `persistentDriveRefusals`, `SandboxDriveMissingError` (a not-ready), `SandboxDriveRefusedError` (an unavailable), `DRIVE_REFUSAL_LIMIT`; `wakeSandbox` stops the box; `provision.ts` records it asleep | `vercel.test.ts` "only this boot's lines count…", "a reexec's second boot.start joins its boot", "counts the persistent refusals in a row…", "a refused boot is a retryable not-ready…", "a refusal no remount can change ends the retries…"; `provision.test.ts` "a drive refusal at create leaves the workspace asleep…" | +| In drive mode, what describes the drive's contents is on the drive: checkout markers, `host.db` and its WAL, `agent-launched`, the owner. | `use_drive_state` in `superset-boot`; `SANDBOX_PATHS.driveState` | boot-twice "host.db and the checkout markers are on the drive, not the root disk", "drive, root rolled back"; real check "unclean stop: host.db passes integrity_check" | +| A drive laid out before is never replaced by the checkout. | `checkout_one` with `drive_has_work` | boot-twice "drive, marker lost: the branch is kept", "drive, fetch failed: nothing on the drive is replaced" | +| A fresh fork's checkout is fetched later too, while it is still the golden's untouched copy: HEAD at the commit the fork had, a clean tree, the branch absent or at that commit. Anything else is the user's and is never fetched over. | `state/golden-head/`, recorded before the first checkout attempt; `untouched_golden` in `checkout_one` (`superset-boot`) | boot-twice "fresh fork, first fetch failed: a later boot fetches the golden copy while it is untouched", "fresh fork, first fetch failed, then the user committed: their copy is kept" | +| A fresh fork of a golden is never recloned: its fetch and `checkout -B` are tried three times, and if they still fail the golden's checkout, dependencies included, is kept as it is (`checkout.fresh.failed`), the box is ready, and a later boot fetches it while it is untouched. The reclone is only for a path with no repository. | `checkout_one` in `superset-boot` (the attempts loop) | boot-twice "fresh fork, fetch failed: never recloned…", "fresh fork of a golden with node_modules, first fetch failed: the dependencies survive and a later boot fetches" | +| A fresh fork checks out its own branch as main does, even if its first session ended before the checkout. | `state/fresh` on the drive, written and synced with the owner at the first layout, removed once the first checkout has run (`checkout` in `superset-boot`) | boot-twice "fresh fork, new branch", "fresh fork, a branch the golden has", "fresh fork, fetch failed", "fresh fork, first session ended before its checkout: the next boot still fetches" | +| A workspace whose snapshot expired loses its own drive, never a golden's or one another row names, and never while the drive is attached. | `markSandboxUnavailable` (`wake.ts`) on a 410 retires the row's names while it is `failed` and still names that box and drive (`retireAndTearDown`); `teardownSandbox` waits for the drive to detach and checks `liveReferences` before the delete, retrying 20 times 30 s apart | wake "a workspace whose snapshot expired …"; teardown "a drive still attached requeues the job and deletes nothing", "a drive a golden environment or another row names is never deleted", "a drive delete that fails is retried"; vercel "a box whose snapshot expired … still lets its drive go" | +| Flag-off boxes run main's checkout code exactly. | `checkout_one` `else` branches | boot-twice first and second boot of the plain box; the diff against main | +| A home subtree is never seeded partially. | `seed_home_subtree` (copy to `.seeding`, then `mv -T`; a failed copy refuses the boot) | boot-twice layout checks only; a failed copy is not simulated | +| One read-write session per drive, and a concurrent wake of the same box does not wait. | `waitForDriveDetached` with `heldByCaller` in `wakeSandbox` | `vercel.test.ts` "a stopped box waits for its drive…", "a drive a concurrent wake already resumed this box with is not foreign"; `drives.test.ts` wait tests | +| A stopped box's drive check and the call that resumes it are one step: no other caller resumes or stops the box in between. A refused box is stopped under the same lock. The lock is keyed on the drive when there is one: a drive has one writer, whatever the box is called. | `ResumeLock` around `wakeSandbox`'s read-to-boot and `provisionSandbox`; `sandboxResumeLock(driveName ?? providerSandboxId)` (`resume-lock.ts`, a transaction-scoped advisory lock; a wake tries and answers "still starting", a provision waits up to 20 s) | `vercel.test.ts` "a second wake while the first resumes does not resume the box…"; `provision.test.ts` "the create holds the lock of the drive, whatever the box is called" | +| Nothing resumes a box outside that one step. After a read, every call goes through the session read (a Session call fails rather than resume a session that ended); a session that ended in between takes the stopped path under the same lock. | `resumeSession` (the only resume), `withLiveSession`, `wakeSandbox`, `provisionSandbox`, the settle's log read through a running session | `vercel.test.ts` "a box found running … whose session ends during the wake is resumed only after the drive check" | +| Two provisions of one row never both run on its drive. | `provisioning -> awake` and `-> failed` only while the row names the job's own box (`transitionCloudWorkspace({ onBox })`); a job that lost to another box tears its own box down with `keepDrive` (stop, wait for the detach, delete the box, never the drive); a new box is created only once its drive has detached | `provision.test.ts` "a provision overtaken by an undo's second provision gives up its box, keeps the drive, and never marks the row"; `vercel.test.ts` "a new box on a drive is created only once a box of another name has let go of the drive" | +| A redelivered provision whose box has stopped waits for the drive and remounts it before anything resumes the box. | `provisionSandbox` (`existing.status !== "running"`) | `vercel.test.ts` "a redelivered provision whose box stopped waits for the drive before anything resumes it" | +| The settle never resumes a box another caller stopped. | `settleSandbox` reads the boot log only from a box it just found running | `vercel.test.ts` settle tests | +| A provision that lost its row to an archive tears nothing down; that archive's reap, keyed on its `archivedAt`, owns the box. A later archive gets its own full grace period. | `provision.ts` (the `!ready` branch); `isStillArchived` and the retiring write in `reap.ts` | `provision.test.ts` "a provision that lost its row to an archive leaves the box to that archive's reap", "a failed create whose row was archived meanwhile tears nothing down" | +| Committing to delete a box and drive retires their names in the same write, so an undo after it gets a new box and a new drive, never the ones being deleted. | `retireAndTearDown` (`teardown.ts`) from the reap's delete stage and the archive fallback; the provision failure's transition to `failed` | `reap.test.ts` "the delete stage retires the box's names before it deletes them…", "an undo that wins the row before the retire leaves the box and drive alone"; `provision.test.ts` "a failed create retires the box's name in the same write…" | +| A box is stopped and its drive detached before either is deleted. Deletes never wait in a request, except promote, which must wait for the source's drive before it can fork it. | `teardownSandbox`, `deleteSandboxAndDriveLater` (`teardown.ts`), route `/api/cloud-workspaces/teardown`; promote's two detach waits (up to 2 minutes each) inside the 300 s tRPC route | `teardown.test.ts` "stops the box, waits for its drive, then deletes…" | +| A teardown never deletes what a workspace row, in any status, or a live environment names. Every real teardown clears the names first, so a name still on a row is that row's: an archived row may be undone onto it, a failed one keeps its box and drive for diagnosis. | `liveReferences` (`teardown-guard.ts`), read before each destructive call | `teardown.test.ts` "a box a revived workspace runs on is left alone", "an unarchived workspace's new box keeps the drive…" | +| A provision fails or stops only its own row: one still `provisioning`, or one this job itself made `awake`. A row another delivery or a wake brought up is never failed, stopped or torn down by it. | `madeAwake` and the `from` of the failure transition in `provision.ts`; the stop, teardown and failed chat moves follow only a transition it won | `provision.test.ts` "a second delivery on the same box that fails after the first brought the row up leaves the live workspace alone" | +| A provision that finds the box's lock held is retried, never failed or stopped for it. | `SandboxBusyError` from `sandboxResumeLock`; `provision.ts` requeues itself every 15 s, at most 10 times, then fails only a row still `provisioning` | `provision.test.ts` "a lock held by another caller requeues the provision…", "a lock still held after the last retry fails only a row still provisioning" | +| A lost provision's box is deleted even while the row's live box holds the kept drive. | `teardownSandbox` with `keepDrive` waits only for the drive to leave this box | `teardown.test.ts` "a lost box goes even while the row's live box holds the kept drive" | +| A failed provision deletes only a drive it forked. | `forkedDrive` in `provision.ts` | `provision.test.ts` "a failed create deletes a drive it forked, never one it found" | +| No teardown failure is silent. | `teardownSandbox` reports to Sentry and requeues any failure; giving up is a Sentry error | `teardown.test.ts` "any other failure is retried, not dropped", "gives up after the last attempt…" | +| A drive nothing names is found and, on request, deleted; the sweep never deletes a drive any row names, archived or not, and checks each again just before its delete. | `drive-sweep.ts` (`orphanCandidates`, `sweepDrives`, `stillOrphaned`), Sandbox workflow `run: drive-sweep`, dry run unless `drive_sweep_delete` | `drive-sweep.test.ts` "any row that names a drive keeps it…", "a drive an unarchive named after the listing is kept…" | +| A failed promote leaks neither the golden nor its drive. | `abandonGolden` (`environment.ts`) | `environment.test.ts` "hands the golden and its forked drive to the retried teardown" | +| A golden carries no workspace identity, on the root disk or the drive, and is stripped only with its drive mounted. | `inheritedIdentityPaths` (includes `SANDBOX_PATHS.driveState`); `stripWorkspaceIdentity` checks `mountpoint` and the exit code | `vercel.test.ts` "forks the source's drive once it is released and strips identity on the drive", "a golden whose drive did not mount is not stripped and fails the promote" | +| Turning the flag off stops new drives. | `buildSandboxClaim` (provision) and `environment.promote` consult `workspaceDrivesEnabled`; the release writes `golden_drive_name = null` | `claim.test.ts` "with the organization's flag off…"; `environment.test.ts` "a workspace whose files are on a drive does not spread drive mode…" | +| Released clients see the same outputs as before this branch. | `withClientStatus` drops `driveName`, `driveLaidOutAt`; `publicEnvironment` drops `goldenDriveName` | `client-status.test.ts` "never carries the drive columns…"; `environment.test.ts` "the output never carries the golden drive…" | +| The drive mount and the agent-tools mount survive each other's updates. | `composeMounts`, `wakeSandbox` | `vercel.test.ts` "the drive and the agent-tools drive share one mounts object", "an agent-tools update keeps the drive the box already has" | + + +## Rollback + +1. Turn `cloud-workspace-drives` off for the organization. From then on a new workspace gets no + drive, even in an environment in drive mode: its box clones and installs onto the root disk, as + before drives existed, and so is slower to create. Promote refuses a workspace whose files are + on a drive. The next release builds the golden without a drive and writes + `golden_drive_name = null`, which takes the environment out of drive mode. +2. Existing workspaces with a drive keep it and keep working: their data lives there. Nothing + moves it back to the root disk; such a workspace is archived and recreated if it must leave + drive mode. +3. Drives that nothing names any more: dispatch the Sandbox workflow with `run: drive-sweep`, + read the list, then run it again with `drive_sweep_delete`. +4. Code rollback: revert the branch's commits. Rows with `drive_name` keep their column, and no + pre-drive boot script may run over a laid-out drive (`drive.downgrade`), so workspaces with a + drive refuse to boot instead of showing an empty `/workspace`. Archive them before such a + revert. + + +## Before customers + +- Schedule the drive sweep and let it delete. Today it is a `workflow_dispatch` job only, dry run + unless `drive_sweep_delete` is set, so a drive whose teardown gave up stays until someone runs it + (the model's NoOrphanForever holds only if it is run). Once it has run dry in production a few + times and listed nothing it should not, run it on a schedule without the dry run. + + +## Follow-ups (out of scope) + +- Environment builds from `[install]` (PR #9282) producing golden drives. +- Migrating existing workspaces. +- Customer environments' checkpoints through `superset environments replace` beyond what promote + already covers. +- A drive variable in `Asleep.tla`: not needed. No status write is new: a create that meets a + drive refusal stops the box and records it asleep, which is `StopBox` and `StopRecord`; a wake + that stops a refused box writes no status, so the row sees it as `Crash`; a persistent refusal + marks the workspace failed through `markSandboxUnavailable`, as a gone box does. The model + passes unchanged (1,624,931 states) and the broken config still fails. +- Environment builds (#9282): `activateGolden` must move `source_ref` and `golden_drive_name` + together, in one row update, the way promote and the release do. A golden and a drive from two + different builds would give every new workspace a checkout that does not match its root disk. +- A sweep of drives named `drive-ws-*` whose sandbox no longer exists. + + +## Validation and Acceptance + +From the repository root: + + bun run lint:fix && bun run typecheck + cd packages/trpc && bun test # drives.test.ts, vercel.test.ts + cd packages/sandbox && bun test + bun run --cwd packages/sandbox image --local + cd packages/sandbox && bun run src/boot-twice.ts # includes the drive pass + .agents/skills/verify-sandbox-asleep/run.sh + +Against real Vercel: dispatch the Sandbox workflow with `run: real-sandbox` and +`real_sandbox_drive: true`. It must print `ok` for every `drive:` line, exit 0, and delete what it +made. `real_sandbox_keep: true` keeps the stopped boxes and the box drive for inspection. On a +failed drive expectation it prints a `drive: diagnosis` block (mounts, drive contents, the conf's +drive key, the run dir, `daemon.json`, the checkout log and the boot log). + + +## Idempotence and Recovery + +Every drive call is safe to repeat: `forkWorkspaceDrive` finds a drive of the same name, a delete +of a missing drive is a no-op, the boot's binds check `mountpoint -q` first. Turning the flag off +stops new golden drives at the next release; an environment already in drive mode stays in it +until a release or promote replaces its golden. + + +## Interfaces and Dependencies + +`packages/trpc/src/lib/sandbox/drives.ts`: `WORKSPACE_DRIVE_MOUNT_PATH`, `workspaceDriveNameFor`, +`goldenDriveNameFor`, `workspaceDriveNameToClaim`, `workspaceDriveMounts`, +`workspaceDrivesEnabled(organizationId)`, `DriveHost`, `vercelDrives()`, `forkWorkspaceDrive`, +`deleteWorkspaceDrive`, `waitForDriveDetached`, `DriveStillAttachedError`. +`golden-drives.ts`: `deleteGoldenDriveIfUnreferenced`. `vercel.ts`: `composeMounts`, +`deleteSandboxAndDrive`, `inheritedIdentityPaths`, and `promoteSandboxToEnvironment` with +`sourceDriveName`. `vercel-credentials.ts`: `credentials()`, moved out of `vercel.ts` so +`drives.ts` and `vercel.ts` do not import each other. From 5c03bd9cf1d82871c21253d437a172a35e1ec950 Mon Sep 17 00:00:00 2001 From: Satya Patel <14907857+saddlepaddle@users.noreply.github.com> Date: Wed, 18 Nov 2026 22:12:11 +0000 Subject: [PATCH 2/3] fix(cloud-workspace): no customer copy names a drive A drive refused for good fails the workspace as host_not_ready ("The workspace did not become ready."), not sandbox_gone. A drive still flushing at a wake is a SandboxNotReadyError ("still starting"). The errors a client can be handed carry the drive in fields, console lines and Sentry extra, not in the message. A promote of a workspace on a drive that fails, or is refused with drives off, says only "This workspace could not be saved as an environment. Try again later.", translated in every locale. The daily expiry probe is dropped: retention moves to its own PR, whose reaper calls retireAndTearDown, as the 410 path here does. --- packages/i18n/locales/cs/messages.po | 6 ++--- packages/i18n/locales/de/messages.po | 6 ++--- packages/i18n/locales/en/messages.po | 6 ++--- packages/i18n/locales/es/messages.po | 6 ++--- packages/i18n/locales/fr/messages.po | 6 ++--- packages/i18n/locales/id/messages.po | 6 ++--- packages/i18n/locales/it/messages.po | 6 ++--- packages/i18n/locales/ja/messages.po | 6 ++--- packages/i18n/locales/ko/messages.po | 6 ++--- packages/i18n/locales/nl/messages.po | 6 ++--- packages/i18n/locales/pl/messages.po | 6 ++--- packages/i18n/locales/pt-BR/messages.po | 6 ++--- packages/i18n/locales/ru/messages.po | 6 ++--- packages/i18n/locales/tr/messages.po | 6 ++--- packages/i18n/locales/uk/messages.po | 6 ++--- packages/i18n/locales/vi/messages.po | 6 ++--- packages/i18n/locales/zh-CN/messages.po | 6 ++--- packages/i18n/locales/zh-TW/messages.po | 6 ++--- packages/i18n/src/server-errors.ts | 4 ++-- packages/trpc/src/lib/sandbox/vercel.test.ts | 3 ++- packages/trpc/src/lib/sandbox/vercel.ts | 24 ++++++++++++------- .../trpc/src/router/cloud-workspace/wake.ts | 12 +++++++++- .../router/environment/environment.test.ts | 20 +++++++++++----- .../src/router/environment/environment.ts | 18 +++++++++++--- plans/20261116-workspace-drive.md | 22 ++++++++++++----- 25 files changed, 130 insertions(+), 81 deletions(-) diff --git a/packages/i18n/locales/cs/messages.po b/packages/i18n/locales/cs/messages.po index 372594aaa63..691c191eeee 100644 --- a/packages/i18n/locales/cs/messages.po +++ b/packages/i18n/locales/cs/messages.po @@ -20867,6 +20867,9 @@ msgstr "Tímto zrušíte přípravu a vrátíte všechny připravené změny. P msgid "This workspace" msgstr "Tento pracovní prostor" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Tento pracovní prostor se nepodařilo uložit jako prostředí. Zkuste to později." + msgid "This workspace has no project to branch from" msgstr "Tento pracovní prostor nemá projekt, ze kterého by šlo větvit" @@ -22633,9 +22636,6 @@ msgstr "Pracovní prostor vytvořen" msgid "Workspace details" msgstr "Podrobnosti pracovního prostoru" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Disky pracovních prostorů jsou pro tuto organizaci vypnuté, takže pracovní prostor, jehož soubory jsou na disku, se nemůže stát prostředím." - msgid "Workspace ID copied" msgstr "ID pracovního prostoru zkopírováno" diff --git a/packages/i18n/locales/de/messages.po b/packages/i18n/locales/de/messages.po index 6143d0ca70f..aa89ad95316 100644 --- a/packages/i18n/locales/de/messages.po +++ b/packages/i18n/locales/de/messages.po @@ -20867,6 +20867,9 @@ msgstr "Damit werden alle gestageden Änderungen aus dem Index genommen und zur msgid "This workspace" msgstr "Dieser Arbeitsbereich" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Dieser Arbeitsbereich konnte nicht als Umgebung gespeichert werden. Versuche es später erneut." + msgid "This workspace has no project to branch from" msgstr "Dieser Workspace hat kein Projekt, von dem aus verzweigt werden kann" @@ -22633,9 +22636,6 @@ msgstr "Arbeitsbereich erstellt" msgid "Workspace details" msgstr "Arbeitsbereichsdetails" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Arbeitsbereich-Laufwerke sind für diese Organisation ausgeschaltet, daher kann ein Arbeitsbereich, dessen Dateien auf einem Laufwerk liegen, nicht zu einer Umgebung werden." - msgid "Workspace ID copied" msgstr "Arbeitsbereich-ID kopiert" diff --git a/packages/i18n/locales/en/messages.po b/packages/i18n/locales/en/messages.po index 496dbf02e99..b1b97096467 100644 --- a/packages/i18n/locales/en/messages.po +++ b/packages/i18n/locales/en/messages.po @@ -20867,6 +20867,9 @@ msgstr "This will unstage and revert all staged changes. Staged new files will b msgid "This workspace" msgstr "This workspace" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "This workspace could not be saved as an environment. Try again later." + msgid "This workspace has no project to branch from" msgstr "This workspace has no project to branch from" @@ -22633,9 +22636,6 @@ msgstr "Workspace created" msgid "Workspace details" msgstr "Workspace details" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." - msgid "Workspace ID copied" msgstr "Workspace ID copied" diff --git a/packages/i18n/locales/es/messages.po b/packages/i18n/locales/es/messages.po index 1c3ab41db32..fdd4154ca59 100644 --- a/packages/i18n/locales/es/messages.po +++ b/packages/i18n/locales/es/messages.po @@ -20867,6 +20867,9 @@ msgstr "Esto quitará de preparados y revertirá todos los cambios preparados. L msgid "This workspace" msgstr "Este espacio de trabajo" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "No se pudo guardar este espacio de trabajo como entorno. Inténtalo de nuevo más tarde." + msgid "This workspace has no project to branch from" msgstr "Este espacio de trabajo no tiene un proyecto desde el que bifurcar" @@ -22633,9 +22636,6 @@ msgstr "Espacio de trabajo creado" msgid "Workspace details" msgstr "Detalles del espacio de trabajo" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Las unidades de espacio de trabajo están desactivadas para esta organización, así que un espacio de trabajo cuyos archivos están en una unidad no puede convertirse en un entorno." - msgid "Workspace ID copied" msgstr "ID del espacio de trabajo copiado" diff --git a/packages/i18n/locales/fr/messages.po b/packages/i18n/locales/fr/messages.po index 19ca75f319e..e0f28091a9d 100644 --- a/packages/i18n/locales/fr/messages.po +++ b/packages/i18n/locales/fr/messages.po @@ -20867,6 +20867,9 @@ msgstr "Cette action désindexera et annulera toutes les modifications indexées msgid "This workspace" msgstr "Cet espace de travail" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Impossible d’enregistrer cet espace de travail comme environnement. Réessayez plus tard." + msgid "This workspace has no project to branch from" msgstr "Cet espace de travail n'a aucun projet depuis lequel dupliquer" @@ -22633,9 +22636,6 @@ msgstr "Espace de travail créé" msgid "Workspace details" msgstr "Détails de l'espace de travail" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Les disques d'espace de travail sont désactivés pour cette organisation : un espace de travail dont les fichiers sont sur un disque ne peut pas devenir un environnement." - msgid "Workspace ID copied" msgstr "ID de l'espace de travail copié" diff --git a/packages/i18n/locales/id/messages.po b/packages/i18n/locales/id/messages.po index e3638f0c5bd..1f20fd63ef3 100644 --- a/packages/i18n/locales/id/messages.po +++ b/packages/i18n/locales/id/messages.po @@ -20867,6 +20867,9 @@ msgstr "Ini akan meng-unstage dan mengembalikan semua perubahan yang di-stage. F msgid "This workspace" msgstr "Workspace ini" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Workspace ini tidak dapat disimpan sebagai lingkungan. Coba lagi nanti." + msgid "This workspace has no project to branch from" msgstr "Ruang kerja ini tidak memiliki proyek untuk dicabangkan" @@ -22633,9 +22636,6 @@ msgstr "Workspace dibuat" msgid "Workspace details" msgstr "Detail workspace" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Drive workspace dinonaktifkan untuk organisasi ini, jadi workspace yang file-nya ada di drive tidak bisa menjadi lingkungan." - msgid "Workspace ID copied" msgstr "ID workspace disalin" diff --git a/packages/i18n/locales/it/messages.po b/packages/i18n/locales/it/messages.po index 1e85d757a96..35cc9c03307 100644 --- a/packages/i18n/locales/it/messages.po +++ b/packages/i18n/locales/it/messages.po @@ -20867,6 +20867,9 @@ msgstr "Questo toglierà dallo stage e annullerà tutte le modifiche in stage. I msgid "This workspace" msgstr "Questo workspace" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Impossibile salvare questo workspace come ambiente. Riprova più tardi." + msgid "This workspace has no project to branch from" msgstr "Questo workspace non ha un progetto da cui diramare" @@ -22633,9 +22636,6 @@ msgstr "Workspace creato" msgid "Workspace details" msgstr "Dettagli del workspace" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "I dischi dei workspace sono disattivati per questa organizzazione, quindi un workspace i cui file si trovano su un disco non può diventare un ambiente." - msgid "Workspace ID copied" msgstr "ID del workspace copiato" diff --git a/packages/i18n/locales/ja/messages.po b/packages/i18n/locales/ja/messages.po index efe1e6e7083..39438ee3a43 100644 --- a/packages/i18n/locales/ja/messages.po +++ b/packages/i18n/locales/ja/messages.po @@ -20867,6 +20867,9 @@ msgstr "ステージ済みの変更をすべてステージ解除して元に戻 msgid "This workspace" msgstr "このワークスペース" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "このワークスペースを環境として保存できませんでした。しばらくしてからもう一度お試しください。" + msgid "This workspace has no project to branch from" msgstr "このワークスペースには分岐元のプロジェクトがありません" @@ -22633,9 +22636,6 @@ msgstr "ワークスペースを作成しました" msgid "Workspace details" msgstr "ワークスペースの詳細" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "この組織ではワークスペースのドライブが無効になっているため、ファイルがドライブ上にあるワークスペースは環境にできません。" - msgid "Workspace ID copied" msgstr "ワークスペースIDをコピーしました" diff --git a/packages/i18n/locales/ko/messages.po b/packages/i18n/locales/ko/messages.po index 2cf53230988..65699d91b40 100644 --- a/packages/i18n/locales/ko/messages.po +++ b/packages/i18n/locales/ko/messages.po @@ -20867,6 +20867,9 @@ msgstr "스테이징된 변경 사항을 모두 스테이징 해제하고 되돌 msgid "This workspace" msgstr "이 워크스페이스" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "이 워크스페이스를 환경으로 저장할 수 없습니다. 나중에 다시 시도하세요." + msgid "This workspace has no project to branch from" msgstr "이 워크스페이스에는 분기할 프로젝트가 없습니다" @@ -22633,9 +22636,6 @@ msgstr "워크스페이스를 만들었습니다" msgid "Workspace details" msgstr "워크스페이스 세부 정보" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "이 조직에서는 워크스페이스 드라이브가 꺼져 있어, 파일이 드라이브에 있는 워크스페이스는 환경이 될 수 없습니다." - msgid "Workspace ID copied" msgstr "워크스페이스 ID 복사됨" diff --git a/packages/i18n/locales/nl/messages.po b/packages/i18n/locales/nl/messages.po index ae8abdb4d9e..6abc8cec65e 100644 --- a/packages/i18n/locales/nl/messages.po +++ b/packages/i18n/locales/nl/messages.po @@ -20867,6 +20867,9 @@ msgstr "Hiermee worden alle gestagede wijzigingen ge-unstaged en teruggedraaid. msgid "This workspace" msgstr "Deze werkruimte" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Deze werkruimte kon niet als omgeving worden opgeslagen. Probeer het later opnieuw." + msgid "This workspace has no project to branch from" msgstr "Deze workspace heeft geen project om vanaf af te splitsen" @@ -22633,9 +22636,6 @@ msgstr "Werkruimte aangemaakt" msgid "Workspace details" msgstr "Werkruimtedetails" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Werkruimteschijven zijn uitgeschakeld voor deze organisatie, dus een werkruimte waarvan de bestanden op een schijf staan, kan geen omgeving worden." - msgid "Workspace ID copied" msgstr "Werkruimte-ID gekopieerd" diff --git a/packages/i18n/locales/pl/messages.po b/packages/i18n/locales/pl/messages.po index 581122cab21..8ee8b616e15 100644 --- a/packages/i18n/locales/pl/messages.po +++ b/packages/i18n/locales/pl/messages.po @@ -20867,6 +20867,9 @@ msgstr "Usunie to z indeksu i cofnie wszystkie zaindeksowane zmiany. Zaindeksowa msgid "This workspace" msgstr "Ten obszar roboczy" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Nie udało się zapisać tego obszaru roboczego jako środowiska. Spróbuj ponownie później." + msgid "This workspace has no project to branch from" msgstr "Ten obszar roboczy nie ma projektu, z którego można się rozgałęzić" @@ -22633,9 +22636,6 @@ msgstr "Obszar roboczy utworzony" msgid "Workspace details" msgstr "Szczegóły obszaru roboczego" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Dyski obszarów roboczych są wyłączone w tej organizacji, więc obszar roboczy, którego pliki są na dysku, nie może stać się środowiskiem." - msgid "Workspace ID copied" msgstr "Skopiowano ID obszaru roboczego" diff --git a/packages/i18n/locales/pt-BR/messages.po b/packages/i18n/locales/pt-BR/messages.po index 28cad998528..95b674bc955 100644 --- a/packages/i18n/locales/pt-BR/messages.po +++ b/packages/i18n/locales/pt-BR/messages.po @@ -20867,6 +20867,9 @@ msgstr "Isso remove do stage e reverte todas as alterações no stage. Arquivos msgid "This workspace" msgstr "Esta área de trabalho" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Não foi possível salvar esta área de trabalho como ambiente. Tente novamente mais tarde." + msgid "This workspace has no project to branch from" msgstr "Este workspace não tem projeto do qual ramificar" @@ -22633,9 +22636,6 @@ msgstr "Área de trabalho criada" msgid "Workspace details" msgstr "Detalhes da área de trabalho" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Os discos de área de trabalho estão desativados para esta organização, então uma área de trabalho cujos arquivos estão em um disco não pode virar um ambiente." - msgid "Workspace ID copied" msgstr "ID da área de trabalho copiado" diff --git a/packages/i18n/locales/ru/messages.po b/packages/i18n/locales/ru/messages.po index b489ee5c6ac..ca64a5a5164 100644 --- a/packages/i18n/locales/ru/messages.po +++ b/packages/i18n/locales/ru/messages.po @@ -20867,6 +20867,9 @@ msgstr "Это снимет индексацию и откатит все про msgid "This workspace" msgstr "Эта рабочая область" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Не удалось сохранить это рабочее пространство как окружение. Повторите попытку позже." + msgid "This workspace has no project to branch from" msgstr "В этом рабочем пространстве нет проекта для ответвления" @@ -22633,9 +22636,6 @@ msgstr "Рабочая область создана" msgid "Workspace details" msgstr "Сведения о рабочей области" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Диски рабочих областей отключены для этой организации, поэтому рабочая область, файлы которой лежат на диске, не может стать окружением." - msgid "Workspace ID copied" msgstr "ID рабочей области скопирован" diff --git a/packages/i18n/locales/tr/messages.po b/packages/i18n/locales/tr/messages.po index 632c9ba7412..1955431dcd9 100644 --- a/packages/i18n/locales/tr/messages.po +++ b/packages/i18n/locales/tr/messages.po @@ -20867,6 +20867,9 @@ msgstr "Bu işlem, hazırlanmış tüm değişiklikleri hazırlıktan çıkarır msgid "This workspace" msgstr "Bu çalışma alanı" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Bu çalışma alanı ortam olarak kaydedilemedi. Daha sonra tekrar deneyin." + msgid "This workspace has no project to branch from" msgstr "Bu çalışma alanının çatallanacak bir projesi yok" @@ -22633,9 +22636,6 @@ msgstr "Çalışma alanı oluşturuldu" msgid "Workspace details" msgstr "Çalışma alanı ayrıntıları" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Bu kuruluş için çalışma alanı sürücüleri kapalı; bu yüzden dosyaları bir sürücüde olan bir çalışma alanı ortama dönüştürülemez." - msgid "Workspace ID copied" msgstr "Çalışma alanı kimliği kopyalandı" diff --git a/packages/i18n/locales/uk/messages.po b/packages/i18n/locales/uk/messages.po index d00c564ad93..052346a5af6 100644 --- a/packages/i18n/locales/uk/messages.po +++ b/packages/i18n/locales/uk/messages.po @@ -20861,6 +20861,9 @@ msgstr "Це зніме з індексу та відкотить усі про msgid "This workspace" msgstr "Цей робочий простір" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Не вдалося зберегти цей робочий простір як середовище. Спробуйте пізніше." + msgid "This workspace has no project to branch from" msgstr "У цього робочого простору немає проєкту, від якого можна створити гілку" @@ -22627,9 +22630,6 @@ msgstr "Робочий простір створено" msgid "Workspace details" msgstr "Відомості про робочий простір" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Диски робочих просторів вимкнено для цієї організації, тому робочий простір, файли якого на диску, не може стати середовищем." - msgid "Workspace ID copied" msgstr "ID робочого простору скопійовано" diff --git a/packages/i18n/locales/vi/messages.po b/packages/i18n/locales/vi/messages.po index 08ecef9adb0..005cf9911e5 100644 --- a/packages/i18n/locales/vi/messages.po +++ b/packages/i18n/locales/vi/messages.po @@ -20867,6 +20867,9 @@ msgstr "Thao tác này sẽ bỏ stage và hoàn nguyên toàn bộ thay đổi msgid "This workspace" msgstr "Workspace này" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Không thể lưu không gian làm việc này thành môi trường. Hãy thử lại sau." + msgid "This workspace has no project to branch from" msgstr "Workspace này không có dự án để tạo nhánh" @@ -22633,9 +22636,6 @@ msgstr "Đã tạo workspace" msgid "Workspace details" msgstr "Chi tiết workspace" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "Ổ đĩa workspace đã bị tắt cho tổ chức này, nên một workspace có tệp nằm trên ổ đĩa không thể trở thành môi trường." - msgid "Workspace ID copied" msgstr "Đã sao chép ID workspace" diff --git a/packages/i18n/locales/zh-CN/messages.po b/packages/i18n/locales/zh-CN/messages.po index 58d4baf1e82..37a38483732 100644 --- a/packages/i18n/locales/zh-CN/messages.po +++ b/packages/i18n/locales/zh-CN/messages.po @@ -20867,6 +20867,9 @@ msgstr "这将取消暂存并还原所有已暂存的更改。已暂存的新文 msgid "This workspace" msgstr "此工作区" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "无法将此工作区保存为环境。请稍后重试。" + msgid "This workspace has no project to branch from" msgstr "此工作区没有可供分叉的项目" @@ -22633,9 +22636,6 @@ msgstr "工作区已创建" msgid "Workspace details" msgstr "工作区详情" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "此组织已关闭工作区驱动器,因此文件位于驱动器上的工作区无法成为环境。" - msgid "Workspace ID copied" msgstr "已复制工作区ID" diff --git a/packages/i18n/locales/zh-TW/messages.po b/packages/i18n/locales/zh-TW/messages.po index e8db5cba7d3..871c0b51a37 100644 --- a/packages/i18n/locales/zh-TW/messages.po +++ b/packages/i18n/locales/zh-TW/messages.po @@ -20867,6 +20867,9 @@ msgstr "這將取消暫存並還原所有已暫存的更改。已暫存的新檔 msgid "This workspace" msgstr "此工作區" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "無法將此工作區儲存為環境。請稍後再試。" + msgid "This workspace has no project to branch from" msgstr "此工作區沒有可供分叉的專案" @@ -22633,9 +22636,6 @@ msgstr "工作區已建立" msgid "Workspace details" msgstr "工作區詳細資料" -msgid "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment." -msgstr "此組織已關閉工作區磁碟,因此檔案位於磁碟上的工作區無法成為環境。" - msgid "Workspace ID copied" msgstr "已複製工作區ID" diff --git a/packages/i18n/src/server-errors.ts b/packages/i18n/src/server-errors.ts index 81cc3b2e08e..55050df12b7 100644 --- a/packages/i18n/src/server-errors.ts +++ b/packages/i18n/src/server-errors.ts @@ -508,11 +508,11 @@ export const serverErrorMessages: Record< "This environment's repositories are fixed; promote a workspace again to change them", }), ), - "serverError.environment.drivesOff": () => + "serverError.environment.couldNotPromote": () => i18n._( msg({ message: - "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment.", + "This workspace could not be saved as an environment. Try again later.", }), ), "serverError.environment.couldNotRecord": () => diff --git a/packages/trpc/src/lib/sandbox/vercel.test.ts b/packages/trpc/src/lib/sandbox/vercel.test.ts index 6a071d53891..149d74b27be 100644 --- a/packages/trpc/src/lib/sandbox/vercel.test.ts +++ b/packages/trpc/src/lib/sandbox/vercel.test.ts @@ -1145,7 +1145,8 @@ describe("settleSandbox with a drive", () => { }).catch((caught: unknown) => caught); expect(error).toBeInstanceOf(SandboxDriveRefusedError); expect(error).toBeInstanceOf(SandboxUnavailableError); - expect(String(error)).toContain(`in ${DRIVE_REFUSAL_LIMIT} boots in a row`); + expect(error).toMatchObject({ boots: DRIVE_REFUSAL_LIMIT }); + expect(String(error)).not.toContain("drive"); }); test("a boot that answers without laying the drive out fails the settle", async () => { diff --git a/packages/trpc/src/lib/sandbox/vercel.ts b/packages/trpc/src/lib/sandbox/vercel.ts index 205781d70d3..fa1b33c63ef 100644 --- a/packages/trpc/src/lib/sandbox/vercel.ts +++ b/packages/trpc/src/lib/sandbox/vercel.ts @@ -38,6 +38,7 @@ import { } from "./credentials"; import { type DriveHost, + DriveStillAttachedError, deleteWorkspaceDrive, goldenDriveNameFor, vercelDrives, @@ -458,10 +459,9 @@ export class SandboxNotReadyError extends Error { /** Another caller holds the box's resume lock: retry later, nothing is wrong with the box. */ export class SandboxBusyError extends SandboxNotReadyError { - constructor(key: string) { - super(key); + constructor(readonly key: string) { + super("a sandbox another caller is starting"); this.name = "SandboxBusyError"; - this.message = `Another caller is resuming or creating the box on ${key}`; } } @@ -478,14 +478,13 @@ export class SandboxDriveMissingError extends SandboxNotReadyError { ) { super(providerSandboxId); this.name = "SandboxDriveMissingError"; - this.message = `Sandbox ${providerSandboxId} did not come up on its drive ${driveName}: ${reason}`; } } /** * The drive refused for good: the same refusal that no remount can change, in - * several boots in a row. The workspace is marked failed with this message; - * its drive and box are kept. + * several boots in a row. The workspace is marked failed; its drive and box + * are kept. */ export class SandboxDriveRefusedError extends SandboxUnavailableError { constructor( @@ -496,7 +495,6 @@ export class SandboxDriveRefusedError extends SandboxUnavailableError { ) { super(providerSandboxId, new Error(reason)); this.name = "SandboxDriveRefusedError"; - this.message = `Sandbox ${providerSandboxId} refused its drive ${driveName} in ${boots} boots in a row: ${reason}`; } } @@ -645,7 +643,12 @@ export async function settleSandbox(args: { reason, boots, ); - Sentry.captureException(refused); + console.warn( + `[sandbox] ${args.providerSandboxId}: ${reason} in ${boots} boots in a row`, + ); + Sentry.captureException(refused, { + extra: { driveName, reason, boots }, + }); return refused; }; const bootId = args.bootId; @@ -1084,6 +1087,11 @@ export async function wakeSandbox(args: { } catch (error) { if (isUnavailable(error)) throw new SandboxUnavailableError(args.providerSandboxId, error); + // The last session is still flushing: the box is starting, as any caller sees it. + if (error instanceof DriveStillAttachedError) { + console.warn(`[sandbox] ${args.providerSandboxId}: ${error.message}`); + throw new SandboxNotReadyError(args.providerSandboxId); + } throw error; } } diff --git a/packages/trpc/src/router/cloud-workspace/wake.ts b/packages/trpc/src/router/cloud-workspace/wake.ts index 2e842e0b2a2..9c1d77aef98 100644 --- a/packages/trpc/src/router/cloud-workspace/wake.ts +++ b/packages/trpc/src/router/cloud-workspace/wake.ts @@ -1,5 +1,6 @@ import * as Sentry from "@sentry/core"; import { db } from "@superset/db/client"; +import type { CloudWorkspaceFailureReason } from "@superset/db/enums"; import { cloudWorkspaces } from "@superset/db/schema"; import { isCloudUsageRefusal } from "@superset/shared/billing"; import { @@ -18,6 +19,7 @@ import { bootPhases, buildSandboxClaim, describeSandbox, + SandboxDriveRefusedError, type SandboxUnavailableError, stopAndSnapshot, stopSandbox, @@ -244,6 +246,14 @@ export async function wakeAfterStop( return wakeCloudWorkspace(row); } +/** A drive that refused reads as any box that would not start. */ +function unavailableReason( + error: SandboxUnavailableError, +): CloudWorkspaceFailureReason { + if (error instanceof SandboxDriveRefusedError) return "host_not_ready"; + return error.snapshotExpired ? "snapshot_expired" : "sandbox_gone"; +} + /** * For a sandbox that is gone or can never resume. An `awake` row nothing can * open would sit in the sidebar forever; failed is the state the client @@ -258,7 +268,7 @@ export async function markSandboxUnavailable( scope: { cloudWorkspaceId: row.id, organizationId: row.organizationId }, from: ["awake", "asleep"], set: { sandboxUrl: null }, - reason: error.snapshotExpired ? "snapshot_expired" : "sandbox_gone", + reason: unavailableReason(error), error, }); if (row.driveName && error.snapshotExpired) { diff --git a/packages/trpc/src/router/environment/environment.test.ts b/packages/trpc/src/router/environment/environment.test.ts index 7381fc238a5..076f1042615 100644 --- a/packages/trpc/src/router/environment/environment.test.ts +++ b/packages/trpc/src/router/environment/environment.test.ts @@ -229,14 +229,18 @@ const failingPromote = { describe("environment.promote that fails", () => { stub(sandbox, failingPromote); - test("hands the golden and its forked drive to the retried teardown", async () => { + test("hands the golden and its forked drive to the retried teardown, and tells the user only that the promote failed", async () => { environment.createdByUserId = "user-1"; - await expect( - caller.environment.promote({ + const failure = await caller.environment + .promote({ cloudWorkspaceId: WORKSPACE_ID, environmentId: ENVIRONMENT_ID, - }), - ).rejects.toThrow("snapshot failed after the fork"); + }) + .catch((error: Error) => error); + expect(failure).toMatchObject({ + message: + "This workspace could not be saved as an environment. Try again later.", + }); expect(abandoned).toHaveLength(1); const [golden] = abandoned; expect(golden?.providerSandboxId).toMatch(/^env-/); @@ -255,7 +259,11 @@ describe("environment.promote with drives off", () => { cloudWorkspaceId: WORKSPACE_ID, environmentId: ENVIRONMENT_ID, }), - ).rejects.toMatchObject({ code: "PRECONDITION_FAILED" }); + ).rejects.toMatchObject({ + code: "PRECONDITION_FAILED", + message: + "This workspace could not be saved as an environment. Try again later.", + }); expect(promoted).toBe(0); expect(abandoned).toEqual([]); }); diff --git a/packages/trpc/src/router/environment/environment.ts b/packages/trpc/src/router/environment/environment.ts index fb361d239cd..f841b708dc0 100644 --- a/packages/trpc/src/router/environment/environment.ts +++ b/packages/trpc/src/router/environment/environment.ts @@ -12,6 +12,7 @@ import { isCloudUsageRefusal, MACHINE_SIZES } from "@superset/shared/billing"; import { SANDBOX_IMAGE_NAME } from "@superset/shared/constants"; import { SANDBOX_REGION_IDS } from "@superset/shared/sandbox/regions"; import type { TRPCRouterRecord } from "@trpc/server"; +import { TRPCError } from "@trpc/server"; import { and, asc, eq, inArray, isNull, or } from "drizzle-orm"; import { z } from "zod"; import { @@ -538,8 +539,8 @@ export const environmentRouter = { throw userError({ code: "PRECONDITION_FAILED", message: - "Workspace drives are turned off for this organization, so a workspace whose files are on a drive cannot become an environment.", - i18nKey: "serverError.environment.drivesOff", + "This workspace could not be saved as an environment. Try again later.", + i18nKey: "serverError.environment.couldNotPromote", }); } const golden = await promoteSandboxToEnvironment({ @@ -552,7 +553,18 @@ export const environmentRouter = { goldenName, workspace.driveName ? goldenDriveNameFor(goldenName) : null, ); - throw error; + if (!workspace.driveName || error instanceof TRPCError) throw error; + // What went wrong names the drive; the user sees the failure any promote can have. + console.error( + `[environment/promote] ${workspace.id} could not be promoted`, + error, + ); + throw userError({ + code: "PRECONDITION_FAILED", + message: + "This workspace could not be saved as an environment. Try again later.", + i18nKey: "serverError.environment.couldNotPromote", + }); }); const { stopped } = golden; const restart = async () => { diff --git a/plans/20261116-workspace-drive.md b/plans/20261116-workspace-drive.md index 7d780638dcc..97b3d1f51b8 100644 --- a/plans/20261116-workspace-drive.md +++ b/plans/20261116-workspace-drive.md @@ -239,12 +239,19 @@ Documented by Vercel or the SDK, not measured: fork fell back to main's reclone, which empties the checkout. A fresh fork has no user work, and the later-boot refetch of an untouched copy covers it, so it retries three times and then keeps the golden's checkout. Date/Author: 2026-11-18, coordinator. -- Decision: a workspace's drive is deleted when its snapshot expires (30 days after last use). - Rationale: the box can never resume, and the drive would bill storage for good. Vercel expires - the snapshot itself; the control plane learns of it only from the 410 a wake, an agent turn or an - automation dispatch gets, so that is where the drive goes, through the retried teardown. A - workspace nobody opens again keeps its drive until the drive sweep runs. Date/Author: 2026-11-18, - Satya. +- Decision: a workspace's drive is deleted when its snapshot expires. Rationale: the box can never + resume, and the drive would bill storage for good. Retention will be ours: a separate reaper + deletes an expired workspace's box, snapshots and drive itself, instead of relying on Vercel's + 30-day `snapshotExpiration`. It calls `retireAndTearDown`, the one function that retires a row's + box and drive names and queues their retried teardown. Until then, and as a fallback after, + the 410 a wake, an agent turn or an automation dispatch gets on an expired snapshot takes + `markSandboxUnavailable` to the same function. Date/Author: 2026-11-18, Satya. +- Decision: a customer never reads that the box has a drive. A drive that refused for good fails + the workspace with `host_not_ready` ("The workspace did not become ready."), a drive still + flushing at a wake answers "still starting", the error messages a client can be handed name no + drive, and a promote of a workspace on a drive that fails, or is refused with drives off, says + "This workspace could not be saved as an environment. Try again later." The detail stays in the + logs and in the errors' fields. Date/Author: 2026-11-18, coordinator. - Decision: a checkout with no marker on a drive is fetched only while it is the golden's untouched copy (`state/golden-head`), else kept (the model's FixFetchIfUntouched). Rationale: removing `state/fresh` after the first attempt keeps a failed first checkout from being refetched over the @@ -533,6 +540,9 @@ never shown as their work. unless `drive_sweep_delete` is set, so a drive whose teardown gave up stays until someone runs it (the model's NoOrphanForever holds only if it is run). Once it has run dry in production a few times and listed nothing it should not, run it on a schedule without the dry run. +- Our own retention (a separate PR): delete an expired workspace's box, snapshots and drive + through `retireAndTearDown`. Until it ships, a drive goes only when someone opens its workspace + after Vercel expired the snapshot. ## Follow-ups (out of scope) From bf7543b4f0add326e28b528cd855b9176ecd4bb7 Mon Sep 17 00:00:00 2001 From: Satya Patel <14907857+saddlepaddle@users.noreply.github.com> Date: Wed, 18 Nov 2026 23:21:24 +0000 Subject: [PATCH 3/3] fix(sandbox): host-service's own env turns the agents' self-updates off DISABLE_AUTOUPDATER, OPENCODE_DISABLE_AUTOUPDATE and PI_SKIP_VERSION_CHECK were only in the login profile. A chat agent is launched from host-service's login-shell snapshot, which falls back to host-service's own env when the probe fails, and that env is the env -i list in superset-boot. The three now ride in that list too. The plan's retention decision names #9987's reaper, which calls retireAndTearDown, and its Before-customers item goes. --- docs/cloud-sandbox-mismatches.md | 4 +++- .../sandbox/bundle/rootfs/usr/local/bin/superset-boot | 2 ++ packages/sandbox/src/boot-twice.ts | 8 ++++++++ plans/20261116-workspace-drive.md | 7 ++----- 4 files changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/cloud-sandbox-mismatches.md b/docs/cloud-sandbox-mismatches.md index 6385138d963..9f379ac1119 100644 --- a/docs/cloud-sandbox-mismatches.md +++ b/docs/cloud-sandbox-mismatches.md @@ -1011,7 +1011,9 @@ OpenCode and Pi update in place, beside their binary. On a box they run from a read-only mount and are pinned by the bundle. `/etc/profile.d/superset.sh` turns off their update checks (`OPENCODE_DISABLE_AUTOUPDATE`, `PI_SKIP_VERSION_CHECK`), the same way `DISABLE_AUTOUPDATER` does for -Claude. They keep their state under `~/.local/share/opencode`, +Claude. `superset-boot` also puts all three in host-service's own env: a chat +agent gets host-service's env, not a terminal's, whenever host-service's +login-shell probe fails. They keep their state under `~/.local/share/opencode`, `~/.config/opencode` and `~/.pi/agent`, as they do on a laptop. **A read-only snapshot mount taken while a writer holds the drive is corrupt.** diff --git a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot index eee15fe1aa5..c13fc864f36 100755 --- a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot +++ b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot @@ -293,6 +293,8 @@ host_env=( "SUPERSET_SANDBOX_IMAGE_TAG=${SUPERSET_SANDBOX_IMAGE_TAG}" "SUPERSET_SANDBOX_PROVIDER=${SUPERSET_SANDBOX_PROVIDER}" "SUPERSET_RUN_DIR=${SUPERSET_RUN_DIR}" "SUPERSET_SANDBOX_BOOT_LOG=${SUPERSET_BOOT_LOG}" "HOST_SERVICE_SECRET=${HOST_SERVICE_SECRET}" + # Also in the login profile; here too, so an agent host-service starts gets them when its shell probe fails. + "DISABLE_AUTOUPDATER=1" "OPENCODE_DISABLE_AUTOUPDATE=1" "PI_SKIP_VERSION_CHECK=1" ) [ -n "$DRIVE_MODE" ] && host_env+=("BUN_INSTALL_CACHE_DIR=${SUPERSET_BUN_STORE}" "npm_config_store_dir=${SUPERSET_PNPM_STORE}") for key in HOST_SERVICE_SENTRY_DSN HOST_SERVICE_SENTRY_ENVIRONMENT SUPERSET_SANDBOX_CREATOR_USER_ID \ diff --git a/packages/sandbox/src/boot-twice.ts b/packages/sandbox/src/boot-twice.ts index b0b73fa2435..deeca68df57 100644 --- a/packages/sandbox/src/boot-twice.ts +++ b/packages/sandbox/src/boot-twice.ts @@ -844,6 +844,14 @@ try { .split("\n") .find((line) => line.startsWith("SUPERSET_SANDBOX_PLUGINS=")) ?? "unset", ); + expect( + "host-service's own env turns the agents' self-updates off, whatever its shell probe finds", + [ + "DISABLE_AUTOUPDATER=1", + "OPENCODE_DISABLE_AUTOUPDATE=1", + "PI_SKIP_VERSION_CHECK=1", + ].every((line) => hostEnv.split("\n").includes(line)), + ); // The control plane's push, so the runner's hook sequencing runs: this // checkout declares no start hook, which the log must say. diff --git a/plans/20261116-workspace-drive.md b/plans/20261116-workspace-drive.md index 97b3d1f51b8..11a79685c67 100644 --- a/plans/20261116-workspace-drive.md +++ b/plans/20261116-workspace-drive.md @@ -240,8 +240,8 @@ Documented by Vercel or the SDK, not measured: the later-boot refetch of an untouched copy covers it, so it retries three times and then keeps the golden's checkout. Date/Author: 2026-11-18, coordinator. - Decision: a workspace's drive is deleted when its snapshot expires. Rationale: the box can never - resume, and the drive would bill storage for good. Retention will be ours: a separate reaper - deletes an expired workspace's box, snapshots and drive itself, instead of relying on Vercel's + resume, and the drive would bill storage for good. Retention will be ours: the reaper of #9987 + (every 6 h) deletes an expired workspace's box, snapshots and drive itself, instead of relying on Vercel's 30-day `snapshotExpiration`. It calls `retireAndTearDown`, the one function that retires a row's box and drive names and queues their retried teardown. Until then, and as a fallback after, the 410 a wake, an agent turn or an automation dispatch gets on an expired snapshot takes @@ -540,9 +540,6 @@ never shown as their work. unless `drive_sweep_delete` is set, so a drive whose teardown gave up stays until someone runs it (the model's NoOrphanForever holds only if it is run). Once it has run dry in production a few times and listed nothing it should not, run it on a schedule without the dry run. -- Our own retention (a separate PR): delete an expired workspace's box, snapshots and drive - through `retireAndTearDown`. Until it ships, a drive goes only when someone opens its workspace - after Vercel expired the snapshot. ## Follow-ups (out of scope)