diff --git a/.agents/skills/verify-sandbox-asleep/SKILL.md b/.agents/skills/verify-sandbox-asleep/SKILL.md index 18fe8968e98..f3b41e9df58 100644 --- a/.agents/skills/verify-sandbox-asleep/SKILL.md +++ b/.agents/skills/verify-sandbox-asleep/SKILL.md @@ -38,12 +38,12 @@ Synced with commit `0ca8daad`. Paths are under `packages/trpc/src/`. | `ProvisionDone` | `provisionCloudWorkspace` in `router/cloud-workspace/provision.ts`: `provisionSandbox`, the transition to `awake`, then `recordSandboxAwake` | | `WakeResume` / `WakeRecord` | `wakeCloudWorkspace` in `router/cloud-workspace/wake.ts`: `wakeSandbox` in `lib/sandbox/vercel.ts`, then `recordSandboxAwake` in `router/cloud-workspace/sandbox-expiry.ts`. A wake that read the box running and resumed it after a timeout stop is the same `WakeResume`: the box was stopped when the call resumed it | | `TopUpExtend` / `TopUpRecord` | `sandbox.keepalive` in `router/sandbox/sandbox.ts`: `topUpSandboxSession`, then `recordSandboxAwake` | -| `StopBox` / `StopRecord` | `stopAndSnapshot` or `stopSandbox`, then `recordSandboxAsleep` in `router/cloud-workspace/sandbox-sleep.ts` (most sites call it through `recordBoxStopped`, which first writes the boot's end reason, a write the model leaves out), from `cloudWorkspace.sleep` (`cloud-workspace.ts`), `restartCloudWorkspace` and `wakeAfterStop` (`wake.ts`; `environment.promote` calls `wakeAfterStop` after the snapshot in `promoteSandboxToEnvironment` stopped the source, so `StopBox` there is the snapshot), a refused `wakeCloudWorkspace` whose own call resumed the box (it records that session awake first, a `WakeRecord`, then stops it; a refusal for a box another wake resumed stops nothing), and the spend-cap stops in `meter.ts` (`settleBoot`, `settleOrganizationNow`, the refused adopt in `adoptRunningBoxes`) | +| `StopBox` / `StopRecord` | `stopAndSnapshot` or `stopSandbox`, then `recordSandboxAsleep` in `router/cloud-workspace/sandbox-sleep.ts` (most sites call it through `recordBoxStopped`, which first writes the boot's end reason, a write the model leaves out), from `cloudWorkspace.sleep` (`cloud-workspace.ts`), `restartCloudWorkspace` and `wakeAfterStop` (`wake.ts`; `environment.promote` calls `wakeAfterStop` after the snapshot in `promoteSandboxToEnvironment` stopped the source, so `StopBox` there is the snapshot), a create whose drive refused (`provision.ts`, after its transition to `awake`), a refused `wakeCloudWorkspace` whose own call resumed the box (it records that session awake first, a `WakeRecord`, then stops it; a refusal for a box another wake resumed stops nothing), and the spend-cap stops in `meter.ts` (`settleBoot`, `settleOrganizationNow`, the refused adopt in `adoptRunningBoxes`) | | `WatchFire` | `watchSandboxExpiry` in `sandbox-expiry.ts`, queued by `queueWatch` through `publishCloudWorkspaceJob` (`jobs.ts`), served by `apps/api/src/app/api/cloud-workspaces/sandbox-expiry/route.ts` | | `ReconcileList` / `ReconcileApply` | `reconcileCloudWorkspaces` in `sandbox-expiry.ts`: `listActiveWorkspaceSandboxes`, then `syncSandboxExpiries` against each listing's `currentSessionId`; a stopped listing goes through `recordStopSeen` in `sandbox-sleep.ts` (`recordSandboxAsleep`, with the end reason first when the recorded expiry has passed) | | `Archive` | `cloudWorkspace.delete` in `cloud-workspace.ts`: the transition to `archived`, then `queueReap` | | `Reap` | the stop stage of `reapArchivedCloudWorkspace` in `reap.ts`, and the `queueReap` fallback that stops at once when the stop cannot be queued. Both call `recordSandboxAsleep`, which on an archived row writes only the expiry and session id | -| `Tick` / `Crash` | Vercel ends a session at its timeout, or a session fails on its own | +| `Tick` / `Crash` | Vercel ends a session at its timeout, or a session fails on its own. A wake that stops a box whose drive refused (`wakeSandbox`) writes no status, so the row sees it as a crash | | `CAS` | `stillStoppedSession` in `sandbox-sleep.ts` (asleep writes) and the new-session rule in `markAwake` (awake writes) | When you add, remove or reorder a writer, change its action here and in `Asleep.tla`, run both diff --git a/.github/workflows/sandbox.yml b/.github/workflows/sandbox.yml index 31d4d56969c..ad0708d52c9 100644 --- a/.github/workflows/sandbox.yml +++ b/.github/workflows/sandbox.yml @@ -24,7 +24,24 @@ on: - real-sandbox - production-release - inventory + - drive-sweep default: real-sandbox + real_sandbox_drive: + description: "real-sandbox only: also run the drive-mode pass (a throwaway golden drive, a box on a fork of it, a stop and a wake)" + type: boolean + default: false + real_sandbox_keep: + description: "real-sandbox only: keep the boxes (stopped) and the box drive for inspection instead of deleting them" + type: boolean + default: false + drive_sweep_delete: + description: "drive-sweep only: delete the orphaned drives it lists (off: a dry run)" + type: boolean + default: false + real_sandbox_work_gb: + description: "real-sandbox drive pass only: GB written under /workspace on the drive box and the plain box" + type: string + default: "4" jobs: boot-twice: @@ -119,7 +136,7 @@ jobs: name: Real sandbox if: github.event_name == 'workflow_dispatch' && inputs.run == 'real-sandbox' runs-on: ubuntu-latest - timeout-minutes: 30 + timeout-minutes: 90 steps: - name: Check out code uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 @@ -146,6 +163,9 @@ jobs: CDN_R2_ENDPOINT: ${{ secrets.CDN_R2_ENDPOINT }} CDN_R2_BUCKET: ${{ secrets.CDN_R2_BUCKET }} CDN_URL: ${{ secrets.CDN_URL }} + REAL_SANDBOX_DRIVE: ${{ inputs.real_sandbox_drive && '1' || '' }} + REAL_SANDBOX_KEEP: ${{ inputs.real_sandbox_keep && '1' || '' }} + REAL_SANDBOX_WORK_GB: ${{ inputs.real_sandbox_work_gb }} run: bun run src/real-sandbox.ts inventory: @@ -173,6 +193,34 @@ jobs: VERCEL_SANDBOX_PROJECT_ID: ${{ secrets.VERCEL_SANDBOX_PROJECT_ID }} run: bun run inventory + drive-sweep: + name: Drive sweep + if: github.event_name == 'workflow_dispatch' && inputs.run == 'drive-sweep' + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Setup Bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 + with: + bun-version-file: .bun-version + + - name: Install dependencies + run: bun install --frozen --ignore-scripts + + - name: List, or delete, drives nothing names + working-directory: packages/sandbox + env: + NEON_API_KEY: ${{ secrets.NEON_API_KEY }} + NEON_PROJECT_ID: ${{ secrets.NEON_PROJECT_ID }} + VERCEL_SANDBOX_TOKEN: ${{ secrets.VERCEL_SANDBOX_TOKEN }} + VERCEL_SANDBOX_TEAM_ID: ${{ secrets.VERCEL_SANDBOX_TEAM_ID }} + VERCEL_SANDBOX_PROJECT_ID: ${{ secrets.VERCEL_SANDBOX_PROJECT_ID }} + DRIVE_SWEEP_DELETE: ${{ inputs.drive_sweep_delete && '--delete' || '' }} + run: bun run drive-sweep --production $DRIVE_SWEEP_DELETE + production-release: name: Production release # A merge that touches the image or the bundle releases it, and the nightly diff --git a/apps/api/src/app/api/cloud-workspaces/provision/route.ts b/apps/api/src/app/api/cloud-workspaces/provision/route.ts index afefc8333f3..3175e8ae40e 100644 --- a/apps/api/src/app/api/cloud-workspaces/provision/route.ts +++ b/apps/api/src/app/api/cloud-workspaces/provision/route.ts @@ -18,6 +18,7 @@ const payloadSchema = z /** Absent when the user typed a name, which the row already holds. */ namingPrompt: z.string().max(CLOUD_AGENT_PROMPT_MAX_LENGTH).optional(), launch: cloudAgentLaunchSchema.optional(), + lockRetries: z.number().int().min(0).optional(), }) .strict(); diff --git a/apps/api/src/app/api/cloud-workspaces/teardown/route.ts b/apps/api/src/app/api/cloud-workspaces/teardown/route.ts new file mode 100644 index 00000000000..ead4e9cbc8f --- /dev/null +++ b/apps/api/src/app/api/cloud-workspaces/teardown/route.ts @@ -0,0 +1,34 @@ +import { teardownSandbox } from "@superset/trpc/cloud-workspace-teardown"; +import { z } from "zod"; +import { verifyQstashRequest } from "@/lib/verifyQstash"; + +export const dynamic = "force-dynamic"; +export const maxDuration = 60; + +const payloadSchema = z + .object({ + providerSandboxId: z.string().min(1), + driveName: z.string().min(1).nullable(), + attempt: z.number().int().min(0), + keepDrive: z.boolean().optional(), + }) + .strict(); + +/** Stops a box, then deletes it and its drive once the drive has detached; requeues itself until then. */ +export async function POST(request: Request): Promise { + const body = await request.text(); + const rejected = await verifyQstashRequest( + request, + body, + "/api/cloud-workspaces/teardown", + ); + if (rejected) return rejected; + + const parsed = payloadSchema.safeParse(JSON.parse(body)); + if (!parsed.success) { + console.error("[cloud-workspaces/teardown] invalid payload", parsed.error); + return Response.json({ error: "Invalid payload" }, { status: 400 }); + } + const outcome = await teardownSandbox(parsed.data); + return Response.json({ ok: true, outcome }); +} diff --git a/docs/cloud-sandbox-mismatches.md b/docs/cloud-sandbox-mismatches.md index 803c699ddca..9f379ac1119 100644 --- a/docs/cloud-sandbox-mismatches.md +++ b/docs/cloud-sandbox-mismatches.md @@ -699,6 +699,56 @@ socket, all describing processes that no longer exist. The boot runner deletes and recreates it first thing; anything that reads a flag there must tolerate a stale one for the milliseconds before that. +**A workspace drive is not a disk the box owns.** In drive mode +(`environments.golden_drive_name` set), `/workspace` and large home subtrees +live on a Vercel drive at `/mnt/superset-drive`, so they never enter the +box's snapshot. Three things a machine someone owns never makes you think +about: + +- *Bind mounts end with the session.* Vercel attaches the drive on every + session (the control plane passes `mounts` on every create and wake), but + the `mount --bind` of `/workspace` and each home subtree is gone after a + stop. `superset-boot` lays them out again on every boot, before + host-service and the checkout read `/workspace` (`drive.layout` in the + boot log). Anything that runs before that sees the root disk's empty + `/workspace`. +- *Hardlinks must stay inside `/workspace`.* Each bind mount is its own + `vfsmount`, and `link(2)` across two of them fails with `EXDEV` even on the + same filesystem. bun and pnpm then copy every package instead of linking + it. So their stores are inside the `/workspace` mount + (`BUN_INSTALL_CACHE_DIR=/workspace/.superset/stores/bun`, + `npm_config_store_dir=/workspace/.superset/stores/pnpm`, in host-service's + env and the login profile), not under `~/.bun` or `~/.cache`. +- *A drive needs detach before the next mount.* After a stop the drive stays + attached while the session flushes what it wrote (6 s after 1 GB, 44 s + after 12 GB), and a mount taken meanwhile reads a corrupt or empty + filesystem. `wakeSandbox` waits for `Drive.get(...).currentSandboxName` to + clear, at most 2 minutes, and fails the wake rather than mount. A sandbox + deleted while it still holds its drive leaves the drive attached for + minutes, so every delete stops the box first (`deleteSandboxAndDrive`). + +A drive has no discard either: deleting a file on it does not lower its +billed size. +A drive that did not mount leaves the root disk's empty `/workspace` in its +place, which would look like the user's work is gone. So the identity file +names the drive (`SUPERSET_SANDBOX_DRIVE`), the drive names its owner +(`/mnt/superset-drive/state/workspace-id`), and `superset-boot` refuses to +continue (`drive.missing`, `drive.foreign`, `drive.unlaid`) before the bundle, +host-service and the checkout. The control plane spots the refusal in this +boot's lines of the boot log within about 2 s and answers "still starting"; +the next wake mounts the drive afresh, and a create records the box asleep. +A refusal no remount can change (another workspace's drive, a laid-out drive +that mounts empty, a boot script that predates the drive) ends after 3 boots +in a row as `failed`, with the box and the drive kept. A boot that answers +without `drive.layout` is refused the same way. +A box on an image whose runner predates the guard gets it after the reexec +into the bundle's runner, still before host-service. + +*The root disk and the drive can disagree.* A stop whose snapshot does not +land rolls the root back and not the drive. So in drive mode the checkout +markers and `host.db` live on the drive, and the checkout never replaces a +non-empty checkout there. + **The control plane's command runs from the image's workdir, which the checkout replaces.** The first boot cloned `/workspace` by deleting and recreating it, and every child of the boot command — host-service, git, @@ -961,7 +1011,9 @@ OpenCode and Pi update in place, beside their binary. On a box they run from a read-only mount and are pinned by the bundle. `/etc/profile.d/superset.sh` turns off their update checks (`OPENCODE_DISABLE_AUTOUPDATE`, `PI_SKIP_VERSION_CHECK`), the same way `DISABLE_AUTOUPDATER` does for -Claude. They keep their state under `~/.local/share/opencode`, +Claude. `superset-boot` also puts all three in host-service's own env: a chat +agent gets host-service's env, not a terminal's, whenever host-service's +login-shell probe fails. They keep their state under `~/.local/share/opencode`, `~/.config/opencode` and `~/.pi/agent`, as they do on a laptop. **A read-only snapshot mount taken while a writer holds the drive is corrupt.** @@ -1076,3 +1128,12 @@ doesn't re-spend a session on it; the next thing to try is attaching to the `win.setSize()` directly rather than simulating it from outside, or watching `app.on("child-process-gone")` / `render-process-gone` across a CDP-reload loop to catch the actual GPU-process death this is presumably downstream of. + +**`sudo` cannot resolve the box's hostname.** A machine someone owns has its +hostname in `/etc/hosts`; a Vercel box does not, so every `sudo` (including +`runCommand({ sudo: true })`) writes `sudo: unable to resolve host : +Name or service not known` to stderr. The command still runs. A check that +reads `stdout + stderr` then sees an extra line and fails a comparison that +was correct; the first real drive check failed 7 expectations that way. +**What we did:** the real-sandbox drive pass compares stdout only and shows +stderr as detail. **Open:** the hostname is still missing from `/etc/hosts`. diff --git a/packages/i18n/locales/cs/messages.po b/packages/i18n/locales/cs/messages.po index 09933a46950..691c191eeee 100644 --- a/packages/i18n/locales/cs/messages.po +++ b/packages/i18n/locales/cs/messages.po @@ -20867,6 +20867,9 @@ msgstr "Tímto zrušíte přípravu a vrátíte všechny připravené změny. P msgid "This workspace" msgstr "Tento pracovní prostor" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Tento pracovní prostor se nepodařilo uložit jako prostředí. Zkuste to později." + msgid "This workspace has no project to branch from" msgstr "Tento pracovní prostor nemá projekt, ze kterého by šlo větvit" diff --git a/packages/i18n/locales/de/messages.po b/packages/i18n/locales/de/messages.po index 6a48fd54ac7..aa89ad95316 100644 --- a/packages/i18n/locales/de/messages.po +++ b/packages/i18n/locales/de/messages.po @@ -20867,6 +20867,9 @@ msgstr "Damit werden alle gestageden Änderungen aus dem Index genommen und zur msgid "This workspace" msgstr "Dieser Arbeitsbereich" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Dieser Arbeitsbereich konnte nicht als Umgebung gespeichert werden. Versuche es später erneut." + msgid "This workspace has no project to branch from" msgstr "Dieser Workspace hat kein Projekt, von dem aus verzweigt werden kann" diff --git a/packages/i18n/locales/en/messages.po b/packages/i18n/locales/en/messages.po index 22835bd432c..b1b97096467 100644 --- a/packages/i18n/locales/en/messages.po +++ b/packages/i18n/locales/en/messages.po @@ -20867,6 +20867,9 @@ msgstr "This will unstage and revert all staged changes. Staged new files will b msgid "This workspace" msgstr "This workspace" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "This workspace could not be saved as an environment. Try again later." + msgid "This workspace has no project to branch from" msgstr "This workspace has no project to branch from" diff --git a/packages/i18n/locales/es/messages.po b/packages/i18n/locales/es/messages.po index 4bb6cf105a8..fdd4154ca59 100644 --- a/packages/i18n/locales/es/messages.po +++ b/packages/i18n/locales/es/messages.po @@ -20867,6 +20867,9 @@ msgstr "Esto quitará de preparados y revertirá todos los cambios preparados. L msgid "This workspace" msgstr "Este espacio de trabajo" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "No se pudo guardar este espacio de trabajo como entorno. Inténtalo de nuevo más tarde." + msgid "This workspace has no project to branch from" msgstr "Este espacio de trabajo no tiene un proyecto desde el que bifurcar" diff --git a/packages/i18n/locales/fr/messages.po b/packages/i18n/locales/fr/messages.po index 1f42b4bba60..e0f28091a9d 100644 --- a/packages/i18n/locales/fr/messages.po +++ b/packages/i18n/locales/fr/messages.po @@ -20867,6 +20867,9 @@ msgstr "Cette action désindexera et annulera toutes les modifications indexées msgid "This workspace" msgstr "Cet espace de travail" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Impossible d’enregistrer cet espace de travail comme environnement. Réessayez plus tard." + msgid "This workspace has no project to branch from" msgstr "Cet espace de travail n'a aucun projet depuis lequel dupliquer" diff --git a/packages/i18n/locales/id/messages.po b/packages/i18n/locales/id/messages.po index 82c2d66b65d..1f20fd63ef3 100644 --- a/packages/i18n/locales/id/messages.po +++ b/packages/i18n/locales/id/messages.po @@ -20867,6 +20867,9 @@ msgstr "Ini akan meng-unstage dan mengembalikan semua perubahan yang di-stage. F msgid "This workspace" msgstr "Workspace ini" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Workspace ini tidak dapat disimpan sebagai lingkungan. Coba lagi nanti." + msgid "This workspace has no project to branch from" msgstr "Ruang kerja ini tidak memiliki proyek untuk dicabangkan" diff --git a/packages/i18n/locales/it/messages.po b/packages/i18n/locales/it/messages.po index 629470d66f5..35cc9c03307 100644 --- a/packages/i18n/locales/it/messages.po +++ b/packages/i18n/locales/it/messages.po @@ -20867,6 +20867,9 @@ msgstr "Questo toglierà dallo stage e annullerà tutte le modifiche in stage. I msgid "This workspace" msgstr "Questo workspace" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Impossibile salvare questo workspace come ambiente. Riprova più tardi." + msgid "This workspace has no project to branch from" msgstr "Questo workspace non ha un progetto da cui diramare" diff --git a/packages/i18n/locales/ja/messages.po b/packages/i18n/locales/ja/messages.po index 4b765ee5866..39438ee3a43 100644 --- a/packages/i18n/locales/ja/messages.po +++ b/packages/i18n/locales/ja/messages.po @@ -20867,6 +20867,9 @@ msgstr "ステージ済みの変更をすべてステージ解除して元に戻 msgid "This workspace" msgstr "このワークスペース" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "このワークスペースを環境として保存できませんでした。しばらくしてからもう一度お試しください。" + msgid "This workspace has no project to branch from" msgstr "このワークスペースには分岐元のプロジェクトがありません" diff --git a/packages/i18n/locales/ko/messages.po b/packages/i18n/locales/ko/messages.po index 062b60ed8a5..65699d91b40 100644 --- a/packages/i18n/locales/ko/messages.po +++ b/packages/i18n/locales/ko/messages.po @@ -20867,6 +20867,9 @@ msgstr "스테이징된 변경 사항을 모두 스테이징 해제하고 되돌 msgid "This workspace" msgstr "이 워크스페이스" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "이 워크스페이스를 환경으로 저장할 수 없습니다. 나중에 다시 시도하세요." + msgid "This workspace has no project to branch from" msgstr "이 워크스페이스에는 분기할 프로젝트가 없습니다" diff --git a/packages/i18n/locales/nl/messages.po b/packages/i18n/locales/nl/messages.po index e551554bd86..6abc8cec65e 100644 --- a/packages/i18n/locales/nl/messages.po +++ b/packages/i18n/locales/nl/messages.po @@ -20867,6 +20867,9 @@ msgstr "Hiermee worden alle gestagede wijzigingen ge-unstaged en teruggedraaid. msgid "This workspace" msgstr "Deze werkruimte" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Deze werkruimte kon niet als omgeving worden opgeslagen. Probeer het later opnieuw." + msgid "This workspace has no project to branch from" msgstr "Deze workspace heeft geen project om vanaf af te splitsen" diff --git a/packages/i18n/locales/pl/messages.po b/packages/i18n/locales/pl/messages.po index 411313bc257..8ee8b616e15 100644 --- a/packages/i18n/locales/pl/messages.po +++ b/packages/i18n/locales/pl/messages.po @@ -20867,6 +20867,9 @@ msgstr "Usunie to z indeksu i cofnie wszystkie zaindeksowane zmiany. Zaindeksowa msgid "This workspace" msgstr "Ten obszar roboczy" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Nie udało się zapisać tego obszaru roboczego jako środowiska. Spróbuj ponownie później." + msgid "This workspace has no project to branch from" msgstr "Ten obszar roboczy nie ma projektu, z którego można się rozgałęzić" diff --git a/packages/i18n/locales/pt-BR/messages.po b/packages/i18n/locales/pt-BR/messages.po index 80174655017..95b674bc955 100644 --- a/packages/i18n/locales/pt-BR/messages.po +++ b/packages/i18n/locales/pt-BR/messages.po @@ -20867,6 +20867,9 @@ msgstr "Isso remove do stage e reverte todas as alterações no stage. Arquivos msgid "This workspace" msgstr "Esta área de trabalho" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Não foi possível salvar esta área de trabalho como ambiente. Tente novamente mais tarde." + msgid "This workspace has no project to branch from" msgstr "Este workspace não tem projeto do qual ramificar" diff --git a/packages/i18n/locales/ru/messages.po b/packages/i18n/locales/ru/messages.po index 814cd29b4b7..ca64a5a5164 100644 --- a/packages/i18n/locales/ru/messages.po +++ b/packages/i18n/locales/ru/messages.po @@ -20867,6 +20867,9 @@ msgstr "Это снимет индексацию и откатит все про msgid "This workspace" msgstr "Эта рабочая область" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Не удалось сохранить это рабочее пространство как окружение. Повторите попытку позже." + msgid "This workspace has no project to branch from" msgstr "В этом рабочем пространстве нет проекта для ответвления" diff --git a/packages/i18n/locales/tr/messages.po b/packages/i18n/locales/tr/messages.po index b8e7b7af79a..1955431dcd9 100644 --- a/packages/i18n/locales/tr/messages.po +++ b/packages/i18n/locales/tr/messages.po @@ -20867,6 +20867,9 @@ msgstr "Bu işlem, hazırlanmış tüm değişiklikleri hazırlıktan çıkarır msgid "This workspace" msgstr "Bu çalışma alanı" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Bu çalışma alanı ortam olarak kaydedilemedi. Daha sonra tekrar deneyin." + msgid "This workspace has no project to branch from" msgstr "Bu çalışma alanının çatallanacak bir projesi yok" diff --git a/packages/i18n/locales/uk/messages.po b/packages/i18n/locales/uk/messages.po index e0cd327ba64..052346a5af6 100644 --- a/packages/i18n/locales/uk/messages.po +++ b/packages/i18n/locales/uk/messages.po @@ -20861,6 +20861,9 @@ msgstr "Це зніме з індексу та відкотить усі про msgid "This workspace" msgstr "Цей робочий простір" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Не вдалося зберегти цей робочий простір як середовище. Спробуйте пізніше." + msgid "This workspace has no project to branch from" msgstr "У цього робочого простору немає проєкту, від якого можна створити гілку" diff --git a/packages/i18n/locales/vi/messages.po b/packages/i18n/locales/vi/messages.po index cd39fe440dd..005cf9911e5 100644 --- a/packages/i18n/locales/vi/messages.po +++ b/packages/i18n/locales/vi/messages.po @@ -20867,6 +20867,9 @@ msgstr "Thao tác này sẽ bỏ stage và hoàn nguyên toàn bộ thay đổi msgid "This workspace" msgstr "Workspace này" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "Không thể lưu không gian làm việc này thành môi trường. Hãy thử lại sau." + msgid "This workspace has no project to branch from" msgstr "Workspace này không có dự án để tạo nhánh" diff --git a/packages/i18n/locales/zh-CN/messages.po b/packages/i18n/locales/zh-CN/messages.po index ff17e6a7592..37a38483732 100644 --- a/packages/i18n/locales/zh-CN/messages.po +++ b/packages/i18n/locales/zh-CN/messages.po @@ -20867,6 +20867,9 @@ msgstr "这将取消暂存并还原所有已暂存的更改。已暂存的新文 msgid "This workspace" msgstr "此工作区" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "无法将此工作区保存为环境。请稍后重试。" + msgid "This workspace has no project to branch from" msgstr "此工作区没有可供分叉的项目" diff --git a/packages/i18n/locales/zh-TW/messages.po b/packages/i18n/locales/zh-TW/messages.po index b750ae9f547..871c0b51a37 100644 --- a/packages/i18n/locales/zh-TW/messages.po +++ b/packages/i18n/locales/zh-TW/messages.po @@ -20867,6 +20867,9 @@ msgstr "這將取消暫存並還原所有已暫存的更改。已暫存的新檔 msgid "This workspace" msgstr "此工作區" +msgid "This workspace could not be saved as an environment. Try again later." +msgstr "無法將此工作區儲存為環境。請稍後再試。" + msgid "This workspace has no project to branch from" msgstr "此工作區沒有可供分叉的專案" diff --git a/packages/i18n/src/server-errors.ts b/packages/i18n/src/server-errors.ts index 2ef4684e458..55050df12b7 100644 --- a/packages/i18n/src/server-errors.ts +++ b/packages/i18n/src/server-errors.ts @@ -508,6 +508,13 @@ export const serverErrorMessages: Record< "This environment's repositories are fixed; promote a workspace again to change them", }), ), + "serverError.environment.couldNotPromote": () => + i18n._( + msg({ + message: + "This workspace could not be saved as an environment. Try again later.", + }), + ), "serverError.environment.couldNotRecord": () => i18n._( msg({ diff --git a/packages/sandbox/README.md b/packages/sandbox/README.md index 7785deac5fb..1d64303fa9a 100644 --- a/packages/sandbox/README.md +++ b/packages/sandbox/README.md @@ -53,7 +53,7 @@ The rule of thumb: if you can answer "where is it on the box", put it there. | `bun run build` | Hashes `bundle/`, renders `contract.sh` from `@superset/shared/sandbox/contract`, derives the step versions, writes `dist/bundle//` and the tarball. Deterministic: the same tree is the same sha on any machine. | | `bun run build --publish` | Also uploads the bundle and any asset the bucket lacks, from the local cache or else its row's `source`, checked against the row's sha256. Refuses to publish a bundle that references an asset the bucket does not have. | | `bun run image` | Builds the image: OS, apt lists, vendor tools, the `ubuntu` user, then installs the bundle at its sha and runs every pass once, so a fresh box finds every hash sidecar and step marker matching. A push logs Docker into the registry itself with a credential minted from `VERCEL_SANDBOX_TOKEN`, so there is no login to renew. `--local` builds `superset-sandbox:local` for the boot-twice test; `--dry` prints the Dockerfile. | -| boot (`superset-boot`) | On the box, as root, run by the control plane on every create and wake, after it has written `sandbox.conf`, with the host secret in its env: clears `/run/superset`, installs the pinned bundle if the hash differs, runs `apply-rootfs` and the steps marked `beforeHost` in `steps.json` (each a hash compare), starts host-service as `ubuntu` with the secret from the boot env, writes `resources.env`, starts `superset-memguard` (ranks every process for the OOM killer and reaps the largest before the kernel has to). When `sandbox.conf` has `SUPERSET_SANDBOX_BOX_LOGS=1`, it also starts `superset-logship`, which sends the boot, memguard, desktop and host-service logs to Sentry Logs from a byte offset saved on disk. Then, detached: `sync-assets` and the other steps, then the desktop, with the checkout beside them. `checkout.ready` waits for both, and the stamp `steps.end` says the steps are done. Once host-service answers, the environment has been pushed and the checkout is ready, the runner asks host-service to run the repo's `start` hook with that environment. | +| boot (`superset-boot`) | On the box, as root, run by the control plane on every create and wake, after it has written `sandbox.conf`, with the host secret in its env: clears `/run/superset`, lays out the workspace drive when one is mounted at `/mnt/superset-drive` (`plans/20261116-workspace-drive.md`), installs the pinned bundle if the hash differs, runs `apply-rootfs` and the steps marked `beforeHost` in `steps.json` (each a hash compare), starts host-service as `ubuntu` with the secret from the boot env, writes `resources.env`, starts `superset-memguard` (ranks every process for the OOM killer and reaps the largest before the kernel has to). When `sandbox.conf` has `SUPERSET_SANDBOX_BOX_LOGS=1`, it also starts `superset-logship`, which sends the boot, memguard, desktop and host-service logs to Sentry Logs from a byte offset saved on disk. Then, detached: `sync-assets` and the other steps, then the desktop, with the checkout beside them. `checkout.ready` waits for both, and the stamp `steps.end` says the steps are done. Once host-service answers, the environment has been pushed and the checkout is ready, the runner asks host-service to run the repo's `start` hook with that environment. | | `bun run release` | Rewrites the host-service and CLI assets from this checkout and the claude, codex, Chrome, opencode and pi rows to their latest upstream release, publishes the bundle, pushes the image under a candidate tag `release-` (unless `--skip-image`), builds the internal golden from it with the environment's `setup` hook, probes a fork of it (`src/environments/probe.ts`) including a stop and wake, and only after the probe passes moves `latest` to the candidate and writes the environment rows. Every image environment resolves `superset-hostsvc` to `latest`, so a failed release changes no box. | | `bun run agent-tools --bundle ` | Writes that bundle's agent CLIs to `agent-tools--`, a drive boxes mount read-only at `/opt/agents` when their organization has the `sandbox-agent-tools-drive` flag. A drive is written once and never again; it starts as a fork of the region's newest earlier drive, so only changed agents are fetched, and an agent that fails its download or `--version` check keeps the earlier drive's version or is left out. The drive carries claude, codex and pdev (also in the image) and opencode and pi (only on the drive). `bun run release` does this for every region, six at a time, before it records the bundle. It fails when a region with an environment has no drive, or a drive without claude, codex or pdev. Afterwards a `--production` release deletes the drives that neither an environment row (archived ones too) nor one of the 3 newest released bundles names; one Vercel project holds every database's drives, so no other release deletes any. | | `bun run src/real-sandbox.ts` | The Sandbox workflow dispatched with `real-sandbox`: one real sandbox from the registry image on this checkout's bundle and its agents drive in the default region, the same probe, a stop and a wake. | diff --git a/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh b/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh index 6eae7dfec23..32c4b127ad6 100644 --- a/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh +++ b/packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh @@ -9,6 +9,8 @@ export DISPLAY="${DISPLAY:-${SUPERSET_DISPLAY:-:1}}" [ -r "${SUPERSET_RUN_DIR:-/run/superset}/desktop.env" ] && . "${SUPERSET_RUN_DIR:-/run/superset}/desktop.env" # The box's memory ceiling and CPUs, written by superset-boot from the cgroup. [ -r "${SUPERSET_RUN_DIR:-/run/superset}/resources.env" ] && . "${SUPERSET_RUN_DIR:-/run/superset}/resources.env" +# The package stores on the workspace drive, written by superset-boot in drive mode. +[ -r "${SUPERSET_RUN_DIR:-/run/superset}/drive.env" ] && . "${SUPERSET_RUN_DIR:-/run/superset}/drive.env" case ":$PATH:" in *":/usr/local/go/bin:"*) ;; *) export PATH="$PATH:/usr/local/go/bin" ;; esac # The agent CLIs are pinned by the bundle (install-agent-clis) or run from the # read-only agents drive; a self-update would drift from the pin and only diff --git a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot index 3077db2b102..c13fc864f36 100755 --- a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot +++ b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot @@ -6,6 +6,8 @@ # # clear /run/superset facts about a boot that is over # read the contract and identity /etc/superset/{contract.sh,sandbox.conf} +# lay out the workspace drive only when the control plane mounted one; +# refuse when sandbox.conf names a drive that is not mounted # install the pinned bundle only when its hash differs from current # apply-rootfs, and the steps each a hash compare per file and step; # marked beforeHost the only steps host-service waits for @@ -18,6 +20,9 @@ # the log on its health route so a boot is measurable from outside. set -uo pipefail . /etc/superset/contract.sh +# superset-boot drive-contract: 1 +# A boot that has laid a drive out re-executes only into a script with the +# line above; one without it predates the drive and would clone over it. # The control plane runs this from the image's workdir, which the checkout # below may replace; every child would inherit a deleted cwd. @@ -51,7 +56,7 @@ if [ -f "${SUPERSET_RUN_DIR}/host-service.pid" ] && kill -0 "$(cat "${SUPERSET_R exit 0 fi -stamp "boot.start pid=$$ bundle=$(cat "${SUPERSET_BUNDLE_ROOT}/current.bundle-hash" 2>/dev/null | cut -c1-12)" +stamp "boot.start pid=$$ bundle=$(cat "${SUPERSET_BUNDLE_ROOT}/current.bundle-hash" 2>/dev/null | cut -c1-12) boot=${SUPERSET_BOOT_ID:-}" # --- per-boot state: a restored disk carries the previous session's pids # and ready flags, which describe processes that no longer exist. @@ -59,6 +64,82 @@ rm -rf "$SUPERSET_RUN_DIR" install -d -m 0755 -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$SUPERSET_RUN_DIR" stamp "run.cleared" +# --- the workspace drive, when the environment is in drive mode: /workspace +# and the large home state live on it, so they never enter the box's +# snapshot. Bind mounts end with the session, so every boot does this, before +# anything reads /workspace. A bind mount, not a symlink: paths must not move. +DRIVE_MODE="" +user_path() { + local dir="$1" part IFS=/ + for part in $2; do + dir="${dir}/${part}" + [ -d "$dir" ] || install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$dir" + done +} +bind_into() { + mountpoint -q "$2" || mount --bind "$1" "$2" || refuse "cannot bind ${1} onto ${2}" +} +# In drive mode, what describes the drive's contents lives on the drive. +use_drive_state() { + SUPERSET_HOST_DB="${SUPERSET_DRIVE_STATE_DIR}/host.db" + SUPERSET_CHECKOUTS_DIR="${SUPERSET_DRIVE_STATE_DIR}/checkouts" +} +# A subtree the drive lacks is seeded from what the image left there, under a +# temporary name, so a boot that dies halfway never leaves a partial copy. +seed_home_subtree() { + local sub="$1" target="${drive}/home/$1" tmp="${drive}/home/$1.seeding" + [ "$sub" = "${sub%/*}" ] || user_path "${drive}/home" "${sub%/*}" + rm -rf "$tmp" + install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$tmp" + if [ -d "${SUPERSET_USER_HOME}/${sub}" ]; then + cp -a "${SUPERSET_USER_HOME}/${sub}/." "${tmp}/" || refuse "cannot seed ${target}" + fi + mv -T "$tmp" "$target" || refuse "cannot seed ${target}" +} +layout_drive() { + local drive="${SUPERSET_DRIVE_DIR:-}" started sub owner sentinel + if [ -z "$drive" ] || ! mountpoint -q "$drive"; then + # Without its drive the box would serve the root disk's empty /workspace as the user's work. + [ -z "${SUPERSET_SANDBOX_DRIVE:-}" ] && return 0 + stamp "drive.missing ${SUPERSET_SANDBOX_DRIVE} is not mounted at ${drive:-the drive path}" + refuse "the workspace drive ${SUPERSET_SANDBOX_DRIVE} is not mounted" + fi + : "${SUPERSET_DRIVE_STATE_DIR:=${drive}/state}" + sentinel="${SUPERSET_DRIVE_STATE_DIR}/workspace-id" + owner="$(cat "$sentinel" 2>/dev/null || true)" + if [ -n "$owner" ] && [ "$owner" != "$SUPERSET_SANDBOX_WORKSPACE_ID" ]; then + stamp "drive.foreign ${SUPERSET_SANDBOX_DRIVE:-} belongs to workspace ${owner}" + refuse "the drive at ${drive} belongs to workspace ${owner}" + fi + if [ -z "$owner" ] && [ -n "${SUPERSET_SANDBOX_DRIVE_LAID_OUT:-}" ]; then + stamp "drive.unlaid ${SUPERSET_SANDBOX_DRIVE:-} carries no workspace id though it was laid out before" + refuse "the drive at ${drive} is not this workspace's laid-out drive" + fi + started="$(date +%s%3N)" + user_path "$drive" workspace + user_path "$drive" home + user_path "$drive" state + install -d -m 0710 "${drive}/docker" + install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$SUPERSET_WORKSPACE_DIR" + bind_into "${drive}/workspace" "$SUPERSET_WORKSPACE_DIR" + for sub in ${SUPERSET_DRIVE_HOME_SUBTREES:-}; do + [ -d "${drive}/home/${sub}" ] || seed_home_subtree "$sub" + user_path "$SUPERSET_USER_HOME" "$sub" + bind_into "${drive}/home/${sub}" "${SUPERSET_USER_HOME}/${sub}" + done + if [ -z "$owner" ]; then + # A fresh fork: its checkout is no one's work until the first checkout has run, in this boot or a later one. + : > "${SUPERSET_DRIVE_STATE_DIR}/fresh" && + printf '%s\n' "$SUPERSET_SANDBOX_WORKSPACE_ID" > "${sentinel}.tmp" && mv -T "${sentinel}.tmp" "$sentinel" && + sync "${SUPERSET_DRIVE_STATE_DIR}/fresh" "$sentinel" "$SUPERSET_DRIVE_STATE_DIR" || refuse "cannot write ${sentinel}" + fi + DRIVE_MODE=1 + use_drive_state + printf 'export BUN_INSTALL_CACHE_DIR=%s npm_config_store_dir=%s\n' "$SUPERSET_BUN_STORE" "$SUPERSET_PNPM_STORE" > "${SUPERSET_RUN_DIR}/drive.env" + stamp "drive.layout ms=$(( $(date +%s%3N) - started ))" +} +layout_drive + # --- the bundle the environment pins. A box with no pin would silently keep # whatever its image or golden baked, for as long as it lives. [ -n "${SUPERSET_BUNDLE_SHA:-}" ] || refuse "SUPERSET_BUNDLE_SHA missing from ${SUPERSET_CONF}; the control plane pins every box" @@ -107,17 +188,35 @@ BUNDLE="${SUPERSET_BUNDLE_ROOT}/current" stamp "rootfs.start"; "$BUNDLE/setup" apply-rootfs >> "$SUPERSET_BOOT_LOG" 2>&1; stamp "rootfs.end" # apply-rootfs may have replaced the contract; re-read it. . /etc/superset/contract.sh +[ -z "$DRIVE_MODE" ] || use_drive_state # It may also have replaced this script. The shell is running the copy the # image baked, so without this a bundle's change to the boot script only # takes effect on the boot after the one that installed it — which is how an # older image's checkout raced a newer bundle's. Once per boot: the guard # stops a bundle whose script never settles from spinning here. +if [ -n "$DRIVE_MODE" ] && ! grep -q '^# superset-boot drive-contract: [1-9]' "$0"; then + stamp "drive.downgrade the installed boot script predates the workspace drive" + refuse "the pinned bundle's boot script does not know the workspace drive" +fi if [ -z "${SUPERSET_BOOT_REEXEC:-}" ] && [ "$(self_hash)" != "$BOOT_HASH_BEFORE" ]; then stamp "boot.reexec into the installed bundle's script" SUPERSET_BOOT_REEXEC=1 exec "$0" "$@" fi stamp "steps.host.start"; "$BUNDLE/setup" run-steps before-host >> "$SUPERSET_BOOT_LOG" 2>&1; stamp "steps.host.end" +# In drive mode, Docker's images and layers go on the drive too. apply-rootfs +# compares only the hash sidecar, so this rewrite stays until the bundle +# ships a new daemon.json, and then it is merged again right here. +configure_docker() { + [ -n "$DRIVE_MODE" ] || return 0 + local want + want="$(jq -c --arg root "${SUPERSET_DRIVE_DIR}/docker" '. + {"data-root": $root}' "${BUNDLE}/rootfs/etc/docker/daemon.json")" || + { stamp "docker.config.failed"; return 0; } + [ "$(jq -c . /etc/docker/daemon.json 2>/dev/null)" = "$want" ] || printf '%s\n' "$want" > /etc/docker/daemon.json + stamp "docker.data-root ${SUPERSET_DRIVE_DIR}/docker" +} +configure_docker + # --- the agent CLIs from the region's agent-tools drive, when the control # plane mounted one. Without it the image's agents stay on PATH. Mounts are # per session, so a link into a mount this session lacks points nowhere. @@ -194,7 +293,10 @@ host_env=( "SUPERSET_SANDBOX_IMAGE_TAG=${SUPERSET_SANDBOX_IMAGE_TAG}" "SUPERSET_SANDBOX_PROVIDER=${SUPERSET_SANDBOX_PROVIDER}" "SUPERSET_RUN_DIR=${SUPERSET_RUN_DIR}" "SUPERSET_SANDBOX_BOOT_LOG=${SUPERSET_BOOT_LOG}" "HOST_SERVICE_SECRET=${HOST_SERVICE_SECRET}" + # Also in the login profile; here too, so an agent host-service starts gets them when its shell probe fails. + "DISABLE_AUTOUPDATER=1" "OPENCODE_DISABLE_AUTOUPDATE=1" "PI_SKIP_VERSION_CHECK=1" ) +[ -n "$DRIVE_MODE" ] && host_env+=("BUN_INSTALL_CACHE_DIR=${SUPERSET_BUN_STORE}" "npm_config_store_dir=${SUPERSET_PNPM_STORE}") for key in HOST_SERVICE_SENTRY_DSN HOST_SERVICE_SENTRY_ENVIRONMENT SUPERSET_SANDBOX_CREATOR_USER_ID \ SUPERSET_SANDBOX_PLUGINS \ SUPERSET_SANDBOX_AGENT SUPERSET_SANDBOX_AGENT_PROMPT SUPERSET_SANDBOX_AGENT_MODEL SUPERSET_SANDBOX_AGENT_EFFORT SUPERSET_SANDBOX_AGENT_MODE SUPERSET_SANDBOX_AGENT_ATTACHMENTS SUPERSET_SANDBOX_AGENT_SURFACE SUPERSET_SANDBOX_AGENT_AUTOMATION_RUN_ID SUPERSET_SANDBOX_AGENT_TERMINAL_ID \ @@ -281,22 +383,64 @@ start_hook() { # path is ".". A marker per path under the state directory says it is done; # a golden has the markers stripped so a fork fetches its branch into the # checkout it inherited. +# A drive laid out in an earlier boot holds the user's work: nothing on it is replaced. +drive_has_work() { [ -n "$DRIVE_MODE" ] && [ ! -e "${SUPERSET_DRIVE_STATE_DIR}/fresh" ]; } +# Still exactly the golden's copy: HEAD at the commit the fresh fork had, nothing changed, and the +# branch absent or at that commit. Then no user work can be in it, and it may be fetched. +untouched_golden() { + local ws="$1" recorded="$2" branch="$3" head + [ -f "$recorded" ] || return 1 + head="$(cat "$recorded")" + [ "$(as_user git -C "$ws" rev-parse HEAD 2>/dev/null)" = "$head" ] || return 1 + [ -z "$(as_user git -C "$ws" status --porcelain 2>/dev/null)" ] || return 1 + case "$(as_user git -C "$ws" rev-parse -q --verify "refs/heads/${branch}" 2>/dev/null || echo none)" in + "$head"|none) return 0 ;; + *) return 1 ;; + esac +} checkout_one() { local url="$1" branch="$2" path="$3" base="$4" [ -n "$base" ] || base="$branch" local ws="${SUPERSET_WORKSPACE_DIR}/${path}" marker="${SUPERSET_CHECKOUTS_DIR}/${path}" if [ "$path" = "." ]; then ws="$SUPERSET_WORKSPACE_DIR"; marker="${SUPERSET_CHECKOUTS_DIR}/@root"; fi if [ -f "$marker" ]; then stamp "checkout.skipped ${path} already done"; return 0; fi - local baked + local baked recorded untouched="" + recorded="${SUPERSET_DRIVE_STATE_DIR:-}/golden-head/${path//\//%}" + [ "$path" = "." ] && recorded="${SUPERSET_DRIVE_STATE_DIR:-}/golden-head/@root" baked="$(as_user git -C "$ws" remote get-url origin 2>/dev/null || true)" if [ "$baked" = "$url" ] && [ -d "$ws/.git" ]; then - if as_user git -C "$ws" fetch -q --depth 1 origin "$base" && as_user git -C "$ws" checkout -q -B "$branch" FETCH_HEAD; then - touch "$marker"; stamp "checkout.fetched ${path} ${branch} from ${base}"; return 0 + if [ -n "$DRIVE_MODE" ] && ! drive_has_work && [ ! -f "$recorded" ]; then + install -d "${recorded%/*}" + as_user git -C "$ws" rev-parse HEAD > "$recorded" 2>/dev/null && sync "$recorded" || rm -f "$recorded" + fi + drive_has_work && untouched_golden "$ws" "$recorded" "$branch" && untouched=1 + # On a drive laid out before, a checkout that already has the branch is the user's work. + if drive_has_work && [ -z "$untouched" ] && as_user git -C "$ws" rev-parse -q --verify "refs/heads/${branch}" > /dev/null; then + touch "$marker"; stamp "checkout.kept ${path} ${branch} is already on the drive"; return 0 + fi + # A fresh fork tries a few times, then keeps the golden's copy: never a reclone over its dependencies. + local attempts=1 attempt + [ -n "$DRIVE_MODE" ] && ! drive_has_work && attempts=3 + for attempt in $(seq 1 "$attempts"); do + if as_user git -C "$ws" fetch -q --depth 1 origin "$base" && as_user git -C "$ws" checkout -q -B "$branch" FETCH_HEAD; then + touch "$marker"; rm -f "$recorded" + stamp "checkout.fetched ${path} ${branch} from ${base}${untouched:+ (an untouched golden copy)}"; return 0 + fi + [ "$attempt" -lt "$attempts" ] && sleep $(( attempt * 2 )) + done + if [ "$attempts" -gt 1 ]; then + stamp "checkout.fresh.failed ${path} ${branch}: the golden's checkout is kept; a later boot fetches it while it is untouched"; return 0 + fi + if drive_has_work; then + stamp "checkout.kept ${path} fetch failed; the checkout on the drive is left as it is"; return 1 fi # A clone a previous session interrupted leaves a repository that can # fetch nothing; start over rather than fail every boot. stamp "checkout.fetch.failed ${path} ${branch}; recloning" fi + if drive_has_work && [ -n "$(find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' ! -name '.superset' -print -quit 2>/dev/null)" ]; then + stamp "checkout.kept ${path} ${ws} is not empty; nothing on the drive is replaced"; return 1 + fi # Clone beside the target and move the result in, the worktree first and # .git last, so the directory becomes a repository in one rename: host-service # seeds the workspace row before the checkout lands and would otherwise find @@ -309,10 +453,19 @@ checkout_one() { || { find "$tmp" -mindepth 1 -delete; as_user git clone -q --depth 1 "$url" "$tmp"; }; then as_user git -C "$tmp" checkout -q -B "$branch" install -d -o "$SUPERSET_USER" -g "$SUPERSET_USER" "$ws" - find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' -exec rm -rf {} + - ( shopt -s dotglob nullglob; for entry in "$tmp"/*; do [ "${entry##*/}" = ".git" ] || mv "$entry" "$ws"/; done ) + if [ -n "$DRIVE_MODE" ]; then + # A fresh fork's golden checkout goes; the package stores stay, and a repository's own .superset/ merges into them. + find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' ! -name '.superset' -exec rm -rf {} + + ( shopt -s dotglob nullglob; for entry in "$tmp"/*; do + [ "${entry##*/}" = ".git" ] && continue + if [ -d "${ws}/${entry##*/}" ]; then cp -a "$entry/." "${ws}/${entry##*/}/" && rm -rf "$entry"; else mv "$entry" "$ws"/; fi + done ) + else + find "$ws" -mindepth 1 -maxdepth 1 ! -name '.clone.*' -exec rm -rf {} + + ( shopt -s dotglob nullglob; for entry in "$tmp"/*; do [ "${entry##*/}" = ".git" ] || mv "$entry" "$ws"/; done ) + fi mv "$tmp/.git" "$ws/.git"; rmdir "$tmp" - touch "$marker"; stamp "checkout.cloned ${path} ${url}"; return 0 + touch "$marker"; rm -f "$recorded"; stamp "checkout.cloned ${path} ${url}"; return 0 fi rm -rf "$tmp"; stamp "checkout.clone.failed ${path}"; return 1 } @@ -325,8 +478,22 @@ checkout() { [ -n "$path" ] || continue checkout_one "$url" "$branch" "$path" "$base" || failed=1 done < <(printf '%s' "$SUPERSET_SANDBOX_REPOSITORIES" | jq -r '.[] | [.url, .branch, .path, (.baseBranch // "")] | @tsv') + # The box was usable once this checkout ran, failed or not: from now on what is on the drive may be the user's. + if [ -n "$DRIVE_MODE" ] && [ -e "${SUPERSET_DRIVE_STATE_DIR}/fresh" ]; then + rm -f "${SUPERSET_DRIVE_STATE_DIR}/fresh" && sync "$SUPERSET_DRIVE_STATE_DIR" && stamp "checkout.fresh.done" + fi + exclude_stores return "$failed" } +# The package stores sit inside a root checkout in drive mode; git must not +# see them. Only stores/: a repository may track a .superset/ of its own. +exclude_stores() { + local exclude="${SUPERSET_WORKSPACE_DIR}/.git/info/exclude" + [ -n "$DRIVE_MODE" ] && [ -d "${SUPERSET_WORKSPACE_DIR}/.git" ] || return 0 + grep -qxF '/.superset/stores/' "$exclude" 2>/dev/null && return 0 + as_user mkdir -p "${exclude%/*}" + printf '/.superset/stores/\n' | as_user tee -a "$exclude" > /dev/null +} # --- the rest of the passes, then the desktop, beside the checkout. # checkout.ready waits for the steps too: the first agent launch, the start diff --git a/packages/sandbox/package.json b/packages/sandbox/package.json index e00d676269f..7fea8d12a3b 100644 --- a/packages/sandbox/package.json +++ b/packages/sandbox/package.json @@ -9,6 +9,7 @@ "pin": "bun run src/pin.ts", "assets": "bun run src/assets/produce.ts", "inventory": "bun run src/inventory.ts", + "drive-sweep": "bun run src/drive-sweep.ts", "release": "bun run src/environments/release.ts", "agent-tools": "bun run src/agent-tools.ts", "record-bundle": "bun run src/environments/record-bundle.ts", diff --git a/packages/sandbox/src/boot-twice.ts b/packages/sandbox/src/boot-twice.ts index 64b37c49a05..deeca68df57 100644 --- a/packages/sandbox/src/boot-twice.ts +++ b/packages/sandbox/src/boot-twice.ts @@ -9,6 +9,9 @@ * * bun run src/boot-twice.ts expects superset-sandbox:local * SANDBOX_IMAGE=... bun run src/boot-twice.ts + * BOOT_TWICE_CA_FILE=/etc/ssl/certs/ca-certificates.crt bun run src/boot-twice.ts + * in a cloud workspace, whose egress firewall re-signs TLS: without + * its CA the checkout cannot reach GitHub * * Exits non-zero on any failed expectation, with the boot log printed. */ @@ -17,14 +20,24 @@ import { readFileSync } from "node:fs"; import { join } from "node:path"; import { renderSandboxConf, + SANDBOX_DRIVE_HOME_SUBTREES, + SANDBOX_DRIVE_STORES, SANDBOX_PATHS, SANDBOX_PORTS, + type SandboxIdentity, } from "@superset/shared/sandbox/contract"; import { BUNDLE_SRC, PACKAGE_ROOT } from "./build"; import { discardingSteps, loadAssets, loadSteps } from "./manifest"; const IMAGE = process.env.SANDBOX_IMAGE ?? "superset-sandbox:local"; const NAME = `superset-boot-twice-${Date.now().toString(36)}`; +const DRIVE_NAME = `${NAME}-drive`; +const CA_MOUNT = process.env.BOOT_TWICE_CA_FILE + ? [ + "-v", + `${process.env.BOOT_TWICE_CA_FILE}:/etc/ssl/certs/ca-certificates.crt:ro`, + ] + : []; const bundleSha = readFileSync( join(PACKAGE_ROOT, "dist", "bundle.sha256"), "utf8", @@ -47,9 +60,19 @@ function docker( } return out; } -const exec = (cmd: string, opts: { user?: string; check?: boolean } = {}) => +const exec = ( + cmd: string, + opts: { user?: string; check?: boolean; on?: string } = {}, +) => docker( - ["exec", ...(opts.user ? ["-u", opts.user] : []), NAME, "bash", "-lc", cmd], + [ + "exec", + ...(opts.user ? ["-u", opts.user] : []), + opts.on ?? NAME, + "bash", + "-lc", + cmd, + ], { check: opts.check }, ); @@ -57,7 +80,7 @@ const exec = (cmd: string, opts: { user?: string; check?: boolean } = {}) => * The way the control plane starts boot: as the image user, through sudo * with the secret preserved from the command's env, never on the argv. */ -function bootAsUser(): string { +function bootAsUser(on = NAME): string { const proc = Bun.spawnSync( [ "docker", @@ -66,7 +89,7 @@ function bootAsUser(): string { "ubuntu", "-e", "HOST_SERVICE_SECRET=boot-twice-secret", - NAME, + on, "sudo", "--preserve-env=HOST_SERVICE_SECRET", "/usr/local/bin/superset-boot", @@ -87,7 +110,7 @@ function expect(label: string, ok: boolean, detail = ""): void { if (!ok) failures.push(label); } -const identity = renderSandboxConf({ +const identityFields: SandboxIdentity = { SUPERSET_SANDBOX_CONTRACT: "1", SUPERSET_BUNDLE_SHA: bundleSha, SUPERSET_API_URL: "https://api.example.invalid", @@ -116,9 +139,561 @@ const identity = renderSandboxConf({ enabled: true, }, ]), -}); +}; +const identity = renderSandboxConf(identityFields); + +/** + * Drive mode, with a tmpfs standing in for the Vercel drive at its mount + * path: the layout, the store variables and Docker's data-root, then a second boot after the session's bind mounts are gone. + */ +function drivePass(): void { + const drive = SANDBOX_PATHS.drive; + const on = DRIVE_NAME; + docker([ + "run", + "-d", + "--name", + on, + "--platform", + "linux/amd64", + "--privileged", + "--user", + "root", + ...CA_MOUNT, + "--tmpfs", + `${drive}:exec,mode=0755`, + IMAGE, + "sleep", + "infinity", + ]); + const driveIdentity: SandboxIdentity = { + ...identityFields, + SUPERSET_SANDBOX_DRIVE: DRIVE_NAME, + }; + const writeConf = (fields: SandboxIdentity) => + docker(["exec", "-i", on, "bash", "-c", `cat > ${SANDBOX_PATHS.conf}`], { + stdin: renderSandboxConf(fields), + }); + writeConf(driveIdentity); + const bootDrive = () => { + bootAsUser(on); + exec( + `for i in $(seq 1 600); do [ -f ${SANDBOX_PATHS.run}/host-service.ready ] && exit 0; sleep 0.1; done; exit 1`, + { check: false, on }, + ); + }; + const homeMounts = SANDBOX_DRIVE_HOME_SUBTREES.map( + (sub) => `${SANDBOX_PATHS.home}/${sub}`, + ); + const allMounted = () => + exec( + [SANDBOX_PATHS.workspace, ...homeMounts] + .map((path) => `mountpoint -q ${path} || echo "not mounted ${path}"`) + .join("; "), + { check: false, on }, + ).trim(); + + bootDrive(); + const log = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive: layout stamped with its milliseconds", + /drive\.layout ms=\d+/.test(log), + log.match(/drive\.layout.*/)?.[0], + ); + expect( + "drive: /workspace and every home subtree are bind mounts", + allMounted() === "", + allMounted(), + ); + expect( + "drive: /workspace is backed by the drive", + /\/workspace/.test( + exec(`findmnt -no FSROOT ${SANDBOX_PATHS.workspace}`, { + check: false, + on, + }), + ), + exec(`findmnt -no SOURCE,FSROOT ${SANDBOX_PATHS.workspace}`, { + check: false, + on, + }).trim(), + ); + expect( + "drive: its directories belong to the sandbox user", + exec( + `stat -c %U ${drive}/workspace ${drive}/home ${drive}/home/.local ${SANDBOX_PATHS.home}/.local ${SANDBOX_PATHS.home}/.cargo`, + { check: false, on }, + ).trim() === Array(5).fill("ubuntu").join("\n"), + ); + const hostEnv = exec( + `tr '\\0' '\\n' < /proc/$(cat ${SANDBOX_PATHS.run}/host-service.pid)/environ`, + { check: false, on }, + ); + expect( + "drive: host-service gets the stores inside /workspace", + hostEnv.includes(`BUN_INSTALL_CACHE_DIR=${SANDBOX_DRIVE_STORES.bun}`) && + hostEnv.includes(`npm_config_store_dir=${SANDBOX_DRIVE_STORES.pnpm}`), + ); + expect( + "drive: a login shell gets them too", + exec("echo $BUN_INSTALL_CACHE_DIR", { + user: "ubuntu", + check: false, + on, + }).trim() === SANDBOX_DRIVE_STORES.bun, + ); + expect( + "drive: Docker's data-root is on the drive, storage driver kept", + /"data-root":"\/mnt\/superset-drive\/docker"/.test( + exec("jq -c . /etc/docker/daemon.json", { check: false, on }), + ) && + /fuse-overlayfs/.test( + exec("cat /etc/docker/daemon.json", { check: false, on }), + ), + exec("cat /etc/docker/daemon.json", { check: false, on }).trim(), + ); + const checkedOut = /ready/.test( + exec( + `for i in $(seq 1 1200); do [ -f ${SANDBOX_PATHS.run}/checkout.ready ] && echo ready && exit 0; sleep 0.1; done; echo missing`, + { check: false, on }, + ), + ); + expect("drive: checkout done", checkedOut); + expect( + "drive: the checkout landed on the drive", + /ok/.test( + exec(`test -d ${drive}/workspace/.git && echo ok`, { check: false, on }), + ), + ); + expect( + "drive: git ignores the stores of a root checkout", + /\/\.superset\/stores\//.test( + exec(`cat ${SANDBOX_PATHS.workspace}/.git/info/exclude`, { + check: false, + on, + }), + ), + ); + + // A new session: no processes and no bind mounts survive; the drive does. + const endSession = () => + exec( + `pkill -u ubuntu -TERM || true; pkill -x superset-memgua || true; sleep 1; pkill -u ubuntu -KILL || true; pkill -x Xvnc || true; rm -rf ${SANDBOX_PATHS.run}; for m in $(findmnt -rn -o TARGET | grep -E '^(${SANDBOX_PATHS.workspace}|${SANDBOX_PATHS.home}/)' | sort -r); do umount -l "$m"; done; : > ${SANDBOX_PATHS.bootLog}`, + { check: false, on }, + ); + endSession(); + bootDrive(); + const log2 = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive, second boot: layout redone", + /drive\.layout ms=\d+/.test(log2), + ); + expect( + "drive, second boot: every bind mount back", + allMounted() === "", + allMounted(), + ); + expect( + "drive, second boot: nothing installed", + /apply-rootfs installed=0/.test(log2), + ); + expect( + "drive, second boot: the checkout is still there", + /checkout\.(skipped|fetched)/.test( + exec( + `for i in $(seq 1 600); do grep -qE 'checkout.(skipped|fetched|cloned|clone)' ${SANDBOX_PATHS.bootLog} && break; sleep 0.1; done; cat ${SANDBOX_PATHS.bootLog}`, + { check: false, on }, + ), + ) || !checkedOut, + log2.match(/checkout\..*/)?.[0], + ); + console.log(`\nboot log (drive, second boot):\n${log2}`); + + const state = SANDBOX_PATHS.driveState; + expect( + "drive: the drive names its workspace", + exec(`cat ${state}/workspace-id`, { check: false, on }).trim() === + identityFields.SUPERSET_SANDBOX_WORKSPACE_ID, + ); + expect( + "drive: host.db and the checkout markers are on the drive, not the root disk", + exec( + `test -f ${state}/host.db && test -f ${state}/checkouts/@root && ! test -e ${SANDBOX_PATHS.checkouts}/@root && echo ok`, + { check: false, on }, + ).trim() === "ok", + exec(`ls ${state} ${state}/checkouts ${SANDBOX_PATHS.state}`, { + check: false, + on, + }), + ); + + // The user's work: a local commit and an uncommitted file. + const ws = SANDBOX_PATHS.workspace; + exec( + `git -C ${ws} -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m user-work && echo user > ${ws}/user-file.txt`, + { user: "ubuntu", on }, + ); + const head = () => + exec(`git -C ${ws} rev-parse HEAD`, { + check: false, + user: "ubuntu", + on, + }).trim(); + const work = head(); + const bootAndCheckout = () => { + bootDrive(); + return exec( + `for i in $(seq 1 1200); do grep -q 'checkout.end' ${SANDBOX_PATHS.bootLog} && break; sleep 0.1; done; cat ${SANDBOX_PATHS.bootLog}`, + { check: false, on }, + ); + }; + const workKept = () => + head() === work && + exec(`cat ${ws}/user-file.txt`, { check: false, on }).trim() === "user"; + + // The root disk rolled back to before the checkout; the drive did not. + endSession(); + exec( + `rm -rf ${SANDBOX_PATHS.checkouts} ${SANDBOX_PATHS.hostDb} ${SANDBOX_PATHS.hostDb}-wal ${SANDBOX_PATHS.hostDb}-shm`, + { on }, + ); + const rolledBack = bootAndCheckout(); + expect( + "drive, root rolled back: checkout skipped, the work kept", + /checkout\.skipped \. /.test(rolledBack) && workKept(), + rolledBack.match(/checkout\..*/g)?.join(" | "), + ); -docker(["rm", "-f", NAME], { check: false }); + // Even with its marker gone, a checkout that has the branch is the user's. + endSession(); + exec(`rm -rf ${state}/checkouts`, { on }); + const markerLost = bootAndCheckout(); + expect( + "drive, marker lost: the branch is kept, not reset", + /checkout\.kept \. main is already on the drive/.test(markerLost) && + workKept(), + markerLost.match(/checkout\..*/g)?.join(" | "), + ); + + // A fetch that fails must never turn into a reclone over the drive. + endSession(); + exec(`rm -rf ${state}/checkouts`, { on }); + exec(`git -C ${drive}/workspace branch -q -m main user-main`, { + user: "ubuntu", + on, + }); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { + url: "https://github.com/paradevai/paradev.git", + branch: "superset/boot-twice-no-such-branch", + baseBranch: "superset/boot-twice-no-such-branch", + path: ".", + }, + ]), + }); + const fetchFailed = bootAndCheckout(); + expect( + "drive, fetch failed: nothing on the drive is replaced", + /checkout\.kept \. fetch failed/.test(fetchFailed) && + !/checkout\.cloned/.test(fetchFailed) && + workKept(), + fetchFailed.match(/checkout\..*/g)?.join(" | "), + ); + exec(`git -C ${drive}/workspace branch -q -m user-main main`, { + user: "ubuntu", + on, + }); + writeConf(driveIdentity); + + // A pinned bundle whose boot script predates the drive: it must never run over the laid-out drive. + endSession(); + const installedBoot = "/usr/local/bin/superset-boot"; + const bundledBoot = `${SANDBOX_PATHS.bundleRoot}/current/rootfs${installedBoot}`; + exec( + `cp ${bundledBoot} /tmp/superset-boot.keep && sed -i '/^# superset-boot drive-contract:/d' ${bundledBoot} && rm -f ${installedBoot}.hash`, + { on }, + ); + const downgraded = bootAsUser(on); + const downgradeLog = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive, older boot script: refused before the reexec, nothing checked out", + /exit=[1-9]/.test(downgraded) && + /drive\.downgrade /.test(downgradeLog) && + !/boot\.reexec|checkout\.start|host\.exec/.test(downgradeLog) && + workKept(), + downgradeLog.match(/drive\.downgrade.*/)?.[0] ?? downgraded.slice(-200), + ); + exec( + `cp /tmp/superset-boot.keep ${bundledBoot} && rm -f ${installedBoot}.hash`, + { + on, + }, + ); + + // A fresh fork of a golden drive: promote stripped its state, and a new workspace boots on it. + const repo = "https://github.com/paradevai/paradev.git"; + const currentBranch = () => + exec(`git -C ${ws} rev-parse --abbrev-ref HEAD`, { + check: false, + user: "ubuntu", + on, + }).trim(); + const forkBoot = (branch: string, baseBranch: string) => { + endSession(); + exec(`rm -rf ${state}`, { on }); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: randomUUID(), + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { url: repo, branch, baseBranch, path: "." }, + ]), + }); + return bootAndCheckout(); + }; + const newBranch = forkBoot("superset/boot-twice-fork", "main"); + expect( + "fresh fork, new branch: fetched from its base, on the branch", + /checkout\.fetched \. superset\/boot-twice-fork from main/.test( + newBranch, + ) && currentBranch() === "superset/boot-twice-fork", + newBranch.match(/checkout\..*/g)?.join(" | "), + ); + const sameBranch = forkBoot("main", "main"); + expect( + "fresh fork, a branch the golden has: reset to its base, not kept", + /checkout\.fetched \. main from main/.test(sameBranch) && + currentBranch() === "main" && + head() !== work, + sameBranch.match(/checkout\..*/g)?.join(" | "), + ); + const fetchFails = forkBoot( + "superset/boot-twice-fork-2", + "superset/boot-twice-no-such-branch", + ); + expect( + "fresh fork, fetch failed: never recloned; the golden's checkout is kept and the box is ready", + /checkout\.fresh\.failed \. superset\/boot-twice-fork-2/.test(fetchFails) && + !/checkout\.cloned/.test(fetchFails) && + currentBranch() === "main" && + exec( + `for i in $(seq 1 600); do [ -f ${SANDBOX_PATHS.run}/checkout.ready ] && echo ready && exit 0; sleep 0.1; done; echo missing`, + { check: false, on }, + ).trim() === "ready", + fetchFails.match(/checkout\..*/g)?.join(" | "), + ); + // A fork of a golden whose checkout carries its installed dependencies, on a new branch. + endSession(); + exec( + `git -C ${drive}/workspace checkout -q main && mkdir -p ${drive}/workspace/node_modules/golden-dep && echo installed > ${drive}/workspace/node_modules/golden-dep/index.js`, + { user: "ubuntu", on }, + ); + exec(`rm -rf ${state}`, { on }); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: randomUUID(), + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { + url: repo, + branch: "superset/boot-twice-fork-deps", + baseBranch: "main", + path: ".", + }, + ]), + }); + const withDeps = bootAndCheckout(); + expect( + "fresh fork of a golden with node_modules, new branch: the dependencies survive", + /checkout\.fetched \. superset\/boot-twice-fork-deps from main/.test( + withDeps, + ) && + exec(`cat ${ws}/node_modules/golden-dep/index.js`, { + check: false, + on, + }).trim() === "installed", + withDeps.match(/checkout\..*/g)?.join(" | "), + ); + + // A fresh fork whose first session ended between its layout and its checkout. + endSession(); + const interrupted = randomUUID(); + exec( + `git -C ${drive}/workspace checkout -q main && git -C ${drive}/workspace -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m stale-golden-copy`, + { user: "ubuntu", on }, + ); + const staleCopy = exec(`git -C ${drive}/workspace rev-parse main`, { + user: "ubuntu", + on, + }).trim(); + exec( + `rm -rf ${state} && mkdir -p ${state} && : > ${state}/fresh && echo ${interrupted} > ${state}/workspace-id && chown -R ubuntu:ubuntu ${state}`, + { on }, + ); + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: interrupted, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { url: repo, branch: "main", baseBranch: "main", path: "." }, + ]), + }); + const resumedFresh = bootAndCheckout(); + expect( + "fresh fork, first session ended before its checkout: the next boot still fetches", + /checkout\.fetched \. main from main/.test(resumedFresh) && + head() !== staleCopy && + exec(`test -e ${state}/fresh && echo still || echo gone`, { + check: false, + on, + }).trim() === "gone", + resumedFresh.match(/checkout\..*/g)?.join(" | "), + ); + + // A fresh fork whose first fetch and clone both fail: a later boot still fetches the untouched golden copy. + endSession(); + const failedFirst = randomUUID(); + const unreachable = "https://example.invalid/paradev.git"; + // The golden is clean: the earlier cases' user file is not part of it. + exec( + `rm -f ${drive}/workspace/user-file.txt && git -C ${drive}/workspace checkout -q main && git -C ${drive}/workspace -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m stale-golden-copy-2 && git -C ${drive}/workspace remote set-url origin ${unreachable}`, + { user: "ubuntu", on }, + ); + const staleCopy2 = exec(`git -C ${drive}/workspace rev-parse main`, { + user: "ubuntu", + on, + }).trim(); + exec(`rm -rf ${state}`, { on }); + const forkOn = (url: string) => + writeConf({ + ...driveIdentity, + SUPERSET_SANDBOX_WORKSPACE_ID: failedFirst, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { url, branch: "main", baseBranch: "main", path: "." }, + ]), + }); + forkOn(unreachable); + const firstAttempt = bootAndCheckout(); + endSession(); + exec(`git -C ${drive}/workspace remote set-url origin ${repo}`, { + user: "ubuntu", + on, + }); + forkOn(repo); + const statusAfterFailure = exec( + `git -C ${drive}/workspace status --porcelain 2>&1 | head -10`, + { check: false, user: "ubuntu", on }, + ).trim(); + const depsAfterFailure = exec( + `cat ${drive}/workspace/node_modules/golden-dep/index.js`, + { check: false, on }, + ).trim(); + const laterBoot = bootAndCheckout(); + expect( + "fresh fork of a golden with node_modules, first fetch failed: the dependencies survive and a later boot fetches", + /checkout\.fresh\.failed/.test(firstAttempt) && + !/checkout\.cloned/.test(firstAttempt) && + depsAfterFailure === "installed" && + exec(`cat ${ws}/node_modules/golden-dep/index.js`, { + check: false, + on, + }).trim() === "installed" && + /checkout\.fetched \. main from main \(an untouched golden copy\)/.test( + laterBoot, + ) && + head() !== staleCopy2, + `${firstAttempt.match(/checkout\..*/g)?.join(" | ")} || ${laterBoot.match(/checkout\..*/g)?.join(" | ")} || status after the failure: ${statusAfterFailure || "clean"}`, + ); + + // The same, but the user committed after the failed first fetch: that copy is theirs now, kept. + endSession(); + exec(`git -C ${drive}/workspace remote set-url origin ${unreachable}`, { + user: "ubuntu", + on, + }); + exec(`rm -rf ${state}`, { on }); + forkOn(unreachable); + bootAndCheckout(); + endSession(); + exec( + `git -C ${drive}/workspace -c user.name=t -c user.email=t@example.invalid commit -q --allow-empty -m user-work-after-failed-fetch && git -C ${drive}/workspace remote set-url origin ${repo}`, + { user: "ubuntu", on }, + ); + const usersCommit = exec(`git -C ${drive}/workspace rev-parse HEAD`, { + user: "ubuntu", + on, + }).trim(); + forkOn(repo); + const touched = bootAndCheckout(); + expect( + "fresh fork, first fetch failed, then the user committed: their copy is kept", + /checkout\.kept \. main is already on the drive/.test(touched) && + head() === usersCommit, + touched.match(/checkout\..*/g)?.join(" | "), + ); + + writeConf(driveIdentity); + + const refuses = (label: string, phase: string) => { + const answer = bootAsUser(on); + const log = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + `${label}: refused before host-service, stamped ${phase}`, + /exit=[1-9]/.test(answer) && + new RegExp(`${phase.replace(".", "\\.")} `).test(log) && + !/host\.exec|checkout\.start/.test(log) && + exec(`mountpoint -q ${ws} && echo bound`, { + check: false, + on, + }).trim() === "", + log.match(/drive\..*/)?.[0] ?? answer.slice(-200), + ); + }; + + // Another workspace's drive. + endSession(); + writeConf({ ...driveIdentity, SUPERSET_SANDBOX_WORKSPACE_ID: randomUUID() }); + refuses("drive foreign", "drive.foreign"); + writeConf(driveIdentity); + + // A drive that mounts empty though the workspace was laid out before. + endSession(); + exec(`umount ${drive} && mount -t tmpfs -o exec,mode=0755 tmpfs ${drive}`, { + on, + }); + writeConf({ ...driveIdentity, SUPERSET_SANDBOX_DRIVE_LAID_OUT: "1" }); + refuses("drive empty", "drive.unlaid"); + + // A session whose drive did not mount, while sandbox.conf still names one. + endSession(); + exec(`umount ${drive}`, { on }); + const refused = bootAsUser(on); + const log3 = exec(`cat ${SANDBOX_PATHS.bootLog}`, { on }); + expect( + "drive missing: boot exits non-zero", + /exit=[1-9]/.test(refused), + refused.slice(-200).replaceAll("\n", " | "), + ); + expect( + "drive missing: stamped drive.missing", + /drive\.missing /.test(log3), + log3.match(/drive\.missing.*/)?.[0], + ); + expect( + "drive missing: no host-service, no checkout", + !/host\.exec|checkout\.start|steps\.start/.test(log3) && + exec(`ls ${SANDBOX_PATHS.run}`, { check: false, on }).trim() === "", + log3, + ); + expect( + "drive missing: /workspace is not bound", + exec(`mountpoint -q ${SANDBOX_PATHS.workspace} && echo bound`, { + check: false, + on, + }).trim() === "", + ); +} + +docker(["rm", "-f", NAME, DRIVE_NAME], { check: false }); // Privileged for the /dev/shm remount and the X socket dir, like the VM. No // network policy here; the checkout needs the internet. docker([ @@ -131,6 +706,7 @@ docker([ "--privileged", "--user", "root", + ...CA_MOUNT, IMAGE, "sleep", "infinity", @@ -268,6 +844,14 @@ try { .split("\n") .find((line) => line.startsWith("SUPERSET_SANDBOX_PLUGINS=")) ?? "unset", ); + expect( + "host-service's own env turns the agents' self-updates off, whatever its shell probe finds", + [ + "DISABLE_AUTOUPDATER=1", + "OPENCODE_DISABLE_AUTOUPDATE=1", + "PI_SKIP_VERSION_CHECK=1", + ].every((line) => hostEnv.split("\n").includes(line)), + ); // The control plane's push, so the runner's hook sequencing runs: this // checkout declares no start hook, which the log must say. @@ -471,8 +1055,10 @@ try { exec(`ls ${SANDBOX_PATHS.run}`, { check: false }), ), ); + + drivePass(); } finally { - docker(["rm", "-f", NAME], { check: false }); + docker(["rm", "-f", NAME, DRIVE_NAME], { check: false }); } if (failures.length) { console.error( diff --git a/packages/sandbox/src/drive-comparison.test.ts b/packages/sandbox/src/drive-comparison.test.ts new file mode 100644 index 00000000000..815b8e0aab6 --- /dev/null +++ b/packages/sandbox/src/drive-comparison.test.ts @@ -0,0 +1,64 @@ +import { describe, expect, test } from "bun:test"; +import { readSentinels, renderComparison } from "./drive-comparison"; + +describe("renderComparison", () => { + test("one column per box, a dash where a box has no measure, LOST for a lost sentinel", () => { + const table = renderComparison([ + { + label: "plain", + measures: { + createToReadyMs: 31_000, + firstStop: { ms: 40_000, snapshotBytes: 7 * 1024 ** 3 }, + sentinels: { "uncommitted file": true }, + }, + }, + { + label: "drive", + measures: { + forkMs: 500, + firstStop: { ms: 12_000, snapshotBytes: 2.11 * 1024 ** 3 }, + sentinels: { "uncommitted file": false }, + }, + }, + ]).split("\n"); + expect(table[0]).toMatch(/plain\s+drive$/); + expect(table.find((row) => row.startsWith("create to host.ready"))).toMatch( + /31\.0 s\s+-$/, + ); + expect(table.find((row) => row.startsWith("drive fork"))).toMatch( + /-\s+0\.5 s$/, + ); + expect( + table.find((row) => row.startsWith("root snapshot, 1st stop")), + ).toMatch(/7\.00 GB\s+2\.11 GB$/); + expect( + table.find((row) => row.startsWith("kept: uncommitted file")), + ).toMatch(/yes\s+LOST$/); + }); +}); + +describe("readSentinels", () => { + test("a sentinel is kept only when it reads back exactly", async () => { + const answers: Record = { + "superset-sentinel-uncommitted.txt": "t0k", + "rev-parse HEAD": "abc", + ".cache/superset-sentinel": "t0k", + ".config/superset-sentinel": "other", + "image inspect": "", + }; + const run = async (command: string) => { + const key = Object.keys(answers).find((part) => command.includes(part)); + const out = key ? (answers[key] ?? "") : ""; + return { out, detail: out }; + }; + expect( + await readSentinels(run, { token: "t0k", commit: "abc", image: "img" }), + ).toEqual({ + "uncommitted file": true, + "local commit": true, + "file in a bound home subtree": true, + "file on the root disk": false, + "docker image": false, + }); + }); +}); diff --git a/packages/sandbox/src/drive-comparison.ts b/packages/sandbox/src/drive-comparison.ts new file mode 100644 index 00000000000..206fc431efa --- /dev/null +++ b/packages/sandbox/src/drive-comparison.ts @@ -0,0 +1,204 @@ +/** + * The same work on a box with a workspace drive and on one without, measured + * the way a person meets it: create, install, stop, wake, and whether what + * they wrote is still there. The REAL_SANDBOX_DRIVE pass of real-sandbox.ts + * runs it; this module holds the work, the sentinels and the table. + */ + +import { SANDBOX_PATHS } from "@superset/shared/sandbox/contract"; + +export const SENTINELS = [ + "uncommitted file", + "local commit", + "file in a bound home subtree", + "file on the root disk", + "docker image", +] as const; +export type SentinelName = (typeof SENTINELS)[number]; + +export interface StopMeasure { + ms: number; + snapshotBytes: number; +} + +export interface BoxMeasures { + createToReadyMs?: number; + forkMs?: number; + installMs?: number; + writeMs?: number; + firstStop?: StopMeasure; + secondStop?: StopMeasure; + wake?: { + detachMs?: number; + mountMs?: number; + resumeMs?: number; + bootMs?: number; + totalMs: number; + }; + /** From the start of the create or wake until git and node_modules answer in /workspace. */ + usableMs?: number; + sentinels?: Partial>; +} + +/** A command on the box: stdout only, since sudo there warns on stderr. */ +export type BoxRun = ( + command: string, + options?: { root?: boolean }, +) => Promise<{ out: string; detail: string }>; + +const WORKSPACE = SANDBOX_PATHS.workspace; +const DOCKER_UP = + "pgrep -x dockerd > /dev/null || (dockerd > /tmp/dockerd.log 2>&1 &); for i in $(seq 1 60); do docker info > /dev/null 2>&1 && break; sleep 1; done"; + +async function timed(run: () => Promise): Promise { + const started = Date.now(); + await run(); + return Date.now() - started; +} + +export async function waitForCheckout(run: BoxRun): Promise { + const ready = await run( + `for i in $(seq 1 600); do [ -f ${SANDBOX_PATHS.run}/checkout.ready ] && echo ready && exit 0; sleep 1; done; echo missing`, + ); + return ready.out === "ready"; +} + +/** A real install, then `gb` GB of incompressible data under /workspace. */ +export async function doWork( + run: BoxRun, + gb: number, +): Promise<{ installMs: number; writeMs: number; installed: string }> { + let installed = ""; + const installMs = await timed(async () => { + installed = ( + await run( + `cd ${WORKSPACE} && bun install --frozen-lockfile > /tmp/bun-install.log 2>&1 && echo installed || tail -5 /tmp/bun-install.log`, + ) + ).out; + }); + const writeMs = await timed(() => + run( + `head -c ${gb * 1024 * 1024 * 1024} /dev/urandom > ${WORKSPACE}/.superset-work-filler && sync`, + ), + ); + return { installMs, writeMs, installed }; +} + +export interface SentinelValues { + token: string; + commit: string; + image: string; +} + +export async function writeSentinels( + run: BoxRun, + token: string, +): Promise { + const commit = await run( + [ + `echo ${token} > ${WORKSPACE}/superset-sentinel-uncommitted.txt`, + `git -C ${WORKSPACE} -c user.name=check -c user.email=check@example.invalid commit -q --allow-empty -m "sentinel ${token}"`, + "mkdir -p ~/.cache ~/.config", + `echo ${token} > ~/.cache/superset-sentinel`, + `echo ${token} > ~/.config/superset-sentinel`, + `git -C ${WORKSPACE} rev-parse HEAD`, + ].join(" && "), + ); + const image = await run( + `${DOCKER_UP}; mkdir -p /tmp/superset-sentinel && echo ${token} > /tmp/superset-sentinel/sentinel && tar -C /tmp/superset-sentinel -c . | docker import - superset-sentinel:check`, + { root: true }, + ); + return { token, commit: commit.out, image: image.out }; +} + +export async function readSentinels( + run: BoxRun, + values: SentinelValues, +): Promise> { + const read = async (command: string, root = false) => + (await run(command, { root })).out; + return { + "uncommitted file": + (await read(`cat ${WORKSPACE}/superset-sentinel-uncommitted.txt`)) === + values.token, + "local commit": + values.commit !== "" && + (await read(`git -C ${WORKSPACE} rev-parse HEAD`)) === values.commit, + "file in a bound home subtree": + (await read("cat ~/.cache/superset-sentinel")) === values.token, + "file on the root disk": + (await read("cat ~/.config/superset-sentinel")) === values.token, + "docker image": + values.image !== "" && + (await read( + `${DOCKER_UP}; docker image inspect -f '{{.Id}}' superset-sentinel:check`, + true, + )) === values.image, + }; +} + +/** Milliseconds from `since` until /workspace answers git and has node_modules. */ +export async function usableSince( + run: BoxRun, + since: number, +): Promise { + const usable = await run( + `for i in $(seq 1 300); do git -C ${WORKSPACE} status --porcelain > /dev/null 2>&1 && test -d ${WORKSPACE}/node_modules && echo usable && exit 0; sleep 1; done; echo never`, + ); + return usable.out === "usable" ? Date.now() - since : undefined; +} + +const seconds = (ms: number | undefined) => + ms === undefined ? "-" : `${(ms / 1000).toFixed(1)} s`; +const gigabytes = (bytes: number | undefined) => + bytes === undefined ? "-" : `${(bytes / 1024 ** 3).toFixed(2)} GB`; + +/** One row per measure, one column per box. */ +export function renderComparison( + columns: Array<{ label: string; measures: BoxMeasures }>, +): string { + const rows: Array<[string, (m: BoxMeasures) => string]> = [ + ["create to host.ready", (m) => seconds(m.createToReadyMs)], + ["drive fork at create", (m) => seconds(m.forkMs)], + ["bun install", (m) => seconds(m.installMs)], + ["write the filler", (m) => seconds(m.writeMs)], + ["root snapshot, 1st stop", (m) => gigabytes(m.firstStop?.snapshotBytes)], + ["stop time, 1st stop", (m) => seconds(m.firstStop?.ms)], + ["wake: detach wait", (m) => seconds(m.wake?.detachMs)], + ["wake: mount update", (m) => seconds(m.wake?.mountMs)], + ["wake: resume", (m) => seconds(m.wake?.resumeMs)], + ["wake: boot to host.ready", (m) => seconds(m.wake?.bootMs)], + ["wake: total", (m) => seconds(m.wake?.totalMs)], + ["until /workspace is usable", (m) => seconds(m.usableMs)], + ["root snapshot, 2nd stop", (m) => gigabytes(m.secondStop?.snapshotBytes)], + ["stop time, 2nd stop", (m) => seconds(m.secondStop?.ms)], + ...SENTINELS.map((name): [string, (m: BoxMeasures) => string] => [ + `kept: ${name}`, + (m) => { + const kept = m.sentinels?.[name]; + return kept === undefined ? "-" : kept ? "yes" : "LOST"; + }, + ]), + ]; + const table = [ + ["", ...columns.map((column) => column.label)], + ...rows.map(([label, cell]) => [ + label, + ...columns.map((column) => cell(column.measures)), + ]), + ]; + const widths = table[0]?.map((_, i) => + Math.max(...table.map((row) => row[i]?.length ?? 0)), + ); + return table + .map((row) => + row + .map((cell, i) => + i === 0 + ? cell.padEnd(widths?.[i] ?? 0) + : cell.padStart(widths?.[i] ?? 0), + ) + .join(" "), + ) + .join("\n"); +} diff --git a/packages/sandbox/src/drive-sweep.test.ts b/packages/sandbox/src/drive-sweep.test.ts new file mode 100644 index 00000000000..5a58da087c5 --- /dev/null +++ b/packages/sandbox/src/drive-sweep.test.ts @@ -0,0 +1,80 @@ +import { describe, expect, test } from "bun:test"; +import { orphanCandidates, type SweepHost, sweepDrives } from "./drive-sweep"; + +const DAY = 24 * 60 * 60 * 1000; +const NOW = 100 * DAY; +const old = NOW - 10 * DAY; + +describe("orphanCandidates", () => { + test("any row that names a drive keeps it, whatever its status or age", () => { + const candidates = orphanCandidates({ + drives: [ + { name: "drive-ws-archived-long-ago", createdAt: old }, + { name: "drive-ws-unnamed", createdAt: old }, + { name: "drive-ws-young", createdAt: NOW - DAY / 2 }, + { name: "drive-ws-held", createdAt: old, attachedTo: "box" }, + { name: "env-drive-archived-env", createdAt: old }, + { name: "env-drive-gone", createdAt: old }, + { name: "agent-tools-iad1-feedfacecafe", createdAt: old }, + ], + named: new Set(["drive-ws-archived-long-ago", "env-drive-archived-env"]), + now: NOW, + minAgeMs: 2 * DAY, + }); + expect(candidates.map((drive) => drive.name)).toEqual([ + "drive-ws-unnamed", + "env-drive-gone", + ]); + }); +}); + +describe("sweepDrives", () => { + test("a drive an unarchive named after the listing is kept: each delete checks again", async () => { + const deleted: string[] = []; + let named = new Set(); + const host: SweepHost = { + listDrives: async () => [ + { name: "drive-ws-a", createdAt: old }, + { name: "drive-ws-b", createdAt: old }, + ], + namedDrives: async () => { + const listedAt = named; + // The user unarchives workspace a right after the listing. + named = new Set(["drive-ws-a"]); + return listedAt; + }, + stillOrphaned: async (name) => !named.has(name), + remove: async (name) => { + deleted.push(name); + }, + }; + const result = await sweepDrives({ + host, + now: NOW, + minAgeMs: 2 * DAY, + remove: true, + log: () => {}, + }); + expect(deleted).toEqual(["drive-ws-b"]); + expect(result).toEqual({ candidates: 2, deleted: 1, failed: 0 }); + }); + + test("a dry run deletes nothing", async () => { + const deleted: string[] = []; + await sweepDrives({ + host: { + listDrives: async () => [{ name: "drive-ws-a", createdAt: old }], + namedDrives: async () => new Set(), + stillOrphaned: async () => true, + remove: async (name) => { + deleted.push(name); + }, + }, + now: NOW, + minAgeMs: 2 * DAY, + remove: false, + log: () => {}, + }); + expect(deleted).toEqual([]); + }); +}); diff --git a/packages/sandbox/src/drive-sweep.ts b/packages/sandbox/src/drive-sweep.ts new file mode 100644 index 00000000000..a57330a775d --- /dev/null +++ b/packages/sandbox/src/drive-sweep.ts @@ -0,0 +1,199 @@ +/** + * Finds workspace and golden drives that nothing names any more: no + * workspace row and no environment row, whatever its status, no sandbox + * attached, and older than a safe age. Lists them; deletes them only with + * --delete, and checks each again just before its delete. + * + * bun run drive-sweep DATABASE_URL, dry run + * bun run drive-sweep --production the production database + * bun run drive-sweep --production --delete deletes what it lists + * --min-age-days default 2 + * + * Needs VERCEL_SANDBOX_TOKEN, VERCEL_SANDBOX_TEAM_ID and VERCEL_SANDBOX_PROJECT_ID. + */ + +const DAY_MS = 24 * 60 * 60 * 1000; +export const SWEPT_PREFIXES = ["drive-", "env-drive-"] as const; + +export interface SweptDrive { + name: string; + createdAt: number; + attachedTo?: string; +} + +/** + * A drive is a candidate when no row names it and no sandbox holds it, and + * it is older than `minAgeMs`: a release writes its golden drive's name only + * once the golden is built. A row that is really torn down has its names + * cleared in the same write (retireAndTearDown), so any row that still names + * a drive, archived or not, may still be undone onto it. + */ +export function orphanCandidates(args: { + drives: SweptDrive[]; + named: ReadonlySet; + now: number; + minAgeMs: number; +}): SweptDrive[] { + return args.drives.filter( + (drive) => + SWEPT_PREFIXES.some((prefix) => drive.name.startsWith(prefix)) && + !args.named.has(drive.name) && + !drive.attachedTo && + args.now - drive.createdAt > args.minAgeMs, + ); +} + +export interface SweepHost { + listDrives(): Promise; + namedDrives(): Promise>; + /** Read at delete time, not from the list: an unarchive may have named or mounted it since. */ + stillOrphaned(name: string): Promise; + remove(name: string): Promise; +} + +export async function sweepDrives(args: { + host: SweepHost; + now: number; + minAgeMs: number; + remove: boolean; + log: (line: string) => void; +}): Promise<{ candidates: number; deleted: number; failed: number }> { + const drives = await args.host.listDrives(); + const candidates = orphanCandidates({ + drives, + named: await args.host.namedDrives(), + now: args.now, + minAgeMs: args.minAgeMs, + }); + args.log(`${drives.length} drives listed, ${candidates.length} orphaned`); + let deleted = 0; + let failed = 0; + for (const drive of candidates) { + const age = ((args.now - drive.createdAt) / DAY_MS).toFixed(1); + if (!args.remove) { + args.log(`would delete ${drive.name} (${age} days old)`); + continue; + } + try { + if (!(await args.host.stillOrphaned(drive.name))) { + args.log( + `kept ${drive.name}: a row names it or a sandbox holds it now`, + ); + continue; + } + await args.host.remove(drive.name); + deleted++; + args.log(`deleted ${drive.name} (${age} days old)`); + } catch (error) { + failed++; + args.log(`::warning::could not delete ${drive.name}: ${error}`); + } + } + if (!args.remove && candidates.length) { + args.log("dry run: pass --delete to delete"); + } + return { candidates: candidates.length, deleted, failed }; +} + +if (import.meta.main) { + const argv = process.argv.slice(2); + const ageFlag = argv.indexOf("--min-age-days"); + const minAgeDays = ageFlag >= 0 ? Number(argv[ageFlag + 1]) : 2; + const credentials = { + token: process.env.VERCEL_SANDBOX_TOKEN ?? "", + teamId: process.env.VERCEL_SANDBOX_TEAM_ID ?? "", + projectId: process.env.VERCEL_SANDBOX_PROJECT_ID ?? "", + }; + if (!credentials.token || !credentials.teamId || !credentials.projectId) { + console.error( + "VERCEL_SANDBOX_TOKEN, VERCEL_SANDBOX_TEAM_ID and VERCEL_SANDBOX_PROJECT_ID are required", + ); + process.exit(64); + } + if (!Number.isFinite(minAgeDays) || minAgeDays < 1) { + console.error("--min-age-days must be at least 1"); + process.exit(64); + } + if (argv.includes("--production")) { + const { useProductionDatabase } = await import( + "./environments/production-database" + ); + console.log(`database: Neon branch ${await useProductionDatabase()}`); + } else if (!process.env.DATABASE_URL) { + console.error("DATABASE_URL is required (or pass --production)"); + process.exit(64); + } + process.env.SKIP_ENV_VALIDATION ??= "1"; + const { db } = await import("@superset/db/client"); + const { cloudWorkspaces, environments } = await import("@superset/db/schema"); + const { eq, isNotNull } = await import("drizzle-orm"); + const { Drive } = await import("@vercel/sandbox"); + + const named = async (name: string) => + ( + await Promise.all([ + db + .select({ id: cloudWorkspaces.id }) + .from(cloudWorkspaces) + .where(eq(cloudWorkspaces.driveName, name)) + .limit(1), + db + .select({ id: environments.id }) + .from(environments) + .where(eq(environments.goldenDriveName, name)) + .limit(1), + ]) + ).some((rows) => rows.length > 0); + const host: SweepHost = { + async listDrives() { + const drives = new Map(); + for (const prefix of SWEPT_PREFIXES) { + for await (const drive of await Drive.list({ + ...credentials, + namePrefix: prefix, + sortBy: "name", + })) { + drives.set(drive.name, { + name: drive.name, + createdAt: drive.createdAt.getTime(), + attachedTo: drive.currentSandboxName, + }); + } + } + return [...drives.values()]; + }, + async namedDrives() { + const [workspaces, goldens] = await Promise.all([ + db + .select({ name: cloudWorkspaces.driveName }) + .from(cloudWorkspaces) + .where(isNotNull(cloudWorkspaces.driveName)), + db + .select({ name: environments.goldenDriveName }) + .from(environments) + .where(isNotNull(environments.goldenDriveName)), + ]); + return new Set( + [...workspaces, ...goldens].flatMap((row) => + row.name ? [row.name] : [], + ), + ); + }, + async stillOrphaned(name) { + if (await named(name)) return false; + const drive = await Drive.get({ ...credentials, name }); + return !drive.currentSandboxName; + }, + async remove(name) { + await (await Drive.get({ ...credentials, name })).delete(); + }, + }; + const { failed } = await sweepDrives({ + host, + now: Date.now(), + minAgeMs: minAgeDays * DAY_MS, + remove: argv.includes("--delete"), + log: console.log, + }); + process.exit(failed ? 1 : 0); +} diff --git a/packages/sandbox/src/environments/probe.ts b/packages/sandbox/src/environments/probe.ts index e1b198bd79d..c5425d7a937 100644 --- a/packages/sandbox/src/environments/probe.ts +++ b/packages/sandbox/src/environments/probe.ts @@ -175,12 +175,40 @@ export async function probeBox(args: ProbeArgs): Promise { /up/, 90, ); + await until( + "websockify listening", + `ss -ltn | grep -q ':${SANDBOX_PORTS.desktop} ' && echo listening`, + /listening/, + 60, + ); const rfb = await rfbHandshake(host, args.hostSecret); check( "desktop stream answers RFB through host-service", rfb.startsWith("RFB "), JSON.stringify(rfb), ); + if (!rfb.startsWith("RFB ")) { + const plainGet = await status( + `${host}/desktop/websockify?token=${encodeURIComponent(args.hostSecret)}`, + ); + const onBox = await run( + [ + `echo '--- listening'`, + `ss -ltn | grep -E ':(${SANDBOX_PORTS.desktop}|${SANDBOX_PORTS.vnc}) '`, + `echo '--- processes'`, + `pgrep -a -f 'websockify|Xvnc'`, + `echo '--- loopback handshake'`, + `curl -s --max-time 5 -o /dev/null -w '%{http_code}' -H 'Connection: Upgrade' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' http://127.0.0.1:${SANDBOX_PORTS.desktop}/`, + `echo; echo '--- desktop.log'`, + `tail -20 ${SANDBOX_PATHS.logs}/desktop.log`, + `echo '--- host-service.log, desktop lines'`, + `grep -i -E 'desktop|websockify' ${SANDBOX_PATHS.logs}/host-service.log | tail -20`, + ].join("; "), + ); + log( + `RFB diagnosis: GET through host-service answered ${plainGet}\n${onBox}`, + ); + } if (args.expectAnthropicRule) { const models = (key: string) => run( diff --git a/packages/sandbox/src/environments/production-database.ts b/packages/sandbox/src/environments/production-database.ts new file mode 100644 index 00000000000..7815aec8304 --- /dev/null +++ b/packages/sandbox/src/environments/production-database.ts @@ -0,0 +1,37 @@ +/** + * Points DATABASE_URL at the Neon project's default branch, the production + * database (NEON_API_KEY, NEON_PROJECT_ID). Returns the branch it chose. + */ +export async function useProductionDatabase(): Promise { + const key = process.env.NEON_API_KEY; + const project = process.env.NEON_PROJECT_ID; + if (!key || !project) { + throw new Error("--production needs NEON_API_KEY and NEON_PROJECT_ID"); + } + const headers = { authorization: `Bearer ${key}` }; + const branchesResponse = await fetch( + `https://console.neon.tech/api/v2/projects/${project}/branches`, + { headers }, + ); + if (!branchesResponse.ok) { + throw new Error(`Neon branches: ${branchesResponse.status}`); + } + const branches = (await branchesResponse.json()) as { + branches: Array<{ id: string; name: string; default?: boolean }>; + }; + const main = branches.branches.find((b) => b.default); + if (!main) throw new Error("no default branch in the Neon project"); + const uriResponse = await fetch( + `https://console.neon.tech/api/v2/projects/${project}/connection_uri?branch_id=${main.id}&database_name=neondb&role_name=neondb_owner&pooled=false`, + { headers }, + ); + if (!uriResponse.ok) { + throw new Error(`Neon connection_uri: ${uriResponse.status}`); + } + const uri = (await uriResponse.json()) as { uri?: string }; + if (!uri.uri) + throw new Error("could not resolve the production connection string"); + process.env.DATABASE_URL = uri.uri; + process.env.DATABASE_URL_UNPOOLED = uri.uri; + return `${main.name} (${main.id})`; +} diff --git a/packages/sandbox/src/environments/release.ts b/packages/sandbox/src/environments/release.ts index 95dd3faf6c6..21f96e697dc 100644 --- a/packages/sandbox/src/environments/release.ts +++ b/packages/sandbox/src/environments/release.ts @@ -51,6 +51,7 @@ import { SANDBOX_REGION_IDS, } from "@superset/shared/sandbox/regions"; import { Sandbox } from "@vercel/sandbox"; +import { useProductionDatabase } from "./production-database"; // The provisioning code imports the API env schema; an operator running this // should not need every API secret to exist locally. @@ -99,31 +100,10 @@ if (!credentials.token || !credentials.teamId || !credentials.projectId) { if (!process.env.SANDBOX_GATE_SECRET) fail("SANDBOX_GATE_SECRET is required (the host secret derives from it)"); if (PRODUCTION) { - const key = process.env.NEON_API_KEY; - const project = process.env.NEON_PROJECT_ID; - if (!key || !project) - fail("--production needs NEON_API_KEY and NEON_PROJECT_ID"); - const headers = { authorization: `Bearer ${key}` }; - const branchesResponse = await fetch( - `https://console.neon.tech/api/v2/projects/${project}/branches`, - { headers }, + const branch = await useProductionDatabase().catch((error: unknown) => + fail(String(error instanceof Error ? error.message : error)), ); - if (!branchesResponse.ok) fail(`Neon branches: ${branchesResponse.status}`); - const branches = (await branchesResponse.json()) as { - branches: Array<{ id: string; name: string; default?: boolean }>; - }; - const main = branches.branches.find((b) => b.default); - if (!main) fail("no default branch in the Neon project"); - const uriResponse = await fetch( - `https://console.neon.tech/api/v2/projects/${project}/connection_uri?branch_id=${main.id}&database_name=neondb&role_name=neondb_owner&pooled=false`, - { headers }, - ); - if (!uriResponse.ok) fail(`Neon connection_uri: ${uriResponse.status}`); - const uri = (await uriResponse.json()) as { uri?: string }; - if (!uri.uri) fail("could not resolve the production connection string"); - process.env.DATABASE_URL = uri.uri; - process.env.DATABASE_URL_UNPOOLED = uri.uri; - log(`database: Neon branch ${main.name} (${main.id})`); + log(`database: Neon branch ${branch}`); } else if (!process.env.DATABASE_URL) { fail("DATABASE_URL is required (or pass --production)"); } @@ -204,11 +184,22 @@ if (SKIP_IMAGE) { // email clients, whose secrets this job does not have. const { deleteSandbox, + deleteSandboxAndDrive, provisionSandbox, stopAndSnapshot, stripWorkspaceIdentity, wakeSandbox, } = await import("@superset/trpc/lib/sandbox/vercel"); +const { + forkWorkspaceDrive, + goldenDriveNameFor, + vercelDrives, + waitForDriveDetached, + workspaceDriveIdentity, + workspaceDriveMounts, + workspaceDriveNameFor, + workspaceDrivesEnabled, +} = await import("@superset/trpc/lib/sandbox/drives"); const { deriveSandboxCredentials } = await import( "@superset/trpc/lib/sandbox/credentials" ); @@ -223,7 +214,11 @@ const setupScript = readFileSync( ); const setupHook = [setupScript]; -function identityFor(workspaceId: string, sourceRef: string): SandboxIdentity { +function identityFor( + workspaceId: string, + sourceRef: string, + driveName: string | null, +): SandboxIdentity { return { SUPERSET_SANDBOX_CONTRACT: String(SANDBOX_CONTRACT_VERSION) as "1", SUPERSET_BUNDLE_SHA: bundle.sha256, @@ -235,12 +230,23 @@ function identityFor(workspaceId: string, sourceRef: string): SandboxIdentity { ]), SUPERSET_SANDBOX_IMAGE_TAG: sourceRef, SUPERSET_SANDBOX_PROVIDER: "vercel", + ...workspaceDriveIdentity(driveName, { laidOut: false }), }; } const golden = `env-internal-${Date.now().toString(36)}`; const goldenWorkspaceId = randomUUID(); const goldenSecret = await sandboxHostSecretFor(goldenWorkspaceId); +// In drive mode the golden boots with its drive from the first boot, so the +// checkout and the installs never touch its root disk. +const drives = vercelDrives(); +const goldenDrive = (await workspaceDrivesEnabled(ORGANIZATION_ID as string)) + ? goldenDriveNameFor(golden) + : null; +if (goldenDrive) { + await drives.create(goldenDrive, REGION); + log(`golden: drive mode; ${goldenDrive} created in ${REGION}`); +} log(`golden: provisioning ${golden} from ${goldenImage}`); // A freshly pushed image sits in `Preparing` while the registry optimises it // for sandboxes, and create answers 409 until then. @@ -256,13 +262,14 @@ for (;;) { region: REGION, }, claim: { - identity: identityFor(goldenWorkspaceId, goldenImage), + identity: identityFor(goldenWorkspaceId, goldenImage, goldenDrive), hostSecret: goldenSecret, managedEnv: {}, networkPolicy: "allow-all", vcpus: GOLDEN_BOX_VCPUS, agentTools: true, }, + mounts: workspaceDriveMounts(goldenDrive), }); break; } catch (error) { @@ -280,7 +287,7 @@ for (;;) { await wakeSandbox({ providerSandboxId: golden, claim: { - identity: identityFor(goldenWorkspaceId, goldenImage), + identity: identityFor(goldenWorkspaceId, goldenImage, goldenDrive), hostSecret: goldenSecret, managedEnv: {}, networkPolicy: "allow-all", @@ -383,8 +390,9 @@ if (failed) fail(`${failed} check(s) failed; ${golden} left for inspection`); // A fork must not come up as the golden's workspace; and the stop is the // snapshot forks start from. -await stripWorkspaceIdentity(goldenBox); +await stripWorkspaceIdentity(goldenBox, { withDrive: goldenDrive !== null }); await stopAndSnapshot(golden); +if (goldenDrive) await waitForDriveDetached(goldenDrive, { drives }); log(`golden: ${golden} stripped, stopped and snapshotted`); // 4. probe, as a workspace @@ -441,21 +449,36 @@ const { networkPolicy, managedEnv } = await deriveSandboxCredentials({ githubToken: null, gitAuthor: { name: "Superset release", email: "noreply@superset.sh" }, }); +const probeDrive = goldenDrive ? workspaceDriveNameFor(probe) : null; const probeClaim = { - identity: identityFor(probeWorkspaceId, golden), + identity: identityFor(probeWorkspaceId, golden, probeDrive), hostSecret: probeSecret, managedEnv, networkPolicy, vcpus: GOLDEN_BOX_VCPUS, agentTools: true, }; +const probeMounts = workspaceDriveMounts(probeDrive); +if (probeDrive) { + await forkWorkspaceDrive({ + name: probeDrive, + region: REGION, + goldenDriveName: goldenDrive, + drives, + }); +} log(`probe: provisioning ${probe} as a fork of ${golden}`); await provisionSandbox({ name: probe, environment: { sourceKind: "fork", sourceRef: golden, region: REGION }, claim: probeClaim, + mounts: probeMounts, +}); +await wakeSandbox({ + providerSandboxId: probe, + claim: probeClaim, + mounts: probeMounts, }); -await wakeSandbox({ providerSandboxId: probe, claim: probeClaim }); let probeFailed = await probeBox({ name: probe, credentials, @@ -500,14 +523,18 @@ if (ENV_FILE) { { const box = await Sandbox.get({ ...credentials, name: probe }); await box.stop(); - await wakeSandbox({ providerSandboxId: probe, claim: probeClaim }); + await wakeSandbox({ + providerSandboxId: probe, + claim: probeClaim, + mounts: probeMounts, + }); probeFailed += await checkWakeLog({ name: probe, credentials, log }); } if (probeFailed) fail( `${probeFailed} probe check(s) failed; ${golden} and ${probe} left for inspection`, ); -await deleteSandbox(probe); +await deleteSandboxAndDrive(probe, probeDrive, drives); log(`probe: ${probe} deleted`); if (!SKIP_IMAGE) { @@ -560,6 +587,7 @@ const fromGolden = { sourceKind: "fork" as const, sourceRef: golden, region: REGION, + goldenDriveName: goldenDrive, bundleSha: bundle.sha256, primaryRepositoryId: monorepo.id, archivedAt: null, @@ -649,6 +677,12 @@ if ( } else { await deleteSandbox(previous.sourceRef); log(`previous golden ${previous.sourceRef} deleted`); + const { deleteGoldenDriveIfUnreferenced } = await import( + "@superset/trpc/lib/sandbox/golden-drives" + ); + await deleteGoldenDriveIfUnreferenced(previous.goldenDriveName, drives); + if (previous.goldenDriveName) + log(`previous golden drive ${previous.goldenDriveName} deleted`); } } log("release done"); diff --git a/packages/sandbox/src/real-sandbox.ts b/packages/sandbox/src/real-sandbox.ts index 1ce5208b71a..bc5ed2d9851 100644 --- a/packages/sandbox/src/real-sandbox.ts +++ b/packages/sandbox/src/real-sandbox.ts @@ -5,10 +5,26 @@ * built here from a throwaway identity. * * bun run src/real-sandbox.ts (VERCEL_SANDBOX_*, SANDBOX_GATE_SECRET, CDN_R2_*) + * REAL_SANDBOX_DRIVE=1 bun run src/real-sandbox.ts * * Publishes the bundle first so the box can fetch it if the image is older, * and its agent-tools drive, which the box mounts. - * Leaves the sandbox up for inspection on failure; deletes it on success. + * + * With REAL_SANDBOX_DRIVE=1, whatever the first box's result, the same work + * runs on a box in drive mode and on a plain box, side by side: the checkout, + * a real `bun install`, REAL_SANDBOX_WORK_GB (default 4) GB written under + * /workspace, persistence sentinels, a stop, a wake, a second stop. The drive + * box also checks the layout, the stores and Docker on the drive. Then + * it is stopped uncleanly (host-service killed, writers still going) and must + * boot with host.db and the checkout intact, and it is promoted to a + * drive-mode golden that a new workspace is created from on a new branch, the + * production path. One table compares them (drive-comparison.ts). A wake + * that is not served from a warm snapshot cache cannot be forced from here. + * + * Every box and drive it made is deleted at the end, pass or fail: the box + * stopped first, each drive once detached. REAL_SANDBOX_KEEP=1 keeps the + * boxes (stopped) and the box drive for inspection instead. Exits 1 when + * any check fails. */ import { randomUUID } from "node:crypto"; @@ -16,10 +32,24 @@ import { GOLDEN_BOX_VCPUS } from "@superset/shared/billing"; import { SANDBOX_IMAGE_NAME } from "@superset/shared/constants"; import { SANDBOX_CONTRACT_VERSION, + SANDBOX_PATHS, + SANDBOX_USER, type SandboxIdentity, } from "@superset/shared/sandbox/contract"; import { DEFAULT_SANDBOX_REGION } from "@superset/shared/sandbox/regions"; -import { Sandbox } from "@vercel/sandbox"; +import { Sandbox, Snapshot } from "@vercel/sandbox"; +import { + type BoxMeasures, + type BoxRun, + doWork, + readSentinels, + renderComparison, + SENTINELS, + type StopMeasure, + usableSince, + waitForCheckout, + writeSentinels, +} from "./drive-comparison"; process.env.SKIP_ENV_VALIDATION ??= "1"; @@ -71,9 +101,14 @@ if (gaps.length) { process.exit(1); } -const { deleteSandbox, provisionSandbox, wakeSandbox } = await import( - "@superset/trpc/lib/sandbox/vercel" -); +const { + deleteSandbox, + deleteSandboxAndDrive, + promoteSandboxToEnvironment, + provisionSandbox, + stopSandbox, + wakeSandbox, +} = await import("@superset/trpc/lib/sandbox/vercel"); const { sandboxHostSecretFor } = await import( "@superset/trpc/lib/sandbox/access" ); @@ -111,62 +146,616 @@ const claim = { agentTools: true, }; -log( - `provisioning ${name} from ${SANDBOX_IMAGE_NAME} on bundle ${bundle.sha256.slice(0, 12)}`, -); // A freshly pushed image sits in `Preparing` while the registry optimises // it, and create answers 409 until then. -const deadline = Date.now() + 20 * 60_000; -for (;;) { - try { - await provisionSandbox({ - name, - environment: { - sourceKind: "image", - sourceRef: SANDBOX_IMAGE_NAME, - region: DEFAULT_SANDBOX_REGION, - }, - claim, - }); - break; - } catch (error) { - const notReady = - String(error).includes("409") || - String(error).includes("image_not_ready"); - if (!notReady || Date.now() > deadline) throw error; - log("image still preparing in the registry, waiting"); - await new Promise((resolve) => setTimeout(resolve, 15_000)); +async function provisionFromImage( + box: string, + mounts?: Parameters[0]["mounts"], + boxClaim: Parameters[0]["claim"] = claim, +): Promise { + const deadline = Date.now() + 20 * 60_000; + for (;;) { + try { + await provisionSandbox({ + name: box, + environment: { + sourceKind: "image", + sourceRef: SANDBOX_IMAGE_NAME, + region: DEFAULT_SANDBOX_REGION, + }, + claim: boxClaim, + mounts, + }); + return; + } catch (error) { + const notReady = + String(error).includes("409") || + String(error).includes("image_not_ready"); + if (!notReady || Date.now() > deadline) throw error; + log("image still preparing in the registry, waiting"); + await new Promise((resolve) => setTimeout(resolve, 15_000)); + } } } -const woken = await wakeSandbox({ providerSandboxId: name, claim }); -if (!woken) throw new Error(`${name} did not wake`); + log( - `host-service answered at ${woken.hostTarget} (${Date.now() - started} ms since start)`, + `provisioning ${name} from ${SANDBOX_IMAGE_NAME} on bundle ${bundle.sha256.slice(0, 12)}`, ); -let failed = await probeBox({ - name, - credentials, - hostSecret, - bundleSha: bundle.sha256, - branch: workingBranch, - primaryPath: ".", - gate: process.env.SANDBOX_GATE_ORIGIN - ? { workspaceId, userId: randomUUID() } - : null, - log, -}); -{ +const withDrivePass = process.env.REAL_SANDBOX_DRIVE === "1"; +const keep = process.env.REAL_SANDBOX_KEEP === "1"; +let failed = 0; +try { + await provisionFromImage(name); + const woken = await wakeSandbox({ providerSandboxId: name, claim }); + if (!woken) throw new Error(`${name} did not wake`); + log( + `host-service answered at ${woken.hostTarget} (${Date.now() - started} ms since start)`, + ); + failed += await probeBox({ + name, + credentials, + hostSecret, + bundleSha: bundle.sha256, + branch: workingBranch, + primaryPath: ".", + gate: process.env.SANDBOX_GATE_ORIGIN + ? { workspaceId, userId: randomUUID() } + : null, + log, + }); const box = await Sandbox.get({ ...credentials, name }); await box.stop(); const wakeStarted = Date.now(); await wakeSandbox({ providerSandboxId: name, claim }); log(`woken again in ${Date.now() - wakeStarted} ms`); failed += await checkWakeLog({ name, credentials, log }); +} catch (error) { + console.error( + `check of ${name} stopped early: ${String(error).slice(0, 300)}`, + ); + failed++; } -if (failed) { - console.error(`${failed} check(s) failed; ${name} left for inspection`); +if (keep) { + await stopSandbox(name) + .then(() => log(`${name} stopped and kept (REAL_SANDBOX_KEEP=1)`)) + .catch((error) => log(`could not stop ${name}: ${error}`)); +} else { + await stopSandbox(name) + .then(() => deleteSandbox(name)) + .then(() => log(`${name} deleted`)) + .catch((error) => log(`could not delete ${name}: ${error}`)); +} + +const driveFailures = withDrivePass ? await drivePass() : 0; +if (failed || driveFailures) { + console.error( + `${failed} check(s) and ${driveFailures} drive check(s) failed`, + ); process.exit(1); } -await deleteSandbox(name); -log(`${name} deleted; all checks passed`); +log("all checks passed"); process.exit(0); + +async function drivePass(): Promise { + const { + forkWorkspaceDrive, + vercelDrives, + waitForDriveDetached, + workspaceDriveIdentity, + workspaceDriveMounts, + goldenDriveNameFor, + workspaceDriveNameFor, + } = await import("@superset/trpc/lib/sandbox/drives"); + + const drives = vercelDrives(); + const workGb = Number(process.env.REAL_SANDBOX_WORK_GB) || 4; + const suffix = Date.now().toString(36); + const goldenDrive = `env-drive-real-check-${suffix}`; + const box = `ws-real-drive-${suffix}`; + const plainBox = `ws-real-plain-${suffix}`; + const goldenName = `env-real-check-${suffix}`; + const goldenBoxDrive = goldenDriveNameFor(goldenName); + const forkBox = `ws-real-golden-${suffix}`; + const boxDrive = workspaceDriveNameFor(box); + const forkDrive = workspaceDriveNameFor(forkBox); + const mounts = workspaceDriveMounts(boxDrive); + const claimWith = (driveName: string | null, laidOut: boolean) => ({ + ...claim, + identity: { + ...identity, + ...workspaceDriveIdentity(driveName, { laidOut }), + }, + }); + const drive = SANDBOX_PATHS.drive; + const expectations = [ + "sandbox.conf names the drive", + "layout stamped", + "host-service ready", + "/workspace is a bind mount of the drive", + "the checkout landed on the drive", + "docker info reports the drive data-root", + "the checkout is still there after a wake", + "the layout was redone on the wake", + "/workspace is a mount point again", + "bun install on the plain box", + "bun install on the drive box", + ...SENTINELS.map((name) => `plain box kept the ${name}`), + ...SENTINELS.map((name) => `drive box kept the ${name}`), + "the drive box's root snapshot is smaller than the plain box's", + "unclean stop: the box boots again", + "unclean stop: host.db passes integrity_check", + "unclean stop: the checkout is intact", + "a workspace of a drive-mode golden boots on its new branch", + "a workspace of a drive-mode golden has the golden's dependencies", + ]; + const reported = new Set(); + let failures = 0; + const check = (label: string, ok: boolean, detail = "") => { + reported.add(label); + log( + `${ok ? "ok " : "FAIL"} drive: ${label}${detail ? `: ${detail.replaceAll("\n", " | ")}` : ""}`, + ); + if (!ok) failures++; + }; + // Only stdout is compared: sudo on the box warns on stderr that it cannot resolve the hostname. + const runOn = + (name: string): BoxRun => + async (command, options = {}) => { + const sandbox = await Sandbox.get({ ...credentials, name }); + const script = Buffer.from(command).toString("base64"); + const result = await sandbox.runCommand({ + cmd: "bash", + args: options.root + ? ["-lc", command] + : [ + "-c", + `echo ${script} | base64 -d | runuser -u ${SANDBOX_USER} -- bash -l`, + ], + sudo: true, + }); + const out = (await result.stdout()).trim(); + const err = (await result.stderr()).trim(); + return { + out, + detail: [out, err && `stderr: ${err}`, `exit=${result.exitCode}`] + .filter(Boolean) + .join("\n"), + }; + }; + const run = runOn(box); + const root = (command: string) => run(command, { root: true }); + const layouts = async () => + Number( + ( + await root( + `grep -c 'drive\\.layout ms=' ${SANDBOX_PATHS.bootLog} || true`, + ) + ).out, + ); + const diagnose = async (when: string, on: BoxRun = run) => { + const report = await on( + [ + `echo '--- findmnt ${drive} and /workspace'`, + `findmnt -no SOURCE,MAJ:MIN,FSROOT,OPTIONS ${drive} ${SANDBOX_PATHS.workspace}`, + `echo '--- ls ${drive} ${drive}/workspace ${drive}/state'`, + `ls -la ${drive} ${drive}/workspace ${drive}/state 2>&1 | head -60`, + `echo '--- ${SANDBOX_PATHS.conf} drive keys'`, + `grep SUPERSET_SANDBOX_DRIVE ${SANDBOX_PATHS.conf}`, + `echo '--- run dir'`, + `ls ${SANDBOX_PATHS.run}`, + `echo '--- /etc/docker/daemon.json'`, + `cat /etc/docker/daemon.json`, + `echo '--- df ${drive}'`, + `df -h ${drive} ${SANDBOX_PATHS.workspace} 2>&1`, + `echo '--- git status of /workspace'`, + `runuser -u ${SANDBOX_USER} -- git -C ${SANDBOX_PATHS.workspace} status --short --branch 2>&1 | head -20`, + `echo '--- checkout.log'`, + `tail -30 ${SANDBOX_PATHS.logs}/checkout.log`, + `echo '--- boot log'`, + `cat ${SANDBOX_PATHS.bootLog}`, + ].join("; "), + { root: true }, + ).catch((error) => ({ detail: `diagnosis failed: ${error}` })); + log(`drive: diagnosis ${when}:\n${report.detail}`); + }; + + const stopMeasured = async (name: string): Promise => { + const before = (await Sandbox.get({ ...credentials, name, resume: false })) + .currentSnapshotId; + const stopStarted = Date.now(); + await stopSandbox(name); + while (Date.now() - stopStarted < 10 * 60_000) { + const now = await Sandbox.get({ ...credentials, name, resume: false }); + if ( + now.status === "stopped" && + now.currentSnapshotId && + now.currentSnapshotId !== before + ) { + const ms = Date.now() - stopStarted; + const snapshot = await Snapshot.get({ + ...credentials, + snapshotId: now.currentSnapshotId, + }); + return { ms, snapshotBytes: snapshot.sizeBytes }; + } + await new Promise((resolve) => setTimeout(resolve, 1000)); + } + throw new Error(`${name} did not snapshot within 10 minutes of its stop`); + }; + + const measures: Record<"plain" | "drive" | "golden", BoxMeasures> = { + plain: {}, + drive: {}, + golden: {}, + }; + const token = randomUUID().replaceAll("-", ""); + + /** Create, work, stop, wake, read the sentinels back, stop again. */ + const measureBox = async (args: { + name: string; + label: "plain" | "drive"; + provisionClaim: typeof claim; + wakeClaim: typeof claim; + mounts?: typeof mounts; + afterFirstBoot?: () => Promise; + beforeStop?: () => Promise; + afterWake?: () => Promise; + }) => { + const m = measures[args.label]; + const boxRun = runOn(args.name); + const createStarted = Date.now(); + await provisionFromImage(args.name, args.mounts, args.provisionClaim); + await wakeSandbox({ + providerSandboxId: args.name, + claim: args.provisionClaim, + mounts: args.mounts, + }); + m.createToReadyMs = Date.now() - createStarted; + if (!(await waitForCheckout(boxRun))) { + throw new Error(`${args.name}: the checkout never became ready`); + } + await args.afterFirstBoot?.(); + const work = await doWork(boxRun, workGb); + m.installMs = work.installMs; + m.writeMs = work.writeMs; + check( + `bun install on the ${args.label} box`, + work.installed === "installed", + work.installed, + ); + const values = await writeSentinels(boxRun, token); + await args.beforeStop?.(); + m.firstStop = await stopMeasured(args.name); + log(`drive: ${args.label} box stopped once`); + const phases: Partial> = {}; + const wakeStarted = Date.now(); + await wakeSandbox({ + providerSandboxId: args.name, + claim: args.wakeClaim, + mounts: args.mounts, + onPhase: (phase, ms) => { + phases[phase] = (phases[phase] ?? 0) + ms; + }, + }); + m.wake = { + detachMs: phases.detach, + mountMs: phases.mount, + resumeMs: phases.resume, + bootMs: phases.boot, + totalMs: Date.now() - wakeStarted, + }; + m.usableMs = await usableSince(boxRun, wakeStarted); + m.sentinels = await readSentinels(boxRun, values); + for (const name of SENTINELS) { + check(`${args.label} box kept the ${name}`, m.sentinels[name] === true); + } + await args.afterWake?.(); + m.secondStop = await stopMeasured(args.name); + log(`drive: ${args.label} box stopped twice`); + }; + + const stoppedAt: string[] = []; + const plain = measureBox({ + name: plainBox, + label: "plain", + provisionClaim: claim, + wakeClaim: claim, + }).catch((error) => { + stoppedAt.push(`plain box: ${String(error).slice(0, 300)}`); + }); + const driveBox = (async () => { + const forkStarted = Date.now(); + await drives.create(goldenDrive, DEFAULT_SANDBOX_REGION); + await forkWorkspaceDrive({ + name: boxDrive, + region: DEFAULT_SANDBOX_REGION, + goldenDriveName: goldenDrive, + drives, + }); + measures.drive.forkMs = Date.now() - forkStarted; + log(`drive: ${boxDrive} forked from ${goldenDrive}; provisioning ${box}`); + let before = 0; + await measureBox({ + name: box, + label: "drive", + provisionClaim: claimWith(boxDrive, false), + wakeClaim: claimWith(boxDrive, true), + mounts, + afterFirstBoot: async () => { + const conf = await root( + `grep -c "^SUPERSET_SANDBOX_DRIVE='${boxDrive}'$" ${SANDBOX_PATHS.conf}`, + ); + check("sandbox.conf names the drive", conf.out === "1", conf.detail); + const bootLog = (await root(`cat ${SANDBOX_PATHS.bootLog}`)).out; + check( + "layout stamped", + /drive\.layout ms=\d+/.test(bootLog), + bootLog.match(/drive\.(layout|missing|foreign|unlaid).*/)?.[0], + ); + check("host-service ready", /host\.ready/.test(bootLog)); + // A bind mount of a subdirectory: the drive's device, at fsroot /workspace. + const backing = await root( + `findmnt -rno MAJ:MIN ${drive}; findmnt -rno MAJ:MIN,FSROOT ${SANDBOX_PATHS.workspace}`, + ); + const [device, workspaceMount] = backing.out.split("\n"); + check( + "/workspace is a bind mount of the drive", + Boolean(device) && workspaceMount === `${device} /workspace`, + backing.detail, + ); + const checkout = await root( + `test -d ${drive}/workspace/.git && echo on-drive`, + ); + check( + "the checkout landed on the drive", + checkout.out === "on-drive", + checkout.detail, + ); + const dockerRoot = await root( + `(dockerd > /tmp/dockerd.log 2>&1 &); for i in $(seq 1 60); do docker info --format '{{.DockerRootDir}}' 2>/dev/null && exit 0; sleep 1; done; tail -5 /tmp/dockerd.log`, + ); + check( + "docker info reports the drive data-root", + dockerRoot.out === `${drive}/docker`, + dockerRoot.detail, + ); + if (failures) await diagnose("after the first boot"); + }, + beforeStop: async () => { + before = await layouts(); + }, + afterWake: async () => { + const kept = await run( + `git -C ${SANDBOX_PATHS.workspace} rev-parse --is-inside-work-tree`, + ); + check( + "the checkout is still there after a wake", + kept.out === "true", + kept.detail, + ); + const after = await layouts(); + check( + "the layout was redone on the wake", + after > before, + `${before} layout stamps before the wake, ${after} after`, + ); + const mounted = await root( + `mountpoint ${SANDBOX_PATHS.workspace} && echo yes`, + ); + check( + "/workspace is a mount point again", + mounted.out.endsWith("yes"), + mounted.detail, + ); + }, + }); + })().catch((error) => { + stoppedAt.push(`drive box: ${String(error).slice(0, 300)}`); + return "failed" as const; + }); + const [, driveOutcome] = await Promise.all([plain, driveBox]); + + if (driveOutcome !== "failed") { + // An unclean stop: host-service killed, writers into host.db and /workspace still going. + try { + await wakeSandbox({ + providerSandboxId: box, + claim: claimWith(boxDrive, true), + mounts, + }); + const writer = Buffer.from( + [ + 'const { Database } = require("bun:sqlite");', + 'process.on("SIGTERM", () => {});', + "const db = new Database(process.argv[2]);", + 'db.exec("pragma journal_mode=wal; create table if not exists superset_probe (i integer, at integer)");', + 'for (let i = 0; ; i++) db.run("insert into superset_probe values (?, ?)", [i, Date.now()]);', + ].join("\n"), + ).toString("base64"); + await run( + `echo ${writer} | base64 -d > /tmp/superset-probe-writer.js && (setsid nohup bun /tmp/superset-probe-writer.js ${SANDBOX_PATHS.driveState}/host.db > /tmp/superset-probe-writer.log 2>&1 &) && (setsid nohup bash -c "trap '' TERM; while true; do head -c 1048576 /dev/urandom >> ${SANDBOX_PATHS.workspace}/.superset-unclean; done" > /dev/null 2>&1 &)`, + ); + await root("sleep 3; pkill -9 -f host-service.js; true"); + await stopMeasured(box); + await wakeSandbox({ + providerSandboxId: box, + claim: claimWith(boxDrive, true), + mounts, + }); + check("unclean stop: the box boots again", true); + const integrity = await run( + `bun -e 'const { Database } = require("bun:sqlite"); console.log(Object.values(new Database(process.argv[1], { readonly: true }).query("pragma integrity_check").get() ?? {})[0])' ${SANDBOX_PATHS.driveState}/host.db`, + ); + check( + "unclean stop: host.db passes integrity_check", + integrity.out === "ok", + integrity.detail, + ); + const checkout = await run( + `git -C ${SANDBOX_PATHS.workspace} fsck --connectivity-only --no-progress > /dev/null && git -C ${SANDBOX_PATHS.workspace} status --porcelain > /dev/null && echo intact`, + ); + check( + "unclean stop: the checkout is intact", + checkout.out === "intact", + checkout.detail, + ); + } catch (error) { + stoppedAt.push(`unclean stop: ${String(error).slice(0, 300)}`); + } + + // The production path: promote the drive box to a drive-mode golden, then a workspace of it. + try { + const golden = await promoteSandboxToEnvironment({ + sourceSandbox: box, + goldenName, + organizationId: identity.SUPERSET_SANDBOX_ORGANIZATION_ID, + sourceDriveName: boxDrive, + drives, + }); + await stopSandbox(box); + const forkWorkspaceId = randomUUID(); + const forkBranch = `superset/real-check-golden-${forkWorkspaceId.slice(0, 8)}`; + const forkClaim = { + ...claim, + hostSecret: await sandboxHostSecretFor(forkWorkspaceId), + identity: { + ...identity, + SUPERSET_SANDBOX_WORKSPACE_ID: forkWorkspaceId, + SUPERSET_SANDBOX_REPOSITORIES: JSON.stringify([ + { + url: "https://github.com/paradevai/paradev.git", + branch: forkBranch, + baseBranch: "main", + path: ".", + }, + ]), + SUPERSET_SANDBOX_IMAGE_TAG: goldenName, + ...workspaceDriveIdentity(forkDrive, { laidOut: false }), + }, + }; + const forkMounts = workspaceDriveMounts(forkDrive); + const createStarted = Date.now(); + await forkWorkspaceDrive({ + name: forkDrive, + region: golden.region, + goldenDriveName: golden.goldenDriveName, + drives, + }); + measures.golden.forkMs = Date.now() - createStarted; + await provisionSandbox({ + name: forkBox, + environment: { + sourceKind: "fork", + sourceRef: goldenName, + region: golden.region, + }, + claim: forkClaim, + mounts: forkMounts, + }); + await wakeSandbox({ + providerSandboxId: forkBox, + claim: forkClaim, + mounts: forkMounts, + }); + measures.golden.createToReadyMs = Date.now() - createStarted; + const forkRun = runOn(forkBox); + measures.golden.usableMs = await usableSince(forkRun, createStarted); + await waitForCheckout(forkRun); + const branch = await forkRun( + `git -C ${SANDBOX_PATHS.workspace} rev-parse --abbrev-ref HEAD`, + ); + check( + "a workspace of a drive-mode golden boots on its new branch", + branch.out === forkBranch, + branch.detail, + ); + const deps = await forkRun( + `test -d ${SANDBOX_PATHS.workspace}/node_modules && echo present`, + ); + check( + "a workspace of a drive-mode golden has the golden's dependencies", + deps.out === "present", + deps.detail, + ); + if (deps.out !== "present") { + await diagnose("of the workspace of the drive-mode golden", forkRun); + } + } catch (error) { + stoppedAt.push(`golden path: ${String(error).slice(0, 300)}`); + } + } + const plainSnapshot = measures.plain.firstStop?.snapshotBytes; + const driveSnapshot = measures.drive.firstStop?.snapshotBytes; + if (plainSnapshot !== undefined && driveSnapshot !== undefined) { + check( + "the drive box's root snapshot is smaller than the plain box's", + driveSnapshot < plainSnapshot, + `${(driveSnapshot / 1024 ** 3).toFixed(2)} GB against ${(plainSnapshot / 1024 ** 3).toFixed(2)} GB`, + ); + } + for (const label of expectations) { + if (!reported.has(label)) { + check( + label, + false, + `not reached${stoppedAt.length ? `: ${stoppedAt.join("; ")}` : ""}`, + ); + } + } + if (failures) await diagnose("at the end"); + log( + `drive: comparison, ${workGb} GB written under /workspace\n${renderComparison( + [ + { label: "plain", measures: measures.plain }, + { label: "drive", measures: measures.drive }, + { label: "workspace of a drive golden", measures: measures.golden }, + ], + )}`, + ); + + const cleanups: Array<[string, () => Promise]> = keep + ? [ + [plainBox, () => stopSandbox(plainBox)], + [box, () => stopSandbox(box)], + [forkBox, () => stopSandbox(forkBox)], + [goldenName, () => stopSandbox(goldenName)], + ] + : [ + [ + plainBox, + () => stopSandbox(plainBox).then(() => deleteSandbox(plainBox)), + ], + [ + `${box} and ${boxDrive}`, + () => deleteSandboxAndDrive(box, boxDrive, drives), + ], + [ + `${forkBox} and ${forkDrive}`, + () => deleteSandboxAndDrive(forkBox, forkDrive, drives), + ], + [ + `${goldenName} and ${goldenBoxDrive}`, + () => deleteSandboxAndDrive(goldenName, goldenBoxDrive, drives), + ], + ]; + for (const [what, cleanup] of cleanups) { + await cleanup() + .then(() => + log( + `drive: ${what} ${keep ? "stopped and kept (REAL_SANDBOX_KEEP=1)" : "deleted"}`, + ), + ) + .catch((error) => { + failures++; + log(`FAIL drive: could not clean up ${what}: ${error}`); + }); + } + await waitForDriveDetached(goldenDrive, { drives }) + .then(() => drives.remove(goldenDrive)) + .then(() => log(`drive: ${goldenDrive} deleted`)) + .catch((error) => { + failures++; + log(`FAIL drive: could not delete ${goldenDrive}: ${error}`); + }); + return failures; +} diff --git a/packages/shared/src/constants.ts b/packages/shared/src/constants.ts index 76cdb18e648..07eb433703b 100644 --- a/packages/shared/src/constants.ts +++ b/packages/shared/src/constants.ts @@ -206,6 +206,13 @@ export const FEATURE_FLAGS = { SANDBOX_BOX_LOGS: "sandbox-box-logs", /** One answer for every box (asked for the subject `sandbox-fleet`, no group): off stops box logs at each box's next wake. */ SANDBOX_BOX_LOGS_FLEET: "sandbox-box-logs-fleet", + /** + * Lets a golden build or a promote put an organization's environment in + * drive mode (`environments.golden_drive_name`). Evaluated per organization. + * Workspaces follow their environment, never this flag. Unloaded or + * offline means no drive. + */ + CLOUD_WORKSPACE_DRIVES: "cloud-workspace-drives", /** * Moving a workspace and its chats to the cloud or to another computer: * shows the Move items, and gates diff --git a/packages/shared/src/sandbox/contract.ts b/packages/shared/src/sandbox/contract.ts index a383b7579a5..2334b528a99 100644 --- a/packages/shared/src/sandbox/contract.ts +++ b/packages/shared/src/sandbox/contract.ts @@ -37,8 +37,56 @@ export const SANDBOX_PATHS = { backgrounds: "/usr/share/backgrounds/superset", /** A read-only snapshot of the region's agent-tools drive, when one is mounted. */ agents: "/opt/agents", + /** The workspace's own drive, read-write, when its environment is in drive mode. */ + drive: "/mnt/superset-drive", + /** + * In drive mode, what describes the drive's contents lives on the drive, so + * a root disk that rolls back cannot disagree with it: `host.db`, the + * checkout markers and `workspace-id`, the drive's owner. + */ + driveState: "/mnt/superset-drive/state", } as const; +/** + * Home subtrees that live on the workspace drive, relative to the home + * directory. Each is bind-mounted from `/home/`. + */ +export const SANDBOX_DRIVE_HOME_SUBTREES = [ + ".cache", + ".local/share", + ".local/state", + ".npm", + ".cargo/registry", + ".rustup", + ".claude", + ".codex/sessions", + "go/pkg", +] as const; + +/** Package stores inside the `/workspace` bind mount: a hardlink across two mounts fails with EXDEV. */ +export const SANDBOX_DRIVE_STORES = { + bun: `${SANDBOX_PATHS.workspace}/.superset/stores/bun`, + pnpm: `${SANDBOX_PATHS.workspace}/.superset/stores/pnpm`, +} as const; + +/** + * Where a path of the laid-out box lives on the drive while no boot has + * bind-mounted it; null for a path on the root disk. + */ +export function sandboxDrivePathOf(path: string): string | null { + const workspace = SANDBOX_PATHS.workspace; + if (path === workspace || path.startsWith(`${workspace}/`)) { + return `${SANDBOX_PATHS.drive}/workspace${path.slice(workspace.length)}`; + } + for (const subtree of SANDBOX_DRIVE_HOME_SUBTREES) { + const mounted = `${SANDBOX_PATHS.home}/${subtree}`; + if (path === mounted || path.startsWith(`${mounted}/`)) { + return `${SANDBOX_PATHS.drive}/home${path.slice(SANDBOX_PATHS.home.length)}`; + } + } + return null; +} + /** * The agent CLIs of one bundle in one region: written once by the release, * then only ever mounted read-only. A drive only mounts in its own region. @@ -111,6 +159,10 @@ export const sandboxIdentitySchema = z.object({ /** The environment row's source (image name or golden), for telemetry. */ SUPERSET_SANDBOX_IMAGE_TAG: z.string().min(1), SUPERSET_SANDBOX_PROVIDER: z.string().min(1), + /** The workspace drive's name; the box refuses to boot when it is not mounted. */ + SUPERSET_SANDBOX_DRIVE: z.string().min(1).optional(), + /** Set after the provision: the drive must already carry this workspace's id. */ + SUPERSET_SANDBOX_DRIVE_LAID_OUT: z.literal("1").optional(), HOST_SERVICE_SENTRY_DSN: z.string().optional(), HOST_SERVICE_SENTRY_ENVIRONMENT: z.string().optional(), /** Starts `superset-logship`, which sends the platform's box logs to Sentry. Absent is off. */ @@ -239,6 +291,11 @@ export function renderContractShell(): string { SUPERSET_DESKTOP_TEMPLATES: SANDBOX_PATHS.desktopTemplates, SUPERSET_BACKGROUNDS_DIR: SANDBOX_PATHS.backgrounds, SUPERSET_AGENTS_DIR: SANDBOX_PATHS.agents, + SUPERSET_DRIVE_DIR: SANDBOX_PATHS.drive, + SUPERSET_DRIVE_STATE_DIR: SANDBOX_PATHS.driveState, + SUPERSET_DRIVE_HOME_SUBTREES: SANDBOX_DRIVE_HOME_SUBTREES.join(" "), + SUPERSET_BUN_STORE: SANDBOX_DRIVE_STORES.bun, + SUPERSET_PNPM_STORE: SANDBOX_DRIVE_STORES.pnpm, SUPERSET_HOST_SERVICE_PORT: SANDBOX_PORTS.hostService, SUPERSET_DESKTOP_PORT: SANDBOX_PORTS.desktop, SUPERSET_VNC_PORT: SANDBOX_PORTS.vnc, diff --git a/packages/trpc/package.json b/packages/trpc/package.json index c444ba70db4..58e052f41e0 100644 --- a/packages/trpc/package.json +++ b/packages/trpc/package.json @@ -136,6 +136,10 @@ "types": "./src/router/cloud-workspace/reap.ts", "default": "./src/router/cloud-workspace/reap.ts" }, + "./cloud-workspace-teardown": { + "types": "./src/router/cloud-workspace/teardown.ts", + "default": "./src/router/cloud-workspace/teardown.ts" + }, "./automation-dispatch": { "types": "./src/router/automation/dispatch.ts", "default": "./src/router/automation/dispatch.ts" diff --git a/packages/trpc/src/lib/sandbox/claim.test.ts b/packages/trpc/src/lib/sandbox/claim.test.ts new file mode 100644 index 00000000000..e8b57417791 --- /dev/null +++ b/packages/trpc/src/lib/sandbox/claim.test.ts @@ -0,0 +1,112 @@ +import { describe, expect, test } from "bun:test"; +import { stub } from "../../../test/stub"; +import * as environments from "../../router/environment/resolve-environment"; +import * as plan from "../billing/plan"; +import * as featureFlags from "../feature-flags"; +import * as access from "./access"; +import { buildSandboxClaim } from "./claim"; +import * as credentials from "./credentials"; +import * as drives from "./drives"; +import * as plugins from "./plugins"; +import * as repositories from "./repositories"; + +let goldenDriveName: string | null = null; +let drivesOn = true; + +stub(environments, { + resolveEnvironment: async () => ({ + id: "env-1", + provider: "vercel", + sourceKind: "fork", + sourceRef: "env-golden", + region: "iad1", + bundleSha: "a".repeat(64), + primaryRepositoryId: null, + goldenDriveName, + envs: {}, + }), +}); +stub(repositories, { + workspaceRepositories: async () => [], + installationTokenFor: async () => null, +}); +stub(plugins, { creatorPlugins: async () => [] }); +stub(access, { sandboxHostSecretFor: async () => "secret" }); +stub(featureFlags, { + organizationFlagEnabled: async () => false, + boxLogsEnabled: async () => false, +}); +stub(credentials, { + deriveSandboxCredentials: async () => ({ + networkPolicy: "allow-all", + managedEnv: {}, + files: [], + }), +}); +stub(plan, { resolveMachineSize: async () => ({ vcpus: 4 }) }); +stub(drives, { workspaceDrivesEnabled: async () => drivesOn }); + +const row = (driveName: string | null, driveLaidOutAt: Date | null = null) => + ({ + id: "00000000-0000-4000-8000-000000000001", + organizationId: "00000000-0000-4000-8000-000000000002", + environmentId: "00000000-0000-4000-8000-000000000003", + createdByUserId: null, + providerSandboxId: "ws-1", + provider: "vercel", + baseBranch: "main", + branch: "superset/ws-1", + machineSize: null, + driveName, + driveLaidOutAt, + }) as unknown as Parameters[0]["row"]; + +describe("buildSandboxClaim drive", () => { + test("an existing workspace gets no drive on a wake when its environment enters drive mode", async () => { + goldenDriveName = "env-drive-env-golden"; + const { claim } = await buildSandboxClaim({ row: row(null) }); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE).toBeUndefined(); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT).toBeUndefined(); + }); + + test("a provision in drive mode claims a new drive, not yet laid out", async () => { + goldenDriveName = "env-drive-env-golden"; + const { claim } = await buildSandboxClaim({ + row: row(null), + provisioning: true, + }); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE).toBe("drive-ws-1"); + expect(claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT).toBeUndefined(); + }); + + test("a wake expects the drive laid out only once the row recorded a boot that laid it out", async () => { + goldenDriveName = null; + const laidOut = await buildSandboxClaim({ + row: row("drive-ws-1", new Date()), + }); + expect(laidOut.claim.identity.SUPERSET_SANDBOX_DRIVE).toBe("drive-ws-1"); + expect(laidOut.claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT).toBe("1"); + const never = await buildSandboxClaim({ row: row("drive-ws-1") }); + expect(never.claim.identity.SUPERSET_SANDBOX_DRIVE).toBe("drive-ws-1"); + expect( + never.claim.identity.SUPERSET_SANDBOX_DRIVE_LAID_OUT, + ).toBeUndefined(); + }); + + test("with the organization's flag off a new workspace gets no drive; one that has a drive keeps it", async () => { + goldenDriveName = "env-drive-env-golden"; + drivesOn = false; + const fresh = await buildSandboxClaim({ + row: row(null), + provisioning: true, + }); + expect(fresh.driveName).toBeNull(); + expect(fresh.claim.identity.SUPERSET_SANDBOX_DRIVE).toBeUndefined(); + const unarchived = await buildSandboxClaim({ + row: row("drive-ws-1"), + provisioning: true, + }); + expect(unarchived.driveName).toBe("drive-ws-1"); + drivesOn = true; + }); +}); diff --git a/packages/trpc/src/lib/sandbox/claim.ts b/packages/trpc/src/lib/sandbox/claim.ts index c89ecfe46b6..4de5983b2cc 100644 --- a/packages/trpc/src/lib/sandbox/claim.ts +++ b/packages/trpc/src/lib/sandbox/claim.ts @@ -35,6 +35,7 @@ import { boxLogsEnabled, organizationFlagEnabled } from "../feature-flags"; import { githubUserConnectionFor, githubUserTokenFor } from "../github-user"; import { sandboxHostSecretFor } from "./access"; import { deriveSandboxCredentials, gitAuthorFor } from "./credentials"; +import { claimedWorkspaceDrive, workspaceDrivesEnabled } from "./drives"; import { creatorPlugins } from "./plugins"; import { readRepoHooks } from "./repo-hooks"; import { @@ -87,11 +88,15 @@ export async function buildSandboxClaim(args: { * are fixed once the box exists), and it costs a GitHub request. */ withRepoHooks?: boolean; + /** A create, the only claim that may give the workspace its environment's drive. */ + provisioning?: boolean; }): Promise<{ claim: SandboxClaim; environment: SandboxEnvironment; repositories: SandboxRepository[]; agentCredentialDigest: string; + /** The drive the claim names; what a provision must fork and mount. */ + driveName: string | null; }> { const [environment, signIns, boxLogs] = await Promise.all([ resolveEnvironment(args.row.environmentId, args.row.organizationId), @@ -137,6 +142,18 @@ export async function buildSandboxClaim(args: { : null; const repositories = toSandboxRepositories(checkouts); + const provisioning = args.provisioning ?? false; + const drive = claimedWorkspaceDrive({ + row: args.row, + goldenDriveName: environment.goldenDriveName, + provisioning, + // The kill switch: with the flag off, a new workspace gets no drive. + drivesEnabled: + provisioning && + !args.row.driveName && + Boolean(environment.goldenDriveName) && + (await workspaceDrivesEnabled(args.row.organizationId)), + }); const identity: SandboxIdentity = { SUPERSET_SANDBOX_CONTRACT: String(SANDBOX_CONTRACT_VERSION) as "1", SUPERSET_API_URL: env.NEXT_PUBLIC_API_URL, @@ -147,6 +164,7 @@ export async function buildSandboxClaim(args: { SUPERSET_SANDBOX_PLUGINS: JSON.stringify(plugins), SUPERSET_SANDBOX_IMAGE_TAG: environment.sourceRef, SUPERSET_SANDBOX_PROVIDER: args.row.provider, + ...drive.identity, SUPERSET_BUNDLE_SHA: environment.bundleSha, ...(env.SENTRY_DSN_SANDBOX ? { @@ -194,8 +212,10 @@ export async function buildSandboxClaim(args: { sourceKind: environment.sourceKind, sourceRef: environment.sourceRef, region: environment.region, + goldenDriveName: environment.goldenDriveName, }, repositories, agentCredentialDigest: agentCredentialDigest(signIns), + driveName: drive.driveName, }; } diff --git a/packages/trpc/src/lib/sandbox/drives.test.ts b/packages/trpc/src/lib/sandbox/drives.test.ts new file mode 100644 index 00000000000..5ada5f501b8 --- /dev/null +++ b/packages/trpc/src/lib/sandbox/drives.test.ts @@ -0,0 +1,212 @@ +import { beforeEach, describe, expect, test } from "bun:test"; +import { renderSandboxConf } from "@superset/shared/sandbox/contract"; +import { + type DriveHost, + DriveStillAttachedError, + deleteWorkspaceDrive, + forkWorkspaceDrive, + WORKSPACE_DRIVE_MOUNT_PATH, + waitForDriveDetached, + workspaceDriveIdentity, + workspaceDriveMounts, + workspaceDriveNameToClaim, +} from "./drives"; + +let drives: Map; +let calls: string[]; + +const host: DriveHost = { + find: async (name) => drives.get(name) ?? null, + create: async (name, region) => { + calls.push(`create:${name}@${region}`); + drives.set(name, { region }); + }, + fork: async (source, name) => { + calls.push(`fork:${source}->${name}`); + const parent = drives.get(source); + if (!parent) throw new Error(`no ${source}`); + drives.set(name, { region: parent.region }); + }, + remove: async (name) => { + if (drives.get(name)?.attachedTo) throw new Error(`${name} attached`); + if (drives.delete(name)) calls.push(`delete:${name}`); + }, +}; + +beforeEach(() => { + drives = new Map(); + calls = []; +}); + +describe("workspaceDriveMounts", () => { + test("mounts the drive read-write at one path", () => { + expect(workspaceDriveMounts("drive-ws-1")).toEqual({ + [WORKSPACE_DRIVE_MOUNT_PATH]: { drive: "drive-ws-1", mode: "read-write" }, + }); + }); + + test("mounts nothing for a workspace without a drive", () => { + expect(workspaceDriveMounts(null)).toBeUndefined(); + expect(workspaceDriveMounts(undefined)).toBeUndefined(); + }); +}); + +describe("workspaceDriveIdentity", () => { + const identity = { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "a".repeat(64), + SUPERSET_API_URL: "https://api.example.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "image", + SUPERSET_SANDBOX_PROVIDER: "vercel", + } as const; + + test("a workspace with a drive names it in sandbox.conf, so the box can refuse without it", () => { + expect( + renderSandboxConf({ + ...identity, + ...workspaceDriveIdentity("drive-ws-1", { laidOut: true }), + }), + ).toContain( + "SUPERSET_SANDBOX_DRIVE='drive-ws-1'\nSUPERSET_SANDBOX_DRIVE_LAID_OUT='1'\n", + ); + }); + + test("a workspace without a drive writes no drive key", () => { + expect(workspaceDriveIdentity(null, { laidOut: true })).toEqual({}); + expect( + renderSandboxConf({ + ...identity, + ...workspaceDriveIdentity(undefined, { laidOut: false }), + }), + ).not.toContain("SUPERSET_SANDBOX_DRIVE"); + }); +}); + +describe("workspaceDriveNameToClaim", () => { + test("an environment in drive mode gives a new workspace a drive", () => { + expect( + workspaceDriveNameToClaim({ + existing: null, + goldenDriveName: "env-drive-env-1", + providerSandboxId: "ws-1", + }), + ).toBe("drive-ws-1"); + }); + + test("an environment without a golden drive gives no drive at all", () => { + expect( + workspaceDriveNameToClaim({ + existing: null, + goldenDriveName: null, + providerSandboxId: "ws-1", + }), + ).toBeNull(); + }); + + test("a re-delivered provision keeps the name already on the row", () => { + expect( + workspaceDriveNameToClaim({ + existing: "drive-ws-old", + goldenDriveName: "env-drive-env-1", + providerSandboxId: "ws-new", + }), + ).toBe("drive-ws-old"); + }); +}); + +describe("forkWorkspaceDrive", () => { + test("forks the golden drive", async () => { + drives.set("golden", { region: "iad1" }); + await forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: "golden", + drives: host, + }); + expect(calls).toEqual(["fork:golden->drive-ws-1"]); + }); + + test("finds the drive a previous delivery made", async () => { + drives.set("golden", { region: "iad1" }); + drives.set("drive-ws-1", { region: "iad1" }); + await forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: "golden", + drives: host, + }); + expect(calls).toEqual([]); + }); + + test("refuses a golden in another region", async () => { + drives.set("golden", { region: "iad1" }); + await expect( + forkWorkspaceDrive({ + name: "drive-ws-1", + region: "sfo1", + goldenDriveName: "golden", + drives: host, + }), + ).rejects.toThrow("is in iad1"); + expect(calls).toEqual([]); + }); + + test("never creates an empty drive", async () => { + await expect( + forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: null, + drives: host, + }), + ).rejects.toThrow("no golden drive"); + await expect( + forkWorkspaceDrive({ + name: "drive-ws-1", + region: "iad1", + goldenDriveName: "gone", + drives: host, + }), + ).rejects.toThrow("does not exist"); + expect(calls).toEqual([]); + }); +}); + +describe("waitForDriveDetached", () => { + test("returns once the previous session lets go of the drive", async () => { + let polls = 0; + const flushing: DriveHost = { + ...host, + find: async () => { + polls++; + return { region: "iad1", attachedTo: polls < 3 ? "ws-1" : undefined }; + }, + }; + await waitForDriveDetached("drive-ws-1", { drives: flushing, pollMs: 1 }); + expect(polls).toBe(3); + }); + + test("fails rather than mount a drive another session still writes", async () => { + drives.set("drive-ws-1", { region: "iad1", attachedTo: "ws-1" }); + await expect( + waitForDriveDetached("drive-ws-1", { + drives: host, + timeoutMs: 5, + pollMs: 1, + }), + ).rejects.toBeInstanceOf(DriveStillAttachedError); + }); +}); + +describe("deleteWorkspaceDrive", () => { + test("deletes the drive, and treats one already gone as deleted", async () => { + drives.set("drive-ws-1", { region: "iad1" }); + await deleteWorkspaceDrive("drive-ws-1", host); + await deleteWorkspaceDrive("drive-ws-1", host); + await deleteWorkspaceDrive(null, host); + expect(calls).toEqual(["delete:drive-ws-1"]); + }); +}); diff --git a/packages/trpc/src/lib/sandbox/drives.ts b/packages/trpc/src/lib/sandbox/drives.ts new file mode 100644 index 00000000000..28b23a0dedc --- /dev/null +++ b/packages/trpc/src/lib/sandbox/drives.ts @@ -0,0 +1,250 @@ +import { FEATURE_FLAGS } from "@superset/shared/constants"; +import { + SANDBOX_PATHS, + type SandboxIdentity, +} from "@superset/shared/sandbox/contract"; +import { + APIError, + Drive, + type SandboxMounts, + type SandboxRegion, +} from "@vercel/sandbox"; +import { credentials } from "./vercel-credentials"; + +/** Vercel allows one read-write mount per drive, so the whole drive sits at one path. */ +export const WORKSPACE_DRIVE_MOUNT_PATH = SANDBOX_PATHS.drive; + +export function workspaceDriveNameFor(providerSandboxId: string): string { + return `drive-${providerSandboxId}`; +} + +/** + * Fork or nothing: a workspace has a drive exactly when its environment is + * in drive mode, because the golden's root disk was built without /workspace. + */ +export function workspaceDriveNameToClaim(args: { + existing: string | null; + goldenDriveName: string | null | undefined; + providerSandboxId: string; +}): string | null { + if (args.existing) return args.existing; + if (!args.goldenDriveName) return null; + return workspaceDriveNameFor(args.providerSandboxId); +} + +export function workspaceDriveIdentity( + driveName: string | null | undefined, + { laidOut }: { laidOut: boolean }, +): Pick< + SandboxIdentity, + "SUPERSET_SANDBOX_DRIVE" | "SUPERSET_SANDBOX_DRIVE_LAID_OUT" +> { + if (!driveName) return {}; + return { + SUPERSET_SANDBOX_DRIVE: driveName, + ...(laidOut ? { SUPERSET_SANDBOX_DRIVE_LAID_OUT: "1" as const } : {}), + }; +} + +/** + * The drive a claim names. Only a provision gives a workspace its + * environment's drive, and only while the organization's flag is on; a wake + * mounts the row's own drive or none, so a workspace made before its + * environment entered drive mode boots exactly as it did. A wake expects the + * drive laid out once the row recorded a boot that laid it out. + */ +export function claimedWorkspaceDrive(args: { + row: { + driveName: string | null; + driveLaidOutAt: Date | null; + providerSandboxId: string; + }; + goldenDriveName: string | null | undefined; + provisioning: boolean; + drivesEnabled: boolean; +}): { + driveName: string | null; + identity: ReturnType; +} { + if (!args.provisioning) { + return { + driveName: args.row.driveName, + identity: workspaceDriveIdentity(args.row.driveName, { + laidOut: args.row.driveLaidOutAt !== null, + }), + }; + } + const driveName = workspaceDriveNameToClaim({ + existing: args.row.driveName, + goldenDriveName: args.drivesEnabled ? args.goldenDriveName : null, + providerSandboxId: args.row.providerSandboxId, + }); + return { + driveName, + identity: workspaceDriveIdentity(driveName, { laidOut: false }), + }; +} + +export function goldenDriveNameFor(goldenName: string): string { + return `env-drive-${goldenName}`; +} + +/** Vercel does not carry mounts from one session to the next, so every create and wake passes these. */ +export function workspaceDriveMounts( + driveName: string | null | undefined, +): SandboxMounts | undefined { + if (!driveName) return undefined; + return { + [WORKSPACE_DRIVE_MOUNT_PATH]: { drive: driveName, mode: "read-write" }, + }; +} + +/** + * Whether a golden build or a promote may put an organization's environment + * in drive mode. Workspaces follow their environment's row, never this. + * Fails closed: PostHog unreachable means no drive. + */ +export async function workspaceDrivesEnabled( + organizationId: string, +): Promise { + // Lazily, so the release scripts that import this need no PostHog key. + const { posthog } = await import("../analytics"); + const enabled = await posthog + .isFeatureEnabled(FEATURE_FLAGS.CLOUD_WORKSPACE_DRIVES, organizationId, { + groups: { organization: organizationId }, + sendFeatureFlagEvents: false, + }) + .catch(() => undefined); + return enabled === true; +} + +/** What drive bookkeeping needs from the provider; `vercelDrives` is the real one. */ +export interface DriveHost { + /** Null when no drive has the name. */ + find(name: string): Promise<{ region: string; attachedTo?: string } | null>; + create(name: string, region: string): Promise; + fork(source: string, name: string): Promise; + /** Throws while the drive is attached to a sandbox. */ + remove(name: string): Promise; +} + +function isNotFound(error: unknown): boolean { + return error instanceof APIError && error.response.status === 404; +} + +export function vercelDrives(): DriveHost { + const get = async (name: string) => { + try { + return await Drive.get({ ...credentials(), name }); + } catch (error) { + if (isNotFound(error)) return null; + throw error; + } + }; + return { + async find(name) { + const drive = await get(name); + return drive + ? { region: drive.region, attachedTo: drive.currentSandboxName } + : null; + }, + async create(name, region) { + await Drive.getOrCreate({ + ...credentials(), + name, + region: region as SandboxRegion, + }); + }, + async fork(source, name) { + const parent = await Drive.get({ ...credentials(), name: source }); + await parent.fork({ name }); + }, + async remove(name) { + await (await get(name))?.delete(); + }, + }; +} + +/** + * The workspace's drive, forked from its environment's golden drive. Forks + * are copy-on-write and near instant. Idempotent on the name, so a + * re-delivered provision finds the drive it already made. + */ +export async function forkWorkspaceDrive(args: { + name: string; + region: string; + goldenDriveName: string | null; + drives?: DriveHost; +}): Promise<"existing" | "forked"> { + const drives = args.drives ?? vercelDrives(); + if (await drives.find(args.name)) return "existing"; + if (!args.goldenDriveName) { + throw new Error( + `Drive ${args.name} is missing and its environment has no golden drive to fork`, + ); + } + const golden = await drives.find(args.goldenDriveName); + if (!golden) { + throw new Error(`Golden drive ${args.goldenDriveName} does not exist`); + } + if (golden.region !== args.region) { + throw new Error( + `Golden drive ${args.goldenDriveName} is in ${golden.region}; the sandbox runs in ${args.region}`, + ); + } + await drives.fork(args.goldenDriveName, args.name); + return "forked"; +} + +/** + * A drive already gone is the state we wanted. Throws while the drive is + * still attached, so the caller stops its sandbox first and retries. + */ +export async function deleteWorkspaceDrive( + name: string | null | undefined, + drives: DriveHost = vercelDrives(), +): Promise { + if (!name) return; + await drives.remove(name); +} + +export const DRIVE_DETACH_TIMEOUT_MS = 2 * 60 * 1000; + +export class DriveStillAttachedError extends Error { + constructor( + readonly driveName: string, + readonly sandboxName: string, + ) { + super( + `Drive ${driveName} is still attached to ${sandboxName}; the previous session has not finished writing it`, + ); + this.name = "DriveStillAttachedError"; + } +} + +/** + * A session that stops keeps its drive attached while it flushes, and a mount + * taken meanwhile reads a corrupt or empty filesystem. + */ +export async function waitForDriveDetached( + driveName: string, + options: { + drives?: DriveHost; + timeoutMs?: number; + pollMs?: number; + /** Whether the sandbox holding the drive is the one about to mount it, already running. */ + heldByCaller?: (attachedTo: string) => Promise; + } = {}, +): Promise<"detached" | "held"> { + const drives = options.drives ?? vercelDrives(); + const deadline = Date.now() + (options.timeoutMs ?? DRIVE_DETACH_TIMEOUT_MS); + for (;;) { + const drive = await drives.find(driveName); + if (!drive?.attachedTo) return "detached"; + if (await options.heldByCaller?.(drive.attachedTo)) return "held"; + if (Date.now() >= deadline) { + throw new DriveStillAttachedError(driveName, drive.attachedTo); + } + await new Promise((resolve) => setTimeout(resolve, options.pollMs ?? 2000)); + } +} diff --git a/packages/trpc/src/lib/sandbox/golden-drives.ts b/packages/trpc/src/lib/sandbox/golden-drives.ts new file mode 100644 index 00000000000..ce7a16d8d2c --- /dev/null +++ b/packages/trpc/src/lib/sandbox/golden-drives.ts @@ -0,0 +1,24 @@ +import { db } from "@superset/db/client"; +import { environments } from "@superset/db/schema"; +import { and, eq, isNull } from "drizzle-orm"; +import { type DriveHost, vercelDrives } from "./drives"; + +/** + * An environment's golden drive goes once no live environment names it. Its + * forks are independent drives and stay readable after it is deleted. + */ +export async function deleteGoldenDriveIfUnreferenced( + name: string | null | undefined, + drives: DriveHost = vercelDrives(), +): Promise { + if (!name) return; + const referenced = await db.query.environments.findFirst({ + where: and( + eq(environments.goldenDriveName, name), + isNull(environments.archivedAt), + ), + columns: { id: true }, + }); + if (referenced) return; + await drives.remove(name); +} diff --git a/packages/trpc/src/lib/sandbox/index.ts b/packages/trpc/src/lib/sandbox/index.ts index ec376ff4943..2f9b6da808c 100644 --- a/packages/trpc/src/lib/sandbox/index.ts +++ b/packages/trpc/src/lib/sandbox/index.ts @@ -26,6 +26,18 @@ export { } from "./box-pressure"; export { buildSandboxClaim } from "./claim"; export { deriveSandboxCredentials } from "./credentials"; +export { + DriveStillAttachedError, + deleteWorkspaceDrive, + forkWorkspaceDrive, + goldenDriveNameFor, + WORKSPACE_DRIVE_MOUNT_PATH, + waitForDriveDetached, + workspaceDriveMounts, + workspaceDriveNameToClaim, + workspaceDrivesEnabled, +} from "./drives"; +export { deleteGoldenDriveIfUnreferenced } from "./golden-drives"; export { listRemoteBranches, type RemoteBranch, @@ -56,6 +68,7 @@ export { type ActiveWorkspaceSandbox, type BoxSession, deleteSandbox, + deleteSandboxAndDrive, describeSandbox, findSandboxSession, HOST_SERVICE_PORT, @@ -65,7 +78,11 @@ export { promoteSandboxToEnvironment, provisionSandbox, pushManagedEnv, + type ResumeLock, + SandboxBusyError, type SandboxClaim, + SandboxDriveMissingError, + SandboxDriveRefusedError, type SandboxEnvironment, SandboxNotReadyError, SandboxUnavailableError, diff --git a/packages/trpc/src/lib/sandbox/vercel-credentials.ts b/packages/trpc/src/lib/sandbox/vercel-credentials.ts new file mode 100644 index 00000000000..3c97aa04053 --- /dev/null +++ b/packages/trpc/src/lib/sandbox/vercel-credentials.ts @@ -0,0 +1,9 @@ +import { env } from "../../env"; + +export function credentials() { + return { + token: env.VERCEL_SANDBOX_TOKEN, + teamId: env.VERCEL_SANDBOX_TEAM_ID, + projectId: env.VERCEL_SANDBOX_PROJECT_ID, + }; +} diff --git a/packages/trpc/src/lib/sandbox/vercel.test.ts b/packages/trpc/src/lib/sandbox/vercel.test.ts index 2141b68005c..149d74b27be 100644 --- a/packages/trpc/src/lib/sandbox/vercel.test.ts +++ b/packages/trpc/src/lib/sandbox/vercel.test.ts @@ -1,26 +1,40 @@ import { afterAll, beforeAll, + beforeEach, describe, expect, setSystemTime, test, } from "bun:test"; +import { SANDBOX_PATHS } from "@superset/shared/sandbox/contract"; import { APIError, Drive, Sandbox } from "@vercel/sandbox"; import { stub } from "../../../test/stub"; import { agentToolsMounts, applySandboxPolicy, + composeMounts, + DRIVE_REFUSAL_LIMIT, deleteSandbox, + deleteSandboxAndDrive, describeSandbox, + driveBootsIn, + driveRefusalIn, findSandboxSession, listActiveWorkspaceSandboxes, listSandboxSessions, + persistentDriveRefusals, promoteSandboxToEnvironment, + provisionSandbox, + type ResumeLock, type SandboxClaim, + SandboxDriveMissingError, + SandboxDriveRefusedError, + SandboxNotReadyError, SandboxUnavailableError, SESSION_TIMEOUT_MS, sandboxExists, + settleSandbox, stopSandbox, topUpSandboxSession, wakeSandbox, @@ -313,7 +327,7 @@ describe("promoteSandboxToEnvironment", () => { let createTags: unknown; const golden = { currentSnapshotId: "snap_golden_0", - runCommand: async () => {}, + runCommand: async () => ({ exitCode: 0 }), stop: async () => {}, }; stub(Sandbox, { @@ -342,6 +356,7 @@ describe("promoteSandboxToEnvironment", () => { expect(await promote()).toEqual({ goldenName: "env-1", region: "iad1", + goldenDriveName: null, stopped: { sessionId: "sess_A" }, }); expect(sourceSnapshot).toBe("snap_now"); @@ -420,7 +435,7 @@ describe("wakeSandbox", () => { SUPERSET_SANDBOX_PROVIDER: "vercel", SUPERSET_BUNDLE_SHA: "a".repeat(64), }; - async function wake(): Promise { + async function _wake(): Promise { const authorized: Box[] = []; await wakeSandbox({ providerSandboxId: "ws-box", @@ -435,22 +450,6 @@ describe("wakeSandbox", () => { return authorized; } - test("a box read running that a call resumed runs beforeResume for the new session", async () => { - running = true; - expect(await wake()).toEqual([ - { - memoryMb: 16_384, - vcpus: 8, - resumedSession: { sessionId: "sess_new", expiresAt: box.expiresAt }, - }, - ]); - }); - - test("a box read stopped runs beforeResume once, before its own resume", async () => { - running = false; - expect(await wake()).toEqual([{ memoryMb: 8192, vcpus: 4 }]); - }); - test("a wake only for a running box leaves a box that stopped since the caller looked", async () => { running = false; calls.length = 0; @@ -485,6 +484,869 @@ describe("listActiveWorkspaceSandboxes", () => { }); }); +describe("wakeSandbox mounts", () => { + const RESUMED = new Error("resumed"); + let status: "running" | "stopped"; + let current: Record; + let events: string[]; + let updates: Array>; + const fake = { + get status() { + return status; + }, + get mounts() { + return current; + }, + vcpus: 4, + region: "iad1", + domain: () => "https://box.invalid", + update: async (params: Record) => { + events.push("update"); + updates.push(params); + }, + extendTimeout: async () => {}, + writeFiles: async () => { + throw RESUMED; + }, + currentSession: () => fake, + }; + stub(Sandbox, { get: async () => fake }); + let attachedTo: string | undefined; + stub(Drive, { + get: async ({ name }: { name: string }) => { + events.push(`drive:${name}`); + // The concurrent wake that took the drive has resumed the box. + if (attachedTo) status = "running"; + return { name, currentSandboxName: attachedTo }; + }, + }); + stub(globalThis, { + fetch: async () => { + throw new Error("no network in tests"); + }, + }); + const drive = { drive: "drive-1", mode: "read-write" }; + const mounts = { [SANDBOX_PATHS.drive]: drive }; + const claim = { vcpus: 4, files: [], networkPolicy: {} } as never; + const withAgentTools = { + ...(claim as object), + agentTools: true, + identity: { SUPERSET_BUNDLE_SHA: `feedfacecafe${"0".repeat(52)}` }, + } as never; + const stopAtResume = async () => { + events.push("resume"); + throw RESUMED; + }; + + beforeEach(() => { + current = {}; + events = []; + updates = []; + attachedTo = undefined; + }); + + test("a drive a concurrent wake already resumed this box with is not foreign", async () => { + status = "stopped"; + attachedTo = "sbx"; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts: mounts as never, + beforeResume: stopAtResume, + }), + ).rejects.toThrow(); + expect(events).not.toContain("resume"); + expect(updates.some((update) => "mounts" in update)).toBe(false); + }); + + test("a stopped box waits for its drive, then gets its mounts, then resumes", async () => { + status = "stopped"; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts: mounts as never, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(events).toEqual(["drive:drive-1", "update", "resume"]); + expect(updates).toEqual([{ mounts }]); + }); + + test("the drive and the agent-tools drive share one mounts object", async () => { + status = "stopped"; + const agents = "agent-tools-iad1-feedfacecafe"; + current = { + [SANDBOX_PATHS.agents]: { drive: "agent-tools-old", mode: "snapshot" }, + }; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim: withAgentTools, + mounts: mounts as never, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(updates).toEqual([ + { + mounts: { + [SANDBOX_PATHS.drive]: drive, + [SANDBOX_PATHS.agents]: { drive: agents, mode: "snapshot" }, + }, + }, + ]); + }); + + test("an agent-tools update keeps the drive the box already has", async () => { + status = "stopped"; + current = { [SANDBOX_PATHS.drive]: drive }; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim: withAgentTools, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(updates[0]?.mounts).toMatchObject({ [SANDBOX_PATHS.drive]: drive }); + }); + + test("a box without a drive gets no mount update", async () => { + status = "stopped"; + await expect( + wakeSandbox({ + providerSandboxId: "sbx", + claim, + beforeResume: stopAtResume, + }), + ).rejects.toBe(RESUMED); + expect(updates).toEqual([]); + }); + + test("a running box keeps the mounts its session started with", async () => { + status = "running"; + await expect( + wakeSandbox({ providerSandboxId: "sbx", claim, mounts: mounts as never }), + ).rejects.toThrow(); + expect(updates).toEqual([{ networkPolicy: {} }]); + expect(events).not.toContain("drive:drive-1"); + }); +}); + +describe("resume under one lock", () => { + let status: "running" | "stopped"; + let attachedTo: string | undefined; + let events: string[]; + let gate: Promise; + let bootLog: string; + let refuse: boolean; + const box = () => ({ + get status() { + return status; + }, + mounts: {}, + vcpus: 4, + domain: () => "https://box.invalid", + expiresAt: undefined, + update: async (params: Record) => { + if (params.mounts) { + events.push("mount"); + await gate; + } + }, + writeFiles: async () => { + events.push("write"); + }, + runCommand: async (command: { env?: Record }) => { + events.push("boot"); + status = "running"; + const id = command.env?.SUPERSET_BOOT_ID ?? ""; + if (refuse) { + bootLog = `1 boot.start pid=1 boot=${id}\n2 drive.missing drive-1 is not mounted at /mnt/superset-drive\n`; + } + }, + readFileToBuffer: async () => Buffer.from(bootLog), + stop: async () => { + events.push("stop"); + status = "stopped"; + }, + }); + stub(Sandbox, { + get: async () => { + const fake = box(); + return { ...fake, currentSession: () => fake }; + }, + }); + stub(Drive, { + get: async ({ name }: { name: string }) => { + events.push(`drive:${attachedTo ?? "detached"}`); + const holder = attachedTo; + // The session that stopped finishes saving and lets go. + attachedTo = undefined; + return { name, currentSandboxName: holder }; + }, + }); + stub(globalThis, { + fetch: async () => { + throw new Error("not answering"); + }, + }); + const mounts = { + [SANDBOX_PATHS.drive]: { drive: "drive-1", mode: "read-write" }, + } as never; + const claim = { + identity: { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "0".repeat(64), + SUPERSET_API_URL: "https://api.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "env-g", + SUPERSET_SANDBOX_PROVIDER: "vercel", + SUPERSET_SANDBOX_DRIVE: "drive-1", + }, + hostSecret: "s", + files: [], + vcpus: 4, + networkPolicy: {}, + } as never; + let holder: string | null; + const lock = + (who: string): ResumeLock => + async (step) => { + if (holder) throw new SandboxNotReadyError("sbx"); + holder = who; + events.push(`lock:${who}`); + try { + return await step(); + } finally { + holder = null; + } + }; + + beforeEach(() => { + status = "stopped"; + attachedTo = undefined; + events = []; + gate = Promise.resolve(); + bootLog = ""; + refuse = false; + holder = null; + }); + + test("a redelivered provision whose box stopped waits for the drive before anything resumes it", async () => { + attachedTo = "sbx"; + await provisionSandbox({ + name: "sbx", + environment: { sourceKind: "fork", sourceRef: "env-g", region: "iad1" }, + claim, + mounts, + exclusive: lock("provision"), + }); + expect(events).toEqual([ + "lock:provision", + "drive:sbx", + "drive:detached", + "mount", + "write", + "boot", + ]); + }); + + test("a second wake while the first resumes does not resume the box; it answers still starting", async () => { + let open = () => {}; + gate = new Promise((resolve) => { + open = resolve; + }); + const first = wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts, + exclusive: lock("first"), + }).catch((error: unknown) => error); + await new Promise((resolve) => setTimeout(resolve, 0)); + const second = await wakeSandbox({ + providerSandboxId: "sbx", + claim, + mounts, + exclusive: lock("second"), + }).catch((error: unknown) => error); + expect(second).toBeInstanceOf(SandboxNotReadyError); + expect(events).toEqual(["lock:first", "drive:detached", "mount"]); + refuse = true; + open(); + const outcome = await first; + expect(outcome).toBeInstanceOf(SandboxDriveMissingError); + // The refused box is stopped under the lock, so no wake can resume it in between. + expect(events).toEqual([ + "lock:first", + "drive:detached", + "mount", + "write", + "boot", + "lock:first", + "stop", + ]); + }); +}); + +describe("a box found running", () => { + const ended = new Error("session ended"); + const booted = new Error("booted on the new session"); + let calls: string[]; + let status: "running" | "stopped"; + let attachedTo: string | undefined; + const endedSession = { + extendTimeout: async () => { + calls.push("old-session:extend"); + // The extension past the plan's cap is refused, and the session ends. + status = "stopped"; + attachedTo = "sbx"; + throw ended; + }, + update: async () => { + calls.push("old-session:policy"); + throw ended; + }, + writeFiles: async () => { + calls.push("old-session:write"); + throw ended; + }, + runCommand: async () => { + calls.push("old-session:boot"); + throw ended; + }, + }; + const newSession = { + sessionId: "sess-new", + vcpus: 8, + extendTimeout: async () => {}, + update: async () => {}, + writeFiles: async () => { + calls.push("new-session:write"); + }, + runCommand: async () => { + calls.push("new-session:boot"); + throw booted; + }, + }; + // A Sandbox method other than a config update would resume a session that ended. + const resumes = (name: string) => async () => { + calls.push(`sandbox:${name}`); + }; + stub(Sandbox, { + get: async ({ resume }: { resume?: boolean }) => { + if (resume) { + calls.push("resume"); + status = "running"; + } + return { + status, + mounts: {}, + vcpus: 4, + expiresAt: new Date(Date.now() + 1000), + domain: () => "https://box.invalid", + extendTimeout: resumes("extend"), + update: async (params: Record) => { + calls.push(params.mounts ? "mount" : "sandbox:update"); + }, + writeFiles: resumes("write"), + runCommand: resumes("boot"), + currentSession: () => (resume ? newSession : endedSession), + }; + }, + }); + stub(Drive, { + get: async ({ name }: { name: string }) => { + calls.push(`drive:${attachedTo ?? "detached"}`); + const holder = attachedTo; + attachedTo = undefined; + return { name, currentSandboxName: holder }; + }, + }); + stub(globalThis, { + fetch: async () => { + throw new Error("not answering"); + }, + }); + + beforeEach(() => { + calls = []; + status = "running"; + attachedTo = undefined; + }); + + const wake = ( + beforeResume?: Parameters[0]["beforeResume"], + ) => + wakeSandbox({ + providerSandboxId: "sbx", + claim: { + identity: { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "0".repeat(64), + SUPERSET_API_URL: "https://api.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: + "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "env-g", + SUPERSET_SANDBOX_PROVIDER: "vercel", + SUPERSET_SANDBOX_DRIVE: "drive-1", + }, + files: [{ path: "/x", content: "y" }], + networkPolicy: {}, + vcpus: 4, + } as never, + mounts: { + [SANDBOX_PATHS.drive]: { drive: "drive-1", mode: "read-write" }, + } as never, + beforeResume, + }); + + test("whose session ends during the wake is resumed only after the drive check", async () => { + await expect(wake()).rejects.toBe(booted); + expect(calls.filter((call) => call.startsWith("sandbox:"))).toEqual([]); + const resumedAt = calls.indexOf("resume"); + expect(calls.slice(0, resumedAt)).toContain("drive:sbx"); + expect(calls.slice(0, resumedAt)).toContain("drive:detached"); + expect(calls.slice(0, resumedAt)).toContain("mount"); + expect(calls.slice(resumedAt)).toEqual([ + "resume", + "new-session:write", + "new-session:write", + "new-session:boot", + ]); + }); + + test("whose session ends during the wake meters the session resumed here before it boots", async () => { + const metered: unknown[] = []; + await expect( + wake(async (box) => { + metered.push(box); + calls.push("metered"); + }), + ).rejects.toBe(booted); + expect(metered).toEqual([ + { + memoryMb: 16_384, + vcpus: 8, + resumedSession: { sessionId: "sess-new", expiresAt: expect.any(Date) }, + }, + ]); + expect(calls.indexOf("metered")).toBeLessThan( + calls.indexOf("new-session:boot"), + ); + }); +}); + +describe("a new box on a drive", () => { + let events: string[]; + let attachedTo: string | undefined; + const session = { + writeFiles: async () => { + events.push("write"); + }, + runCommand: async () => { + events.push("boot"); + }, + }; + stub(Sandbox, { + get: async () => { + throw new APIError(new Response(null, { status: 404 })); + }, + fork: async () => { + events.push("create"); + return { + domain: () => "https://box.invalid", + expiresAt: undefined, + currentSession: () => session, + }; + }, + }); + stub(Drive, { + get: async ({ name }: { name: string }) => { + events.push(`drive:${attachedTo ?? "detached"}`); + const holder = attachedTo; + attachedTo = undefined; + return { name, currentSandboxName: holder }; + }, + }); + + beforeEach(() => { + events = []; + attachedTo = undefined; + }); + + test("is created only once a box of another name has let go of the drive", async () => { + // An undo's second provision: the first job's box still holds the drive. + attachedTo = "ws-box-1"; + await provisionSandbox({ + name: "ws-box-2", + environment: { sourceKind: "fork", sourceRef: "env-g", region: "iad1" }, + claim: { + identity: { + SUPERSET_SANDBOX_CONTRACT: "1", + SUPERSET_BUNDLE_SHA: "0".repeat(64), + SUPERSET_API_URL: "https://api.invalid", + SUPERSET_SANDBOX_WORKSPACE_ID: "00000000-0000-4000-8000-000000000001", + SUPERSET_SANDBOX_ORGANIZATION_ID: + "00000000-0000-4000-8000-000000000002", + SUPERSET_SANDBOX_REPOSITORIES: "[]", + SUPERSET_SANDBOX_IMAGE_TAG: "env-g", + SUPERSET_SANDBOX_PROVIDER: "vercel", + }, + hostSecret: "s", + files: [], + vcpus: 4, + networkPolicy: {}, + } as never, + mounts: { + [SANDBOX_PATHS.drive]: { drive: "drive-1", mode: "read-write" }, + } as never, + }); + expect(events).toEqual([ + "drive:ws-box-1", + "drive:detached", + "create", + "write", + "boot", + ]); + }); +}); + +describe("composeMounts", () => { + test("merges parts, a later part winning a shared path, and drops empty", () => { + const a = { "/a": { drive: "a", mode: "read-write" as const } }; + const b = { "/a": { drive: "b", mode: "snapshot" as const } }; + expect(composeMounts(a, undefined, b)).toEqual(b); + expect(composeMounts(undefined, {})).toBeUndefined(); + }); +}); + +describe("drive refusals in the boot log", () => { + const boot = (at: number, id: string, ...lines: string[]) => + [`${at} boot.start pid=1 bundle=abc boot=${id}`, ...lines].join("\n"); + + test("only this boot's lines count, so a previous boot's refusal never stops a healthy box", () => { + const log = [ + boot( + 1, + "old", + "2 drive.missing drive-1 is not mounted at /mnt/superset-drive", + ), + boot(3, "new", "4 run.cleared"), + ].join("\n"); + expect(driveRefusalIn(log, "old")).toBe( + "drive.missing drive-1 is not mounted at /mnt/superset-drive", + ); + expect(driveRefusalIn(log, "new")).toBeNull(); + expect(driveRefusalIn(log, "not-started-yet")).toBeNull(); + }); + + test("a reexec's second boot.start joins its boot", () => { + const log = boot( + 1, + "b", + "2 boot.reexec into the installed bundle's script", + "3 boot.start pid=2 bundle=abc boot=b", + "4 drive.foreign drive-1 belongs to workspace w", + ); + expect(driveBootsIn(log)).toHaveLength(1); + expect(driveRefusalIn(log, "b")).toBe( + "drive.foreign drive-1 belongs to workspace w", + ); + }); + + test("counts the persistent refusals in a row; a missing drive or a laid-out boot ends the run", () => { + const foreign = (at: number, id: string) => + boot(at, id, `${at + 1} drive.foreign d belongs to workspace w`); + const log = [ + boot(1, "a", "2 drive.layout ms=3", "3 host.exec 1.0"), + foreign(10, "b"), + boot(20, "c", "21 host.exec 1.0"), + foreign(30, "d"), + ].join("\n"); + expect(persistentDriveRefusals(log)).toBe(3); + expect( + persistentDriveRefusals( + `${log}\n${boot(40, "e", "41 drive.missing d is not mounted")}`, + ), + ).toBe(0); + }); +}); + +describe("settleSandbox with a drive", () => { + let health: unknown = null; + let bootLog = ""; + stub(globalThis, { + fetch: async (url: string) => { + if (String(url).endsWith("/trpc/health.check") && health !== null) { + return new Response(JSON.stringify(health)); + } + if (String(url).endsWith("/trpc/sandbox.setEnvironment")) { + return new Response("{}"); + } + throw new Error("not answering"); + }, + }); + const box = { + status: "running", + readFileToBuffer: async () => Buffer.from(bootLog), + } as never; + const claim = { + identity: { SUPERSET_SANDBOX_DRIVE: "drive-1" }, + hostSecret: "secret", + managedEnv: {}, + } as never; + const answering = (phases: string[]) => ({ + result: { + data: { + json: { + sandboxBoot: { + stamps: phases.map((phase, at) => ({ phase, at })), + }, + }, + }, + }, + }); + + test("a refused boot is a retryable not-ready, found without the full host wait", async () => { + health = null; + bootLog = + "1 boot.start pid=1 boot=this\n2 drive.missing drive-1 is not mounted at /mnt/superset-drive\n"; + const started = Date.now(); + const error = await settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + bootId: "this", + getSession: async () => box, + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SandboxDriveMissingError); + expect(error).toBeInstanceOf(SandboxNotReadyError); + expect(Date.now() - started).toBeLessThan(1_000); + }); + + test("a refusal no remount can change ends the retries after several boots in a row", async () => { + health = null; + bootLog = Array.from( + { length: DRIVE_REFUSAL_LIMIT }, + (_, i) => + `${i * 10} boot.start pid=1 boot=b${i}\n${i * 10 + 1} drive.unlaid drive-1 carries no workspace id`, + ).join("\n"); + const error = await settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + bootId: `b${DRIVE_REFUSAL_LIMIT - 1}`, + getSession: async () => box, + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(SandboxDriveRefusedError); + expect(error).toBeInstanceOf(SandboxUnavailableError); + expect(error).toMatchObject({ boots: DRIVE_REFUSAL_LIMIT }); + expect(String(error)).not.toContain("drive"); + }); + + test("a boot that answers without laying the drive out fails the settle", async () => { + health = answering(["boot.start", "host.ready"]); + bootLog = ""; + await expect( + settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + getSession: async () => box, + }), + ).rejects.toBeInstanceOf(SandboxDriveMissingError); + }); + + test("a boot that laid the drive out settles", async () => { + health = answering(["boot.start", "drive.layout", "host.ready"]); + bootLog = ""; + const answer = await settleSandbox({ + providerSandboxId: "sbx", + hostTarget: "https://box.invalid", + claim, + getSession: async () => box, + }); + expect(answer.stamps.map((stamp) => stamp.phase)).toContain("drive.layout"); + }); +}); + +describe("promoteSandboxToEnvironment", () => { + let events: string[]; + let created: Record | null; + let goldenStatus: string; + let goldenSnapshot: string; + let stripExit: number; + const source = { + status: "running", + region: "iad1", + vcpus: 4, + currentSession: () => ({ + sessionId: "sess-source", + snapshot: async () => { + events.push("snapshot source"); + return { snapshotId: "snap-1" }; + }, + }), + }; + const golden = { + get status() { + return goldenStatus; + }, + get currentSnapshotId() { + return goldenSnapshot; + }, + runCommand: async (command: { args: string[] }) => { + events.push(`strip ${command.args.join(" ")}`); + return { exitCode: stripExit }; + }, + stop: async () => { + goldenStatus = "stopped"; + goldenSnapshot = "snap-golden"; + }, + }; + stub(Sandbox, { + get: async ({ name }: { name: string }) => + name === "ws-source" ? source : golden, + create: async (params: Record) => { + created = params; + return golden; + }, + }); + const drives = { + find: async (name: string) => { + events.push(`find ${name}`); + return { region: "iad1" }; + }, + create: async () => {}, + fork: async (from: string, to: string) => { + events.push(`fork ${from} ${to}`); + }, + remove: async () => {}, + }; + beforeEach(() => { + events = []; + created = null; + goldenStatus = "running"; + goldenSnapshot = "snap-0"; + stripExit = 0; + }); + + test("a golden whose drive did not mount is not stripped and fails the promote", async () => { + stripExit = 3; + await expect( + promoteSandboxToEnvironment({ + sourceSandbox: "ws-source", + goldenName: "env-g", + organizationId: "org_1", + sourceDriveName: "drive-ws-source", + drives, + }), + ).rejects.toThrow(`${SANDBOX_PATHS.drive} is not mounted`); + const strip = events.find((event) => event.startsWith("strip")) ?? ""; + expect(strip).toContain(`mountpoint -q ${SANDBOX_PATHS.drive} || exit 3;`); + }); + + test("forks the source's drive once it is released, strips identity on the drive, and leaves the source's restart to the caller", async () => { + const result = await promoteSandboxToEnvironment({ + sourceSandbox: "ws-source", + goldenName: "env-g", + organizationId: "org_1", + sourceDriveName: "drive-ws-source", + drives, + }); + expect(result.goldenDriveName).toBe("env-drive-env-g"); + expect(created?.mounts).toEqual({ + [SANDBOX_PATHS.drive]: { drive: "env-drive-env-g", mode: "read-write" }, + }); + const strip = events.find((event) => event.startsWith("strip")) ?? ""; + expect(strip).toContain(`${SANDBOX_PATHS.drive}/workspace/.env`); + expect(strip).toContain(`${SANDBOX_PATHS.drive}/home/.codex/sessions`); + expect(strip).toContain(SANDBOX_PATHS.driveState); + expect(events).toEqual([ + "snapshot source", + "find drive-ws-source", + "fork drive-ws-source env-drive-env-g", + strip, + "find env-drive-env-g", + ]); + expect(result.stopped).toEqual({ sessionId: "sess-source" }); + }); + + test("a source without a drive leaves the environment out of drive mode", async () => { + const result = await promoteSandboxToEnvironment({ + sourceSandbox: "ws-source", + goldenName: "env-g", + organizationId: "org_1", + drives, + }); + expect(result.goldenDriveName).toBeNull(); + expect(created).not.toHaveProperty("mounts"); + expect(events.some((event) => event.startsWith("fork"))).toBe(false); + expect(events.join("\n")).not.toContain(SANDBOX_PATHS.drive); + }); +}); + +describe("deleteSandboxAndDrive", () => { + let events: string[]; + let status: string; + let expired = false; + stub(Sandbox, { + get: async () => { + if (expired) throw new APIError(new Response(null, { status: 410 })); + return box(); + }, + }); + const box = () => ({ + get status() { + return status; + }, + currentSession: () => ({ sessionId: "s" }), + stop: async () => { + events.push("stop"); + status = "stopped"; + }, + delete: async () => { + events.push("delete sandbox"); + }, + }); + const drives = { + find: async () => { + events.push("find"); + return { region: "iad1" }; + }, + create: async () => {}, + fork: async () => {}, + remove: async (name: string) => { + events.push(`delete ${name}`); + }, + }; + + test("stops the box and waits for the drive before deleting either", async () => { + events = []; + status = "running"; + await deleteSandboxAndDrive("ws-1", "drive-ws-1", drives); + expect(events).toEqual([ + "stop", + "find", + "delete sandbox", + "delete drive-ws-1", + ]); + }); + + test("a box whose snapshot expired, which cannot be fetched, still lets its drive go", async () => { + events = []; + expired = true; + await deleteSandboxAndDrive("ws-1", "drive-ws-1", drives); + expired = false; + expect(events).toEqual(["find", "delete drive-ws-1"]); + }); +}); + describe("topUpSandboxSession", () => { let expiresAt: Date; const extended: number[] = []; diff --git a/packages/trpc/src/lib/sandbox/vercel.ts b/packages/trpc/src/lib/sandbox/vercel.ts index b79ff3809e9..fa1b33c63ef 100644 --- a/packages/trpc/src/lib/sandbox/vercel.ts +++ b/packages/trpc/src/lib/sandbox/vercel.ts @@ -10,6 +10,7 @@ * ticket-checking gate in front of every port (`access.ts`). */ +import { randomUUID } from "node:crypto"; import * as Sentry from "@sentry/core"; import { renderSandboxConf, @@ -18,6 +19,7 @@ import { SANDBOX_PUBLISHED_PORTS, type SandboxIdentity, sandboxAgentToolsDrive, + sandboxDrivePathOf, } from "@superset/shared/sandbox/contract"; import { APIError, @@ -26,14 +28,24 @@ import { Sandbox, type SandboxMounts, type SandboxRegion, + type Session, } from "@vercel/sandbox"; -import { env } from "../../env"; import { bootStampsFrom, type HostAnswer } from "./boot-trace"; import { CODEX_AUTH_PATH, CODEX_SYSTEM_CONFIG_PATH, type SandboxFile, } from "./credentials"; +import { + type DriveHost, + DriveStillAttachedError, + deleteWorkspaceDrive, + goldenDriveNameFor, + vercelDrives, + waitForDriveDetached, + workspaceDriveMounts, +} from "./drives"; +import { credentials } from "./vercel-credentials"; export const HOST_SERVICE_PORT = SANDBOX_PORTS.hostService; /** @@ -53,14 +65,6 @@ const GOLDEN_SESSION_TIMEOUT_MS = 30 * 60 * 1000; const BOOT_COMMAND = "/usr/local/bin/superset-boot"; const HOST_PROBE_TIMEOUT_MS = 1_500; -function credentials() { - return { - token: env.VERCEL_SANDBOX_TOKEN, - teamId: env.VERCEL_SANDBOX_TEAM_ID, - projectId: env.VERCEL_SANDBOX_PROJECT_ID, - }; -} - function isNotFound(error: unknown): boolean { return error instanceof APIError && error.response.status === 404; } @@ -79,10 +83,14 @@ export class SandboxUnavailableError extends Error { } get snapshotExpired(): boolean { - return this.cause instanceof APIError && this.cause.response.status === 410; + return isSnapshotExpired(this.cause); } } +function isSnapshotExpired(error: unknown): boolean { + return error instanceof APIError && error.response.status === 410; +} + function isUnavailable(error: unknown): boolean { return ( error instanceof APIError && @@ -158,6 +166,8 @@ export interface SandboxEnvironment { sourceKind: "image" | "fork"; sourceRef: string; region: string; + /** Set when the environment is in drive mode; see `drives.ts`. */ + goldenDriveName?: string | null; } /** Everything the box needs to become one workspace; nothing of it is a create-time env. */ @@ -242,6 +252,21 @@ export function agentToolsDriveName( : null; } +/** One `mounts` object for a session; a later part wins a path both name. */ +export function composeMounts( + ...parts: Array +): SandboxMounts | undefined { + const mounts = Object.assign({}, ...parts.filter(Boolean)) as SandboxMounts; + return Object.keys(mounts).length > 0 ? mounts : undefined; +} + +function readWriteDriveAt(mounts: SandboxMounts, path: string): string | null { + const mount = mounts[path]; + if (!mount) return null; + if (mount instanceof Drive) return mount.name; + return mount.mode === "read-write" ? mount.drive : null; +} + function publishedPorts(extra: readonly number[] = []): number[] { return [...new Set([...SANDBOX_PUBLISHED_PORTS, ...extra])]; } @@ -253,7 +278,7 @@ function publishedPorts(extra: readonly number[] = []): number[] { * without a size limit and so the runner's inputs never change shape. */ async function writeIdentity( - sandbox: Sandbox, + sandbox: Pick, identity: SandboxIdentity, ): Promise { await sandbox.writeFiles([ @@ -271,7 +296,7 @@ async function writeIdentity( * started there. They hold placeholders only. */ async function writeSignInFiles( - sandbox: Sandbox, + sandbox: Pick, files: SandboxFile[] = [], ): Promise { if (files.length === 0) return; @@ -285,13 +310,19 @@ async function writeSignInFiles( * argv. The runner refuses to stack a second host-service on a live one, so * a wake that races a wake is harmless. */ -async function runBoot(sandbox: Sandbox, hostSecret: string): Promise { +async function runBoot( + sandbox: Pick, + hostSecret: string, +): Promise { + // Stamped on boot.start, so a wait reads only this boot's lines of the log. + const bootId = randomUUID(); await sandbox.runCommand({ cmd: "sudo", - args: ["--preserve-env=HOST_SERVICE_SECRET", BOOT_COMMAND], + args: ["--preserve-env=HOST_SERVICE_SECRET,SUPERSET_BOOT_ID", BOOT_COMMAND], detached: true, - env: { HOST_SERVICE_SECRET: hostSecret }, + env: { HOST_SERVICE_SECRET: hostSecret, SUPERSET_BOOT_ID: bootId }, }); + return bootId; } /** @@ -306,12 +337,16 @@ export async function provisionSandbox(args: { claim: SandboxClaim; /** A golden under construction keeps its snapshots until its row goes. */ kind?: "workspace" | "environment"; + mounts?: SandboxMounts; + exclusive?: ResumeLock; }): Promise<{ providerSandboxId: string; sandboxUrl: string; hostTarget: string; expiresAt: Date | null; sessionId: string | null; + /** Ties a wait for host-service to this boot's lines of the boot log. */ + bootId: string; }> { const kind = args.kind ?? "workspace"; const config = { @@ -327,53 +362,89 @@ export async function provisionSandbox(args: { keepLastSnapshots: { count: 1 }, tags: sandboxTags(kind, args.claim.identity), }; - const mounts = await agentToolsMounts( - agentToolsDriveName(args.claim, args.environment.region), + const mounts = composeMounts( + args.mounts, + await agentToolsMounts( + agentToolsDriveName(args.claim, args.environment.region), + ), ); - const sandbox = - (await getSandboxToWake(args.name)) ?? - (await Sentry.startSpan( - { - name: - args.environment.sourceKind === "fork" ? "fork" : "create from image", - op: "sandbox.provider", - attributes: { - "sandbox.name": args.name, - "sandbox.source": args.environment.sourceRef, - "sandbox.region": args.environment.region, - "sandbox.vcpus": args.claim.vcpus, + return (args.exclusive ?? unlocked)(async () => { + const existing = await getSandboxToWake(args.name); + const drive = mounts ? readWriteDriveAt(mounts, SANDBOX_PATHS.drive) : null; + let live: { sandbox: Sandbox; session: Session }; + if (existing?.status === "running") { + live = { sandbox: existing, session: existing.currentSession() }; + } else if (existing) { + // A redelivered provision whose box has stopped: its drive may still be flushing. + const outcome = drive + ? await waitForDriveDetached(drive, { + heldByCaller: heldByThisRunningBox(args.name), + }) + : "detached"; + if (outcome === "detached" && mounts) await existing.update({ mounts }); + live = await resumeSession(args.name); + } else { + // A drive has one writer whatever the box is called: a box of another name may still hold it. + if (drive) await waitForDriveDetached(drive); + const created = await Sentry.startSpan( + { + name: + args.environment.sourceKind === "fork" + ? "fork" + : "create from image", + op: "sandbox.provider", + attributes: { + "sandbox.name": args.name, + "sandbox.source": args.environment.sourceRef, + "sandbox.region": args.environment.region, + "sandbox.vcpus": args.claim.vcpus, + }, }, - }, + () => + args.environment.sourceKind === "fork" + ? // A fork copies the golden's region; a snapshot only exists where + // it was taken. + Sandbox.fork({ + ...config, + ...(mounts ? { mounts } : {}), + sourceSandbox: args.environment.sourceRef, + resources: { vcpus: args.claim.vcpus }, + }) + : Sandbox.create({ + ...config, + ...(mounts ? { mounts } : {}), + image: args.environment.sourceRef, + region: args.environment.region as SandboxRegion, + resources: { vcpus: args.claim.vcpus }, + }), + ); + live = { sandbox: created, session: created.currentSession() }; + } + const booted = await Sentry.startSpan( + { name: "start boot", op: "sandbox" }, () => - args.environment.sourceKind === "fork" - ? // A fork copies the golden's region; a snapshot only exists where - // it was taken. - Sandbox.fork({ - ...config, - ...(mounts ? { mounts } : {}), - sourceSandbox: args.environment.sourceRef, - resources: { vcpus: args.claim.vcpus }, - }) - : Sandbox.create({ - ...config, - ...(mounts ? { mounts } : {}), - image: args.environment.sourceRef, - region: args.environment.region as SandboxRegion, - resources: { vcpus: args.claim.vcpus }, - }), - )); - await Sentry.startSpan({ name: "start boot", op: "sandbox" }, async () => { - await writeIdentity(sandbox, args.claim.identity); - await writeSignInFiles(sandbox, args.claim.files); - await runBoot(sandbox, args.claim.hostSecret); + withLiveSession({ + providerSandboxId: args.name, + mounts, + live, + use: async (current) => { + await writeIdentity(current.session, args.claim.identity); + await writeSignInFiles(current.session, args.claim.files); + return runBoot(current.session, args.claim.hostSecret); + }, + }), + ); + const bootId = booted.result; + const { sandbox } = booted.live; + return { + providerSandboxId: args.name, + sandboxUrl: sandbox.domain(HOST_SERVICE_PORT), + hostTarget: sandbox.domain(HOST_SERVICE_PORT), + expiresAt: sandbox.expiresAt ?? null, + sessionId: sessionIdOf(sandbox), + bootId, + }; }); - return { - providerSandboxId: args.name, - sandboxUrl: sandbox.domain(HOST_SERVICE_PORT), - hostTarget: sandbox.domain(HOST_SERVICE_PORT), - expiresAt: sandbox.expiresAt ?? null, - sessionId: sessionIdOf(sandbox), - }; } const HOST_READY_TIMEOUT_MS = 60_000; @@ -386,12 +457,131 @@ export class SandboxNotReadyError extends Error { } } +/** Another caller holds the box's resume lock: retry later, nothing is wrong with the box. */ +export class SandboxBusyError extends SandboxNotReadyError { + constructor(readonly key: string) { + super("a sandbox another caller is starting"); + this.name = "SandboxBusyError"; + } +} + +/** + * The box did not come up on the drive its identity names: the drive did not + * mount, belongs to another workspace, or the boot never laid it out. The + * data is intact, so it is a not-ready the next wake retries, never a failure. + */ +export class SandboxDriveMissingError extends SandboxNotReadyError { + constructor( + providerSandboxId: string, + readonly driveName: string, + readonly reason: string, + ) { + super(providerSandboxId); + this.name = "SandboxDriveMissingError"; + } +} + +/** + * The drive refused for good: the same refusal that no remount can change, in + * several boots in a row. The workspace is marked failed; its drive and box + * are kept. + */ +export class SandboxDriveRefusedError extends SandboxUnavailableError { + constructor( + providerSandboxId: string, + readonly driveName: string, + readonly reason: string, + readonly boots: number, + ) { + super(providerSandboxId, new Error(reason)); + this.name = "SandboxDriveRefusedError"; + } +} + +/** Boots in a row that a refusal no remount can change ends the retries. */ +export const DRIVE_REFUSAL_LIMIT = 3; + +const STAMP = /^(\d+) ([a-z][a-z0-9-]*(?:\.[a-z0-9-]+)+)(?: (.*))?$/; + +interface DriveBoot { + id: string | null; + refusal: string | null; + laidOut: boolean; + hostStarted: boolean; +} + +/** + * The boot log is append-only across sessions. One entry per boot; a + * reexec's second `boot.start` carries the same boot id and joins its boot. + */ +export function driveBootsIn(bootLog: string): DriveBoot[] { + const boots: DriveBoot[] = []; + for (const line of bootLog.split("\n")) { + const match = STAMP.exec(line.trim()); + if (!match) continue; + const [, , phase, detail = ""] = match; + if (phase === "boot.start") { + const id = /(?:^| )boot=(\S+)/.exec(detail)?.[1] ?? null; + const last = boots.at(-1); + if (!(last && id && last.id === id)) { + boots.push({ id, refusal: null, laidOut: false, hostStarted: false }); + } + continue; + } + const boot = boots.at(-1); + if (!boot) continue; + if (/^drive\.(missing|foreign|unlaid|downgrade)$/.test(phase ?? "")) { + boot.refusal ??= `${phase} ${detail}`.trim(); + } else if (phase === "drive.layout") { + boot.laidOut = true; + } else if (phase === "host.exec") { + boot.hostStarted = true; + } + } + return boots; +} + +/** This boot's refusal, if it refused; another boot's line never counts. */ +export function driveRefusalIn(bootLog: string, bootId: string): string | null { + const boot = driveBootsIn(bootLog) + .reverse() + .find((entry) => entry.id === bootId); + return boot?.refusal ?? null; +} + +/** + * How many of the last boots in a row failed in a way a remount cannot fix: + * a drive of another workspace, a drive with no owner, a boot script that + * predates the drive, or a boot that started host-service without a layout. + */ +export function persistentDriveRefusals(bootLog: string): number { + let count = 0; + for (const boot of driveBootsIn(bootLog).reverse()) { + const persistent = boot.refusal + ? !boot.refusal.startsWith("drive.missing") + : boot.hostStarted && !boot.laidOut; + if (!persistent) break; + count++; + } + return count; +} + +/** How often a wait for host-service also reads the boot log for a drive refusal. */ +const DRIVE_REFUSAL_POLL_MS = 2_000; + async function waitForHostService( target: string, providerSandboxId: string, + refusal?: () => Promise, ): Promise { const deadline = Date.now() + HOST_READY_TIMEOUT_MS; + let nextRefusalCheck = 0; while (Date.now() < deadline) { + if (refusal && Date.now() >= nextRefusalCheck) { + const refused = await refusal(); + if (refused) throw refused; + nextRefusalCheck = Date.now() + DRIVE_REFUSAL_POLL_MS; + } const body: unknown = await fetch(`${target}/trpc/health.check`, { signal: AbortSignal.timeout(HOST_READY_POLL_MS * 6), }) @@ -416,11 +606,76 @@ export async function settleSandbox(args: { providerSandboxId: string; hostTarget: string; claim: SandboxClaim; + /** The boot this settle waits for; without one, no refusal is read from the log. */ + bootId?: string; + /** The running session to read the boot log from, fetched fresh for each read; null when the box is not running. */ + getSession?: () => Promise | null>; }): Promise { + const driveName = args.claim.identity.SUPERSET_SANDBOX_DRIVE; + const getSession = + args.getSession ?? + (async () => { + const box = await getSandbox(args.providerSandboxId); + return box?.status === "running" ? box.currentSession() : null; + }); + const bootLog = async () => { + // Through the session: a read from a box another caller stopped never resumes it. + const session = await getSession(); + if (!session) return ""; + const log = await session + .readFileToBuffer({ path: SANDBOX_PATHS.bootLog }) + .catch(() => null); + return log?.toString("utf8") ?? ""; + }; + const driveFailure = (driveName: string, reason: string, log: string) => { + const boots = persistentDriveRefusals(log); + if (boots < DRIVE_REFUSAL_LIMIT) { + console.warn(`[sandbox] ${args.providerSandboxId}: ${reason}`); + return new SandboxDriveMissingError( + args.providerSandboxId, + driveName, + reason, + ); + } + const refused = new SandboxDriveRefusedError( + args.providerSandboxId, + driveName, + reason, + boots, + ); + console.warn( + `[sandbox] ${args.providerSandboxId}: ${reason} in ${boots} boots in a row`, + ); + Sentry.captureException(refused, { + extra: { driveName, reason, boots }, + }); + return refused; + }; + const bootId = args.bootId; + const refusal = + driveName && bootId + ? async () => { + const log = await bootLog(); + const refused = driveRefusalIn(log, bootId); + return refused ? driveFailure(driveName, refused, log) : null; + } + : undefined; const answer = await waitForHostService( args.hostTarget, args.providerSandboxId, + refusal, ); + // A pinned bundle whose boot predates the drive layout serves the root disk's empty /workspace. + if ( + driveName && + !answer.stamps.some((stamp) => stamp.phase === "drive.layout") + ) { + throw driveFailure( + driveName, + "the boot answered without laying the drive out", + await bootLog(), + ); + } await pushManagedEnv( args.hostTarget, args.claim.hostSecret, @@ -548,21 +803,96 @@ async function hostServiceAnswers(hostTarget: string): Promise { * moved), the credential rules are re-applied (a token may have aged), and * the managed environment is pushed again (a fresh session holds none). */ +/** + * Runs `step` alone among everything that may resume this sandbox, and throws + * SandboxNotReadyError when another caller holds it. A stopped box's drive + * check and the call that resumes it are one step under it, so no two + * sessions ever hold the drive. + */ +export type ResumeLock = (step: () => Promise) => Promise; +const unlocked: ResumeLock = (step) => step(); + +/** + * The one call in this module that resumes a stopped box; it runs under the + * resume lock, after the drive check. Every other call goes through the + * returned Session: a Sandbox method resumes a session that ended, a Session + * method fails instead. + */ +async function resumeSession( + providerSandboxId: string, +): Promise<{ sandbox: Sandbox; session: Session }> { + const sandbox = await Sandbox.get({ + ...credentials(), + name: providerSandboxId, + resume: true, + }); + return { sandbox, session: sandbox.currentSession() }; +} + +/** + * Runs `use` on the box's session as it was read. If that session ended in + * between, a Session call fails rather than resume it; the box then takes + * the stopped path, under the caller's lock: wait for the drive, remount, + * resume, and `use` runs once more on the new session. + */ +async function withLiveSession(args: { + providerSandboxId: string; + mounts: SandboxMounts | undefined; + live: { sandbox: Sandbox; session: Session }; + use: (live: { sandbox: Sandbox; session: Session }) => Promise; + onResumed?: (live: { sandbox: Sandbox; session: Session }) => Promise; +}): Promise<{ result: T; live: { sandbox: Sandbox; session: Session } }> { + try { + return { result: await args.use(args.live), live: args.live }; + } catch (error) { + const now = await getSandbox(args.providerSandboxId); + if (!now || now.status === "running") throw error; + const drive = args.mounts + ? readWriteDriveAt(args.mounts, SANDBOX_PATHS.drive) + : null; + const outcome = drive + ? await waitForDriveDetached(drive, { + heldByCaller: heldByThisRunningBox(args.providerSandboxId), + }) + : "detached"; + if (outcome === "detached" && args.mounts) { + await now.update({ mounts: args.mounts }); + } + const live = await resumeSession(args.providerSandboxId); + await args.onResumed?.(live); + return { result: await args.use(live), live }; + } +} + +/** A drive attached to this very sandbox, running again, is held by the session another caller resumed. */ +function heldByThisRunningBox(providerSandboxId: string) { + return async (attachedTo: string) => + attachedTo === providerSandboxId && + (await getSandbox(providerSandboxId))?.status === "running"; +} + +export type WakePhase = "detach" | "mount" | "resume" | "boot"; + export async function wakeSandbox(args: { providerSandboxId: string; claim: SandboxClaim; + /** Replaces the box's mounts before a stopped session resumes; a running one keeps its own. */ + mounts?: SandboxMounts; /** A stopped box is left stopped, and the wake answers null. */ onlyIfRunning?: boolean; /** * Runs once the box is known to be stopped, before anything resumes it; or, - * for a box read running that a call here resumed (`resumedSession`), once - * those calls end. + * for a box read running whose session ended and was resumed here + * (`resumedSession`), once that resume is done. */ beforeResume?: (box: { memoryMb: number; vcpus: number; resumedSession?: { sessionId: string | null; expiresAt: Date | null }; }) => Promise; + /** How long each part of the wake took, for a check that measures it. */ + onPhase?: (phase: WakePhase, ms: number) => void; + exclusive?: ResumeLock; }): Promise<{ hostTarget: string; /** The session was stopped, so every process on the box started from this claim. */ @@ -573,111 +903,182 @@ export async function wakeSandbox(args: { expiresAt: Date | null; sessionId: string | null; } | null> { - try { - let wasRunning = false; - let resumed = null as { vcpus: number; sessionId: string | null } | null; - const sandbox = await getSandboxToWake( - args.providerSandboxId, - async (box) => { - if (!wasRunning) return; - const session = box.currentSession(); - resumed = { vcpus: session.vcpus, sessionId: session.sessionId }; - }, - ); - if (!sandbox) throw new SandboxUnavailableError(args.providerSandboxId); - const hostTarget = sandbox.domain(HOST_SERVICE_PORT); - wasRunning = sandbox.status === "running"; - if (args.onlyIfRunning && !wasRunning) return null; - let vcpus = sandbox.vcpus ?? args.claim.vcpus; - if (!wasRunning && vcpus !== args.claim.vcpus) { - await Sentry.startSpan({ name: "resize", op: "sandbox.provider" }, () => - sandbox - .update({ resources: { vcpus: args.claim.vcpus } }) - .then(() => { - vcpus = args.claim.vcpus; - }) - .catch((error) => - console.warn( - `[sandbox] resize to ${args.claim.vcpus} vCPUs failed for ${args.providerSandboxId}`, - error, - ), - ), - ); + const timed = async (phase: WakePhase, run: () => Promise) => { + const started = Date.now(); + try { + return await run(); + } finally { + args.onPhase?.(phase, Date.now() - started); } - if (!wasRunning) { - const { [SANDBOX_PATHS.agents]: previous, ...others } = + }; + const exclusive = args.exclusive ?? unlocked; + try { + const resumed = await exclusive(async () => { + const sandbox = await getSandboxToWake(args.providerSandboxId); + if (!sandbox) throw new SandboxUnavailableError(args.providerSandboxId); + const hostTarget = sandbox.domain(HOST_SERVICE_PORT); + const wasRunning = sandbox.status === "running"; + if (args.onlyIfRunning && !wasRunning) return null; + let vcpus = sandbox.vcpus ?? args.claim.vcpus; + let resumedByAnotherWake = false; + const { [SANDBOX_PATHS.agents]: previousAgents, ...kept } = sandbox.mounts ?? {}; const agents = await agentToolsMounts( agentToolsDriveName(args.claim, sandbox.region), ); - if (agents || previous) { - await sandbox - .update({ mounts: { ...others, ...agents } }) - .catch((error) => - console.warn( - `[sandbox] mount update failed for ${args.providerSandboxId}`, - error, - ), + const mounts = composeMounts(kept, args.mounts, agents) ?? {}; + if (!wasRunning) { + const drive = readWriteDriveAt(mounts, SANDBOX_PATHS.drive); + if (drive) { + const outcome = await Sentry.startSpan( + { name: "drive detach", op: "sandbox.provider" }, + () => + timed("detach", () => + waitForDriveDetached(drive, { + heldByCaller: heldByThisRunningBox(args.providerSandboxId), + }), + ), ); - } - } - if (!wasRunning && args.beforeResume) { - await args.beforeResume({ memoryMb: vcpus * MEMORY_MB_PER_VCPU, vcpus }); - } - if (wasRunning) { - const remaining = (sandbox.expiresAt?.getTime() ?? 0) - Date.now(); - if (remaining < EXTEND_BELOW_MS) { - // Past the plan's per-session cap the extension is refused; the - // session then ends and the next open resumes it. - await sandbox.extendTimeout(SESSION_TIMEOUT_MS).catch(() => {}); - } - } - // A box that is already serving needs neither a fresh identity file nor - // another boot. The policy still goes out: a credential may have aged. - const serving = wasRunning && (await hostServiceAnswers(hostTarget)); - await Sentry.startSpan( - { name: "apply claim", op: "sandbox.provider" }, - () => - Promise.all([ - sandbox - .update({ networkPolicy: args.claim.networkPolicy }) - .catch((error) => + // A concurrent wake resumed this box with its drive meanwhile. + resumedByAnotherWake = outcome === "held"; + } + if (!resumedByAnotherWake && vcpus !== args.claim.vcpus) { + await Sentry.startSpan( + { name: "resize", op: "sandbox.provider" }, + () => + sandbox + .update({ resources: { vcpus: args.claim.vcpus } }) + .then(() => { + vcpus = args.claim.vcpus; + }) + .catch((error) => + console.warn( + `[sandbox] resize to ${args.claim.vcpus} vCPUs failed for ${args.providerSandboxId}`, + error, + ), + ), + ); + } + if ( + !resumedByAnotherWake && + (agents || previousAgents || args.mounts) + ) { + const update = Sentry.startSpan( + { name: "mount", op: "sandbox.provider" }, + () => timed("mount", () => sandbox.update({ mounts })), + ); + // Without its drive the workspace would boot on an empty /workspace. + if (args.mounts) await update; + else + await update.catch((error) => console.warn( - `[sandbox] policy update failed for ${args.providerSandboxId}`, + `[sandbox] mount update failed for ${args.providerSandboxId}`, error, ), - ), - serving ? null : writeIdentity(sandbox, args.claim.identity), - writeSignInFiles(sandbox, args.claim.files), - ]), - ); - if (!serving) { - await Sentry.startSpan({ name: "start boot", op: "sandbox" }, () => - runBoot(sandbox, args.claim.hostSecret), - ); - } - if (resumed && args.beforeResume) { - await args.beforeResume({ - memoryMb: resumed.vcpus * MEMORY_MB_PER_VCPU, - vcpus: resumed.vcpus, - resumedSession: { - sessionId: resumed.sessionId, - expiresAt: sandbox.expiresAt ?? null, + ); + } + } + const running = wasRunning || resumedByAnotherWake; + if (!running && args.beforeResume) { + await args.beforeResume({ + memoryMb: vcpus * MEMORY_MB_PER_VCPU, + vcpus, + }); + } + const resumeStarted = Date.now(); + const first = wasRunning + ? { sandbox, session: sandbox.currentSession() } + : await resumeSession(args.providerSandboxId); + const { result, live } = await withLiveSession({ + providerSandboxId: args.providerSandboxId, + mounts, + live: first, + onResumed: async (resumedLive) => { + await args.beforeResume?.({ + memoryMb: resumedLive.session.vcpus * MEMORY_MB_PER_VCPU, + vcpus: resumedLive.session.vcpus, + resumedSession: { + sessionId: resumedLive.session.sessionId, + expiresAt: resumedLive.sandbox.expiresAt ?? null, + }, + }); + }, + use: async (current) => { + if (current === first && wasRunning) { + const remaining = (sandbox.expiresAt?.getTime() ?? 0) - Date.now(); + if (remaining < EXTEND_BELOW_MS) { + // Past the plan's per-session cap the extension is refused; the + // session then ends and the next open resumes it. + await current.session + .extendTimeout(SESSION_TIMEOUT_MS) + .catch(() => {}); + } + } + // A box that is already serving needs neither a fresh identity file nor + // another boot. The policy still goes out: a credential may have aged. + const serving = + current === first && + running && + (await hostServiceAnswers(hostTarget)); + await Sentry.startSpan( + { name: "apply claim", op: "sandbox.provider" }, + () => + Promise.all([ + current.session + .update({ networkPolicy: args.claim.networkPolicy }) + .catch((error) => + console.warn( + `[sandbox] policy update failed for ${args.providerSandboxId}`, + error, + ), + ), + serving + ? null + : writeIdentity(current.session, args.claim.identity), + writeSignInFiles(current.session, args.claim.files), + ]), + ); + const bootId = serving + ? undefined + : await Sentry.startSpan( + { name: "start boot", op: "sandbox" }, + () => runBoot(current.session, args.claim.hostSecret), + ); + return { serving, bootId, resumedHere: current !== first }; }, }); - } + const { serving, bootId } = result; + const booted = !running || result.resumedHere; + args.onPhase?.("resume", Date.now() - resumeStarted); + return { ...live, hostTarget, booted, serving, bootId }; + }); + if (!resumed) return null; + const { sandbox, session, hostTarget, booted, serving, bootId } = resumed; const answer = await Sentry.startSpan( { name: "settle", op: "sandbox" }, () => - settleSandbox({ - providerSandboxId: args.providerSandboxId, - hostTarget, - claim: args.claim, - }), - ); + timed("boot", () => + settleSandbox({ + providerSandboxId: args.providerSandboxId, + hostTarget, + claim: args.claim, + bootId, + }), + ), + ).catch(async (error) => { + // Stopped: the next wake mounts the drive afresh, and a refused box bills nothing. + if ( + error instanceof SandboxDriveMissingError || + error instanceof SandboxDriveRefusedError + ) { + // Under the lock, so no caller resumes the box between its check and this stop. + await exclusive(() => session.stop()).catch(() => {}); + } + throw error; + }); return { hostTarget, - booted: !wasRunning, + booted, rebooted: !serving, answer, expiresAt: sandbox.expiresAt ?? null, @@ -686,6 +1087,11 @@ export async function wakeSandbox(args: { } catch (error) { if (isUnavailable(error)) throw new SandboxUnavailableError(args.providerSandboxId, error); + // The last session is still flushing: the box is starting, as any caller sees it. + if (error instanceof DriveStillAttachedError) { + console.warn(`[sandbox] ${args.providerSandboxId}: ${error.message}`); + throw new SandboxNotReadyError(args.providerSandboxId); + } throw error; } } @@ -713,13 +1119,47 @@ const INHERITED_IDENTITY = [ CODEX_SYSTEM_CONFIG_PATH, ]; +/** Agent sessions are the person's; on a drive they would reach every fork. */ +const DRIVE_IDENTITY = [ + `${SANDBOX_PATHS.home}/.claude/projects`, + `${SANDBOX_PATHS.home}/.codex/sessions`, +]; + +/** + * The paths to remove so a fork starts clean. With a drive, also where those + * paths sit on the drive itself, for a box no boot has laid out. + */ +export function inheritedIdentityPaths(withDrive: boolean): string[] { + if (!withDrive) return INHERITED_IDENTITY; + const paths = [...INHERITED_IDENTITY, ...DRIVE_IDENTITY]; + const onDrive = paths.flatMap((path) => sandboxDrivePathOf(path) ?? []); + return [...paths, ...onDrive, SANDBOX_PATHS.driveState]; +} + /** Removes what a box wrote for the workspace it was, so a fork starts clean. */ -export async function stripWorkspaceIdentity(sandbox: Sandbox): Promise { - await sandbox.runCommand({ - cmd: "rm", - args: ["-rf", ...INHERITED_IDENTITY], +export async function stripWorkspaceIdentity( + sandbox: Pick, + options: { withDrive?: boolean } = {}, +): Promise { + const withDrive = options.withDrive ?? false; + // Unmounted, the rm would miss the drive, and every fork would refuse it as another workspace's. + const result = await sandbox.runCommand({ + cmd: "bash", + args: [ + "-c", + `${withDrive ? `mountpoint -q ${SANDBOX_PATHS.drive} || exit 3; ` : ""}rm -rf -- "$@"`, + "strip", + ...inheritedIdentityPaths(withDrive), + ], sudo: true, }); + if (result.exitCode !== 0) { + throw new Error( + result.exitCode === 3 + ? `${SANDBOX_PATHS.drive} is not mounted; the golden's drive keeps the source's identity` + : `stripping the workspace identity exited ${result.exitCode}`, + ); + } } /** @@ -745,25 +1185,39 @@ export async function stopAndSnapshot( * would start from the source's last stop) and created with an empty env. * Taking that snapshot ends the source's session. The source stays stopped: * the caller starts it again through a wake, which meters the new session. + * + * A source with a drive gives the golden a fork of it, taken once the source + * has released it; the golden drive is stripped and released the same way. */ export async function promoteSandboxToEnvironment(args: { sourceSandbox: string; goldenName: string; organizationId: string; + sourceDriveName?: string | null; + drives?: DriveHost; }): Promise<{ goldenName: string; region: string; + goldenDriveName: string | null; /** The session the snapshot ended; null when the source was not running. */ stopped: { sessionId: string | null } | null; }> { + const drives = args.drives ?? vercelDrives(); const source = await getSandbox(args.sourceSandbox); if (!source) throw new SandboxUnavailableError(args.sourceSandbox); const stopped = source.status === "running" ? { sessionId: sessionIdOf(source) } : null; + const snapshotId = await snapshotNow(source); + let goldenDriveName: string | null = null; + if (args.sourceDriveName) { + goldenDriveName = goldenDriveNameFor(args.goldenName); + await waitForDriveDetached(args.sourceDriveName, { drives }); + await drives.fork(args.sourceDriveName, goldenDriveName); + } const golden = await Sandbox.create({ ...credentials(), name: args.goldenName, - source: { type: "snapshot", snapshotId: await snapshotNow(source) }, + source: { type: "snapshot", snapshotId }, ports: publishedPorts(), timeout: GOLDEN_SESSION_TIMEOUT_MS, region: source.region as SandboxRegion, @@ -773,12 +1227,21 @@ export async function promoteSandboxToEnvironment(args: { snapshotExpiration: 0, keepLastSnapshots: { count: 1 }, tags: environmentTags(args.organizationId), + ...(goldenDriveName + ? { mounts: workspaceDriveMounts(goldenDriveName) } + : {}), }); - await stripWorkspaceIdentity(golden); + await stripWorkspaceIdentity(golden, { withDrive: goldenDriveName !== null }); const created = golden.currentSnapshotId; await golden.stop(); await waitForStopSnapshot(args.goldenName, created); - return { goldenName: args.goldenName, region: source.region, stopped }; + if (goldenDriveName) await waitForDriveDetached(goldenDriveName, { drives }); + return { + goldenName: args.goldenName, + region: source.region, + goldenDriveName, + stopped, + }; } /** @@ -837,7 +1300,10 @@ export async function sandboxExists( export async function stopSandbox( providerSandboxId: string, ): Promise<{ sessionId: string | null }> { - const sandbox = await getSandbox(providerSandboxId); + const sandbox = await getSandbox(providerSandboxId).catch((error) => { + if (isSnapshotExpired(error)) return null; + throw error; + }); if (!sandbox) return { sessionId: null }; const sessionId = sessionIdOf(sandbox); if (sandbox.status === "running") await sandbox.stop(); @@ -989,10 +1455,34 @@ export async function listActiveWorkspaceSandboxes( return found; } -/** Best-effort: a sandbox already gone is the state we wanted. */ +/** + * Best-effort: a sandbox already gone is the state we wanted. One whose + * snapshot expired cannot be fetched to delete; it holds no session and no + * snapshot, and its name is retired. + */ export async function deleteSandbox(providerSandboxId: string): Promise { - const sandbox = await getSandbox(providerSandboxId); + const sandbox = await getSandbox(providerSandboxId).catch((error) => { + if (isSnapshotExpired(error)) return null; + throw error; + }); if (!sandbox) return; // Snapshots outlive a sandbox by default and keep billing storage. await sandbox.delete({ deleteOrphanSnapshots: true }); } + +/** + * A sandbox deleted while it holds its drive read-write leaves the drive + * attached for minutes, so the box is stopped and the drive released first. + * Throws while the drive is still attached; the caller retries. + */ +export async function deleteSandboxAndDrive( + providerSandboxId: string, + driveName: string | null | undefined, + drives: DriveHost = vercelDrives(), +): Promise { + if (!driveName) return deleteSandbox(providerSandboxId); + await stopSandbox(providerSandboxId); + await waitForDriveDetached(driveName, { drives }); + await deleteSandbox(providerSandboxId); + await deleteWorkspaceDrive(driveName, drives); +} diff --git a/packages/trpc/src/router/cloud-workspace/client-status.test.ts b/packages/trpc/src/router/cloud-workspace/client-status.test.ts index 7f358b44713..0c3c5918d2e 100644 --- a/packages/trpc/src/router/cloud-workspace/client-status.test.ts +++ b/packages/trpc/src/router/cloud-workspace/client-status.test.ts @@ -1,5 +1,5 @@ import { describe, expect, test } from "bun:test"; -import { cloudWorkspaceStatusFor } from "./client-status"; +import { cloudWorkspaceStatusFor, withClientStatus } from "./client-status"; const desktop = (version: string) => ({ product: "desktop" as const, version }); @@ -30,3 +30,15 @@ describe("cloudWorkspaceStatusFor", () => { ).toBe("asleep"); }); }); + +describe("withClientStatus output", () => { + test("never carries the drive columns, which released clients never saw", () => { + const out = withClientStatus(null, { + id: "ws-1", + status: "awake" as const, + driveName: "drive-ws-1", + driveLaidOutAt: new Date(), + }); + expect(out).toEqual({ id: "ws-1", status: "awake" }); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/client-status.ts b/packages/trpc/src/router/cloud-workspace/client-status.ts index 1aaa16165ef..1ed6fa4c930 100644 --- a/packages/trpc/src/router/cloud-workspace/client-status.ts +++ b/packages/trpc/src/router/cloud-workspace/client-status.ts @@ -38,9 +38,17 @@ export function cloudWorkspaceStatusFor( return status; } +/** Only the control plane reads these; released clients never saw them. */ +type DriveColumns = "driveName" | "driveLaidOutAt"; + export function withClientStatus( client: ApiClientInfo | null, row: Row, -): Row { - return { ...row, status: cloudWorkspaceStatusFor(client, row.status) }; +): Omit { + const { + driveName: _driveName, + driveLaidOutAt: _driveLaidOutAt, + ...rest + } = row as Row & Partial>; + return { ...rest, status: cloudWorkspaceStatusFor(client, row.status) }; } diff --git a/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts b/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts index c9bd47c194d..3cc9b9b1a28 100644 --- a/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts +++ b/packages/trpc/src/router/cloud-workspace/cloud-workspace.ts @@ -37,7 +37,6 @@ import { logControl } from "../../lib/control-log"; import { callMoves } from "../../lib/moves"; import { nudge, readWorkspaceForwards } from "../../lib/realtime"; import { - deleteSandbox, describeSandbox, HOST_SERVICE_PORT, listRemoteBranches, @@ -90,6 +89,7 @@ import { queueProvision, startCloudWorkspace, } from "./start"; +import { retireAndTearDown } from "./teardown"; import { transitionCloudWorkspace } from "./transition"; import { markSandboxUnavailable, @@ -916,7 +916,13 @@ export const cloudWorkspaceRouter = { meterError, ), ); - await deleteSandbox(row.providerSandboxId); + await retireAndTearDown( + row, + and( + eq(cloudWorkspaces.status, "archived"), + eq(cloudWorkspaces.deletedAt, archivedAt), + ), + ); await reapWorkspaceFiles(row.id); }); } diff --git a/packages/trpc/src/router/cloud-workspace/drive-laid-out.ts b/packages/trpc/src/router/cloud-workspace/drive-laid-out.ts new file mode 100644 index 00000000000..ddb4eaf700f --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/drive-laid-out.ts @@ -0,0 +1,18 @@ +import { db } from "@superset/db/client"; +import { cloudWorkspaces } from "@superset/db/schema"; +import { and, eq, isNull, sql } from "drizzle-orm"; + +/** Once a boot answered with the drive laid out, every later wake expects the drive to name this workspace. */ +export async function recordDriveLaidOut( + cloudWorkspaceId: string, +): Promise { + await db + .update(cloudWorkspaces) + .set({ driveLaidOutAt: sql`now()` }) + .where( + and( + eq(cloudWorkspaces.id, cloudWorkspaceId), + isNull(cloudWorkspaces.driveLaidOutAt), + ), + ); +} diff --git a/packages/trpc/src/router/cloud-workspace/provision-drive.test.ts b/packages/trpc/src/router/cloud-workspace/provision-drive.test.ts new file mode 100644 index 00000000000..1b432fc1366 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/provision-drive.test.ts @@ -0,0 +1,288 @@ +import { beforeEach, describe, expect, test } from "bun:test"; +import { db } from "@superset/db/client"; +import { stub } from "../../../test/stub"; +import * as realtime from "../../lib/realtime"; +import * as sandbox from "../../lib/sandbox"; +import * as chatMoves from "../chat-move/utils/queueChatMove"; +import * as laidOut from "./drive-laid-out"; +import * as jobs from "./jobs"; +import * as meter from "./meter"; +import { provisionCloudWorkspace } from "./provision"; +import * as resumeLock from "./resume-lock"; +import * as expiry from "./sandbox-expiry"; +import * as sleep from "./sandbox-sleep"; +import * as teardown from "./teardown"; +import * as transition from "./transition"; + +const ROW = { + id: "00000000-0000-4000-8000-000000000001", + organizationId: "00000000-0000-4000-8000-000000000002", + createdByUserId: null, + status: "provisioning", + providerSandboxId: "ws-box", + driveName: null as string | null, + driveLaidOutAt: null as Date | null, +}; + +let row: typeof ROW; +let events: string[]; +let settle: () => Promise; +let create: () => Promise; +let fork: "existing" | "forked"; +let wins: (to: string) => boolean; +let current: { status: string; providerSandboxId?: string } | undefined; +let lockKeys: string[]; +let rowBox: string; +// The row's status as the transitions see it: another delivery or a wake may have moved it. +let rowStatus: string; +let authorize: () => Promise<{ bootId: string | null }>; +let requeued: unknown[]; + +let reads = 0; +stub(db.query.cloudWorkspaces, { + findFirst: async () => (reads++ === 0 ? row : current), +}); +stub(db, { + update: () => ({ + set: (values: Record) => ({ + where: async () => { + events.push(`update ${Object.keys(values).join(",")}`); + }, + }), + }), +}); +stub(sandbox, { + buildSandboxClaim: async () => ({ + claim: { vcpus: 4 }, + environment: { region: "iad1", goldenDriveName: "env-drive-g" }, + agentCredentialDigest: "d", + driveName: "drive-ws-box", + }), + forkWorkspaceDrive: async () => fork, + provisionSandbox: async () => { + await create(); + return { + providerSandboxId: "ws-box", + sandboxUrl: "https://box.invalid", + hostTarget: "https://box.invalid", + expiresAt: null, + sessionId: "session-1", + bootId: "boot-1", + }; + }, + settleSandbox: async () => settle(), + stopSandbox: async () => { + events.push("stop"); + return { sessionId: "session-1" }; + }, + recordBootSpans: () => {}, +}); +stub(meter, { + authorizeBoot: async () => authorize(), + attachBootSession: async () => {}, + releaseBoot: async () => {}, +}); +type Move = { + from: string[]; + to: string; + set?: Record; + onBox?: string; +}; +function move(args: Move): boolean { + if (!wins(args.to)) return false; + if (!args.from.includes(rowStatus)) return false; + if (args.onBox !== undefined && args.onBox !== rowBox) return false; + events.push( + `to ${args.to}${args.set && "providerSandboxId" in args.set ? ` box=${args.set.providerSandboxId === "ws-box" ? "same" : "new"}` : ""}`, + ); + rowStatus = args.to; + return true; +} +stub(transition, { + transitionCloudWorkspace: async (args: Move) => move(args), + failCloudWorkspace: async (args: Omit) => + move({ ...args, to: "failed" }), +}); +stub(sleep, { + recordSandboxAsleep: async () => { + events.push("asleep"); + }, +}); +stub(expiry, { recordSandboxAwake: async () => {} }); +stub(laidOut, { + recordDriveLaidOut: async () => { + events.push("laid out"); + }, +}); +stub(teardown, { + deleteSandboxAndDriveLater: async (args: { + driveName: string | null; + keepDrive?: boolean; + }) => { + events.push( + `teardown drive=${args.driveName}${args.keepDrive ? " kept" : ""}`, + ); + }, +}); +stub(chatMoves, { + queueChatMovesTo: async () => {}, + failChatMovesTo: async () => {}, +}); +stub(realtime, { nudge: () => {} }); +stub(jobs, { + publishCloudWorkspaceJob: async (job: { body: unknown }) => { + requeued.push(job.body); + }, +}); +stub(resumeLock, { + sandboxResumeLock: (key: string) => { + lockKeys.push(key); + return (step: () => Promise) => step(); + }, +}); + +beforeEach(() => { + row = { ...ROW }; + events = []; + settle = async () => ({ stamps: [], answeredAt: 0 }); + create = async () => {}; + fork = "forked"; + wins = () => true; + current = undefined; + reads = 0; + lockKeys = []; + rowBox = "ws-box"; + rowStatus = "provisioning"; + authorize = async () => ({ bootId: null }); + requeued = []; +}); + +describe("provisionCloudWorkspace with a drive", () => { + test("records the drive laid out once the boot answered", async () => { + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "provisioned", + ); + expect(events).toContain("laid out"); + }); + + test("a drive refusal at create leaves the workspace asleep for the next wake, never failed", async () => { + settle = async () => { + throw new sandbox.SandboxDriveMissingError( + "ws-box", + "drive-ws-box", + "drive.missing", + ); + }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "provisioned", + ); + expect(events).toContain("stop"); + expect(events).toContain("asleep"); + expect(events).not.toContain("to failed"); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + }); + + test("a second delivery on the same box that fails after the first brought the row up leaves the live workspace alone", async () => { + // The first delivery made the row awake and the user works; this one is refused by the spend cap. + rowStatus = "awake"; + authorize = async () => { + throw new Error("spend cap reached"); + }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "failed", + ); + expect(rowStatus).toBe("awake"); + expect(events).not.toContain("stop"); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + expect(events.some((event) => event.startsWith("to "))).toBe(false); + }); + + test("a lock held by another caller requeues the provision; nothing is stopped or failed", async () => { + create = async () => { + throw new sandbox.SandboxBusyError("drive-ws-box"); + }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "retrying", + ); + expect(requeued).toEqual([{ cloudWorkspaceId: ROW.id, lockRetries: 1 }]); + expect(events).not.toContain("stop"); + expect(events.some((event) => event.startsWith("to "))).toBe(false); + }); + + test("a lock still held after the last retry fails only a row still provisioning", async () => { + create = async () => { + throw new sandbox.SandboxBusyError("drive-ws-box"); + }; + await provisionCloudWorkspace({ + cloudWorkspaceId: ROW.id, + lockRetries: 10, + }); + expect(requeued).toEqual([]); + expect(events).toContain("to failed box=same"); + }); + + test("a provision overtaken by an undo's second provision gives up its box, keeps the drive, and never marks the row", async () => { + // Deleted between claim and create, then undone: the row names another box now. + rowBox = "ws-box-2"; + current = { status: "provisioning", providerSandboxId: "ws-box-2" }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "skipped", + ); + expect(events).toContain("teardown drive=drive-ws-box kept"); + expect(events.some((event) => event.startsWith("to "))).toBe(false); + expect(events).not.toContain("laid out"); + }); + + test("the create holds the lock of the drive, whatever the box is called", async () => { + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(lockKeys).toEqual(["drive-ws-box"]); + }); + + test("a provision that lost its row to an archive leaves the box to that archive's reap", async () => { + wins = (to) => to !== "awake"; + current = { status: "archived", providerSandboxId: "ws-box" }; + expect(await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id })).toBe( + "skipped", + ); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + expect(events).not.toContain("stop"); + }); + + test("a failed create retires the box's name in the same write that fails the row, then tears it down", async () => { + create = async () => { + throw new Error("create refused"); + }; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events).toEqual([ + "update driveName", + "to failed box=new", + "teardown drive=drive-ws-box", + ]); + }); + + test("a failed create whose row was archived meanwhile tears nothing down", async () => { + create = async () => { + throw new Error("create refused"); + }; + wins = (to) => to !== "failed"; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events.some((event) => event.startsWith("teardown"))).toBe(false); + }); + + test("a failed create deletes a drive it forked, never one it found", async () => { + create = async () => { + throw new Error("create refused"); + }; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events).toContain("teardown drive=drive-ws-box"); + + events = []; + reads = 0; + rowStatus = "provisioning"; + row = { ...ROW, driveName: "drive-ws-box" }; + fork = "existing"; + await provisionCloudWorkspace({ cloudWorkspaceId: ROW.id }); + expect(events).toContain("teardown drive=null"); + expect(events).toContain("to failed box=new"); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/provision.ts b/packages/trpc/src/router/cloud-workspace/provision.ts index 8bc3b4c365c..9bea9312dc6 100644 --- a/packages/trpc/src/router/cloud-workspace/provision.ts +++ b/packages/trpc/src/router/cloud-workspace/provision.ts @@ -1,4 +1,3 @@ -import { randomBytes } from "node:crypto"; import * as Sentry from "@sentry/core"; import { db } from "@superset/db/client"; import type { CloudWorkspaceFailureReason } from "@superset/db/enums"; @@ -7,25 +6,35 @@ import { cloudUsageRefusalKey } from "@superset/shared/billing"; import type { CloudAgentLaunch } from "@superset/shared/cloud-agent-launch"; import { CLOUD_WORKSPACE_PROVISION_TRANSACTION } from "@superset/shared/constants"; import { eq } from "drizzle-orm"; -import { controlTags } from "../../lib/control-log"; +import { type ControlScope, controlTags } from "../../lib/control-log"; import { nudge } from "../../lib/realtime"; import { bootPhases, buildSandboxClaim, - deleteSandbox, + forkWorkspaceDrive, provisionSandbox, + SandboxBusyError, + SandboxDriveMissingError, + SandboxDriveRefusedError, + type SandboxEnvironment, SandboxNotReadyError, settleSandbox, stopSandbox, + workspaceDriveMounts, } from "../../lib/sandbox"; import { failChatMovesTo, queueChatMovesTo, } from "../chat-move/utils/queueChatMove"; +import { recordDriveLaidOut } from "./drive-laid-out"; import { generateCloudWorkspaceName } from "./generate-name"; +import { publishCloudWorkspaceJob } from "./jobs"; import { attachBootSession, authorizeBoot, boxFor, releaseBoot } from "./meter"; +import { sandboxResumeLock } from "./resume-lock"; import { recordSandboxAwake } from "./sandbox-expiry"; -import { recordBootTrace } from "./sandbox-sleep"; +import { nextSandboxNameFor } from "./sandbox-name"; +import { recordBootTrace, recordSandboxAsleep } from "./sandbox-sleep"; +import { deleteSandboxAndDriveLater } from "./teardown"; import { failCloudWorkspace, transitionCloudWorkspace } from "./transition"; export const FALLBACK_NAME = "Cloud workspace"; @@ -35,14 +44,7 @@ export function sandboxNameFor(cloudWorkspaceId: string): string { return `ws-${cloudWorkspaceId.replaceAll("-", "").slice(0, 24)}`; } -/** - * A later box for the same row. Vercel refuses a deleted sandbox's name - * with `image_not_ready` for a few minutes after the delete. - */ -export function nextSandboxNameFor(cloudWorkspaceId: string): string { - const prefix = cloudWorkspaceId.replaceAll("-", "").slice(0, 16); - return `ws-${prefix}${randomBytes(4).toString("hex")}`; -} +export { nextSandboxNameFor } from "./sandbox-name"; export interface ProvisionCloudWorkspaceInput { cloudWorkspaceId: string; @@ -53,8 +55,16 @@ export interface ProvisionCloudWorkspaceInput { namingPrompt?: string; /** A built-in agent to run once the sandbox is up; see cloud-agent-launch. */ launch?: CloudAgentLaunch; + /** How many times this provision found the box's resume lock held and requeued itself. */ + lockRetries?: number; } +/** A lock held this long by others means something is stuck: the row is failed as for any error. */ +const PROVISION_LOCK_RETRIES = 10; +const PROVISION_LOCK_RETRY_SECONDS = 15; + +type CloudWorkspaceRow = typeof cloudWorkspaces.$inferSelect; + function provisionFailureReason(error: unknown): CloudWorkspaceFailureReason { switch (cloudUsageRefusalKey(error)) { case "serverError.cloudWorkspace.usageLimitReached": @@ -62,14 +72,48 @@ function provisionFailureReason(error: unknown): CloudWorkspaceFailureReason { case "serverError.cloudWorkspace.cloudNeedsPaidPlan": return "needs_paid_plan"; } - return error instanceof SandboxNotReadyError + return error instanceof SandboxNotReadyError || + error instanceof SandboxDriveRefusedError ? "host_not_ready" : "create_failed"; } +/** + * The name is on the row before the drive exists, so a delete that races + * this provision still finds it. An unarchived workspace keeps its drive when + * it still exists; a drive forked afresh has not been laid out yet. + */ +async function claimWorkspaceDrive( + row: CloudWorkspaceRow, + driveName: string, + environment: SandboxEnvironment, +): Promise<"existing" | "forked"> { + if (!row.driveName) { + await db + .update(cloudWorkspaces) + .set({ driveName }) + .where(eq(cloudWorkspaces.id, row.id)); + } + const outcome = await Sentry.startSpan({ name: "drive", op: "sandbox" }, () => + forkWorkspaceDrive({ + name: driveName, + region: environment.region, + goldenDriveName: environment.goldenDriveName ?? null, + }), + ); + if (outcome === "forked" && row.driveLaidOutAt) { + await db + .update(cloudWorkspaces) + .set({ driveLaidOutAt: null }) + .where(eq(cloudWorkspaces.id, row.id)); + } + return outcome; +} + export type ProvisionCloudWorkspaceOutcome = | "provisioned" | "skipped" + | "retrying" | "failed"; /** @@ -111,6 +155,10 @@ async function provision( where: eq(cloudWorkspaces.id, input.cloudWorkspaceId), }); if (!row) return "skipped"; + const scope: ControlScope = { + cloudWorkspaceId: row.id, + organizationId: row.organizationId, + }; if (row.status !== "provisioning") return "skipped"; const providerSandboxId = row.providerSandboxId; @@ -128,11 +176,25 @@ async function provision( }, ); let bootId: string | null = null; + let driveName = row.driveName; + // Only a drive this provision forked may go with a failed box; an unarchived workspace's is its work. + let forkedDrive = false; + // Another delivery on the same box name, or a wake, may have brought the row up: only this job's own row is its to fail. + let madeAwake = false; try { - const { claim, environment, agentCredentialDigest } = - await Sentry.startSpan({ name: "claim", op: "sandbox" }, () => - buildSandboxClaim({ row, launch: input.launch, withRepoHooks: true }), - ); + const { + claim, + environment, + agentCredentialDigest, + driveName: claimedDrive, + } = await Sentry.startSpan({ name: "claim", op: "sandbox" }, () => + buildSandboxClaim({ + row, + launch: input.launch, + withRepoHooks: true, + provisioning: true, + }), + ); ({ bootId } = await Sentry.startSpan( { name: "authorize boot", op: "billing" }, () => @@ -145,31 +207,61 @@ async function provision( traceId: span.spanContext().traceId, }), )); - if (bootId) span.setAttribute("boot.id", bootId); + if (bootId) { + scope.bootId = bootId; + span.setAttribute("boot.id", bootId); + } + driveName = claimedDrive; + if (driveName) { + forkedDrive = + (await claimWorkspaceDrive(row, driveName, environment)) === "forked"; + } const sandbox = await Sentry.startSpan( { name: "create", op: "sandbox" }, - () => provisionSandbox({ name: providerSandboxId, environment, claim }), + () => + provisionSandbox({ + name: providerSandboxId, + environment, + claim, + mounts: workspaceDriveMounts(driveName), + exclusive: sandboxResumeLock(driveName ?? providerSandboxId, { + wait: true, + }), + }), ); const ready = await transitionCloudWorkspace({ id: row.id, from: ["provisioning"], to: "awake", + onBox: providerSandboxId, set: { providerSandboxId: sandbox.providerSandboxId, sandboxUrl: sandbox.sandboxUrl, bootAgentCredentialDigest: agentCredentialDigest, }, }); + madeAwake = ready; if (!ready) { - // Deleted while the box was being made: the delete won the row, so - // the box it never knew about goes with it. A duplicate delivery of - // this job loses the row to the first one and must leave the box be: - // the name is per workspace, so it is the same box. + // Deleted while the box was being made. The box is the row's, and the + // archive's own reap stops and deletes it after that archive's grace + // period: an undo, more work and a later archive must not lose it to + // this job. A duplicate delivery loses the row to the first one. const current = await db.query.cloudWorkspaces.findFirst({ where: eq(cloudWorkspaces.id, row.id), - columns: { status: true }, + columns: { status: true, providerSandboxId: true }, }); - if (current?.status === "archived") { + // An undo gave the row another box and a second provision: this box is no row's, and must not keep the drive. + const overtaken = + current !== undefined && + current.providerSandboxId !== providerSandboxId; + if (overtaken) { + if (bootId) await releaseBoot(bootId, "provision_abandoned"); + await deleteSandboxAndDriveLater({ + providerSandboxId, + driveName, + keepDrive: true, + }); + } else if (current?.status === "archived") { await failChatMovesTo(row.id, "The cloud workspace was deleted").catch( (error) => console.error( @@ -177,7 +269,6 @@ async function provision( error, ), ); - await deleteSandbox(providerSandboxId); if (bootId) await releaseBoot(bootId, "provision_abandoned"); } await naming.catch(() => {}); @@ -192,13 +283,7 @@ async function provision( } if (bootId) { await attachBootSession({ bootId, providerSandboxId }).catch((error) => - Sentry.captureException(error, { - tags: controlTags({ - cloudWorkspaceId: row.id, - organizationId: row.organizationId, - bootId, - }), - }), + Sentry.captureException(error, { tags: controlTags(scope) }), ); } // The box is booting; the environment it needs arrives once host-service @@ -211,25 +296,71 @@ async function provision( providerSandboxId, hostTarget: sandbox.hostTarget, claim, + bootId: sandbox.bootId, }), ); + if (driveName) await recordDriveLaidOut(row.id); await queueChatMovesTo(row.id).catch((error) => console.error( `[cloud-workspace] could not start the chat moving into ${row.id}`, error, ), ); - await recordBootTrace( - { cloudWorkspaceId: row.id, organizationId: row.organizationId, bootId }, - span, - bootPhases(answer), - ); + await recordBootTrace(scope, span, bootPhases(answer)); await naming.catch((error) => console.error(`[cloud-workspace] naming failed for ${row.id}`, error), ); return "provisioned"; } catch (error) { await naming.catch(() => {}); + // Another caller holds the box's lock: try again later, and touch nothing meanwhile. + const lockRetries = input.lockRetries ?? 0; + if ( + error instanceof SandboxBusyError && + lockRetries < PROVISION_LOCK_RETRIES + ) { + if (bootId) { + await releaseBoot(bootId, "provision_abandoned").catch((releaseError) => + console.error( + `[cloud-workspace] could not release boot ${bootId}`, + releaseError, + ), + ); + } + await publishCloudWorkspaceJob({ + path: "/api/cloud-workspaces/provision", + body: { ...input, lockRetries: lockRetries + 1 }, + delaySeconds: PROVISION_LOCK_RETRY_SECONDS, + runLocally: provisionCloudWorkspace, + }); + return "retrying"; + } + // The drive did not come up but its data is intact: the box sleeps, and the next wake mounts it afresh. + if (error instanceof SandboxDriveMissingError && madeAwake) { + const stopped = await stopSandbox(providerSandboxId).catch(() => ({ + sessionId: null, + })); + await recordSandboxAsleep(row.id, stopped); + nudge(row.organizationId, "cloud_workspaces"); + if (bootId) { + await releaseBoot(bootId, "provision_failed").catch((releaseError) => + console.error( + `[cloud-workspace] could not release boot ${bootId}`, + releaseError, + ), + ); + } + await queueChatMovesTo(row.id).catch((moveError) => + console.error( + `[cloud-workspace] could not start the chat moving into ${row.id}`, + moveError, + ), + ); + console.warn( + `[cloud-workspace] ${row.id} left asleep: its drive did not come up at create`, + ); + return "provisioned"; + } if (bootId) { await releaseBoot(bootId, "provision_failed").catch((releaseError) => console.error( @@ -243,33 +374,48 @@ async function provision( // away, it costs storage rather than compute meanwhile, and a delete // from the sidebar still removes it. Any other failure must not leak // a box, since billing started at provision. - const keepForDiagnosis = error instanceof SandboxNotReadyError; - await (keepForDiagnosis ? stopSandbox : deleteSandbox)( - providerSandboxId, - ).catch((teardownError) => { + const keepForDiagnosis = + error instanceof SandboxNotReadyError || + error instanceof SandboxDriveRefusedError; + // A box that goes is retired with the transition, so nothing can wake it while it is deleted. + const changed = await failCloudWorkspace({ + scope, + from: madeAwake ? ["provisioning", "awake", "asleep"] : ["provisioning"], + onBox: providerSandboxId, + set: keepForDiagnosis + ? { providerSandboxId } + : { + providerSandboxId: nextSandboxNameFor(row.id), + ...(forkedDrive ? { driveName: null, driveLaidOutAt: null } : {}), + }, + reason: provisionFailureReason(error), + error, + }); + const teardownFailed = (teardownError: unknown) => console.error( `[cloud-workspace] ${keepForDiagnosis ? "could not stop" : "leaked"} sandbox ${providerSandboxId}`, teardownError, ); - }); - await failCloudWorkspace({ - scope: { - cloudWorkspaceId: row.id, - organizationId: row.organizationId, - bootId, - }, - from: ["provisioning", "awake", "asleep"], - set: keepForDiagnosis ? { providerSandboxId } : {}, - reason: provisionFailureReason(error), - error, - }); - await failChatMovesTo(row.id, "The cloud workspace could not start").catch( - (moveError) => + if (changed && keepForDiagnosis) { + await stopSandbox(providerSandboxId).catch(teardownFailed); + } + if (changed && !keepForDiagnosis) { + await deleteSandboxAndDriveLater({ + providerSandboxId, + driveName: forkedDrive ? driveName : null, + }).catch(teardownFailed); + } + if (changed) { + await failChatMovesTo( + row.id, + "The cloud workspace could not start", + ).catch((moveError) => console.error( `[cloud-workspace] could not fail the chat moves into ${row.id}`, moveError, ), - ); + ); + } console.error(`[cloud-workspace] provisioning failed for ${row.id}`, error); return "failed"; } diff --git a/packages/trpc/src/router/cloud-workspace/reap.test.ts b/packages/trpc/src/router/cloud-workspace/reap.test.ts index b200c936b57..3d4c903e7f8 100644 --- a/packages/trpc/src/router/cloud-workspace/reap.test.ts +++ b/packages/trpc/src/router/cloud-workspace/reap.test.ts @@ -28,6 +28,22 @@ let onArchiveHook: () => void = () => {}; let reapedFiles: string[] = []; stub(db.query.cloudWorkspaces, { findFirst: () => Promise.resolve(row) }); +// Whether the retiring write still finds the row as the reap read it. +let retires = true; +let retiredTo: Record | null = null; +stub(db, { + update: () => ({ + set: (values: Record) => ({ + where: () => ({ + returning: async () => { + if (!retires) return []; + retiredTo = values; + return [{ id: "row" }]; + }, + }), + }), + }), +}); let recordedAsleep: Array<[string, { sessionId: string | null }]> = []; let endedBoots: unknown[] = []; stub(sandboxSleep, { @@ -187,6 +203,8 @@ describe("reapArchivedCloudWorkspace", () => { reapedFiles = []; recordedAsleep = []; endedBoots = []; + retires = true; + retiredTo = null; }); test("the stop stage ends the stopped session's boot as archived", async () => { @@ -260,6 +278,36 @@ describe("reapArchivedCloudWorkspace", () => { expect(sandboxCalls).toEqual(["delete:ws-box"]); }); + test("the delete stage retires the box's names before it deletes them, so an undo gets a new box and drive", async () => { + row = { ...(row as Row), driveName: "drive-ws-box" } as Row; + expect( + await reapArchivedCloudWorkspace({ ...input, stage: "delete" }), + ).toBe("reaped"); + expect(retiredTo).toMatchObject({ driveName: null, driveLaidOutAt: null }); + expect(retiredTo?.providerSandboxId).not.toBe("ws-box"); + expect(queued).toContainEqual( + expect.objectContaining({ + path: "/api/cloud-workspaces/teardown", + body: { + providerSandboxId: "ws-box", + driveName: "drive-ws-box", + attempt: 0, + }, + }), + ); + }); + + test("an undo that wins the row before the retire leaves the box and drive alone", async () => { + row = { ...(row as Row), driveName: "drive-ws-box" } as Row; + retires = false; + expect( + await reapArchivedCloudWorkspace({ ...input, stage: "delete" }), + ).toBe("skipped"); + expect(sandboxCalls).toEqual([]); + expect(queued).toEqual([]); + expect(reapedFiles).toEqual([]); + }); + test("the delete stage removes the files sent to the workspace", async () => { await reapArchivedCloudWorkspace({ ...input, stage: "delete" }); expect(reapedFiles).toEqual([`file-of-${input.cloudWorkspaceId}`]); diff --git a/packages/trpc/src/router/cloud-workspace/reap.ts b/packages/trpc/src/router/cloud-workspace/reap.ts index 1f815c4a577..271bb126be2 100644 --- a/packages/trpc/src/router/cloud-workspace/reap.ts +++ b/packages/trpc/src/router/cloud-workspace/reap.ts @@ -1,10 +1,9 @@ import { db } from "@superset/db/client"; import { cloudWorkspaces } from "@superset/db/schema"; -import { eq } from "drizzle-orm"; +import { and, eq } from "drizzle-orm"; import { logControl } from "../../lib/control-log"; import { detachAll, reapOrphanFiles } from "../../lib/files"; import { - deleteSandbox, describeSandbox, rerunBoxStartHook, runBoxArchiveHook, @@ -14,6 +13,7 @@ import { import { publishCloudWorkspaceJob } from "./jobs"; import { queueSettleWorkspace, settleWorkspaceNow } from "./meter"; import { recordBootEnd, recordBoxStopped } from "./sandbox-sleep"; +import { retireAndTearDown } from "./teardown"; /** How long an archived workspace keeps its box running, so an undo finds it live. */ export const ARCHIVE_STOP_DELAY_SECONDS = 60; @@ -96,7 +96,15 @@ export async function reapArchivedCloudWorkspace( await settleWorkspaceNow(row.id).catch((error) => console.error(`[cloud-workspace] ${row.id} final settle failed`, error), ); - await deleteSandbox(row.providerSandboxId); + // The names go with this archive's deletion; an undo after it gets a new box and drive. + const retired = await retireAndTearDown( + row, + and( + eq(cloudWorkspaces.status, "archived"), + eq(cloudWorkspaces.deletedAt, new Date(input.archivedAt)), + ), + ); + if (!retired) return "skipped"; await reapWorkspaceFiles(row.id); logControl("archive.deleted", { cloudWorkspaceId: row.id, diff --git a/packages/trpc/src/router/cloud-workspace/resume-lock.ts b/packages/trpc/src/router/cloud-workspace/resume-lock.ts new file mode 100644 index 00000000000..cc50824927c --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/resume-lock.ts @@ -0,0 +1,41 @@ +import { dbWs } from "@superset/db/client"; +import { sql } from "drizzle-orm"; +import { type ResumeLock, SandboxBusyError } from "../../lib/sandbox"; + +/** How long a provision waits for another holder before it requeues itself. */ +const PROVISION_LOCK_WAIT = "20s"; + +/** + * One caller at a time resumes or creates a box on this key: the drive's name + * when there is one, since a drive has one writer whatever the box is + * called, else the box's. A transaction-scoped advisory lock, held while the + * step runs. Another holder throws SandboxBusyError: a wake answers "still + * starting" at once, and its retry sees the box as the first left it; a + * provision waits a moment, then requeues itself rather than fail its row. + */ +export function sandboxResumeLock( + key: string, + { wait = false }: { wait?: boolean } = {}, +): ResumeLock { + const lockKey = `sandbox-resume:${key}`; + return (step) => + dbWs.transaction(async (tx) => { + if (wait) { + await tx.execute( + sql`select set_config('lock_timeout', ${PROVISION_LOCK_WAIT}, true)`, + ); + await tx + .execute(sql`select pg_advisory_xact_lock(hashtext(${lockKey}))`) + .catch(() => { + throw new SandboxBusyError(key); + }); + return step(); + } + const result = await tx.execute( + sql`select pg_try_advisory_xact_lock(hashtext(${lockKey})) as locked`, + ); + const [row] = result.rows as Array<{ locked: boolean }>; + if (!row?.locked) throw new SandboxBusyError(key); + return step(); + }); +} diff --git a/packages/trpc/src/router/cloud-workspace/sandbox-name.ts b/packages/trpc/src/router/cloud-workspace/sandbox-name.ts new file mode 100644 index 00000000000..4c306b21e32 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/sandbox-name.ts @@ -0,0 +1,10 @@ +import { randomBytes } from "node:crypto"; + +/** + * A later box for the same row. Vercel refuses a deleted sandbox's name + * with `image_not_ready` for a few minutes after the delete. + */ +export function nextSandboxNameFor(cloudWorkspaceId: string): string { + const prefix = cloudWorkspaceId.replaceAll("-", "").slice(0, 16); + return `ws-${prefix}${randomBytes(4).toString("hex")}`; +} diff --git a/packages/trpc/src/router/cloud-workspace/teardown-guard.ts b/packages/trpc/src/router/cloud-workspace/teardown-guard.ts new file mode 100644 index 00000000000..cd3987b2743 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/teardown-guard.ts @@ -0,0 +1,57 @@ +import { db } from "@superset/db/client"; +import { cloudWorkspaces, environments } from "@superset/db/schema"; +import { and, eq, isNull } from "drizzle-orm"; + +/** + * Whether a workspace, in any status, or a live environment still names the + * sandbox or the drive. Every real teardown clears the row's names in the + * write that commits to it (retireAndTearDown), so a name still on a row is + * that row's: an archived row may be undone onto it, and a failed one keeps + * its box and drive for diagnosis with the work in them. + */ +export async function liveReferences(args: { + providerSandboxId: string; + driveName: string | null; +}): Promise<{ sandbox: boolean; drive: boolean }> { + const [workspaceBox, goldenBox, workspaceDrive, goldenDrive] = + await Promise.all([ + db + .select({ id: cloudWorkspaces.id }) + .from(cloudWorkspaces) + .where(eq(cloudWorkspaces.providerSandboxId, args.providerSandboxId)) + .limit(1), + db + .select({ id: environments.id }) + .from(environments) + .where( + and( + eq(environments.sourceRef, args.providerSandboxId), + isNull(environments.archivedAt), + ), + ) + .limit(1), + args.driveName + ? db + .select({ id: cloudWorkspaces.id }) + .from(cloudWorkspaces) + .where(eq(cloudWorkspaces.driveName, args.driveName)) + .limit(1) + : [], + args.driveName + ? db + .select({ id: environments.id }) + .from(environments) + .where( + and( + eq(environments.goldenDriveName, args.driveName), + isNull(environments.archivedAt), + ), + ) + .limit(1) + : [], + ]); + return { + sandbox: workspaceBox.length > 0 || goldenBox.length > 0, + drive: workspaceDrive.length > 0 || goldenDrive.length > 0, + }; +} diff --git a/packages/trpc/src/router/cloud-workspace/teardown.test.ts b/packages/trpc/src/router/cloud-workspace/teardown.test.ts new file mode 100644 index 00000000000..5cc817575f2 --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/teardown.test.ts @@ -0,0 +1,229 @@ +import { beforeEach, describe, expect, test } from "bun:test"; +import { stub } from "../../../test/stub"; +import * as sandbox from "../../lib/sandbox"; +import * as jobs from "./jobs"; +import { + deleteSandboxAndDriveLater, + TEARDOWN_ATTEMPTS, + teardownSandbox, +} from "./teardown"; +import * as guard from "./teardown-guard"; + +let calls: string[]; +let attached: boolean; +let queued: Array<{ body: unknown; delaySeconds?: number }>; +let live: { sandbox: boolean; drive: boolean }; +let stopFails: boolean; +let holder: string; +let driveDeleteFails: boolean; + +stub(sandbox, { + stopSandbox: async (id: string) => { + if (stopFails) throw new Error("provider down"); + calls.push(`stop:${id}`); + return { sessionId: null }; + }, + waitForDriveDetached: async ( + name: string, + options: { heldByCaller?: (attachedTo: string) => Promise } = {}, + ) => { + calls.push(`wait:${name}`); + if (!attached) return "detached"; + if (await options.heldByCaller?.(holder)) return "held"; + throw new sandbox.DriveStillAttachedError(name, holder); + }, + deleteSandbox: async (id: string) => { + calls.push(`delete:${id}`); + }, + deleteWorkspaceDrive: async (name: string | null) => { + if (driveDeleteFails) throw new Error("drive delete refused"); + if (name) calls.push(`delete-drive:${name}`); + }, +}); +stub(guard, { liveReferences: async () => live }); +stub(jobs, { + publishCloudWorkspaceJob: async (args: { + body: unknown; + delaySeconds?: number; + }) => { + queued.push({ body: args.body, delaySeconds: args.delaySeconds }); + }, +}); + +beforeEach(() => { + calls = []; + attached = false; + queued = []; + live = { sandbox: false, drive: false }; + stopFails = false; + holder = "box"; + driveDeleteFails = false; +}); + +describe("teardownSandbox", () => { + test("stops the box, waits for its drive, then deletes the box and the drive", async () => { + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("deleted"); + expect(calls).toEqual([ + "stop:box", + "wait:drive-box", + "delete:box", + "delete-drive:drive-box", + ]); + }); + + test("a drive still attached requeues the job and deletes nothing", async () => { + attached = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 3, + }), + ).toBe("retrying"); + expect(calls).toEqual(["stop:box", "wait:drive-box"]); + expect(queued).toEqual([ + { + body: { providerSandboxId: "box", driveName: "drive-box", attempt: 4 }, + delaySeconds: 30, + }, + ]); + }); + + test("a box a revived workspace runs on is left alone", async () => { + live = { sandbox: true, drive: true }; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("kept"); + expect(calls).toEqual([]); + }); + + test("an unarchived workspace's new box keeps the drive; only the old box goes", async () => { + live = { sandbox: false, drive: true }; + expect( + await teardownSandbox({ + providerSandboxId: "old-box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("kept"); + expect(calls).toEqual(["stop:old-box", "wait:drive-box", "delete:old-box"]); + }); + + test("a box of the row's lost provision goes; the drive the row runs on is never deleted, failed or not", async () => { + // The row has since gone failed, and still names the drive with the user's work on it. + live = { sandbox: false, drive: false }; + expect( + await teardownSandbox({ + providerSandboxId: "lost-box", + driveName: "drive-box", + attempt: 0, + keepDrive: true, + }), + ).toBe("deleted"); + expect(calls).toEqual([ + "stop:lost-box", + "wait:drive-box", + "delete:lost-box", + ]); + }); + + test("a lost box goes even while the row's live box holds the kept drive", async () => { + attached = true; + holder = "live-box"; + expect( + await teardownSandbox({ + providerSandboxId: "lost-box", + driveName: "drive-box", + attempt: 0, + keepDrive: true, + }), + ).toBe("deleted"); + expect(calls).toEqual([ + "stop:lost-box", + "wait:drive-box", + "delete:lost-box", + ]); + expect(queued).toEqual([]); + }); + + test("any other failure is retried, not dropped", async () => { + stopFails = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("retrying"); + expect(queued).toHaveLength(1); + }); + + test("a drive a golden environment or another row names is never deleted", async () => { + live = { sandbox: false, drive: true }; + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-golden", + attempt: 0, + }); + expect(calls).not.toContain("delete-drive:drive-golden"); + }); + + test("a drive delete that fails is retried", async () => { + driveDeleteFails = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: 0, + }), + ).toBe("retrying"); + expect(queued).toEqual([ + { + body: { providerSandboxId: "box", driveName: "drive-box", attempt: 1 }, + delaySeconds: 30, + }, + ]); + }); + + test("gives up after the last attempt instead of requeuing forever", async () => { + attached = true; + expect( + await teardownSandbox({ + providerSandboxId: "box", + driveName: "drive-box", + attempt: TEARDOWN_ATTEMPTS - 1, + }), + ).toBe("abandoned"); + expect(queued).toEqual([]); + }); +}); + +describe("deleteSandboxAndDriveLater", () => { + test("a box without a drive is deleted now; one with a drive goes to the job", async () => { + await deleteSandboxAndDriveLater({ + providerSandboxId: "plain", + driveName: null, + }); + await deleteSandboxAndDriveLater({ + providerSandboxId: "box", + driveName: "drive-box", + }); + expect(calls).toEqual(["delete:plain"]); + expect(queued).toEqual([ + { + body: { providerSandboxId: "box", driveName: "drive-box", attempt: 0 }, + delaySeconds: undefined, + }, + ]); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/teardown.ts b/packages/trpc/src/router/cloud-workspace/teardown.ts new file mode 100644 index 00000000000..1b4322058ee --- /dev/null +++ b/packages/trpc/src/router/cloud-workspace/teardown.ts @@ -0,0 +1,167 @@ +import * as Sentry from "@sentry/core"; +import { db } from "@superset/db/client"; +import { cloudWorkspaces } from "@superset/db/schema"; +import { and, eq, type SQL } from "drizzle-orm"; +import { + DriveStillAttachedError, + deleteSandbox, + deleteWorkspaceDrive, + stopSandbox, + waitForDriveDetached, +} from "../../lib/sandbox"; +import { publishCloudWorkspaceJob } from "./jobs"; +import { nextSandboxNameFor } from "./sandbox-name"; +import { liveReferences } from "./teardown-guard"; + +export interface TeardownSandboxInput { + providerSandboxId: string; + driveName: string | null; + attempt: number; + /** The drive is only waited for, never deleted: another box of the row runs on it. */ + keepDrive?: boolean; +} + +/** One attempt fits a short request; a drive still flushing is tried again later. */ +export const TEARDOWN_DETACH_WAIT_MS = 20_000; +export const TEARDOWN_RETRY_SECONDS = 30; +export const TEARDOWN_ATTEMPTS = 20; + +export type TeardownOutcome = "deleted" | "kept" | "retrying" | "abandoned"; + +/** + * Stops the box, waits briefly for its drive to detach, then deletes the box + * and the drive. A drive still attached, or any other failure, requeues the + * job rather than wait in the caller. Before each destructive call it reads + * the rows again: what a live workspace or environment names is kept. + */ +export async function teardownSandbox( + input: TeardownSandboxInput, +): Promise { + try { + return await teardownOnce(input); + } catch (error) { + Sentry.captureException(error, { extra: { ...input } }); + return retryOrAbandon(input, error); + } +} + +async function teardownOnce( + input: TeardownSandboxInput, +): Promise { + if ((await liveReferences(input)).sandbox) return kept(input, "sandbox"); + await stopSandbox(input.providerSandboxId); + if (input.driveName) { + try { + await waitForDriveDetached(input.driveName, { + timeoutMs: TEARDOWN_DETACH_WAIT_MS, + // A kept drive only has to leave this box; the row's other box may hold it for good. + ...(input.keepDrive + ? { + heldByCaller: async (attachedTo: string) => + attachedTo !== input.providerSandboxId, + } + : {}), + }); + } catch (error) { + if (!(error instanceof DriveStillAttachedError)) throw error; + return retryOrAbandon(input, error); + } + } + const live = await liveReferences(input); + if (live.sandbox) return kept(input, "sandbox"); + await deleteSandbox(input.providerSandboxId); + if (input.keepDrive) return "deleted"; + if (live.drive) return kept(input, "drive"); + if ((await liveReferences(input)).drive) return kept(input, "drive"); + await deleteWorkspaceDrive(input.driveName); + return "deleted"; +} + +function kept( + input: TeardownSandboxInput, + what: "sandbox" | "drive", +): TeardownOutcome { + console.log( + `[cloud-workspace] teardown kept the ${what} of ${input.providerSandboxId}: a live row names it`, + ); + return "kept"; +} + +async function retryOrAbandon( + input: TeardownSandboxInput, + error: unknown, +): Promise { + if (input.attempt + 1 >= TEARDOWN_ATTEMPTS) { + Sentry.captureMessage( + `cloud workspace teardown gave up on ${input.providerSandboxId} and its drive ${input.driveName}`, + { level: "error", extra: { ...input, error: String(error) } }, + ); + return "abandoned"; + } + await queueTeardown( + { ...input, attempt: input.attempt + 1 }, + TEARDOWN_RETRY_SECONDS, + ); + return "retrying"; +} + +function queueTeardown( + input: TeardownSandboxInput, + delaySeconds?: number, +): Promise { + return publishCloudWorkspaceJob({ + path: "/api/cloud-workspaces/teardown", + body: input, + delaySeconds, + runLocally: teardownSandbox, + }); +} + +/** A box without a drive is deleted now, as before; one with a drive through the retried job. */ +export async function deleteSandboxAndDriveLater(args: { + providerSandboxId: string; + driveName: string | null | undefined; + keepDrive?: boolean; +}): Promise { + if (!args.driveName) return deleteSandbox(args.providerSandboxId); + await queueTeardown({ + providerSandboxId: args.providerSandboxId, + driveName: args.driveName, + attempt: 0, + ...(args.keepDrive ? { keepDrive: true } : {}), + }); +} + +/** + * Takes the row's box and drive names out of use in the same write that + * commits to deleting them, then queues their teardown. An unarchive after it + * finds a name that does not exist and gets a new box and a new drive, never + * the ones being deleted. Returns false when the row no longer matches + * `condition` or names another box: whatever changed it owns the names now. + */ +export async function retireAndTearDown( + row: { id: string; providerSandboxId: string; driveName: string | null }, + condition: SQL | undefined, +): Promise { + const [retired] = await db + .update(cloudWorkspaces) + .set({ + providerSandboxId: nextSandboxNameFor(row.id), + driveName: null, + driveLaidOutAt: null, + }) + .where( + and( + eq(cloudWorkspaces.id, row.id), + eq(cloudWorkspaces.providerSandboxId, row.providerSandboxId), + condition, + ), + ) + .returning({ id: cloudWorkspaces.id }); + if (!retired) return false; + await deleteSandboxAndDriveLater({ + providerSandboxId: row.providerSandboxId, + driveName: row.driveName, + }); + return true; +} diff --git a/packages/trpc/src/router/cloud-workspace/transition.ts b/packages/trpc/src/router/cloud-workspace/transition.ts index 7c26d5cfbe4..f1a94d8fc8f 100644 --- a/packages/trpc/src/router/cloud-workspace/transition.ts +++ b/packages/trpc/src/router/cloud-workspace/transition.ts @@ -16,6 +16,8 @@ type Move = { id: string; from: readonly CloudWorkspaceStatus[]; set?: Partial; + /** Only while the row still names this box: an unarchive may have given it another. */ + onBox?: string; }; async function moveCloudWorkspace( @@ -28,6 +30,9 @@ async function moveCloudWorkspace( and( eq(cloudWorkspaces.id, args.id), inArray(cloudWorkspaces.status, [...args.from]), + args.onBox === undefined + ? undefined + : eq(cloudWorkspaces.providerSandboxId, args.onBox), ), ) .returning({ id: cloudWorkspaces.id }); @@ -60,16 +65,18 @@ export async function failCloudWorkspace(args: { set?: Partial; reason: CloudWorkspaceFailureReason; error: unknown; -}): Promise { + onBox?: string; +}): Promise { const { scope } = args; const changed = await moveCloudWorkspace({ id: scope.cloudWorkspaceId, from: args.from, to: "failed", + onBox: args.onBox, set: { ...args.set, failureReason: args.reason, failedAt: new Date() }, }); nudge(scope.organizationId, "cloud_workspaces"); - if (!changed) return; + if (!changed) return false; const refused = isCloudUsageRefusal(args.error); logControl( "workspace.failed", @@ -80,4 +87,5 @@ export async function failCloudWorkspace(args: { if (!refused) { Sentry.captureException(args.error, { tags: controlTags(scope) }); } + return true; } diff --git a/packages/trpc/src/router/cloud-workspace/wake.test.ts b/packages/trpc/src/router/cloud-workspace/wake.test.ts index b40ad509d43..30fcde46bb2 100644 --- a/packages/trpc/src/router/cloud-workspace/wake.test.ts +++ b/packages/trpc/src/router/cloud-workspace/wake.test.ts @@ -9,6 +9,7 @@ import * as sandbox from "../../lib/sandbox"; import * as meter from "./meter"; import * as sandboxExpiry from "./sandbox-expiry"; import * as sandboxSleep from "./sandbox-sleep"; +import * as teardown from "./teardown"; import { markSandboxUnavailable, restartCloudWorkspace, @@ -440,3 +441,83 @@ describe("markSandboxUnavailable", () => { }); }); }); + +describe("a workspace whose snapshot expired", () => { + const retired: Array<{ + providerSandboxId: string; + driveName: string | null; + }> = []; + stub(realtime, { nudge: () => {} }); + stub(db, rowUpdate(true)); + stub(Sentry, { captureException: () => "" }); + stub(teardown, { + retireAndTearDown: async (row: { + providerSandboxId: string; + driveName: string | null; + }) => { + retired.push({ + providerSandboxId: row.providerSandboxId, + driveName: row.driveName, + }); + return true; + }, + }); + beforeEach(() => { + retired.length = 0; + }); + const row = { ...ROW, driveName: "drive-ws-box" } as Row; + const unavailable = (status: number) => + new sandbox.SandboxUnavailableError( + "ws-box", + new APIError(new Response(null, { status })), + ); + + test("retires its names and queues the teardown of its box and drive", async () => { + await markSandboxUnavailable(row, unavailable(410)); + expect(retired).toEqual([ + { providerSandboxId: "ws-box", driveName: "drive-ws-box" }, + ]); + }); + + test("a box deleted out from under the row keeps the drive", async () => { + await markSandboxUnavailable(row, unavailable(404)); + expect(retired).toEqual([]); + }); + + test("a drive refused for good is kept for diagnosis", async () => { + await markSandboxUnavailable( + row, + new sandbox.SandboxDriveRefusedError( + "ws-box", + "drive-ws-box", + "drive.foreign", + 3, + ), + ); + expect(retired).toEqual([]); + }); + + test("a workspace without a drive is left as it was", async () => { + await markSandboxUnavailable({ ...row, driveName: null }, unavailable(410)); + expect(retired).toEqual([]); + }); +}); + +describe("a wake of a workspace without a drive", () => { + const wakes: Array<{ mounts?: unknown }> = []; + stub(sandbox, { + buildSandboxClaim: async () => ({ claim: {}, agentCredentialDigest: "" }), + wakeSandbox: async (args: { mounts?: unknown }) => { + wakes.push(args); + throw new Error("wake not under test"); + }, + }); + + test("mounts nothing, whatever its environment", async () => { + await expect( + wakeCloudWorkspace({ ...ROW, driveName: null } as Row), + ).rejects.toThrow("wake not under test"); + expect(wakes).toHaveLength(1); + expect(wakes[0]?.mounts).toBeUndefined(); + }); +}); diff --git a/packages/trpc/src/router/cloud-workspace/wake.ts b/packages/trpc/src/router/cloud-workspace/wake.ts index 2d136ff6a1f..9c1d77aef98 100644 --- a/packages/trpc/src/router/cloud-workspace/wake.ts +++ b/packages/trpc/src/router/cloud-workspace/wake.ts @@ -1,5 +1,6 @@ import * as Sentry from "@sentry/core"; import { db } from "@superset/db/client"; +import type { CloudWorkspaceFailureReason } from "@superset/db/enums"; import { cloudWorkspaces } from "@superset/db/schema"; import { isCloudUsageRefusal } from "@superset/shared/billing"; import { @@ -7,7 +8,7 @@ import { CLOUD_WORKSPACE_WAKE_TRANSACTION, } from "@superset/shared/constants"; import { waitUntil } from "@vercel/functions"; -import { eq } from "drizzle-orm"; +import { and, eq } from "drizzle-orm"; import { type ControlScope, controlTags, @@ -18,22 +19,27 @@ import { bootPhases, buildSandboxClaim, describeSandbox, + SandboxDriveRefusedError, type SandboxUnavailableError, stopAndSnapshot, stopSandbox, wakeSandbox, + workspaceDriveMounts, } from "../../lib/sandbox"; +import { recordDriveLaidOut } from "./drive-laid-out"; import { attachBootSession, authorizeBoot, queueSettleWorkspace, } from "./meter"; +import { sandboxResumeLock } from "./resume-lock"; import { recordSandboxAwake } from "./sandbox-expiry"; import { recordBootTrace, recordBoxStopped, recordSandboxAsleep, } from "./sandbox-sleep"; +import { retireAndTearDown } from "./teardown"; import { failCloudWorkspace } from "./transition"; type CloudWorkspaceRow = typeof cloudWorkspaces.$inferSelect; @@ -108,6 +114,8 @@ async function wake( providerSandboxId: row.providerSandboxId, claim, onlyIfRunning: adoptSignIns, + mounts: workspaceDriveMounts(row.driveName), + exclusive: sandboxResumeLock(row.driveName ?? row.providerSandboxId), beforeResume: async ({ resumedSession, ...box }) => { resumed = resumedSession; ({ bootId } = await Sentry.startSpan( @@ -193,6 +201,8 @@ async function wake( .where(eq(cloudWorkspaces.id, row.id)); } if (expiresAt) await recordSandboxAwake(row.id, { expiresAt, sessionId }); + // settleSandbox only answers for a drive whose boot laid it out. + if (row.driveName && !row.driveLaidOutAt) await recordDriveLaidOut(row.id); if (rebooted) { logControl("wake.completed", scope, { ms: Date.now() - wakeStartedAt, @@ -236,6 +246,14 @@ export async function wakeAfterStop( return wakeCloudWorkspace(row); } +/** A drive that refused reads as any box that would not start. */ +function unavailableReason( + error: SandboxUnavailableError, +): CloudWorkspaceFailureReason { + if (error instanceof SandboxDriveRefusedError) return "host_not_ready"; + return error.snapshotExpired ? "snapshot_expired" : "sandbox_gone"; +} + /** * For a sandbox that is gone or can never resume. An `awake` row nothing can * open would sit in the sidebar forever; failed is the state the client @@ -250,7 +268,21 @@ export async function markSandboxUnavailable( scope: { cloudWorkspaceId: row.id, organizationId: row.organizationId }, from: ["awake", "asleep"], set: { sandboxUrl: null }, - reason: error.snapshotExpired ? "snapshot_expired" : "sandbox_gone", + reason: unavailableReason(error), error, }); + if (row.driveName && error.snapshotExpired) { + await retireAndTearDown( + row, + and( + eq(cloudWorkspaces.status, "failed"), + eq(cloudWorkspaces.driveName, row.driveName), + ), + ).catch((teardownError) => + console.error( + `[cloud-workspace] ${row.id} teardown after snapshot expiry`, + teardownError, + ), + ); + } } diff --git a/packages/trpc/src/router/environment/environment.test.ts b/packages/trpc/src/router/environment/environment.test.ts index 76404214510..076f1042615 100644 --- a/packages/trpc/src/router/environment/environment.test.ts +++ b/packages/trpc/src/router/environment/environment.test.ts @@ -6,6 +6,7 @@ import * as guards from "../../lib/cloud-guards"; import * as sandbox from "../../lib/sandbox"; import { userError } from "../../trpc"; import * as access from "../cloud-workspace/access"; +import * as teardown from "../cloud-workspace/teardown"; import * as wake from "../cloud-workspace/wake"; import * as integration from "../integration/utils"; import * as currentBundle from "./current-bundle"; @@ -56,9 +57,26 @@ stub(access, { status: "awake", baseBranch: "main", branch: "main", + providerSandboxId: "ws-box", + driveName: "drive-ws-box", }), }); -stub(sandbox, { workspaceRepositories: async () => [] }); +let drivesOn = true; +let promoted = 0; +let abandoned: Array<{ providerSandboxId: string; driveName?: string | null }> = + []; +stub(sandbox, { + workspaceRepositories: async () => [], + workspaceDrivesEnabled: async () => drivesOn, +}); +stub(teardown, { + deleteSandboxAndDriveLater: async (args: { + providerSandboxId: string; + driveName?: string | null; + }) => { + abandoned.push(args); + }, +}); stub(plan, { resolveMachineSize: async (_: string, name: string | null | undefined) => name === "large" || name == null @@ -95,6 +113,9 @@ beforeEach(() => { }; isAdmin = false; written = []; + abandoned = []; + drivesOn = true; + promoted = 0; }); describe("environment.update machineSize", () => { @@ -111,6 +132,17 @@ describe("environment.update machineSize", () => { expect(written).toEqual([{ machineSize: null }]); }); + test("the output never carries the golden drive, which released clients never saw", async () => { + isAdmin = true; + environment.goldenDriveName = "env-drive-env-g"; + const updated = await caller.environment.update({ + id: ENVIRONMENT_ID, + machineSize: null, + }); + expect(updated).not.toHaveProperty("goldenDriveName"); + expect(updated).toMatchObject({ id: ENVIRONMENT_ID }); + }); + test("refuses a size the plan does not offer", async () => { environment.createdByUserId = "user-1"; await expect( @@ -187,3 +219,52 @@ describe("environment.promote of a running workspace", () => { expect(order).toEqual(["save", "restart"]); }); }); + +const failingPromote = { + promoteSandboxToEnvironment: async () => { + promoted++; + throw new Error("snapshot failed after the fork"); + }, +}; + +describe("environment.promote that fails", () => { + stub(sandbox, failingPromote); + test("hands the golden and its forked drive to the retried teardown, and tells the user only that the promote failed", async () => { + environment.createdByUserId = "user-1"; + const failure = await caller.environment + .promote({ + cloudWorkspaceId: WORKSPACE_ID, + environmentId: ENVIRONMENT_ID, + }) + .catch((error: Error) => error); + expect(failure).toMatchObject({ + message: + "This workspace could not be saved as an environment. Try again later.", + }); + expect(abandoned).toHaveLength(1); + const [golden] = abandoned; + expect(golden?.providerSandboxId).toMatch(/^env-/); + expect(golden?.driveName).toBe(`env-drive-${golden?.providerSandboxId}`); + expect(written).toEqual([]); + }); +}); + +describe("environment.promote with drives off", () => { + stub(sandbox, failingPromote); + test("a workspace whose files are on a drive does not spread drive mode to a new golden", async () => { + environment.createdByUserId = "user-1"; + drivesOn = false; + await expect( + caller.environment.promote({ + cloudWorkspaceId: WORKSPACE_ID, + environmentId: ENVIRONMENT_ID, + }), + ).rejects.toMatchObject({ + code: "PRECONDITION_FAILED", + message: + "This workspace could not be saved as an environment. Try again later.", + }); + expect(promoted).toBe(0); + expect(abandoned).toEqual([]); + }); +}); diff --git a/packages/trpc/src/router/environment/environment.ts b/packages/trpc/src/router/environment/environment.ts index 060826df00e..f841b708dc0 100644 --- a/packages/trpc/src/router/environment/environment.ts +++ b/packages/trpc/src/router/environment/environment.ts @@ -12,6 +12,7 @@ import { isCloudUsageRefusal, MACHINE_SIZES } from "@superset/shared/billing"; import { SANDBOX_IMAGE_NAME } from "@superset/shared/constants"; import { SANDBOX_REGION_IDS } from "@superset/shared/sandbox/regions"; import type { TRPCRouterRecord } from "@trpc/server"; +import { TRPCError } from "@trpc/server"; import { and, asc, eq, inArray, isNull, or } from "drizzle-orm"; import { z } from "zod"; import { @@ -22,18 +23,22 @@ import { } from "../../lib/cloud-guards"; import { controlTags } from "../../lib/control-log"; import { + deleteGoldenDriveIfUnreferenced, deleteSandbox, + goldenDriveNameFor, loadRepositories, primaryRepository, promoteSandboxToEnvironment, RepositoryError, sortRepositories, + workspaceDrivesEnabled, workspaceRepositories, } from "../../lib/sandbox"; import { personProcedure, userError } from "../../trpc"; import { loadVisibleWorkspace } from "../cloud-workspace/access"; import { isLiveStatus } from "../cloud-workspace/live"; import { assertMachineSizeOffered } from "../cloud-workspace/start"; +import { deleteSandboxAndDriveLater } from "../cloud-workspace/teardown"; import { wakeAfterStop } from "../cloud-workspace/wake"; import { verifyOrgAdmin } from "../integration/utils"; import { currentBundleSha } from "./current-bundle"; @@ -251,6 +256,29 @@ async function loadPromotable( return { workspace, source, checkouts }; } +/** Only the control plane reads the golden drive; released clients never saw it. */ +function publicEnvironment( + row: Row, +): Omit { + const { goldenDriveName: _goldenDriveName, ...rest } = row; + return rest; +} + +/** A golden nothing will point at, and its drive: deleted through the retried job. */ +async function abandonGolden( + goldenName: string, + goldenDriveName: string | null, +): Promise { + await deleteSandboxAndDriveLater({ + providerSandboxId: goldenName, + driveName: goldenDriveName, + }).catch((cleanup: unknown) => + console.error( + `[environment/promote] could not delete unused golden ${goldenName}`, + cleanup, + ), + ); +} export const environmentRouter = { secrets: secretsRouter, @@ -284,7 +312,7 @@ export const environmentRouter = { projectIdsByEnvironment(ids), ]); return rows.map((row) => ({ - ...row, + ...publicEnvironment(row), repositories: repos.get(row.id) ?? [], projectIds: projects.get(row.id) ?? [], })); @@ -300,7 +328,7 @@ export const environmentRouter = { projectIdsByEnvironment([row.id]), ]); return { - ...row, + ...publicEnvironment(row), repositories: repos.get(row.id) ?? [], projectIds: projects.get(row.id) ?? [], }; @@ -374,7 +402,10 @@ export const environmentRouter = { await db.delete(environments).where(eq(environments.id, row.id)); throw error; } - return { ...row, primaryRepositoryId: input.primaryRepositoryId }; + return { + ...publicEnvironment(row), + primaryRepositoryId: input.primaryRepositoryId, + }; }), linkProject: personProcedure @@ -500,10 +531,40 @@ export const environmentRouter = { const limit = target ? null : await assertEnvironmentCapacity(workspace.organizationId); + // The kill switch: with drives off, no promote spreads drive mode to a new golden. + if ( + workspace.driveName && + !(await workspaceDrivesEnabled(workspace.organizationId)) + ) { + throw userError({ + code: "PRECONDITION_FAILED", + message: + "This workspace could not be saved as an environment. Try again later.", + i18nKey: "serverError.environment.couldNotPromote", + }); + } const golden = await promoteSandboxToEnvironment({ sourceSandbox: workspace.providerSandboxId, goldenName, organizationId: workspace.organizationId, + sourceDriveName: workspace.driveName, + }).catch(async (error: unknown) => { + await abandonGolden( + goldenName, + workspace.driveName ? goldenDriveNameFor(goldenName) : null, + ); + if (!workspace.driveName || error instanceof TRPCError) throw error; + // What went wrong names the drive; the user sees the failure any promote can have. + console.error( + `[environment/promote] ${workspace.id} could not be promoted`, + error, + ); + throw userError({ + code: "PRECONDITION_FAILED", + message: + "This workspace could not be saved as an environment. Try again later.", + i18nKey: "serverError.environment.couldNotPromote", + }); }); const { stopped } = golden; const restart = async () => { @@ -528,6 +589,7 @@ export const environmentRouter = { sourceKind: "fork" as const, sourceRef: goldenName, region: golden.region, + goldenDriveName: golden.goldenDriveName, bundleSha: await currentBundleSha(), primaryRepositoryId, }; @@ -616,25 +678,22 @@ export const environmentRouter = { return saved; }) .catch(async (error: unknown) => { - await deleteSandbox(goldenName).catch((cleanup: unknown) => - console.error( - `[environment/promote] could not delete unused golden ${goldenName}`, - cleanup, - ), - ); + await abandonGolden(goldenName, golden.goldenDriveName); await restart(); throw error; }); if (target?.sourceKind === "fork") { - await deleteSandbox(target.sourceRef).catch((error: unknown) => - console.error( - `[environment/promote] ${target.id} replaced; could not delete its previous golden ${target.sourceRef}`, - error, - ), - ); + await deleteSandbox(target.sourceRef) + .then(() => deleteGoldenDriveIfUnreferenced(target.goldenDriveName)) + .catch((error: unknown) => + console.error( + `[environment/promote] ${target.id} replaced; could not delete its previous golden ${target.sourceRef}`, + error, + ), + ); } await restart(); - return row; + return publicEnvironment(row); }), update: personProcedure @@ -707,14 +766,14 @@ export const environmentRouter = { : {}), }; if (Object.keys(patch).length === 0) { - return loadEnvironment(input.id, ctx); + return publicEnvironment(await loadEnvironment(input.id, ctx)); } const [row] = await db .update(environments) .set(patch) .where(eq(environments.id, input.id)) .returning(); - return row; + return row && publicEnvironment(row); }), archive: personProcedure @@ -729,11 +788,13 @@ export const environmentRouter = { .returning({ sourceKind: environments.sourceKind, sourceRef: environments.sourceRef, + goldenDriveName: environments.goldenDriveName, }); // A golden is one environment's alone, and an archived environment // never forks from it again; without this it bills storage forever. if (archived?.sourceKind === "fork") { await deleteSandbox(archived.sourceRef); + await deleteGoldenDriveIfUnreferenced(archived.goldenDriveName); } return { archived: true }; }), diff --git a/packages/trpc/src/router/environment/resolve-environment.ts b/packages/trpc/src/router/environment/resolve-environment.ts index cc869a25f06..8afb86dc2ba 100644 --- a/packages/trpc/src/router/environment/resolve-environment.ts +++ b/packages/trpc/src/router/environment/resolve-environment.ts @@ -12,6 +12,8 @@ export interface ResolvedEnvironment { region: string; bundleSha: string; primaryRepositoryId: string | null; + /** Set when the environment is in drive mode: every workspace's drive forks from it. */ + goldenDriveName: string | null; envs: Record; } @@ -60,6 +62,7 @@ export async function resolveEnvironment( region: row.region, bundleSha: row.bundleSha, primaryRepositoryId: row.primaryRepositoryId, + goldenDriveName: row.goldenDriveName, envs, }; } diff --git a/plans/20261116-workspace-drive.md b/plans/20261116-workspace-drive.md new file mode 100644 index 00000000000..11a79685c67 --- /dev/null +++ b/plans/20261116-workspace-drive.md @@ -0,0 +1,597 @@ +# Put a cloud workspace's repository and state on its own Vercel drive + +This ExecPlan is a living document. The sections `Progress`, `Surprises & Discoveries`, +`Decision Log`, and `Outcomes & Retrospective` must be kept up to date as work proceeds. + +Reference: this plan follows the conventions in `AGENTS.md`, `packages/trpc/AGENTS.md`, +`packages/trpc/src/lib/sandbox/AGENTS.md` and `.agents/commands/create-plan.md`. + +Status (2026-11-17): coded end to end behind the environment's drive mode and the organization +flag `cloud-workspace-drives`. Every Vercel fact this plan relies on was measured on real Vercel. +The code itself has not yet run against real Vercel: the `REAL_SANDBOX_DRIVE=1` pass of the +Sandbox workflow's `real-sandbox` dispatch is the check for that. + + +## Purpose / Big Picture + +A cloud workspace is a Vercel sandbox (a "box"). When a box stops, Vercel takes a snapshot of its +whole disk, and the next wake restores that snapshot before anything runs. The snapshot is a +block-level, full copy of the disk. It is not a delta over the golden it came from, and blocks +that a deleted file freed stay in it, because the disk has no discard. A typical box snapshot is +about 7 GB, and some are 14 to 27 GB. A wake restores at about 1 GB/s, and the restore is almost +never cached, because every workspace's snapshot is unique. + +About 7 GB of a box is `/workspace` (node_modules is about 5 GB and 213k files; `.git`; build +output) plus state under the home directory. A Vercel *drive* is a persistent block device that +is attached to a sandbox at a mount path and is not part of the sandbox's snapshot. With the +repository and the large home state on a drive, the root snapshot holds only the system: measured +2.11 GB, the same as a bare box from the image. + +The drive is *alongside* the persistent box. The root disk keeps its snapshot exactly as today, +so everything a person does on the box still persists, wherever they write it. There is no rule +for users to learn; the drive only moves the large, known directories off the root disk. + +A person sees it as a faster wake (the provider "resume" span in the `cloud_workspace.wake` +Sentry transaction gets shorter) and `git status` in `/workspace` still works with every file +present. + + +## Terms + +A *golden* is a stopped sandbox whose snapshot new workspaces of one environment fork from. It is +the `environments.source_ref` of an environment with `source_kind = 'fork'`. An *environment* is +the row in `environments` (`packages/db/src/schema/schema.ts`). + +A *drive* is Vercel's `Drive` (`@vercel/sandbox` 4.0.0, `dist/drive.d.ts`): a name unique in the +project, a region, `Drive.get`, `Drive.getOrCreate`, `drive.fork({ name })`, `drive.delete()` and +`currentSandboxName` (the sandbox that holds it now). A sandbox attaches drives through `mounts`, +a record from an absolute path to `{ drive: name, mode: "read-write" | "snapshot" }`. + +*Drive mode*: an environment is in drive mode when `environments.golden_drive_name` is set. Its +golden was built with that drive mounted from its first boot, and every workspace of it gets a +fork of that drive. + +A *bind mount* is the Linux `mount --bind` that `superset-boot` does as root to make a directory +of the drive appear at `/workspace` or under `/home/ubuntu`. + + +## Facts this plan relies on + +Measured on Vercel on 2026-11-16 (default image, `iad1`): + +- A drive is a real ext4 block device. inotify and hardlinks within one mount work on it. +- At stop, processes get SIGTERM and at least 10 s before the snapshot is taken. +- A snapshot is a block-level, full copy of the disk. Deleting a file does not shrink a later + snapshot. Only never writing the bytes, or overwriting freed space with zeros, shrinks it. + +Measured on Vercel on 2026-11-17, on our released image: + +- `drive.fork` is copy-on-write and near instant: 0.5 s for a 12 GB drive of 213k files, a fork + of a fork 0.4 s. Forks are independent: a parent can be deleted while forks exist, and the + forks stay readable. +- The full layout works on a box from our image: the drive mounted at `/mnt/superset-drive`; + `mount --bind` of `/workspace` and the home subtrees took 11 ms (29 ms after a wake). Bind + mounts do not survive a session. +- `bun install` with `BUN_INSTALL_CACHE_DIR=/workspace/.superset/stores/bun` hardlinked all + 212,973 files. A hardlink from `/workspace` into another bind mount of the same drive fails with + `EXDEV`. +- `dockerd --data-root /mnt/superset-drive/docker` (storage driver `fuse-overlayfs`) pulled and + ran an image, also after a wake. +- Stop 12 s, wake 1.9 s. Afterwards the root snapshot was 2.11 GB, identical to a bare box. +- A drive must be stopped and detached before anything else mounts it: a mount taken while a + writer still held it read-write saw a corrupt or empty filesystem. After a stop, poll + `Drive.get(...).currentSandboxName` until it is empty. +- Never delete a running sandbox that holds a drive read-write: a writer deleted while running + left the drive attached for minutes. Stop it first. +- A drive has no discard. Stop time grows with what the session wrote to the drive: 44 s after + 12 GB, 6 s after 1 GB. +- `Drive.list` rejects `namePrefix` unless `sortBy: "name"`. + +Documented by Vercel or the SDK, not measured: + +- One read-write mount per drive at a time. Read-only snapshot mounts can be shared. +- A drive's region must equal the sandbox's region. A sandbox has at most 4 mounts. +- Prices: drive storage $0.05 per GB-month, reads $0.0015 per GB, writes $0.004 per GB. Snapshot + storage $0.08 per GB-month. + + +## Progress + +- [x] (2026-11-16) Milestone 1: columns, drives module, mounts on create and wake, drive + deletion, unit tests. +- [x] (2026-11-17) Milestone 2: the open questions about Vercel, measured by hand (above). +- [x] (2026-11-17) Rebased on main; the drive mount and the agent-tools mount (PR #9844) are one + `mounts` object (`composeMounts` in `vercel.ts`). +- [x] (2026-11-17) Milestone 3: `superset-boot` lays out the drive, exports the store variables, + points Docker at the drive. boot-twice covers it with a tmpfs at + the mount path. +- [x] (2026-11-17) Milestone 4: the release builds the internal golden in drive mode when the + organization's flag is on; promote carries a source's drive to the golden. +- [x] (2026-11-17) Milestone 5: a wake waits for the drive to detach (2 minutes, then fails); + every delete stops the box and waits before it deletes the sandbox and the drive; a golden + drive is deleted once no live environment names it. +- [x] (2026-11-17) Rebased on main (#9861, #9862). The boot refuses a missing drive. The + `REAL_SANDBOX_DRIVE=1` pass runs whatever the first box's result, names every drive + expectation, and deletes what it made unless a drive expectation failed. +- [ ] Milestone 6: run the Sandbox workflow's `real-sandbox` dispatch with `real_sandbox_drive`; + then a release with the flag on for the internal organization; compare wake spans and snapshot + sizes for a week; then other organizations. + + +## Surprises & Discoveries + +- Observation: the first real-Vercel drive run (run 44151145824, 2026-11-17) failed 7 of 10 drive + expectations. All 7 were bugs in the check, not the product. `sudo` on the box writes `unable to + resolve host ` to stderr, and the check compared stdout and stderr together. Also, + findmnt reports a bind mount of a subdirectory as `/dev/vdc[/workspace]`, so its SOURCE never + equals the drive's. The log showed the product doing the right thing: `drive.layout ms=50`, + `/workspace` from `/dev/vdc[/workspace]` and docker's root at + `/mnt/superset-drive/docker`. The check now compares stdout only, and compares MAJ:MIN plus + FSROOT for the bind. + +- Observation: hardlinks do not work across two mount points, even on the same filesystem. + `link(2)` returns `EXDEV` when the two paths are on different `vfsmount`s, and a bind mount is a + separate `vfsmount`. Measured on 2026-11-17. So the bun and pnpm stores live inside the + `/workspace` bind mount. +- Observation: `superset-boot` runs `chown -R` on `/workspace` when its owner is not `ubuntu`. On + a drive, that walks 213k files. The layout creates every drive directory as `ubuntu`, and each + missing component of a home subtree on the root disk too (`install -d -o` only owns the last + one). +- Observation: a repository may track a `.superset/` of its own (the config directory before the + rename to `.paradev/`). So the checkout's exclude names only `/.superset/stores/`, and a reclone + merges a repository's `.superset/` into the one the stores live in instead of failing the move. +- Observation: the image's `superset-boot` runs first, and an older one has no layout. The layout + is idempotent and runs again after the reexec into the bundle's runner, so a box on an older + image is laid out on its first boot of a bundle that has it, before the steps and host-service. + + +## Decision Log + +- Decision: the drive is *alongside* the persistent box, never a replacement for its snapshot. + Rationale: everything a user does still persists; there is no rule to learn about where files + must go. Date/Author: 2026-11-17, product owner. +- Decision: drive mode is per *environment*, set by `environments.golden_drive_name`, not per + user. Rationale: the golden box and the golden drive must agree. If the golden's root disk still + held `/workspace`, every fork would keep carrying those 5 GB in its snapshot under the bind + mount, and the drive would buy nothing. So in drive mode the golden is built with the drive + mounted from the first boot. Date/Author: 2026-11-17, product owner. +- Decision: one flag, per organization, `cloud-workspace-drives` (PostHog group `organization`, + same pattern as `agent-tools.ts`), read only by the release when it builds a golden. It never + decides anything for a workspace. Rationale: a workspace follows its environment's row; a flag + that changes must not strand a workspace without its files. Date/Author: 2026-11-17, product + owner. +- Decision: fork or nothing. A workspace of an environment in drive mode gets + `goldenDrive.fork(...)`; a workspace of any other environment gets no drive at all. There is no + empty drive. Rationale: an empty drive under a golden whose root disk has the repository hides + the repository; under a golden whose root disk has none, it re-clones and re-installs. + Date/Author: 2026-11-17, product owner. +- Decision: promote carries a source's drive whatever the flag says. Rationale: a source in drive + mode has no `/workspace` on its root disk, so a golden without a fork of its drive would be + empty. The flag still gates the only place that starts drive mode, the release. A human may + prefer promote to refuse when the flag is off. Date/Author: 2026-11-17, agent; kept as coded + by the coordinator on 2026-11-17. +- Decision: the identity file names the workspace's drive (`SUPERSET_SANDBOX_DRIVE`), and + `superset-boot` refuses with `drive.missing` when it is set and the drive is not a mount point. + Rationale: a failed mount must never present the root disk's empty `/workspace` as the user's + work. A wake that finds the refusal in the boot log fails the workspace instead of answering + "still starting". Date/Author: 2026-11-17, coordinator. +- Decision: the drive is mounted once, at `/mnt/superset-drive` (`SANDBOX_PATHS.drive`), and + `superset-boot` bind-mounts subdirectories of it. Rationale: one read-write mount per drive, at + most 4 mounts per sandbox, and `/workspace` itself may be the primary checkout. Date/Author: + 2026-11-16, agent. +- Decision: a workspace drive is `drive-`; a golden drive is + `env-drive-`. The workspace's name is written to `cloud_workspaces.drive_name` + before the fork, so a delete that races the provision finds it. Date/Author: 2026-11-16, agent. +- Decision: `golden_drive_name` is read with the rest of the environment in `resolveEnvironment`, + the same read that picks the golden, so a box and its drive come from the same golden even when + a promote replaces it meanwhile. Date/Author: 2026-11-17, agent. +- Decision: a wake of a stopped box waits for its drive to be detached, at most 2 minutes + (`DRIVE_DETACH_TIMEOUT_MS`), then fails with `DriveStillAttachedError` rather than mount. A + failed drive mount update throws; a failed agent-tools mount update only warns. Date/Author: + 2026-11-17, agent. +- Decision: no swap on the drive. Since #9924 cloud boxes have no swap at all, and memguard reaps + early enough without it; the drive has no swap directory. memguard's cleanup of old + `/var/swap` files is main's and covers every box that could have one: no drive box ran the + earlier drive swap, since no environment has been in drive mode. Date/Author: 2026-11-18, + coordinator. +- Decision: Docker's data-root moves to the drive by `superset-boot` rewriting + `/etc/docker/daemon.json` after `apply-rootfs`, merged from the bundle's own file. Rationale: + `apply-rootfs` compares only the hash sidecar, so the rewrite stays until a bundle changes + `daemon.json`, and then the next boot merges again. Date/Author: 2026-11-17, agent. +- Decision: in drive mode `host.db` moves to the drive with the checkout markers + (`/mnt/superset-drive/state`). Rationale: its workspace and project rows name the worktrees on + the drive, and `agent-launched` beside it says whether the agent already ran in them. On the + root disk, a rollback that the drive does not share would make host-service forget a worktree or + launch the agent twice. Terminal sessions in it describe processes, which no snapshot keeps + anyway. Date/Author: 2026-11-17, agent, at the coordinator's request. +- Decision: the drive names its owner (`state/workspace-id`, synced), and the row records the + first boot that laid the drive out (`cloud_workspaces.drive_laid_out_at`, migration 0197). A + wake's claim says the drive was laid out before (`SUPERSET_SANDBOX_DRIVE_LAID_OUT`) only once + that column is set. Rationale: the earlier rule (every wake claims it) made a workspace whose + first session died before its layout refuse every wake as `drive.unlaid`, and a create that met a + transient mount failure could only fail. Replaces the rule of 2026-11-17 morning. + Date/Author: 2026-11-17, agent, at the coordinator's request. +- Decision: a drive refusal is retried, not failed: the box is stopped and the next wake or the + user's first open mounts the drive afresh. A refusal no remount can change (`drive.foreign`, + `drive.unlaid`, `drive.downgrade`, or host-service up without a layout) ends after 3 boots in a + row, counted from the boot log, with the workspace `failed` and a Sentry report. + Date/Author: 2026-11-17, agent. +- Decision: an unarchived workspace reuses its box and drive while their names are not retired, + that is inside the 7-day grace period. The reap's delete stage retires both names in the same + write that commits to deleting them, so an undo after it gets a new box and a fresh drive; the + user's deletion had completed. The teardown job still reads the rows before every destructive + call and keeps what a live row names. A drive forked afresh clears `drive_laid_out_at`. + Rationale: the drive lifecycle model (PR #9914) found a teardown keyed on names deleting a box + an undo had just brought back. Date/Author: 2026-11-17, agent. +- Decision: a provision that lost its row to an archive no longer tears down. Rationale: it cannot + tell the archive it lost to from a later one after an undo; that archive's reap can, by its + `archivedAt`. The cost is that a box the user never saw lives until that reap's stop and + delete stages. Date/Author: 2026-11-17, agent. +- Decision: "fresh fork" is a file on the drive (`state/fresh`), written and synced with the + owner at the first layout, and removed once the first checkout has run, whether it succeeded or + failed. Rationale: an environment variable lived for one boot only, so a first session that + ended before its checkout left the next boot treating the golden's copy as the user's work (the + model's FixFreshDurable). The file goes after a failed checkout too, unlike the model's + proposal: once that checkout has run the box was usable, and a later fresh-path reclone or + `checkout -B` could replace what the user did meanwhile. Date/Author: 2026-11-18, agent. +- Decision: a fresh fork of a golden never recloned. Rationale: the real check lost a golden's + dependencies once in six runs on a fork whose fetch or `checkout -B` failed, because the fresh + fork fell back to main's reclone, which empties the checkout. A fresh fork has no user work, and + the later-boot refetch of an untouched copy covers it, so it retries three times and then keeps + the golden's checkout. Date/Author: 2026-11-18, coordinator. +- Decision: a workspace's drive is deleted when its snapshot expires. Rationale: the box can never + resume, and the drive would bill storage for good. Retention will be ours: the reaper of #9987 + (every 6 h) deletes an expired workspace's box, snapshots and drive itself, instead of relying on Vercel's + 30-day `snapshotExpiration`. It calls `retireAndTearDown`, the one function that retires a row's + box and drive names and queues their retried teardown. Until then, and as a fallback after, + the 410 a wake, an agent turn or an automation dispatch gets on an expired snapshot takes + `markSandboxUnavailable` to the same function. Date/Author: 2026-11-18, Satya. +- Decision: a customer never reads that the box has a drive. A drive that refused for good fails + the workspace with `host_not_ready` ("The workspace did not become ready."), a drive still + flushing at a wake answers "still starting", the error messages a client can be handed name no + drive, and a promote of a workspace on a drive that fails, or is refused with drives off, says + "This workspace could not be saved as an environment. Try again later." The detail stays in the + logs and in the errors' fields. Date/Author: 2026-11-18, coordinator. +- Decision: a checkout with no marker on a drive is fetched only while it is the golden's untouched + copy (`state/golden-head`), else kept (the model's FixFetchIfUntouched). Rationale: removing + `state/fresh` after the first attempt keeps a failed first checkout from being refetched over the + user's work, but alone it left stale code after a failed first fetch; HEAD, a clean tree and the + branch together prove nobody has worked in it. Date/Author: 2026-11-18, model's author and agent. +- Decision: a provision that finds the lock held requeues itself (15 s, at most 10 times) and + touches nothing; after the last try it fails only a row still `provisioning`. A provision fails + or stops a row only if the row is still `provisioning` or this job made it `awake` (the model's + FixProvLockRetry and FixFailOwnOnly). Date/Author: 2026-11-18, model's author and agent. +- Decision: a fresh fork is recognised by its drive having no owner at the first layout, not by + the golden strip removing local branches (the model's FixFresh). Rationale: promote and the + release both strip the drive's state, owner included, so every fork's first boot is fresh and + takes main's fetch and `checkout -B` whatever branches the golden has; removing branches would + add nothing. Date/Author: 2026-11-17, agent. +- Decision: the control plane does not refuse to pin a pre-drive bundle to a workspace with a + drive. Rationale: nothing records which bundle shas know the drive, and both sides already + refuse: the boot will not reexec into a script without the drive-contract line, and a boot that + answers without `drive.layout` is never treated as ready. Date/Author: 2026-11-17, agent. +- Decision: the kill switch is the organization flag, read at provision and promote. It cannot + stop forks inside an environment already in drive mode without leaving their workspaces empty, + so a provision with the flag off gives the workspace no drive at all, and its box clones and + installs onto the root disk. Date/Author: 2026-11-17, agent. +- Decision: do not migrate existing workspaces. See Migration. Date/Author: 2026-11-16, agent; + confirmed out of scope 2026-11-17. +- Decision (rejected): wake by forking the golden. A wake would fork the environment's golden + afresh and mount the workspace's drive, instead of resuming the workspace's own snapshot. + Rejected on 2026-11-17 by the product owner in favour of a drive alongside the persistent box. + Without the workspace's own snapshot, whatever a person wrote outside the drive would be gone, + and that is a rule to learn. It is not a goal of this plan. + + +## Outcomes & Retrospective + +As of 2026-11-17: the code compiles, the unit tests pass, the model check still passes, and +boot-twice covers the layout in a container. Nothing in this branch has run against real Vercel. +No environment is in drive mode, so no workspace changes until a release runs with the flag on. + + +## Context and Orientation + +Affected: the API (`apps/api`, which serves the tRPC routers), the packages `db`, `trpc`, +`shared` and `sandbox`, and the Sandbox workflow. No tRPC input or output changes. + +- Create: `provisionCloudWorkspace` (`packages/trpc/src/router/cloud-workspace/provision.ts`) + builds the claim, which now carries the environment's `goldenDriveName`; if set, it writes the + workspace's drive name to the row, forks the golden drive (`forkWorkspaceDrive`), and passes the + drive's mount to `provisionSandbox`. +- Boot: `superset-boot` (`packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot`). +- Wake: `wakeCloudWorkspace` (`wake.ts`) passes `workspaceDriveMounts(row.driveName)` to + `wakeSandbox` (`packages/trpc/src/lib/sandbox/vercel.ts`). +- Delete: the reap's delete stage (`reap.ts`), the `cloudWorkspace.delete` fallback and the + provision teardowns call `deleteSandboxAndDrive`. +- Promote, replace and archive: `environment.promote` and `environment.archive` in + `packages/trpc/src/router/environment/environment.ts`; `promoteSandboxToEnvironment` in + `vercel.ts`. +- Release: `packages/sandbox/src/environments/release.ts`. + + +## The layout on the box (coded) + +The drive is mounted by Vercel at `/mnt/superset-drive`. It holds: + + /mnt/superset-drive/ + workspace/ bind-mounted on /workspace + .superset/stores/ bun, pnpm stores (inside the /workspace mount, for hardlinks) + home/ subtrees bind-mounted under /home/ubuntu + .cache .local/share .local/state .npm .cargo/registry .rustup + .claude .codex/sessions go/pkg + docker/ dockerd's data-root + +The home subtrees are `SANDBOX_DRIVE_HOME_SUBTREES` in `packages/shared/src/sandbox/contract.ts`, +rendered into `contract.sh` as `SUPERSET_DRIVE_HOME_SUBTREES`; `sandboxDrivePathOf` maps a box +path to its place on the drive. A subtree the drive does not have yet is seeded once from what the +image left at that path. + +`superset-boot` does the layout as root right after it clears `/run/superset`, before the bundle +passes, host-service and the checkout. Each bind is `mountpoint -q || mount --bind ...`, and +a failed bind refuses the boot. It stamps `drive.layout ms=`. If `/mnt/superset-drive` is not a +mount point, nothing changes: that is every box outside drive mode. + +In drive mode the boot also: + +- writes `BUN_INSTALL_CACHE_DIR=/workspace/.superset/stores/bun` and + `npm_config_store_dir=/workspace/.superset/stores/pnpm` to `/run/superset/drive.env` (sourced by + `/etc/profile.d/superset.sh`) and into host-service's env; +- adds `/.superset/stores/` to `.git/info/exclude` of a root checkout; +- rewrites `/etc/docker/daemon.json` with `"data-root": "/mnt/superset-drive/docker"`. + +What stays on the root disk: `/etc/superset`, `/opt/superset`, `/usr`, `/var/lib/superset` +(host.db, checkout markers), `/var/log/superset`, `/home/ubuntu` itself with its dotfiles, +`~/.config`, `~/.gitconfig`, `~/.paradev`, and the sign-in files the control plane writes on every +wake (`~/.codex/auth.json`, `/etc/codex/config.toml`). + + +## Golden drive per environment (coded) + +The release (`release.ts`), when `cloud-workspace-drives` is on for the internal organization: + +1. creates `env-drive-` in the region; +2. provisions the golden with it mounted read-write, so the boot lays it out and the checkout and + the setup hook's installs land on the drive; +3. strips identity (with the drive paths), stops the golden, waits for the drive to detach; +4. probes a workspace on a fork of the golden drive, stop and wake included, then deletes the + probe and its drive; +5. writes `golden_drive_name` with `source_ref` in the same row update; +6. deletes the previous golden and, once no live environment names it, its golden drive. + +When the flag is off, the release runs as before and writes `golden_drive_name = null`. + +Promote (`promoteSandboxToEnvironment`), when the source workspace has a drive: + +1. snapshots the source, which ends its session; waits for the source drive to detach; +2. forks it to `env-drive-`; +3. creates the golden from the snapshot with the fork mounted read-write; +4. strips `INHERITED_IDENTITY` and the agent session directories (`~/.claude/projects`, + `~/.codex/sessions`), both at their box paths and at their drive paths, because no boot has + laid out the golden; +5. stops the golden and waits for its drive to detach; +6. restarts a source that was running, with its own drive in its mounts. + +The environment row gets `golden_drive_name` with `source_ref`. A source without a drive promotes +as before, and the environment leaves drive mode. A replaced or archived environment's golden +drive is deleted after its golden sandbox, once no live environment names it +(`deleteGoldenDriveIfUnreferenced`). Forks stay readable after their parent is deleted. + + +## Mounts on create and wake (coded) + +`composeMounts` merges the workspace drive's mount and the agent-tools mount into the one +`mounts` object a create or a wake passes. On a wake of a stopped box, `wakeSandbox` starts from +the box's current mounts without the agent-tools entry, adds the workspace drive and the current +agent-tools drive, waits for the drive at `/mnt/superset-drive` to be detached, and calls +`update({ mounts })` once. A running box is not touched. + + +## Single writer and the sleep state machine (coded) + +A stop leaves the drive attached while the session flushes. `wakeSandbox` polls +`currentSandboxName` every 2 s and fails after 2 minutes with `DriveStillAttachedError`. The wake +then throws before anything resumes, and the row stays as it was. + +`Asleep.tla` does not need a new state: the wait writes no status, expiry or session id; it +delays `WakeResume` or fails the wake before it. The model still passes unchanged (1,624,931 +distinct states) and the broken config still fails. The single-writer property itself (at most +one session holds the drive) is outside the model; the unit tests in `drives.test.ts` and +`vercel.test.ts` cover the wait. A drive variable in the model is a possible follow-up. + + +## Delete and archive (coded) + +`deleteSandboxAndDrive` stops the box, waits for the drive to detach, deletes the sandbox, then +the drive. It is used by the reap's delete stage, the `cloudWorkspace.delete` fallback and both +provision teardowns. A drive that is still attached after 2 minutes throws; the reap's delivery +retries. Archive keeps the drive for the 7-day grace period, with the stopped sandbox. + + +## Migration of existing workspaces + +Never, in this plan. Existing workspaces and environments not in drive mode are untouched. Freed +blocks stay in a snapshot, so copying `/workspace` to a drive and removing it would not shrink +the snapshot; only zero-filling free space would, which is not acceptable on a wake. + + +## Billing + +Per typical workspace with 7 GB of `/workspace` and home state, per month: + + today: 7 GB snapshot x $0.08 = $0.56 + drive: up to 7 GB drive x $0.05 = $0.35 (forks are copy-on-write; shared blocks + likely bill once, not measured) + + root snapshot 2.11 GB x $0.08 = $0.17 + = at most about $0.52 + +Per event: a bun install that writes node_modules, 5 GB x $0.004 = $0.02. The main gain is the wake: about 5 GB less to restore (the root +snapshot went from ~7 GB to 2.11 GB), roughly 5 s at the measured rate. + + +## Regions + +A drive must be in the sandbox's region, so a golden, its golden drive and every workspace drive +of an environment are in `environments.region`. `forkWorkspaceDrive` refuses a golden drive in +another region. + + +## Failure modes + +- Fork fails at provision: the provision fails as any other create failure does; the row goes to + `failed`, and the sandbox and the drive go to the teardown job. +- The drive is still attached on a wake: the wake waits 2 minutes, then fails with a clear error. + A drive held by this very sandbox, running again, means a concurrent wake resumed it; the wake + goes on as for a running box. +- Mount update fails on a wake: the wake throws. +- A bind or a home seed fails at boot: the boot refuses, so nothing starts on the root disk's empty + `/workspace`. +- The drive does not mount, belongs to another workspace, or mounts without an owner though the + wake says it was laid out: `superset-boot` stamps `drive.missing`, `drive.foreign` or + `drive.unlaid` and exits non-zero before the bundle, host-service and the checkout. The control + plane reads the boot log every 2 s while it waits for host-service and throws + `SandboxDriveMissingError`, a `SandboxNotReadyError`: the wake stops the box and answers "still + starting", and the next wake mounts the drive afresh; a create records the box asleep instead + of failing it. Only this boot's lines count (the boot id on `boot.start`). A refusal no remount + can change ends after 3 boots in a row: the workspace goes `failed` with the reason, Sentry gets + it, and the box and drive are kept. +- A pinned bundle whose boot script predates the drive: a drive-mode boot refuses + (`drive.downgrade`) before it would reexec into it. +- An unclean stop (host-service killed, writers still running): the real check's probe requires + the box to boot with `host.db` passing `integrity_check` and the checkout passing `git fsck`. + A cold wake, one the snapshot cache did not serve, cannot be forced from the check; its wake + times are warm. +- host-service answers without a `drive.layout` stamp for a workspace with a drive (a pinned + bundle older than the layout): the same retryable error, so its empty `/workspace` is never + shown as ready. +- The root disk rolls back (a stop snapshot that did not land) and the drive does not: the + checkout markers and `host.db` are on the drive, so they still describe it. A checkout that + already has the workspace's branch is kept; a failed fetch keeps the checkout; a clone goes only + into an empty `/workspace`. +- Drive delete fails: the teardown job retries every 30 s, 20 times, then logs and gives up. + + +## Invariants + +The owner's two requirements: nothing a user does is ever lost, and an empty `/workspace` is +never shown as their work. + +| Invariant | Enforced in | Test | +| --- | --- | --- | +| A wake never gives a workspace a drive its row does not name; only a provision consults the environment, and only with the organization's flag on. | `claimedWorkspaceDrive` (`drives.ts`) via `buildSandboxClaim({ provisioning })` (`claim.ts`); `wake()` mounts `workspaceDriveMounts(row.driveName)` (`wake.ts`) | `claim.test.ts` "an existing workspace gets no drive on a wake…", "with the organization's flag off a new workspace gets no drive…"; `wake.test.ts` "a workspace without a drive is woken with no mounts" | +| A box whose identity names a drive starts nothing that reads `/workspace` unless that drive is mounted. | `layout_drive` in `superset-boot` (`drive.missing`) | boot-twice "drive missing: …" (4 checks) | +| The mounted drive is this workspace's. | `layout_drive` (`drive.foreign`; `drive.unlaid` when `SUPERSET_SANDBOX_DRIVE_LAID_OUT` is set); the owner is written and synced at the first layout; a wake's claim sets the key only once `cloud_workspaces.drive_laid_out_at` is set | boot-twice "drive foreign", "drive empty", "the drive names its workspace"; `claim.test.ts` "a wake expects the drive laid out only once the row recorded a boot that laid it out" | +| A laid-out drive is recorded durably. | `recordDriveLaidOut` (`drive-laid-out.ts`) after a settle that saw `drive.layout`, in `provision.ts` and `wake.ts`; a fresh fork at provision clears it | `provision.test.ts` "records the drive laid out once the boot answered" | +| A drive-mode boot never runs a boot script that predates the drive. | the `drive-contract` line and the `drive.downgrade` refusal before the reexec in `superset-boot` | boot-twice "drive, older boot script: refused before the reexec, nothing checked out" | +| The control plane never treats a box with a drive as ready unless its boot laid the drive out. | `settleSandbox` (`vercel.ts`) requires `drive.layout` | `vercel.test.ts` "a boot that answers without laying the drive out fails the settle" | +| A drive refusal is read from this boot only, found within about 2 s, and retried by the next wake or create; one no remount can change ends after 3 boots in a row with a clear `failed` and a Sentry report. | `runBoot` boot id, `driveRefusalIn`, `persistentDriveRefusals`, `SandboxDriveMissingError` (a not-ready), `SandboxDriveRefusedError` (an unavailable), `DRIVE_REFUSAL_LIMIT`; `wakeSandbox` stops the box; `provision.ts` records it asleep | `vercel.test.ts` "only this boot's lines count…", "a reexec's second boot.start joins its boot", "counts the persistent refusals in a row…", "a refused boot is a retryable not-ready…", "a refusal no remount can change ends the retries…"; `provision.test.ts` "a drive refusal at create leaves the workspace asleep…" | +| In drive mode, what describes the drive's contents is on the drive: checkout markers, `host.db` and its WAL, `agent-launched`, the owner. | `use_drive_state` in `superset-boot`; `SANDBOX_PATHS.driveState` | boot-twice "host.db and the checkout markers are on the drive, not the root disk", "drive, root rolled back"; real check "unclean stop: host.db passes integrity_check" | +| A drive laid out before is never replaced by the checkout. | `checkout_one` with `drive_has_work` | boot-twice "drive, marker lost: the branch is kept", "drive, fetch failed: nothing on the drive is replaced" | +| A fresh fork's checkout is fetched later too, while it is still the golden's untouched copy: HEAD at the commit the fork had, a clean tree, the branch absent or at that commit. Anything else is the user's and is never fetched over. | `state/golden-head/`, recorded before the first checkout attempt; `untouched_golden` in `checkout_one` (`superset-boot`) | boot-twice "fresh fork, first fetch failed: a later boot fetches the golden copy while it is untouched", "fresh fork, first fetch failed, then the user committed: their copy is kept" | +| A fresh fork of a golden is never recloned: its fetch and `checkout -B` are tried three times, and if they still fail the golden's checkout, dependencies included, is kept as it is (`checkout.fresh.failed`), the box is ready, and a later boot fetches it while it is untouched. The reclone is only for a path with no repository. | `checkout_one` in `superset-boot` (the attempts loop) | boot-twice "fresh fork, fetch failed: never recloned…", "fresh fork of a golden with node_modules, first fetch failed: the dependencies survive and a later boot fetches" | +| A fresh fork checks out its own branch as main does, even if its first session ended before the checkout. | `state/fresh` on the drive, written and synced with the owner at the first layout, removed once the first checkout has run (`checkout` in `superset-boot`) | boot-twice "fresh fork, new branch", "fresh fork, a branch the golden has", "fresh fork, fetch failed", "fresh fork, first session ended before its checkout: the next boot still fetches" | +| A workspace whose snapshot expired loses its own drive, never a golden's or one another row names, and never while the drive is attached. | `markSandboxUnavailable` (`wake.ts`) on a 410 retires the row's names while it is `failed` and still names that box and drive (`retireAndTearDown`); `teardownSandbox` waits for the drive to detach and checks `liveReferences` before the delete, retrying 20 times 30 s apart | wake "a workspace whose snapshot expired …"; teardown "a drive still attached requeues the job and deletes nothing", "a drive a golden environment or another row names is never deleted", "a drive delete that fails is retried"; vercel "a box whose snapshot expired … still lets its drive go" | +| Flag-off boxes run main's checkout code exactly. | `checkout_one` `else` branches | boot-twice first and second boot of the plain box; the diff against main | +| A home subtree is never seeded partially. | `seed_home_subtree` (copy to `.seeding`, then `mv -T`; a failed copy refuses the boot) | boot-twice layout checks only; a failed copy is not simulated | +| One read-write session per drive, and a concurrent wake of the same box does not wait. | `waitForDriveDetached` with `heldByCaller` in `wakeSandbox` | `vercel.test.ts` "a stopped box waits for its drive…", "a drive a concurrent wake already resumed this box with is not foreign"; `drives.test.ts` wait tests | +| A stopped box's drive check and the call that resumes it are one step: no other caller resumes or stops the box in between. A refused box is stopped under the same lock. The lock is keyed on the drive when there is one: a drive has one writer, whatever the box is called. | `ResumeLock` around `wakeSandbox`'s read-to-boot and `provisionSandbox`; `sandboxResumeLock(driveName ?? providerSandboxId)` (`resume-lock.ts`, a transaction-scoped advisory lock; a wake tries and answers "still starting", a provision waits up to 20 s) | `vercel.test.ts` "a second wake while the first resumes does not resume the box…"; `provision.test.ts` "the create holds the lock of the drive, whatever the box is called" | +| Nothing resumes a box outside that one step. After a read, every call goes through the session read (a Session call fails rather than resume a session that ended); a session that ended in between takes the stopped path under the same lock. | `resumeSession` (the only resume), `withLiveSession`, `wakeSandbox`, `provisionSandbox`, the settle's log read through a running session | `vercel.test.ts` "a box found running … whose session ends during the wake is resumed only after the drive check" | +| Two provisions of one row never both run on its drive. | `provisioning -> awake` and `-> failed` only while the row names the job's own box (`transitionCloudWorkspace({ onBox })`); a job that lost to another box tears its own box down with `keepDrive` (stop, wait for the detach, delete the box, never the drive); a new box is created only once its drive has detached | `provision.test.ts` "a provision overtaken by an undo's second provision gives up its box, keeps the drive, and never marks the row"; `vercel.test.ts` "a new box on a drive is created only once a box of another name has let go of the drive" | +| A redelivered provision whose box has stopped waits for the drive and remounts it before anything resumes the box. | `provisionSandbox` (`existing.status !== "running"`) | `vercel.test.ts` "a redelivered provision whose box stopped waits for the drive before anything resumes it" | +| The settle never resumes a box another caller stopped. | `settleSandbox` reads the boot log only from a box it just found running | `vercel.test.ts` settle tests | +| A provision that lost its row to an archive tears nothing down; that archive's reap, keyed on its `archivedAt`, owns the box. A later archive gets its own full grace period. | `provision.ts` (the `!ready` branch); `isStillArchived` and the retiring write in `reap.ts` | `provision.test.ts` "a provision that lost its row to an archive leaves the box to that archive's reap", "a failed create whose row was archived meanwhile tears nothing down" | +| Committing to delete a box and drive retires their names in the same write, so an undo after it gets a new box and a new drive, never the ones being deleted. | `retireAndTearDown` (`teardown.ts`) from the reap's delete stage and the archive fallback; the provision failure's transition to `failed` | `reap.test.ts` "the delete stage retires the box's names before it deletes them…", "an undo that wins the row before the retire leaves the box and drive alone"; `provision.test.ts` "a failed create retires the box's name in the same write…" | +| A box is stopped and its drive detached before either is deleted. Deletes never wait in a request, except promote, which must wait for the source's drive before it can fork it. | `teardownSandbox`, `deleteSandboxAndDriveLater` (`teardown.ts`), route `/api/cloud-workspaces/teardown`; promote's two detach waits (up to 2 minutes each) inside the 300 s tRPC route | `teardown.test.ts` "stops the box, waits for its drive, then deletes…" | +| A teardown never deletes what a workspace row, in any status, or a live environment names. Every real teardown clears the names first, so a name still on a row is that row's: an archived row may be undone onto it, a failed one keeps its box and drive for diagnosis. | `liveReferences` (`teardown-guard.ts`), read before each destructive call | `teardown.test.ts` "a box a revived workspace runs on is left alone", "an unarchived workspace's new box keeps the drive…" | +| A provision fails or stops only its own row: one still `provisioning`, or one this job itself made `awake`. A row another delivery or a wake brought up is never failed, stopped or torn down by it. | `madeAwake` and the `from` of the failure transition in `provision.ts`; the stop, teardown and failed chat moves follow only a transition it won | `provision.test.ts` "a second delivery on the same box that fails after the first brought the row up leaves the live workspace alone" | +| A provision that finds the box's lock held is retried, never failed or stopped for it. | `SandboxBusyError` from `sandboxResumeLock`; `provision.ts` requeues itself every 15 s, at most 10 times, then fails only a row still `provisioning` | `provision.test.ts` "a lock held by another caller requeues the provision…", "a lock still held after the last retry fails only a row still provisioning" | +| A lost provision's box is deleted even while the row's live box holds the kept drive. | `teardownSandbox` with `keepDrive` waits only for the drive to leave this box | `teardown.test.ts` "a lost box goes even while the row's live box holds the kept drive" | +| A failed provision deletes only a drive it forked. | `forkedDrive` in `provision.ts` | `provision.test.ts` "a failed create deletes a drive it forked, never one it found" | +| No teardown failure is silent. | `teardownSandbox` reports to Sentry and requeues any failure; giving up is a Sentry error | `teardown.test.ts` "any other failure is retried, not dropped", "gives up after the last attempt…" | +| A drive nothing names is found and, on request, deleted; the sweep never deletes a drive any row names, archived or not, and checks each again just before its delete. | `drive-sweep.ts` (`orphanCandidates`, `sweepDrives`, `stillOrphaned`), Sandbox workflow `run: drive-sweep`, dry run unless `drive_sweep_delete` | `drive-sweep.test.ts` "any row that names a drive keeps it…", "a drive an unarchive named after the listing is kept…" | +| A failed promote leaks neither the golden nor its drive. | `abandonGolden` (`environment.ts`) | `environment.test.ts` "hands the golden and its forked drive to the retried teardown" | +| A golden carries no workspace identity, on the root disk or the drive, and is stripped only with its drive mounted. | `inheritedIdentityPaths` (includes `SANDBOX_PATHS.driveState`); `stripWorkspaceIdentity` checks `mountpoint` and the exit code | `vercel.test.ts` "forks the source's drive once it is released and strips identity on the drive", "a golden whose drive did not mount is not stripped and fails the promote" | +| Turning the flag off stops new drives. | `buildSandboxClaim` (provision) and `environment.promote` consult `workspaceDrivesEnabled`; the release writes `golden_drive_name = null` | `claim.test.ts` "with the organization's flag off…"; `environment.test.ts` "a workspace whose files are on a drive does not spread drive mode…" | +| Released clients see the same outputs as before this branch. | `withClientStatus` drops `driveName`, `driveLaidOutAt`; `publicEnvironment` drops `goldenDriveName` | `client-status.test.ts` "never carries the drive columns…"; `environment.test.ts` "the output never carries the golden drive…" | +| The drive mount and the agent-tools mount survive each other's updates. | `composeMounts`, `wakeSandbox` | `vercel.test.ts` "the drive and the agent-tools drive share one mounts object", "an agent-tools update keeps the drive the box already has" | + + +## Rollback + +1. Turn `cloud-workspace-drives` off for the organization. From then on a new workspace gets no + drive, even in an environment in drive mode: its box clones and installs onto the root disk, as + before drives existed, and so is slower to create. Promote refuses a workspace whose files are + on a drive. The next release builds the golden without a drive and writes + `golden_drive_name = null`, which takes the environment out of drive mode. +2. Existing workspaces with a drive keep it and keep working: their data lives there. Nothing + moves it back to the root disk; such a workspace is archived and recreated if it must leave + drive mode. +3. Drives that nothing names any more: dispatch the Sandbox workflow with `run: drive-sweep`, + read the list, then run it again with `drive_sweep_delete`. +4. Code rollback: revert the branch's commits. Rows with `drive_name` keep their column, and no + pre-drive boot script may run over a laid-out drive (`drive.downgrade`), so workspaces with a + drive refuse to boot instead of showing an empty `/workspace`. Archive them before such a + revert. + + +## Before customers + +- Schedule the drive sweep and let it delete. Today it is a `workflow_dispatch` job only, dry run + unless `drive_sweep_delete` is set, so a drive whose teardown gave up stays until someone runs it + (the model's NoOrphanForever holds only if it is run). Once it has run dry in production a few + times and listed nothing it should not, run it on a schedule without the dry run. + + +## Follow-ups (out of scope) + +- Environment builds from `[install]` (PR #9282) producing golden drives. +- Migrating existing workspaces. +- Customer environments' checkpoints through `superset environments replace` beyond what promote + already covers. +- A drive variable in `Asleep.tla`: not needed. No status write is new: a create that meets a + drive refusal stops the box and records it asleep, which is `StopBox` and `StopRecord`; a wake + that stops a refused box writes no status, so the row sees it as `Crash`; a persistent refusal + marks the workspace failed through `markSandboxUnavailable`, as a gone box does. The model + passes unchanged (1,624,931 states) and the broken config still fails. +- Environment builds (#9282): `activateGolden` must move `source_ref` and `golden_drive_name` + together, in one row update, the way promote and the release do. A golden and a drive from two + different builds would give every new workspace a checkout that does not match its root disk. +- A sweep of drives named `drive-ws-*` whose sandbox no longer exists. + + +## Validation and Acceptance + +From the repository root: + + bun run lint:fix && bun run typecheck + cd packages/trpc && bun test # drives.test.ts, vercel.test.ts + cd packages/sandbox && bun test + bun run --cwd packages/sandbox image --local + cd packages/sandbox && bun run src/boot-twice.ts # includes the drive pass + .agents/skills/verify-sandbox-asleep/run.sh + +Against real Vercel: dispatch the Sandbox workflow with `run: real-sandbox` and +`real_sandbox_drive: true`. It must print `ok` for every `drive:` line, exit 0, and delete what it +made. `real_sandbox_keep: true` keeps the stopped boxes and the box drive for inspection. On a +failed drive expectation it prints a `drive: diagnosis` block (mounts, drive contents, the conf's +drive key, the run dir, `daemon.json`, the checkout log and the boot log). + + +## Idempotence and Recovery + +Every drive call is safe to repeat: `forkWorkspaceDrive` finds a drive of the same name, a delete +of a missing drive is a no-op, the boot's binds check `mountpoint -q` first. Turning the flag off +stops new golden drives at the next release; an environment already in drive mode stays in it +until a release or promote replaces its golden. + + +## Interfaces and Dependencies + +`packages/trpc/src/lib/sandbox/drives.ts`: `WORKSPACE_DRIVE_MOUNT_PATH`, `workspaceDriveNameFor`, +`goldenDriveNameFor`, `workspaceDriveNameToClaim`, `workspaceDriveMounts`, +`workspaceDrivesEnabled(organizationId)`, `DriveHost`, `vercelDrives()`, `forkWorkspaceDrive`, +`deleteWorkspaceDrive`, `waitForDriveDetached`, `DriveStillAttachedError`. +`golden-drives.ts`: `deleteGoldenDriveIfUnreferenced`. `vercel.ts`: `composeMounts`, +`deleteSandboxAndDrive`, `inheritedIdentityPaths`, and `promoteSandboxToEnvironment` with +`sourceDriveName`. `vercel-credentials.ts`: `credentials()`, moved out of `vercel.ts` so +`drives.ts` and `vercel.ts` do not import each other.