Skip to content

feat(cloud-workspace): workspace drive per environment, flag-gated (draft) - #9847

Draft
saddlepaddle wants to merge 2 commits into
superset/workspace-drive-columnsfrom
superset/thin-workspace-drive-draft-c4ad5f56
Draft

saddlepaddle wants to merge 2 commits into
superset/workspace-drive-columnsfrom
superset/thin-workspace-drive-draft-c4ad5f56

Conversation

@saddlepaddle

@saddlepaddle saddlepaddle commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Draft. Stacked on #9915 (the database columns), which lands first. Detail: plans/20261116-workspace-drive.md.

What it does

  • A cloud workspace's repository, dependencies and large home folders move to their own Vercel drive. The box's disk keeps everything else, so nothing a person does moves or is lost.
  • The box's snapshot then holds only the system: about 2.8 GB instead of about 7.9 GB, so a wake restores less.
  • It is off until an organization's cloud-workspace-drives flag is on and a release builds that environment's template with a drive. Every other workspace is untouched.

How it works

create (provision)
  if the environment has a template drive and the org flag is on:
      record drive-<box> on the row; fork the template drive (copy-on-write, ~0.5 s)
  under the drive's lock: wait until no box holds the drive, then create the box with it mounted
  mark the row ready only if it still names this job's box (else: drop this box, keep the drive)
  answer ready only once the boot has laid the drive out; then record "laid out" on the row

boot (on the box)
  drive named in sandbox.conf but not mounted, another workspace's, or empty though laid out before
      -> refuse; start nothing
  lay out: /workspace and the big home folders bind-mounted from the drive;
           its state (checkout markers, host.db) kept on the drive too
  fresh fork -> fetch the branch as today; a drive with work on it -> never replaced
  never re-run with an older boot script that predates the drive

wake
  under the drive's lock: a stopped box waits for its drive to be released, remounts, resumes
  every other call cannot resume a box; a refusal is retried on the next open,
  and only one that cannot heal (3 boots in a row) marks the workspace failed, box and drive kept

stop    as today; the drive is released after the box stops

delete + undo
  within the 7 days: undo resumes the same box and drive
  at the end: the row gets new names in the same write that queues the teardown,
              so an undo after it gets a fresh box and drive, never the ones being deleted
  teardown: stop, wait for release, delete box, then drive; never what any row still names

What a customer sees

  • A faster wake, and their files where they left them. Nothing to learn: the paths are the same.
  • If a drive fails to attach, "still starting", then it works on the next open. They never see an empty workspace presented as theirs.
  • Nothing they read names a drive. A drive that refuses for good shows as "The workspace did not become ready.", and a drive still flushing at a wake shows as "still starting". A promote of a workspace on a drive that fails, or is refused with the flag off, says "This workspace could not be saved as an environment. Try again later." (translated).
  • Create is unchanged when the flag is off. With the flag off in a drive-mode environment, a new workspace clones and installs as it did before drives (slower).

Evidence

Real Vercel, REAL_SANDBOX_DRIVE=1: the same work on a drive box and a plain box (clone, bun install, 4 GB written), five persistence sentinels, an unclean stop, and a workspace created from a drive-mode template.

Head Root snapshot, drive vs plain Wake, drive vs plain Sentinels kept Unclean stop / template path
e560e42 2.58 GB vs 7.86 GB 5.4 s vs 7.0 s 5 of 5 not yet probed
e0263ca 2.79 GB vs 7.88 GB 5 of 5 ok / ok
707eabd 2.76 GB vs 7.86 GB 5 of 5 ok / ok
3f7b5ec running

The wake times are warm: a wake that the snapshot cache does not serve cannot be forced from the check.

The TLA+ model of the drive lifecycle (#9914) checks 10 rules: nothing lost, never empty shown as work, one writer per drive, no teardown of a live workspace, no change without a drive, never stuck, a fresh fork fetches, a glitch is retried, no drive left behind, only create gives a drive. Its author re-derived it from this code at 3f7b5ec, where 7 held. This head adds the three fixes the model proposed for the other three (FixFailOwnOnly, FixProvLockRetry, FixFetchIfUntouched), each with a unit test or boot-twice case that plays its trace.

  • Unmodified ./run.sh (the model's design with those fixes on): every scenario passes.
  • Unmodified ./run.sh head: still the 3f7b5ec code, so those three rules fail until the model is re-derived from this head.

Unit tests replay each trace the model found. boot-twice covers the box side in Docker.

Risks and rollback

  • No swap. Cloud boxes have had no swap since fix(sandbox): no swap on cloud boxes; memguard reaps early enough without it #9924, and the drive has none either.
  • Kill switch: turn cloud-workspace-drives off. New workspaces then get no drive, promote refuses a drive workspace, and the next release builds the template without one. Workspaces that already have a drive keep it and keep working. The full procedure is in the plan, under "Rollback".
  • Code revert: workspaces with a drive refuse to boot rather than show an empty /workspace. Archive them before such a revert.
  • Costs: drive storage ($0.05/GB-month) plus writes ($0.004/GB). A per-workspace total is not measured yet. A box whose create lost to a delete lives until that delete's 7-day reap.
  • Not measured: cold wakes; billing at scale; whether Vercel itself refuses a second mount of an attached drive. The code does not rely on that refusal.

Before internal enable

Before customers

  • Our own retention (separate PR), through retireAndTearDown. Until it ships, a drive goes only when someone opens its workspace after the snapshot expired.
  • Schedule the drive sweep and let it delete, after a few clean dry runs.
  • Measure drive cost per workspace, and decide how it is billed.
  • The items in docs/cloud-sandbox-considerations.md.
  • A cold-wake measurement.
  • Environments built from [install] (feat(environments): build cloud environments from [install] (SUPER-3017) #9282) must set source_ref and golden_drive_name together.

Decisions that are yours

  1. Flag off in a drive-mode environment: new workspaces get no drive and clone onto the box's disk (slower), rather than being refused. Keep this degrade, or refuse?
  2. Promote with the flag off: refused for a workspace whose files are on a drive. This reverses the earlier "promote always carries the drive". Confirm.
  3. Undo after the 7-day grace: a fresh box and drive, because the deletion has completed. Confirm.
  4. A drive that cannot heal: after 3 boots in a row the workspace shows failed with the reason; its box and drive are kept. Keep 3?
  5. Existing workspaces are not moved to drives; only new ones in a drive-mode environment get one. Confirm.

Snapshot expiry deletes the drive

Retention will be ours, in a separate PR: a reaper deletes an expired workspace's box, snapshots and drive, instead of relying on Vercel's 30-day snapshotExpiration.

This PR keeps the fallback. When a wake, an agent turn or an automation dispatch gets a 410 for an expired snapshot, markSandboxUnavailable calls retireAndTearDown. That function retires the row's box and drive names, only while the row is failed and still names that box and drive. It then queues the retried teardown, which waits for the drive to detach and never deletes a drive that a row or a golden environment names. The new reaper calls the same function:

// packages/trpc/src/router/cloud-workspace/teardown.ts (export "@superset/trpc/cloud-workspace-teardown")
retireAndTearDown(
  row: { id: string; providerSandboxId: string; driveName: string | null },
  condition: SQL | undefined, // extra WHERE for the retiring UPDATE, e.g. the status the reaper read
): Promise<boolean> // false: the row no longer matches, or names another box; nothing queued

Checks on this head

This head is #9847 rebased on main (e0f54fa) through #9915 (now 0199), squashed into one commit and resolved by hand. The earlier 33 commits are kept locally as backup/9847-pre-rebase.

Ran:

  • lint:fix.
  • Tests: trpc 1160, db 59, api 269, sandbox 89.
  • Typecheck: trpc, db, api, sandbox and i18n. The whole-repo run passed 35 of 39 tasks.
  • check:i18n: the new message is translated in every locale.
  • image --local, and boot-twice with BOOT_TWICE_CA_FILE: all expectations met.
  • The model, unmodified at 98667ea: ./run.sh head, results in the coordinator thread.

Not run:

  • Typecheck for desktop, host-service and cli: tsc is killed for running out of memory on this 7 GB box. None of them is changed here.
  • The real-Vercel check on this head, and a real 410.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

🔗 Preview Links

Service Status Link
Neon Database (Neon) ❌ View Branch
Vercel API (Vercel) ✅ Open Preview
Vercel Web (Vercel) ✅ Open Preview
Vercel Agents Web (Vercel) ➖ Not set up
Vercel Marketing (Vercel) ✅ Open Preview
Vercel Docs (Vercel) ✅ Open Preview

Preview updates automatically with new commits

@expo

expo Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

No installed development build can load this pull request as an update: its native fingerprint is 08a5e03ce5fb and no development build has it. Add the mobile-build label to build one.

@saddlepaddle
saddlepaddle force-pushed the superset/thin-workspace-drive-draft-c4ad5f56 branch from 6cbde6d to cf0d1a5 Compare November 17, 2026 02:40
@saddlepaddle saddlepaddle changed the title spike(cloud-workspace): per-workspace Vercel drive — design + flagged first slice feat(cloud-workspace): workspace drive per environment, flag-gated (draft) Nov 17, 2026
@saddlepaddle
saddlepaddle force-pushed the superset/thin-workspace-drive-draft-c4ad5f56 branch from cf0d1a5 to b68a549 Compare November 17, 2026 03:11
@saddlepaddle
saddlepaddle force-pushed the superset/thin-workspace-drive-draft-c4ad5f56 branch from e0263ca to 707eabd Compare November 18, 2026 00:02
@saddlepaddle
saddlepaddle changed the base branch from main to superset/workspace-drive-columns November 18, 2026 00:02
saddlepaddle added a commit that referenced this pull request Nov 18, 2026
Adds three nullable columns, no defaults, nothing reads them yet:
- environments.golden_drive_name: the Vercel drive an environment's
  workspace drives fork from (0196).
- cloud_workspaces.drive_name: the drive mounted on every session of a
  workspace's box (0196).
- cloud_workspaces.drive_laid_out_at: when a boot first answered with the
  drive laid out (0197).

Split from #9847 so the migrations land before the code that uses them.
@saddlepaddle
saddlepaddle force-pushed the superset/workspace-drive-columns branch from 99b0fd8 to 8bec0b7 Compare November 18, 2026 03:01
saddlepaddle added a commit that referenced this pull request Nov 18, 2026
Makes what #9847 now enforces into code guards with file and line (the
drive-keyed resume lock, Session-only use of a running box, the create's
drive wait, onBox transitions, the lost job's keepDrive teardown, the
any-row teardown guard, the sweep's named-drive rule, state/fresh cleared
on attempt), models the provision's lock timeout, the drive wait under
the lock, duplicate deliveries and a failed first fetch. Seven rules hold;
two real bugs and the fresh-fork trade remain, each with a fix. Reviews
the model patch on superset/drive-model-9847-local-patch: takes what the
code does, rejects the narrowed head_of. SKILL.md: a product change must
not edit the model in the same PR.
saddlepaddle added a commit that referenced this pull request Nov 18, 2026
Adds three nullable columns, no defaults, nothing reads them yet:
- environments.golden_drive_name: the Vercel drive an environment's
  workspace drives fork from.
- cloud_workspaces.drive_name: the drive mounted on every session of a
  workspace's box.
- cloud_workspaces.drive_laid_out_at: when a boot first answered with the
  drive laid out.

One migration, 0197, generated by drizzle-kit on main after #9933's 0196.
Split from #9847 so the migration lands before the code that uses it.
@saddlepaddle
saddlepaddle force-pushed the superset/workspace-drive-columns branch from 8bec0b7 to 7b454fc Compare November 18, 2026 03:14
@saddlepaddle
saddlepaddle force-pushed the superset/thin-workspace-drive-draft-c4ad5f56 branch from 3f7b5ec to 1a96ee6 Compare November 18, 2026 03:48
The workspace drive work of #9847 as one commit on main after 78a831b
(#9915, migration 0199), resolved by hand against main's M5b failure
reasons, M7s logging, M8 metering, M9 non-resuming handles, and the
agent-tools drive. The previous 33 commits are kept locally as
backup/9847-pre-rebase.

Merged with main:
- wakeSandbox keeps the resume lock, the drive detach, the composed
  mounts and withLiveSession, and takes main's onlyIfRunning and its
  non-resuming handles; a session withLiveSession resumes mid-wake is
  reported to beforeResume as resumedSession before it boots, so it is
  metered as M9 meters it.
- promoteSandboxToEnvironment takes main's shape: the snapshot ends the
  source's session and the caller's wakeAfterStop starts it again; the
  drive fork and the golden drive's strip stay.
- provision fails through failCloudWorkspace, which now takes onBox and
  answers whether it moved the row, so a provision still fails only its
  own row.
- A delete or a failed create still retires the names (retireAndTearDown).
A drive refused for good fails the workspace as host_not_ready ("The
workspace did not become ready."), not sandbox_gone. A drive still
flushing at a wake is a SandboxNotReadyError ("still starting"). The
errors a client can be handed carry the drive in fields, console lines
and Sentry extra, not in the message. A promote of a workspace on a
drive that fails, or is refused with drives off, says only "This
workspace could not be saved as an environment. Try again later.",
translated in every locale.

The daily expiry probe is dropped: retention moves to its own PR, whose
reaper calls retireAndTearDown, as the 410 path here does.

This branch had an error being deployed

1 failed deployment
preview — 5c03bd9c Deployed Nov 18, 2026 by saddlepaddle via Deploy Database (Neon) #15252
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant