Skip to content

feat(cloud-workspace): own workspace retention with a daily reaper (draft) - #9987

Draft
saddlepaddle wants to merge 4 commits into
mainfrom
superset/own-workspace-retention-reaper-49e68bae
Draft

saddlepaddle wants to merge 4 commits into
mainfrom
superset/own-workspace-retention-reaper-49e68bae

Conversation

@saddlepaddle

@saddlepaddle saddlepaddle commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Draft. Plan: plans/done/20261118-1500-cloud-workspace-retention-reaper.md.

What it does

  • New workspace boxes are created with snapshotExpiration: 0, as goldens already were. Vercel no longer deletes a workspace's snapshot; we decide when it goes.
  • A job that runs every 6 hours, reap-retention, does the deleting:
    • A workspace that is asleep and unused for 30 days becomes failed / snapshot_expired. That is the same state a 410 gives today, so customers see the same retention and the same failed screen.
    • First retireAndTearDown(row, condition) gives the row a new box name, so nothing can wake the box being deleted, and queues the old box's retried teardown. It has feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847's name, signature and module (@superset/trpc/cloud-workspace-teardown), box-only, so feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847's file replaces it as is.
    • Then the row fails. A wake between the two writes finds no box and fails the row itself.
    • Every box, with a drive or without, goes through the retried teardown job (deleteSandboxAndDriveLater). Main deleted boxes inline, without retry.
  • In production, the same job also deletes an archived box still there a day after its 7-day grace. Vercel's 30-day expiry used to clean up after a lost archive delete; with snapshotExpiration: 0 that backstop is gone. It goes through the archive's own delete stage: reapArchivedCloudWorkspace({stage: "delete", archivedAt: deleted_at}), or reapUndatedArchivedCloudWorkspace (archived, deleted_at null, updated_at older than the cutoff) for the 71 rows with no archive date. At most 20 a run, reported as found / alreadyDeleted / deleted / skipped / deferred / failed.
  • The same job queues teardown for orphan boxes in production. An orphan is a tagged box that no workspace row and no unarchived environment names, settled and at least 7 days old.
  • Each run logs and returns what it retired, queued, skipped and deferred. It also writes a retention.run Sentry log line and checks in to the reap-retention cron monitor, so a reaper that stopped is noticed.

How it works

reap-retention every 6 hours (QStash '0 */6 * * *', Sentry.withMonitor)
  cutoff = now - 30d
  rows = asleep vercel rows, not archived, idleSince(cutoff), oldest first, limit 200
    idleSince: created_at < cutoff, sandbox_expires_at null or < cutoff,
               and no boot that is open or started/stopped after cutoff
  for row:
    box running now (provider read)            -> skippedRunning
    retireAndTearDown(row, status = asleep and still idleSince(cutoff), not archived):
      UPDATE ... SET provider_sandbox_id = nextSandboxNameFor(id)
        WHERE id and provider_sandbox_id = old name and condition
      no row matched                           -> skippedChanged
      else deleteSandboxAndDriveLater(old name) -> queue teardown job
    failCloudWorkspace from asleep, still idleSince(cutoff):
      status failed, reason snapshot_expired, sandbox_url null
    pause 250 ms
  if VERCEL_ENV == production:
    boxes = one Vercel listing of every box
    archived backstop: rows archived with deleted_at (or, with none, updated_at) < now - 8d
      found = count; lingering = those whose box is in the listing; alreadyDeleted = found - lingering
      first 20 lingering: reapArchivedCloudWorkspace(delete, archivedAt) or reapUndatedArchivedCloudWorkspace
    orphans: boxes tagged kind=workspace or kind=environment
    keep: status stopped/failed/aborted, created and status-updated > 7 days ago,
          not retired this run, not named by any workspace row (any status)
          or unarchived environment source_ref
    deleteSandboxAndDriveLater for the first 200, 250 ms apart; the rest are counted as deferred
  after 200 s a run starts no new work; what is left is counted as deferred

teardown(box) job, retried up to 20 times, 30 s apart
  a row names it -> kept
  stop; a row names it -> kept
  delete (with its snapshots)

The run pauses 250 ms after each teardown it queues: at most 4 a second, far below Vercel's 20 deletions a second per team. The seam takes no delay, so the pacing is in the loop.

Sizing: 4 runs of 200 is 800 a day. Production made a median 156 workspaces a day and at most 725 in the last 30 days, so a normal day clears about 5 times over and even the peak day clears within one day. A backlog takes several runs.

Safety

  • Boot rows are written before Vercel resumes a box, so a wake in flight keeps its box. The reaper uses the latest of the boot rows, sandbox_expires_at and created_at, so when they disagree it deletes later, never earlier.
  • Archive and its 7-day grace period are unchanged. Archived rows are never selected, and the teardown never deletes a box that any row, in any status, still names.
  • Orphan reaping runs only when VERCEL_ENV is production. Every deployment shares one Vercel sandbox project, so a preview or dev database would see production's boxes as orphans.

Drives: a seam for #9847

Main has no workspace drives, and no row names one. Until #9847's drive columns exist, any drive would look orphaned, so this PR reaps boxes only. teardownSandbox uses #9847's route (/api/cloud-workspaces/teardown), payload shape and name, with driveName typed null. #9847's drive-aware version replaces it there.

Existing boxes

@vercel/sandbox 4.0.0 has sandbox.update({ snapshotExpiration }), and the call wakes nothing. It changes the default for snapshots taken from then on; the SDK has no call that changes an existing snapshot's expiry. bun run backfill-snapshot-expiration (packages/sandbox) is a dry run by default. With --apply it updates 2 boxes a second. Not run. Until a box's next stop, the 410 path stays its fallback.

State machine

The model is not edited. The reaper's status write maps to Archive: a Live row (here only asleep) moves to a status that is not Live, and the box is unchanged. Its guard is the boot row, which plays the role of "wake" \notin inflight. The teardown that follows is like Reap, except that it deletes the box. One gap: with billing off there is no boot row, so only the provider read guards a wake in flight. A wake that loses that race finds the row failed, the same as a 410 today.

Production numbers (read-only, 2026-11-18)

  • Asleep vercel workspaces: 857. Asleep and unused for more than 30 days: 0. Within 30 days: 857. The reaper's own SQL agrees: 0 at 30 days, 0 at 7, 518 at 2.
  • By boot rows alone, 188 are unused for more than 7 days, and the oldest last boot is 2026-10-22.
  • A one-off write on 2026-11-16 moved sandbox_expires_at past the last boot on 412 asleep rows. The reaper reaches those no earlier than 2026-12-16; until then Vercel's own expiry still applies to them.

Archived rows (read-only, 2026-11-18)

  • 6,203 archived; 3,553 past grace + 1 day (71 of them with no deleted_at). A random sample of 20 archived rows idle for more than 30 days: all 20 boxes are 404, and the one name checked has no snapshots left. A control box returned 200.

Before it runs

  • Create the QStash schedule 0 */6 * * * for /api/cloud-workspaces/jobs/reap-retention in Upstash. It can also be fired once from the "Publish Cloud Workspace Job" workflow.
  • Run the backfill dry, then decide whether to run --apply.

Checks

  • bun test in packages/trpc: 1110 pass. In packages/sandbox: 85 pass.
  • bun run lint; turbo typecheck for trpc, api and sandbox.
  • New tests:
    • retention: "retires each idle workspace, a pause apart", "a box that runs a session now is skipped, not retired", "a row woken or changed after it was read keeps its box", "queues only old, settled boxes no row names, and not one it just retired", "outside production it never looks for orphans", "queues at most a batch of orphans a run and counts the rest", "stops starting work when its time is up and leaves the rest for the next run", "retires the box name while the row is asleep and idle, then fails the row as an expired snapshot", "deletes an archived box still there past its grace through the archive's own delete, by the guard that fits the row", "an archived row unarchived or changed before its delete is counted as skipped"; the production-only test now also covers archived rows.
    • reap: "deletes the box of an archived row with no archive date only while it is still archived and unchanged".
    • teardown: "stops, then deletes a box no row names", "keeps a box a row names by the time it would delete it", "a failure requeues the next attempt, and the last attempt gives up", "gives the row a new box name and queues the old box's teardown only when the update matched", "a box without a drive goes through the retried job too, not an inline delete".
    • vercel: "a workspace box's snapshots never expire on Vercel's clock".
    • backfill: "changes only boxes that still expire, paced, and only with --apply".
  • Not run: the TLA+ model (unchanged), and anything against real Vercel.

New workspace boxes no longer expire their snapshots on Vercel's clock. A
daily QStash job (reap-retention) fails a workspace asleep and unused for 30
days as snapshot_expired, the state the 410 path gives, retires its box name
in the same update, and queues a retried box teardown. In production it also
tears down tagged boxes no row or unarchived environment names, 7 days old,
one delete a second. Counts go to logs, Sentry logs and a cron monitor.

The teardown is box-only with #9847's route and payload, so its drive
version replaces it. A dry-run-by-default backfill sets snapshotExpiration 0
on existing boxes (not run).
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

🔗 Preview Links

Service Status Link
Neon Database (Neon) ✅ View Branch
Vercel API (Vercel) ✅ Open Preview
Vercel Web (Vercel) ✅ Open Preview
Vercel Agents Web (Vercel) ➖ Not set up
Vercel Marketing (Vercel) ✅ Open Preview
Vercel Docs (Vercel) ✅ Open Preview

Preview updates automatically with new commits

@expo

expo Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Preview update published for development build 0dd268df (fingerprint 08a5e03ce5fb). Open the build on your phone and scan:

Update group e00b1d4f

…batch

Four runs of 200 clear 800 a day: production made a median 156 workspaces
a day and at most 725. A run stops starting work after 200 s and reports
the rest as deferred, so a backlog takes several runs inside the route's
300 s.
With snapshotExpiration 0, Vercel no longer cleans up after a lost archive
delete. In production each run lists every box once. An archived row past
grace + 1 day whose box is still listed is deleted through the archive's own
delete stage, or through a new guard (archived, no deleted_at, updated_at
older) for rows archived before deleted_at existed. At most 20 a run,
counted as found / alreadyDeleted / deleted / skipped / deferred / failed.
#9847's seam

The idle stage takes the box name off the row with retireAndTearDown(row,
condition), the name, signature and module #9847 uses, then fails the row.
Every box, with a drive or not, goes through the retried teardown job, and
the run paces teardowns in its loop since the seam takes no delay.

This branch was previously deployed

1 inactive deployment
preview — 72fa4bb7 Deployed Nov 18, 2026 by saddlepaddle via Deploy Marketing #15253
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant