Repository navigation
feat(cloud-workspace): own workspace retention with a daily reaper (draft) - #9987
Draft
saddlepaddle wants to merge 4 commits into
Draft
saddlepaddle wants to merge 4 commits into
saddlepaddle wants to merge 4 commits into
Conversation
New workspace boxes no longer expire their snapshots on Vercel's clock. A daily QStash job (reap-retention) fails a workspace asleep and unused for 30 days as snapshot_expired, the state the 410 path gives, retires its box name in the same update, and queues a retried box teardown. In production it also tears down tagged boxes no row or unarchived environment names, 7 days old, one delete a second. Counts go to logs, Sentry logs and a cron monitor. The teardown is box-only with #9847's route and payload, so its drive version replaces it. A dry-run-by-default backfill sets snapshotExpiration 0 on existing boxes (not run).
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
🚀 Preview Deployment🔗 Preview Links
Preview updates automatically with new commits |
|
Preview update published for development build |
…batch Four runs of 200 clear 800 a day: production made a median 156 workspaces a day and at most 725. A run stops starting work after 200 s and reports the rest as deferred, so a backlog takes several runs inside the route's 300 s.
With snapshotExpiration 0, Vercel no longer cleans up after a lost archive delete. In production each run lists every box once. An archived row past grace + 1 day whose box is still listed is deleted through the archive's own delete stage, or through a new guard (archived, no deleted_at, updated_at older) for rows archived before deleted_at existed. At most 20 a run, counted as found / alreadyDeleted / deleted / skipped / deferred / failed.
#9847's seam The idle stage takes the box name off the row with retireAndTearDown(row, condition), the name, signature and module #9847 uses, then fails the row. Every box, with a drive or not, goes through the retried teardown job, and the run paces teardowns in its loop since the seam takes no delay.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Plan:
plans/done/20261118-1500-cloud-workspace-retention-reaper.md.What it does
snapshotExpiration: 0, as goldens already were. Vercel no longer deletes a workspace's snapshot; we decide when it goes.reap-retention, does the deleting:failed/snapshot_expired. That is the same state a 410 gives today, so customers see the same retention and the same failed screen.retireAndTearDown(row, condition)gives the row a new box name, so nothing can wake the box being deleted, and queues the old box's retried teardown. It has feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847's name, signature and module (@superset/trpc/cloud-workspace-teardown), box-only, so feat(cloud-workspace): workspace drive per environment, flag-gated (draft) #9847's file replaces it as is.deleteSandboxAndDriveLater). Main deleted boxes inline, without retry.snapshotExpiration: 0that backstop is gone. It goes through the archive's own delete stage:reapArchivedCloudWorkspace({stage: "delete", archivedAt: deleted_at}), orreapUndatedArchivedCloudWorkspace(archived,deleted_atnull,updated_atolder than the cutoff) for the 71 rows with no archive date. At most 20 a run, reported as found / alreadyDeleted / deleted / skipped / deferred / failed.retention.runSentry log line and checks in to thereap-retentioncron monitor, so a reaper that stopped is noticed.How it works
The run pauses 250 ms after each teardown it queues: at most 4 a second, far below Vercel's 20 deletions a second per team. The seam takes no delay, so the pacing is in the loop.
Sizing: 4 runs of 200 is 800 a day. Production made a median 156 workspaces a day and at most 725 in the last 30 days, so a normal day clears about 5 times over and even the peak day clears within one day. A backlog takes several runs.
Safety
sandbox_expires_atandcreated_at, so when they disagree it deletes later, never earlier.VERCEL_ENVis production. Every deployment shares one Vercel sandbox project, so a preview or dev database would see production's boxes as orphans.Drives: a seam for #9847
Main has no workspace drives, and no row names one. Until #9847's drive columns exist, any drive would look orphaned, so this PR reaps boxes only.
teardownSandboxuses #9847's route (/api/cloud-workspaces/teardown), payload shape and name, withdriveNametypednull. #9847's drive-aware version replaces it there.Existing boxes
@vercel/sandbox4.0.0 hassandbox.update({ snapshotExpiration }), and the call wakes nothing. It changes the default for snapshots taken from then on; the SDK has no call that changes an existing snapshot's expiry.bun run backfill-snapshot-expiration(packages/sandbox) is a dry run by default. With--applyit updates 2 boxes a second. Not run. Until a box's next stop, the 410 path stays its fallback.State machine
The model is not edited. The reaper's status write maps to
Archive: a Live row (here onlyasleep) moves to a status that is not Live, and the box is unchanged. Its guard is the boot row, which plays the role of"wake" \notin inflight. The teardown that follows is likeReap, except that it deletes the box. One gap: with billing off there is no boot row, so only the provider read guards a wake in flight. A wake that loses that race finds the rowfailed, the same as a 410 today.Production numbers (read-only, 2026-11-18)
sandbox_expires_atpast the last boot on 412 asleep rows. The reaper reaches those no earlier than 2026-12-16; until then Vercel's own expiry still applies to them.Archived rows (read-only, 2026-11-18)
deleted_at). A random sample of 20 archived rows idle for more than 30 days: all 20 boxes are 404, and the one name checked has no snapshots left. A control box returned 200.Before it runs
0 */6 * * *for/api/cloud-workspaces/jobs/reap-retentionin Upstash. It can also be fired once from the "Publish Cloud Workspace Job" workflow.--apply.Checks
bun testin packages/trpc: 1110 pass. In packages/sandbox: 85 pass.bun run lint;turbo typecheckfor trpc, api and sandbox.