Repository navigation
docs(sandbox): TLA+ model of the workspace drive lifecycle (draft) - #9914
Draft
saddlepaddle wants to merge 6 commits into
Draft
saddlepaddle wants to merge 6 commits into
saddlepaddle wants to merge 6 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…ck to its resume Found by the drive lifecycle model (#9914), rule SingleWriter: - A redelivered provision found its box stopped and resumed it with the drive, through writeFiles and the boot command, without waiting for the previous session to release the drive. provisionSandbox now waits for the detach and remounts before anything resumes a box it finds stopped. - Two concurrent wakes both passed the detach check; the first resumed, was refused and stopped the box, and the second resumed it while the stopping session still held the drive. wakeSandbox now runs from its status read to its boot command under a ResumeLock, and a refused box is stopped under the same lock. The control plane's lock (sandboxResumeLock) is a transaction-scoped pg_try_advisory_xact_lock per sandbox; a caller that finds it held answers "still starting" rather than queue. Scripts run unlocked. - The settle's refusal poll fetches the box fresh and reads its boot log only while it is running, so a read never resumes a box another caller stopped.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…etires the names first Found by the drive lifecycle model (#9914): - NoWorkLost: a provision still running while the user archived, unarchived, worked and archived again tore everything down at once, skipping the second grace period. This existed on main too. A provision that lost its row to an archive now tears nothing down: that archive's reap, which knows its own archivedAt, stops and deletes the box after its grace period, and a later archive gets its own. - NoTeardownOfLive: a teardown keyed on names could delete a box that an undo had just brought back. Committing to delete now retires the names in the same write (retireAndTearDown): the row gets a fresh box name and no drive, so an undo after that point gets a new box and a new drive and never the ones being deleted. The reap's delete stage, the archive fallback and a failed provision's transition to failed all retire first, and tear down only when their write won. - nextSandboxNameFor moves to sandbox-name.ts so teardown.ts does not import provision.ts. The plan records the decisions, including why the first-layout check covers the model's FixFresh and the golden strip keeps local branches.
saddlepaddle
force-pushed
the
superset/thin-workspace-drive-draft-c4ad5f56
branch
from
November 18, 2026 00:02
e0263ca to
707eabd
Compare
10 tasks
saddlepaddle
force-pushed
the
superset/drive-lifecycle-tla-model-596243e4
branch
2 times, most recently
from
November 18, 2026 01:11
eda25ab to
7bb2d6d
Compare
Contributor
🚀 Preview Deployment🔗 Preview Links
Preview updates automatically with new commits |
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
Found by the drive lifecycle model (#9914), rule SingleWriter (FixRunningResume). A wake or provision that read its box as running then called Sandbox methods (extendTimeout, update, writeFiles, runCommand), and the SDK resumes a session that ended in between, with no drive check. - resumeSession (Sandbox.get with resume: true) is the only call that resumes a box, and it runs under the resume lock after the drive check. - After a read, every call goes through the session read: a Session method fails instead of resuming. withLiveSession then re-reads the box; if its session ended, it waits for the drive, remounts and resumes under the same lock, and runs the calls once more on the new session. - The resize of a stopped box moves after the drive check, and the settle reads the boot log through a running session only. - A new box with a drive is created only once the drive has detached: a box of another name may still hold it (FixOneProvision, with the next commit).
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…lled Found by the drive lifecycle model (#9914), rule SingleWriter (FixOneProvision). A delete while a provision was between its claim and its create, then an undo, gave the row a new box name and a second provision with the same drive; the first job won provisioning -> awake and wrote its own box back, and both boxes mounted the drive, each under its own box-name lock. - provisioning -> awake and -> failed now require the row to name the job's own box (transitionCloudWorkspace onBox). - A job that lost to another box tears its own box down; the teardown keeps the drive, which the provisioning row names. - sandboxResumeLock is keyed on the drive when there is one. A wake still tries and answers "still starting"; a provision waits up to 20 s, since a missed lock would otherwise fail its row.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…d checks again before each delete Found by the drive lifecycle model (#9914), rules NoWorkLost, NoTeardownOfLive and NoStuckForever (FixSweepNamed). The sweep treated a row archived past its grace as owning nothing; with the reap's delete stage late, an unarchive resumed the box and the sweep deleted its drive from a list built earlier. - Any workspace or environment row that names a drive keeps it, whatever its status: the reap's delete stage clears the names when it really tears a row down (retireAndTearDown), so the grace exception is gone. - Each delete reads the rows and the drive's attachment again first.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
Found by the drive lifecycle model (#9914), rule FreshForkIsFetched (FixFreshDurable). SUPERSET_DRIVE_FRESH lived for one boot, so a first session that ended between the layout and the checkout made the next boot keep a branch the golden already had, unfetched. The first layout now writes state/fresh on the drive, synced with the owner, and the checkout removes it once it has run, whether it succeeded or not: after that the box was usable and what is on the drive may be the user's. boot-twice adds a fork whose first session ended before its checkout.
saddlepaddle
force-pushed
the
superset/drive-lifecycle-tla-model-596243e4
branch
from
November 18, 2026 03:12
7bb2d6d to
ce06e70
Compare
saddlepaddle
had a problem deploying
to
preview
November 18, 2026 03:13 — with
GitHub Actions
Failure
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…ck to its resume Found by the drive lifecycle model (#9914), rule SingleWriter: - A redelivered provision found its box stopped and resumed it with the drive, through writeFiles and the boot command, without waiting for the previous session to release the drive. provisionSandbox now waits for the detach and remounts before anything resumes a box it finds stopped. - Two concurrent wakes both passed the detach check; the first resumed, was refused and stopped the box, and the second resumed it while the stopping session still held the drive. wakeSandbox now runs from its status read to its boot command under a ResumeLock, and a refused box is stopped under the same lock. The control plane's lock (sandboxResumeLock) is a transaction-scoped pg_try_advisory_xact_lock per sandbox; a caller that finds it held answers "still starting" rather than queue. Scripts run unlocked. - The settle's refusal poll fetches the box fresh and reads its boot log only while it is running, so a read never resumes a box another caller stopped.
saddlepaddle
force-pushed
the
superset/thin-workspace-drive-draft-c4ad5f56
branch
from
November 18, 2026 03:48
3f7b5ec to
1a96ee6
Compare
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…etires the names first Found by the drive lifecycle model (#9914): - NoWorkLost: a provision still running while the user archived, unarchived, worked and archived again tore everything down at once, skipping the second grace period. This existed on main too. A provision that lost its row to an archive now tears nothing down: that archive's reap, which knows its own archivedAt, stops and deletes the box after its grace period, and a later archive gets its own. - NoTeardownOfLive: a teardown keyed on names could delete a box that an undo had just brought back. Committing to delete now retires the names in the same write (retireAndTearDown): the row gets a fresh box name and no drive, so an undo after that point gets a new box and a new drive and never the ones being deleted. The reap's delete stage, the archive fallback and a failed provision's transition to failed all retire first, and tear down only when their write won. - nextSandboxNameFor moves to sandbox-name.ts so teardown.ts does not import provision.ts. The plan records the decisions, including why the first-layout check covers the model's FixFresh and the golden strip keeps local branches.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
Found by the drive lifecycle model (#9914), rule SingleWriter (FixRunningResume). A wake or provision that read its box as running then called Sandbox methods (extendTimeout, update, writeFiles, runCommand), and the SDK resumes a session that ended in between, with no drive check. - resumeSession (Sandbox.get with resume: true) is the only call that resumes a box, and it runs under the resume lock after the drive check. - After a read, every call goes through the session read: a Session method fails instead of resuming. withLiveSession then re-reads the box; if its session ended, it waits for the drive, remounts and resumes under the same lock, and runs the calls once more on the new session. - The resize of a stopped box moves after the drive check, and the settle reads the boot log through a running session only. - A new box with a drive is created only once the drive has detached: a box of another name may still hold it (FixOneProvision, with the next commit).
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…lled Found by the drive lifecycle model (#9914), rule SingleWriter (FixOneProvision). A delete while a provision was between its claim and its create, then an undo, gave the row a new box name and a second provision with the same drive; the first job won provisioning -> awake and wrote its own box back, and both boxes mounted the drive, each under its own box-name lock. - provisioning -> awake and -> failed now require the row to name the job's own box (transitionCloudWorkspace onBox). - A job that lost to another box tears its own box down; the teardown keeps the drive, which the provisioning row names. - sandboxResumeLock is keyed on the drive when there is one. A wake still tries and answers "still starting"; a provision waits up to 20 s, since a missed lock would otherwise fail its row.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…d checks again before each delete Found by the drive lifecycle model (#9914), rules NoWorkLost, NoTeardownOfLive and NoStuckForever (FixSweepNamed). The sweep treated a row archived past its grace as owning nothing; with the reap's delete stage late, an unarchive resumed the box and the sweep deleted its drive from a list built earlier. - Any workspace or environment row that names a drive keeps it, whatever its status: the reap's delete stage clears the names when it really tears a row down (retireAndTearDown), so the grace exception is gone. - Each delete reads the rows and the drive's attachment again first.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
Found by the drive lifecycle model (#9914), rule FreshForkIsFetched (FixFreshDurable). SUPERSET_DRIVE_FRESH lived for one boot, so a first session that ended between the layout and the checkout made the next boot keep a branch the golden already had, unfetched. The first layout now writes state/fresh on the drive, synced with the owner, and the checkout removes it once it has run, whether it succeeded or not: after that the box was usable and what is on the drive may be the user's. boot-twice adds a fork whose first session ended before its checkout.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…a held lock Found by the drive lifecycle model (#9914), re-derived from 3f7b5ec: - NoWorkLost (FixFailOwnOnly). Two provisions on one box name (a delivery queued before a delete plus the undo's, or a QStash duplicate): one brought the row awake and the user worked; the other then threw (a spend-cap refusal in authorizeBoot) and its catch failed the row from provisioning, awake or asleep with onBox, which matched because both share the box name, and deleted the box. A provision now fails a row only while it is still provisioning, or after this very job made it awake; the stop, the teardown and the failed chat moves follow only a transition it won. - TransientIsRetried (FixProvLockRetry). The provision's lock wait timed out into SandboxNotReadyError, handled as a box kept for diagnosis: the box stopped and the row failed. A held lock is now SandboxBusyError (a not-ready, so a wake still answers "still starting"), and a provision requeues itself every 15 s, at most 10 times, touching nothing; after that it fails only a row still provisioning. - A lost provision's keepDrive teardown waited for the drive to leave every box, so the lost box leaked while the live box held the drive. It now waits only for this box.
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…den's untouched copy Found by the drive lifecycle model (#9914), rule FreshForkIsFetched (FixFetchIfUntouched). state/fresh goes after the first checkout attempt, so a failed first fetch left the golden's same-named branch kept as the user's work, unfetched, for good. A fresh fork records the golden's commit per repository before its first checkout attempt (state/golden-head). A later boot fetches a checkout with no marker only while it is exactly that copy: HEAD at the recorded commit, a clean tree, and the branch absent or at the same commit. Anything else is kept; a failed fetch keeps it too, never recloning. The record goes once a checkout succeeds. boot-twice adds a fork whose first fetch and clone both fail, then is fetched on a later boot, and one where the user committed in between, whose copy is kept.
saddlepaddle
force-pushed
the
superset/drive-lifecycle-tla-model-596243e4
branch
from
November 18, 2026 04:30
ce06e70 to
98667ea
Compare
saddlepaddle
had a problem deploying
to
preview
November 18, 2026 04:30 — with
GitHub Actions
Failure
… nobody opens again; no customer copy names a drive Vercel expires a workspace's snapshot 30 days after last use, and only a 410 tells the control plane. A workspace nobody opens again never gets that 410 from a wake, so its drive stayed. expireWorkspaceDrives, run once a day by a QStash schedule on /api/cloud-workspaces/jobs/expire-drives, probes at most 25 workspaces on a drive, asleep or failed, whose box last stopped more than 30 days ago, oldest first, one a second, and sends a 410 down markSandboxUnavailable, the wake's own path. The candidate query is in packages/db and is tested against pglite. A customer never reads that the box has a drive: a drive refused for good fails the workspace as host_not_ready, a drive still flushing at a wake is a not-ready, the errors a client can be handed carry the drive in fields rather than the message, and a promote of a workspace on a drive that fails or is refused with drives off says only that it could not be saved as an environment.
saddlepaddle
force-pushed
the
superset/thin-workspace-drive-draft-c4ad5f56
branch
from
November 18, 2026 22:42
592bc80 to
a29d6f1
Compare
Drive.tla checks the drive lifecycle against the owner's three requirements in five scenarios. At e560e42 eight of ten rules fail; each failure is a real bug with a trace and a proposed fix, and with every fix on all rules hold. Broken configs prove each rule can fail.
Models the teardown's row checks, the retirement of names, the resume lock, the downgrade refusal, fresh-fork detection, the provision retry, drive_laid_out_at, the terminal refusal and the drive sweep as the code does them now, and adds a scenario with two provision jobs for one row. Five rules hold; five fail on four real bugs, each with a trace and a proposed fix.
Makes what #9847 now enforces into code guards with file and line (the drive-keyed resume lock, Session-only use of a running box, the create's drive wait, onBox transitions, the lost job's keepDrive teardown, the any-row teardown guard, the sweep's named-drive rule, state/fresh cleared on attempt), models the provision's lock timeout, the drive wait under the lock, duplicate deliveries and a failed first fetch. Seven rules hold; two real bugs and the fresh-fork trade remain, each with a fix. Reviews the model patch on superset/drive-model-9847-local-patch: takes what the code does, rejects the narrowed head_of. SKILL.md: a product change must not edit the model in the same PR.
FailOwnOnly (madeAwake), ProvLockRetry (SandboxBusyError requeue) and FetchIfUntouched (state/golden-head) are now code guards with file and line; the keepDrive teardown waits only for its own box. A failed refetch of an untouched copy keeps it for the next boot, as the code does. All ten rules hold in every scenario and at deeper bounds; each new guard turned off brings back the bug it closed. SKILL.md states what the guarantee covers.
saddlepaddle
force-pushed
the
superset/thin-workspace-drive-draft-c4ad5f56
branch
from
November 18, 2026 23:15
a29d6f1 to
5c03bd9
Compare
…om the code at a29d6f1 Adds the platform's 30-day snapshot expiry, the 410 path of markSandboxUnavailable (stale row read, fail, retire, death between them), the daily expiry probe, and sandboxExists throwing on a 410 in unarchive. New rules: NoDeleteWhileMountable (action property) and ExpiredDriveDeleted (liveness under FairSpecCron). At the code the ten earlier rules and NoDeleteWhileMountable hold; ExpiredDriveDeleted fails for a row stranded in provisioning, which the probe never takes. FixCronStranded closes it. Co-Authored-By: Claude <noreply@anthropic.com>
saddlepaddle
force-pushed
the
superset/drive-lifecycle-tla-model-596243e4
branch
from
November 18, 2026 23:43
98667ea to
23be3cc
Compare
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A TLA+ model of the workspace drive lifecycle, re-derived from the code at
1a96ee6b(head of #9847's branch). Only.agents/skills/verify-sandbox-drive/changes; no product code.The rules (for the product owner)
Your three requirements: nothing a user does is ever lost; an empty
/workspaceis never shown as their work; a workspace without a drive behaves exactly as today. The model checks them as these rules:Results at head
1a96ee6bAll 10 rules hold, from the model's own derivation of the code: every guard the model relies on is in the code at a file and line (listed in
SKILL.md), and no proposed fix is turned on. They also hold at deeper bounds (two provisions with two redeliveries, two failed fetches with four sessions, four sessions in the lifecycle, three concurrent wakes).1a96ee6b?The three fixes claimed for this round, checked against the code
madeAwakeis set from its own→ awaketransition (provision.ts:185,239), the failure isfrom: madeAwake ? [provisioning, awake, asleep] : [provisioning](:371), and the stop, teardown and failed chat moves follow only a transition it won (:381-395). A redelivery is a new execution: it starts withmadeAwakefalse and returnsskippedat once if the row is no longer provisioning (:159), so it can never fail a row another execution brought up.SandboxBusyError(vercel.ts:388,resume-lock.ts:30,38), requeued every 15 s withlockRetries + 1, touching nothing (provision.ts:315-331). After 10 retries (about 6 minutes) it fails a row still provisioning; the model assumes a holder lets go sooner (it waits at most 2 minutes for a drive).state/golden-headanduntouched_golden(superset-boot:376-386,394-402). A failed refetch keeps the checkout and never reclones.Also checked: the lost provision's box no longer leaks (its
keepDriveteardown waits only for its own box,teardown.ts:60); swap is gone from the drive. Each of the three new guards, turned off, brings back the bug it closed (./run.sh bugs). Every broken config fails its rule (./run.sh broken).What the guarantee covers, and what it does not
The rules hold for one workspace on one environment, within the bounds below, and on these assumptions:
Sessioncall on an ended session fails rather than resumes; a mount can fail and the session starts with nothing mounted; an abort rolls back the root disk and not the drive; Vercel's answer to a mount that names a deleted drive. The model does not rely on Vercel refusing a second read-write mount.HEAD, a cleangit statusand the branch. A user's gitignored file at a path the fetched base tracks could be overwritten on such a boot. Not modeled.Bounds
One workspace, one environment, sandbox ids 1 and 2 plus a retired name, drives
drive-1anddrive-2, one reprovision, at most 3 sessions,DRIVE_REFUSAL_LIMIT = 3. Seven scenarios; the pass takes about 100 s on 8 cores,headabout 13 minutes:Drive.cfgDrive-rearchive.cfgDrive-race.cfgDrive-boot.cfgDrive-env.cfgDrive-twoprov.cfgDrive-fresh.cfgDeeper bounds, run by hand, no error: twoprov with 2 redeliveries (8,780,864 states), fresh with 2 failed fetches, 3 wakes and 4 sessions (113,896), lifecycle with 4 sessions (4,695,594), race with 3 wakes and 2 redeliveries (121,408).
Where the model and the code differ
madeAwakeis the job'sownfield, set by its own transition and cleared by a redelivery.liveReferences's environment checks are not modeled.How to run
SKILL.mdkeeps the rule that a product change must not edit the model in the same PR.