Repository navigation
feat(sandbox): agent CLIs only from the agents drive (PR B, draft) - #9911
Draft
saddlepaddle wants to merge 3 commits into
Draft
saddlepaddle wants to merge 3 commits into
saddlepaddle wants to merge 3 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
🚀 Preview Deployment🔗 Preview Links
Preview updates automatically with new commits |
|
Preview update published for development build |
saddlepaddle
force-pushed
the
superset/thin-agents-only-from-the-drive-draft-a37d8481
branch
from
November 18, 2026 02:35
52cb11c to
70c5f1f
Compare
3 tasks
saddlepaddle
changed the base branch from
main
to
superset/agent-tools-drive-release
November 18, 2026 02:36
claude, codex and pdev become drive-only asset rows like opencode and pi; install-agent-clis and the sandbox-agent-tools-drive flag are removed, so every create and wake mounts the drive of the box's region. Drive.get, the create and the wake's mounts update are retried on 429 and 5xx; after that the create or wake fails with "Agents are unavailable in <region> right now; try again" (serverError.cloudWorkspace.agentsUnavailable) and a Sentry event with the region, bundle and drive. superset-boot links every agent the drive's manifest lists over whatever is on PATH, removes the link of one the drive lacks, never reads an older golden's disk copies, and refuses a boot that finds no manifest. SANDBOX_ROOT_DISK_AGENTS names where older bundles put the agents; the release's golden checks, the probe and boot-twice fail when one is on disk. Image (bun run image --local): 4.89 GB -> 4.10 GB.
saddlepaddle
force-pushed
the
superset/thin-agents-only-from-the-drive-draft-a37d8481
branch
from
November 18, 2026 21:55
f8bbe5c to
e04534e
Compare
saddlepaddle
changed the base branch from
superset/agent-tools-drive-release
to
main
November 18, 2026 21:55
saddlepaddle
added a commit
that referenced
this pull request
Nov 18, 2026
…from the image (SUPER-3017) createBox runs the root-disk agent check on a forked box. When it prints anything, the fork is deleted and the build claims a fresh slot: at most 10 fresh builds at once and 40 started per hour, under an advisory lock. A daily build waits up to an hour for a slot; past the caps, or for any other trigger, the build forks as before and the next run checks again. A claimed build is recorded as kind=fresh. The daily run starts the busiest environments first (most cloud workspaces in the last week). The check command is inlined until #9911 exports ROOT_DISK_AGENTS_COMMAND.
2 of 6 tasks
A customer never sees the drive. AgentToolsUnavailableError carries the existing "The cloud workspace could not start" as its message, a wake maps it to the existing "Cloud workspace is still starting" like SandboxNotReadyError, and an automation records cloud_not_ready. The drive, region and bundle stay in Sentry and the server log. The agentsUnavailable message and its translations are removed, and the box's refusal no longer names the drive.
…image real-sandbox boots the registry's latest image, which a release of main built with the agents on its disk, so a clean root disk is not this checkout's to promise there. The probe then checks that the bundle runs no agent-installing step and logs the image's leftovers; the strict check stays for a release that pushed its own image (not --skip-image).
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it does
dest,install-agent-clisis deleted, and thesandbox-agent-tools-driveflag and its plumbing are removed, so every create, wake and promote restart mounts the drive of the box's region./opt/agentsis refused before host-service starts./usr/local/binis replaced with the drive's, so a leftover link cannot win.Pseudocode
Failure path for a box without the manifest: the existing boot-failure path.
refusestampsboot.refusedand exits beforestart_host_service, so host-service never answers. The control plane'swaitForHostServicethen throwsSandboxNotReadyErrorafterHOST_READY_TIMEOUT_MS(60 s):failedand keeps the box stopped for diagnosis (keepForDiagnosis).TIMEOUT "Cloud workspace is still starting".The checkout-failure path was not used. It runs after host-service is up, so it would open a box without agents.
What a customer sees
/opt/agents.failedworkspace (reasoncreate_failed), which the desktop shows with its existing "Couldn't start cloud workspace"; chats moving into it get the existing "The cloud workspace could not start";cloud_not_ready;agent-tools drive not mounted: <reason>, tagssandbox.agent_tools_drive,sandbox.region,sandbox.bundle);Trade-off, plainly: a Vercel drive API outage, or a missing drive in a region, now blocks every create and wake in that region. On main, the box would open with its disk agents; in the earlier version of this PR, it would open without agents. Running boxes are unaffected.
Goldens that still hold agents
Every production golden today holds the agents on its disk. No drive-only bundle has shipped yet, and every golden descends from an image that installed them. Deleting the files would not shrink a snapshot (no discard), and every fork carries the bytes, so each such golden has to be built fresh from the image once.
Count (Neon project
Superset, default branch, read-only SELECT):source_kind = fork), liveAll rows show bundle
46b0f9720162. The release moves every row to the newest bundle, so a row'sbundle_shadoes not say what its golden was built from, and grouping by bundle gives one group. The test therefore has to be a marker on the golden's disk: any ofSANDBOX_ROOT_DISK_AGENTS(/opt/codex,/opt/pdev,media/claude-*,steps/install-agent-clis.version). It also catches a golden promoted later from a workspace that still descends from an old golden.Paths that make a golden (main, #9282, #9906):
release.tsinternal goldenSandbox.createfrom the imageenvironments replace,promoteSandboxToEnvironment)startEnvironmentBuild→rebuildEnvironment(manual,daily,repositories)kind = moves || sourceKind === "image" ? "fresh" : "incremental"; incremental ={ kind: "fork", golden: environment.sourceRef }createBox(build-steps.ts:97)build.kind === "incremental" && sourceKind === "fork"choiceWhere the fix belongs: #9282 (and #9906 on top of it), not this PR. Main has no build path that forks a golden. I did not edit those branches. The change they need:
rebuildEnvironment(feat(environments): build cloud environments from [install] (SUPER-3017) #9282) andcreateBox(feat(environments): run environment builds on Vercel Workflow (SUPER-3017) #9906), right after an incremental build box is forked, runROOT_DISK_AGENTS_COMMAND(exported by this PR from@superset/shared/sandbox/contract).{ kind: "image", region }), and record the build askind: "fresh".[install]in.paradev/config.toml. A promote-only environment without[install]cannot be rebuilt fresh, so its golden keeps the copies until the owner saves a checkpoint from a workspace made from the image (decision 3).What one fresh rebuild costs. The nearest measurement is the release's internal golden, built fresh from the image on 8 vCPUs every night: provision, checkout, setup hook (
bun installof this monorepo), checks, strip, stop and snapshot. It took 122–145 s in the last four production releases, about 16–19 vCPU-minutes. A customer's build takes that box time plus their own[install]time, which I have no data for (environment_buildsonly exists in #9282). Each rebuild also writes one new full golden snapshot; #9282 keeps 5 goldens, so storage grows until older ones are pruned.Rate. All 116 at once would be about 928 vCPUs concurrently, plus 116 clones and dependency installs at once, against GitHub and package registries. Proposal: let #9282's daily job do it, with at most 10 fresh builds running at once and at most 40 started per hour, environments with the most live workspaces first (iad1 and sfo1 hold 558 of the 820). All 116 then finish in about 3 hours of one daily run. If daily runs should stay light, cap fresh rebuilds at 25 a day, which takes 5 days.
Check that a new golden has none.
release.tsgolden checks:no agent CLI on the golden's root disk.probeBox(release probe andreal-sandbox):no agent CLI on the box's root disk, so none in its golden.image: no agent CLI on the root disknow uses the same list, including the marker.Evidence
real-sandboxon359163d2(run 44409700108): every check passes exceptdesktop stream answers RFB through host-service, which fails on main too. Passes include claude, codex and pdev running from the agents drive,the bundle installs no agent CLI on the root disk, andwake: agents linked from the remounted drive. real-sandbox boots the registry'slatestimage, which a main release built with agents on its disk; that image's leftovers (/opt/codex,/opt/pdev, theinstall-agent-clismarker) are logged asinfo. The strictno agent CLI on the box's root diskcheck runs only when the box's image was built from this checkout (a release without--skip-image).bun run image --local): 4.89 GB → 4.10 GB. The bundle layer went from 1.52 GB to 721.7 MB. Re-measured on this head: 4.10 GB.ced887cc:Boot twicepass (2m47s, with the real checkout), plusLint,Test,Typecheck,SherifandValepass.bun testonced887cc(one test per behaviour):agentToolsMounts > retries a drive lookup that failed with 429 or 5xxagentToolsMounts > a drive that stays unavailable, is missing or a writer holds fails like any box that will not start; region, bundle and drive go only to Sentry(asserts the message is the existing generic one and matches none of /drive|agent|sfo1/)classifyDispatchError > a box whose agents drive cannot be mounted is a cloud box that is not readywakeSandbox > mounts the drive of the box's own region, not its environment'swakeSandbox > a drive that cannot be mounted fails the wake before the box resumeswakeSandbox > a mounts update is retried on 5xx, and one that keeps failing fails the wakeprovisionSandbox > a create that failed with 5xx is retried with the drive mountedprovisionSandbox > a drive that cannot be mounted fails the create, and no box is madepromoteSandboxToEnvironment > snapshots a running source through its session and leaves its restart to the caller(pass) covers it, since promote never boots a box and the restart is the wake tested above. My own promote test is removed.session:naming,health.check … sandbox mode) fail here on main's code, and I did not investigate them.src/boot-twice.ts, run locally on this head:every agent on a login shell's PATH runs from the drivedrive: listed agents linkeddrive: boot ignores a leftover disk copydrive: an agent the drive lacks is missing alone, with no fallback to a disk copyno manifest: boot refused before host-service starts, so the open failsimage: no agent CLI on the root disk(now including theinstall-agent-clismarker)Boot twicejob is the real check.src/runner-check.ts: all expectations met.bun run lint:fixclean; typecheck 42/42 (turbo run typecheck --concurrency=2, afterbun installfor a dependency main added; the default concurrency was killed for memory here);bun run check:i18npasses, and this PR no longer changes any catalog (theagentsUnavailablemessage and its translations are gone).bun run release,real-sandbox,agent-tools(no credentials).Risks and rollback
agent-tools drive not mountedSentry issue.sandbox-agent-tools-driveat 100% until this has soaked, so a rollback of only the API to a pre-merge build mounts the drive for everyone through the flag.production-release: environments move to a bundle withdestrows andinstall-agent-clis.sync-assetsfetches claude, codex and pdev back onto the disk. A box from an old golden whose marker still matches already has the copies, untouched by this PR.Needs before merge
release --productionruns have published 19 regions since; each release fails if a needed region lacks claude, codex or pdev.sandbox-agent-tools-driveat 100% and soaked. This PR deletes the flag.ced887cc.real-sandboxdispatch on this head.update({ mounts }). Whichever lands second rebases; not done here.Decisions for the product owner
[install]cannot be rebuilt fresh. Should we (a) leave their goldens holding about 0.8 GB of inert agents (the image difference this PR measured) until the owner saves a checkpoint from an image-made workspace, (b) tell those owners, or (c) block their saves until they add[install]? Recommend (a) with (b): it costs storage, not correctness.(Decided by Satya: agents come only from the drive; an open without the drive fails; goldens must not hold agents.)