Repository navigation
feat(environments): run environment builds on Vercel Workflow (SUPER-3017) - #9906
Draft
saddlepaddle wants to merge 4 commits into
Draft
saddlepaddle wants to merge 4 commits into
saddlepaddle wants to merge 4 commits into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
saddlepaddle
had a problem deploying
to
preview
November 17, 2026 21:53 — with
GitHub Actions
Failure
saddlepaddle
had a problem deploying
to
preview
November 17, 2026 21:53 — with
GitHub Actions
Failure
Contributor
🚀 Preview Deployment🔗 Preview Links
Preview updates automatically with new commits |
saddlepaddle
had a problem deploying
to
preview
November 17, 2026 21:58 — with
GitHub Actions
Failure
saddlepaddle
force-pushed
the
superset/super-3017-environment-builds-all
branch
from
November 18, 2026 18:49
d5171cb to
352fd4a
Compare
…3017) A build was one QStash job with a 285 s budget, so a slow [install] failed with "ran out of time". It is now a durable workflow run: the box does the long work, steps only start a detached command or check it, and the workflow sleeps 5 s between checks. One product limit for the whole build (BUILD_LIMIT_MS), enforced by the workflow and by `timeout` on the box. - environmentBuildWorkflow + steps in packages/trpc reuse #9282's prepareBuild, createBuildBox, succeedBuild, failBuild, pruneGoldens. - Steps are safe to retry: the box is found by name, a command starts once per key (atomic mkdir on the box), and a swap that already committed is not done again. - Secrets never enter the run's event log; each step derives them. - The running command's last 40 lines go to environment_builds.log (migration 0200, a nullable column on the table #9282 adds in 0199). - A build row marked failed stops its run at the next step. - STALE_BUILD_MS moves past the new limit. - ENVIRONMENT_BUILD_FAKE_SANDBOX=1 runs builds on an in-memory box in dev and test; ignored when NODE_ENV is production. - The QStash route /api/environments/build is gone; start() replaces it.
Satya's call: 30 minutes was too long; the old ceiling was about 5. STALE_BUILD_MS stays the limit plus 10 minutes (now 20), and the customer message reads from the constant.
saddlepaddle
force-pushed
the
superset/env-builds-on-vercel-workflow-b817c77c
branch
from
November 18, 2026 18:52
1a8643f to
088222d
Compare
2 of 6 tasks
…from the image (SUPER-3017) createBox runs the root-disk agent check on a forked box. When it prints anything, the fork is deleted and the build claims a fresh slot: at most 10 fresh builds at once and 40 started per hour, under an advisory lock. A daily build waits up to an hour for a slot; past the caps, or for any other trigger, the build forks as before and the next run checks again. A claimed build is recorded as kind=fresh. The daily run starts the busiest environments first (most cloud workspaces in the last week). The check command is inlined until #9911 exports ROOT_DISK_AGENTS_COMMAND.
A build that switches to fresh after its fork had agent CLIs creates its box as envb-<id>-fresh, so a delete of the fork that Vercel has not finished cannot collide with it. Every step reads the name from the build's kind, so a retry still finds the box by name. Cleanup and stale expiry delete both names.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #9282 (base
superset/super-3017-environment-builds-all). Draft; do not merge.What changed
A build was one QStash-delivered function (
maxDuration 300,JOB_MS285 s). Now it is a Vercel Workflow run.environmentBuildWorkflow("use workflow") and 11"use step"functions inpackages/trpc/src/lib/sandbox/environment-build/build-workflow.tsandbuild-steps.ts. The step bodies are feat(environments): build cloud environments from [install] (SUPER-3017) #9282's own functions:prepareBuild,createBuildBox,checkoutScript,succeedBuild/activateGolden,failBuild,pruneGoldens.startCommandstarts the checkout or[install]detached on the box and returns.readCommandis a short step that reads the command's exit file and the last 40 lines of its output, and saves those lines to the build row. The workflow sleeps 5 s between reads. Stop and snapshot work the same way:stopBox, then a loop ofsnapshotTakenand sleep.vercel.ts: the waitingrunBuildCommandandsnapshotBuildBoxare replaced bystartBuildCommand,readBuildCommand,buildBoxSnapshotId,stopBuildBoxandbuildBoxSnapshotTaken. The box-side scripts are insandbox/build-command.ts.BUILD_LIMIT_MS= 10 minutes (Satya's call; the old ceiling was about 5), inlimits.ts. Customer message: "The build ran for more than 10 minutes and was stopped". The workflow enforces it while it waits, andtimeouton the box enforces it on the command.STALE_BUILD_MSgoes from 10 min to limit + 10 min (20 min). Without that change, a long build would be marked stale and its box deleted while it was still running.startEnvironmentBuildcallsstart(environmentBuildWorkflow, [buildId], { attributes: { buildId, environmentId } })in place ofpublishCloudWorkspaceJob.apps/api/src/app/api/environments/build/route.ts(the QStash target) is deleted. feat(environments): build cloud environments from [install] (SUPER-3017) #9282 is unmerged, so no QStash deliveries to it are in flight.apps/api/next.config.ts:withSentryConfig(withWorkflow(config), …).turbo.jsonc: build outputs includesrc/app/.well-known/workflow/**. Deps:workflow@5.5.0inapps/apiandpackages/trpc. 5.6.x is newer than the repo's 3-dayminimum-release-age. This adds about 730 lines tobun.lock, becauseworkflowdepends on every framework adapter.environment_builds.log text(nullable), migration0200_environment_build_log.sql, generated bydrizzle-kit generateafter feat(environments): build cloud environments from [install] (SUPER-3017) #9282's0199_environment_builds. Migration checklist: the table is new in feat(environments): build cloud environments from [install] (SUPER-3017) #9282 and not in production (0 rows). Adding a nullable column with no default is a metadata-only change. This PR changes no tRPC procedure, sologis not read anywhere yet.ENVIRONMENT_BUILD_FAKE_SANDBOX=1(trpcenv.ts, optional, empty in.env.local.example) makes the steps usefake-build-box.ts, an in-memory box. It is ignored whenNODE_ENV=production. It is not set in any deployed environment, so no GitHub secret was added.Workflow, as pseudocode
Retry safety:
createBoxis found by name on retry (feat(environments): build cloud environments from [install] (SUPER-3017) #9282'sgetSandboxcheck).mkdiris atomic on the box). Its exit code is written by rename.activateBuildreturnstruewith no second swap when the build is alreadysucceededwith this golden. Its guard is the golden read when the build started, as in feat(environments): build cloud environments from [install] (SUPER-3017) #9282, not the one read at activation.failBuildis guarded onstatus = 'running'.Expected failures are
FatalErrors (BuildFailure extends FatalError), so they are not retried. Anything else is retried by the runtime, 3 retries by default; after that comes the one cleanup step.Secrets never enter the run's event log. Step inputs and outputs are only build ids, the golden ref, a snapshot id and command tails. Each step that needs the network policy or the environment's variables derives them again with
prepareBuild.Drives (out of scope):
prepareBuildDriveis the placeholder step for laying out the drive before the clone. The detach wait belongs in thesnapshotTakenloop.activateBuild→activateGoldenmust move the snapshot ref and the golden drive name in one update.Cancel/supersede: #9282 has neither. With this PR, setting a running build's row to failed stops its run at the next
readCommandorstopBox. That step throwsFatalError("The build is no longer running"), and the cleanup step deletes the box. This was verified live (below). A future cancel button can write the row, or callgetRun(runId).cancel()after finding the run by itsbuildIdattribute.The five answers
Sources: the docs bundled in the
workflowpackage (node_modules/workflow/docs, the same text as useworkflow.dev, which now redirects to workflow-sdk.dev), https://vercel.com/docs/workflows, /pricing, /concepts, /retention, and https://vercel.com/docs/functions/limitations. Plus what I ran.1. Limits
/.well-known/workflow/v1/flow. There is no separate step route any more. The builder writessrc/app/.well-known/workflow/v1/config.jsonwith"maxDuration": "max"; I saw it in my localnext build. On Pro or Enterprise with Fluid compute, "max" is 800 s (1800 s "extended maximum", beta).maxDurationexports. Our steps take about 0.1–2 s each (measured below).sleep: "No limit". Each sleep hop is ≤ 23 h on Vercel Queues.2. Availability and pricing (Pro/Enterprise)
experimental_hook,experimental_retention,experimentalRoutePrefix.VERCEL_DEPLOYMENT_ID, and without it "every run fails before its first step" (local world,EROFS). I did not check this setting; see "not run".3. Fit with apps/api
next.config.tscomposes aswithSentryConfig(withWorkflow(config), …). It runs undernext dev(Turbopack) andnext buildwith the Lingui SWC plugin and React compiler on.packages/trpc. Discovery follows imports from thestart()call. The dev compile reported98 steps, 1 workflow, andmanifest.jsonlistspackages/trpc/src/lib/sandbox/environment-build/build-workflow.ts#environmentBuildWorkflowand all 11 steps. NotranspilePackagesoroutputFileTracingRootwas needed.@vercel/sandbox's own"use step"methods (Command#wait,FileSystem#…). The SDK ships them; they are registered but unused.bun testruns the workflow as plain functions, since the directives do nothing without the compiler. The repo'snextruns under Node as before.ƒ /.well-known/workflow/v1/flowandƒ /.well-known/workflow/v1/webhook/[token], generated intoapps/api/src/app/.well-known/workflow/v1/(self-gitignored).config.jsonholds the queue trigger{"type":"queue/v2beta","topic":"__wkf_workflow_*"}.ƒ /api/environments/buildis gone.proxy.ts's matcher does not touch/.well-known/workflow, so no change is needed there.withWorkflowadded/.swctoapps/api/.gitignore.vercel buildin GitHub Actions and thenvercel deploy --prebuilt. The world is chosen at runtime: the compiled bundle hasWORKFLOW_TARGET_WORLD || (VERCEL_DEPLOYMENT_ID ? "vercel" : "local"), so a CI build without a deployment id should not bake in the local world. Only a preview deployment can confirm this.4. Starting, observing, deployments
start(environmentBuildWorkflow, [buildId], { attributes })returns{ runId: "wrun_…" }as soon as the run is queued. It replaces the QStash publish.queueDailyBuilds. That function callsstartEnvironmentBuild, which callsstart()for each environment; 1,000 runs/s is far above our count.attributes: { buildId, environmentId }are plaintext and searchable. The attributes came back on the local run:{'buildId': 'aaf83f51-…', 'environmentId': '0000f00d-…'}. No run-id column was added.npx workflow inspect runs --backend vercel; locally,npx workflow weborinspect.deploymentId: "latest"opts in to the newest code.5. Local dev and tests
next dev,withWorkflowselects the local world: SQLite in.next/workflow-data, with queue messages delivered back to the dev server over HTTP.PORT, and our root.envsetsPORT=3007(next.configloads.envwithoverride: true). To run apps/api on another port, setWORKFLOW_LOCAL_BASE_URL=http://localhost:<port>.@workflow/vitest, withtestTimeout: 60_000). That fits neither bun nor the repo's no-waiting rule. Sobuild-workflow.test.tscalls the workflow as a plain function:workflow.sleepto advance a fake clock (Date.nowis spied).@vercel/sandbox'sSandbox.get/fork/create, so feat(environments): build cloud environments from [install] (SUPER-3017) #9282's realcreateBuildBoxruns, and stubs the other box calls.FatalError, up to 4 attempts.Product implications (what a customer sees)
[install]that took more than 180 s used to fail; now it succeeds.running/succeeded/failed), triggers, kinds and Builds table columns. The tRPC output is unchanged.environment_builds.log) now updates every ~5 s while a build runs. No UI or tRPC reads it yet; that is a follow-up, kept out of the migration PR on purpose. On failure,errorstill has the reason plus the last 40 lines.Run for real, locally
next dev, local world,ENVIRONMENT_BUILD_FAKE_SANDBOX=1.br-flat-river-af4vg7vl. It is a child of production, not production. I applied 0186 and 0187 there by hand, then dropped them afterwards.ParadevAI/acme-demo, the public fixture branchsuper-3017-install-fixture, which has an[install]). These rows were deleted afterwards.startEnvironmentBuild({ trigger: "manual" }).Trace of a successful run, from the
next devlog. EachPOST /flow 200is one invocation that ended at asleepand was later resumed by a queue message:npx workflow inspect runs:wrun_01M7NQGQ959VEGRS8KW7VBV8W4 environmentBuildWorkflow C(ompleted). That is 75 events: 20 steps created/started/completed, 6wait_created/wait_completedpairs, and the run's own 3. Event tail:The build row afterwards:
status succeeded,commit_sha fff…f(from the fake checkout),golden_ref envb-aaf83f51…,log= the last[install]lines, 35.7 s from start to finish. The environment row moved tosource_kind fork,source_ref envb-aaf83f51….Second run, a mid-build cancel: while
[install]was running, I set the row tofailed/ "Cancelled by test". The log shows[workflow-sdk] Encountered FatalError while executing step, bubbling up to parent workflow — FatalError: The build is no longer running, then[fake-build-box] delete envb-cd134288….inspect steps -dshows the cleanup step got["cd134288-…", "The build is no longer running"]. So a step'sFatalErrormessage reaches the workflow'scatchintact on the real runtime, not only in unit tests.Checks
Ran, all passing:
bun run lint:fix(no fixes left)bun run typecheck(42/42 tasks)bun testinpackages/trpc(869 pass),apps/api(240 pass),packages/db(23 pass)New tests (bun, no real waiting; the workflow file runs in 0.5 s):
[install](past the old 285 s, inside the 10-minute limit) succeeds[install]exits 2 → failed with its tail, no stop, box deletedfork,read×2 andactivate, then retried → one box, one swapbuild-command.test.tsruns the box-side script in real bash: a second start doesn't run it again, and the exit code and tail are read backI checked that the double-swap test fails when the idempotent branch in
activateBuildis removed.Local
next buildof apps/api: compiled successfully, and the route list includes the two workflow functions. The command then exited 1 with Next's ownSecurityFatalError("Your version of Next.js is affected by a known security vulnerability"). That is a gate onnext@16.4.1itself and has nothing to do with this change. Note: apps/api is on 16.4.1, not 16.4.5.Not run, or not run against real services:
runCommand, the box-sidetimeout/mkdirscript on a real box,stop()and snapshot timing were not exercised against Vercel. They were exercised in bash locally and with the fake box..superset/config.toml. Because of the synthetic installation row,installationTokenFormay have made one GitHub App token request for an installation that doesn't exist; it fell back to no token, as public repositories do.check:i18nandcheck:pluginswere not run: no Lingui strings or plugins changed.Preview deployment
Workflow has no supported option to make the generated
maxDurationa number:@workflow/nexthard-codes'max', up to the latest 5.1.2. The lowest Vercel CLI that accepts it is 50.31.0. Its@vercel/build-utils13.8.0 adds|| maxDuration === "max", while 50.29.0 / 13.7.0 rejects it, and its changelog reads "SupportmaxDuration: 'max'" (vercel/vercel#15217).mainnow has that bump for all workflows (#9980), so this PR no longer changes CI.Results on head
088222dc(rebased onto #9282352fd4a9, which is onmain9f04fe88):0199_environment_builds, then0200_environment_build_logWorkflow routes in the API preview (checked on the previous head, with CLI 50.31.0):
ƒ /.well-known/workflow/v1/flowandƒ /.well-known/workflow/v1/webhook/[token].x-matched-path: /.well-known/workflow/v1/webhook/[token], and its handler answers 404 for an unknown token.x-matched-path: /404. That would fit a queue-triggered consumer, but it is unconfirmed without Vercel team access.Goldens with agent CLIs on the root disk
createBoxruns the root-disk agent check on a forked box. If the check prints anything, the build deletes the fork and starts from the image, withkind=fresh. There is no new column: the next run checks again, so this fixes itself over time.claimFreshRebuild).ROOT_DISK_AGENTS_COMMANDis inlined invercel.tswith a TODO. When feat(sandbox): agent CLIs only from the agents drive (PR B, draft) #9911 merges, swap it for the import from@superset/shared/sandbox/contract.envb-<id>-fresh, so a delete of the fork that Vercel has not finished can't collide with it. Each step reads the name from the build'skind, so a retry still finds the box. Cleanup and stale expiry delete both names.Needs before merge
ROOT_DISK_AGENTS_COMMANDfor the export from feat(sandbox): agent CLIs only from the agents drive (PR B, draft) #9911 once feat(sandbox): agent CLIs only from the agents drive (PR B, draft) #9911 merges.mainin ci: bump the Vercel CLI to 50.31.0 #9980.main, its migration regenerated as0199_environment_builds; this PR's as0200_environment_build_log.br-hidden-hat-afj4fcvw, copied from production on 2026-11-02) fails on main's0195:unsafe use of new value "archived" of enum type cloud_workspace_status. CI replays 0187–0199 in one transaction, and 0195 adds an enum value and then uses it. It needs a reset from production or a fresh branch. I did not touch it.