Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions docs/cloud-sandbox-mismatches.md
Original file line number Diff line number Diff line change
Expand Up @@ -945,16 +945,23 @@ recorded against the API and a runtime failure against the sandbox, and that id
is the only thing that joins the two halves of one broken workspace.

**A drive mount lasts one session and a fork does not inherit it — open.**
For organizations with the `sandbox-agent-tools-drive` PostHog flag, the agent
CLIs come from the bundle's `agent-tools-<region>-<bundle sha 12>` drive,
mounted read-only at `/opt/agents`. A machine someone owns keeps a mount; a
sandbox gets one only from `Sandbox.create`, `Sandbox.fork` or
`sandbox.update({ mounts })` before the session starts, so `provisionSandbox`
and `wakeSandbox` both pass it. A drive lives in one region and mounts only in
that region. A session without the mount leaves links in `/usr/local/bin`
that point nowhere; `superset-boot` re-runs `install-agent-clis` to put the
image's agents back. OpenCode and Pi exist only on the drive, so for them it
deletes the link instead.
The agent CLIs come only from the bundle's `agent-tools-<region>-<bundle sha 12>`
drive, mounted read-only at `/opt/agents`; a new box's own disk has none, and
an older golden's disk copies are left alone and never used. A machine someone
owns keeps a mount; a sandbox gets one only from `Sandbox.create`,
`Sandbox.fork` or `sandbox.update({ mounts })` before the session starts, so
`provisionSandbox` and `wakeSandbox` pass it, and a promote's restart is a
wake. A drive lives in one region and mounts only there, so it is named by
the box's own region, and the release writes one in every region. A drive
that cannot be mounted after a few tries on 429 and 5xx fails the create or
the wake the way any box that will not start does ("The cloud workspace could
not start"; a wake answers "Cloud workspace is still starting"): a customer
never sees the drive. The drive, region and bundle go only to Sentry and the
server log. A promote keeps its golden. A boot that finds no
manifest at `/opt/agents` is refused before host-service starts. **Open:** a
running session whose host-service died is booted again without a remount,
since a mount cannot be added to a live session; if that session has no
drive, its boots are refused until the box is stopped and opened again.

**An agent on the drive cannot update itself.** On a machine someone owns,
OpenCode and Pi update in place, beside their binary. On a box they run from
Expand Down
8 changes: 4 additions & 4 deletions packages/sandbox/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ checklist is `plans/20260913-sandbox-implementation-checklist.md`.
A step lists the archives it unpacks under `discards` in `steps.json`: the
runner deletes them once the step succeeds, keeps the `.hash` sidecar so
`sync-assets` does not fetch them again, and fetches them again only when
the step has to re-run. A file the box runs in place (`claude`, `superset`)
the step has to re-run. A file the box runs in place (`superset`)
is never discarded.
- **An agent only the agents drive carries** (`opencode`, `pi`) is a row
with no `dest`. `assets.tsv` leaves it out, so neither the image nor a
- **An agent CLI** (`claude`, `codex`, `pdev`, `opencode`, `pi`) is a row
with no `dest`: only the agents drive carries it. `assets.tsv` leaves it out, so neither the image nor a
boot fetches it; `bun run build --publish` still mirrors it to the bucket,
and `bun run agent-tools` fetches it from there by sha. Its pin goes into
the bundle's `drive-assets.tsv`, which no box reads, so a bump alone still
Expand Down Expand Up @@ -55,7 +55,7 @@ The rule of thumb: if you can answer "where is it on the box", put it there.
| `bun run image` | Builds the image: OS, apt lists, vendor tools, the `ubuntu` user, then installs the bundle at its sha and runs every pass once, so a fresh box finds every hash sidecar and step marker matching. A push logs Docker into the registry itself with a credential minted from `VERCEL_SANDBOX_TOKEN`, so there is no login to renew. `--local` builds `superset-sandbox:local` for the boot-twice test; `--dry` prints the Dockerfile. |
| boot (`superset-boot`) | On the box, as root, run by the control plane on every create and wake, after it has written `sandbox.conf`, with the host secret in its env: clears `/run/superset`, installs the pinned bundle if the hash differs, runs `apply-rootfs` and the steps marked `beforeHost` in `steps.json` (each a hash compare), starts host-service as `ubuntu` with the secret from the boot env, writes `resources.env`, starts `superset-memguard` (ranks every process for the OOM killer and reaps the largest before the kernel has to). When `sandbox.conf` has `SUPERSET_SANDBOX_BOX_LOGS=1`, it also starts `superset-logship`, which sends the boot, memguard, desktop and host-service logs to Sentry Logs from a byte offset saved on disk. Then, detached: `sync-assets` and the other steps, then the desktop, with the checkout beside them. `checkout.ready` waits for both, and the stamp `steps.end` says the steps are done. Once host-service answers, the environment has been pushed and the checkout is ready, the runner asks host-service to run the repo's `start` hook with that environment. |
| `bun run release` | Rewrites the host-service and CLI assets from this checkout and the claude, codex, Chrome, opencode and pi rows to their latest upstream release, publishes the bundle, pushes the image under a candidate tag `release-<bundle sha>` (unless `--skip-image`), builds the internal golden from it with the environment's `setup` hook, probes a fork of it (`src/environments/probe.ts`) including a stop and wake, and only after the probe passes moves `latest` to the candidate and writes the environment rows. Every image environment resolves `superset-hostsvc` to `latest`, so a failed release changes no box. |
| `bun run agent-tools --bundle <sha>` | Writes that bundle's agent CLIs to `agent-tools-<region>-<sha 12>`, a drive boxes mount read-only at `/opt/agents` when their organization has the `sandbox-agent-tools-drive` flag. A drive is written once and never again; it starts as a fork of the region's newest earlier drive, so only changed agents are fetched, and an agent that fails its download or `--version` check keeps the earlier drive's version or is left out. The drive carries claude, codex and pdev (also in the image) and opencode and pi (only on the drive). `bun run release` does this for every region, six at a time, before it records the bundle. It fails when a region with an environment has no drive, or a drive without claude, codex or pdev. Afterwards a `--production` release deletes the drives that neither an environment row (archived ones too) nor one of the 3 newest released bundles names; one Vercel project holds every database's drives, so no other release deletes any. |
| `bun run agent-tools --bundle <sha>` | Writes that bundle's agent CLIs to `agent-tools-<region>-<sha 12>`, a drive every box mounts read-only at `/opt/agents`; a box has no other agent CLIs, so a create or wake whose drive cannot be mounted fails, and a boot without the drive's manifest is refused. A drive is written once and never again; it starts as a fork of the region's newest earlier drive, so only changed agents are fetched, and an agent that fails its download or `--version` check keeps the earlier drive's version or is left out. The drive carries claude, codex, pdev, opencode and pi. `bun run release` does this for every region, six at a time, before it records the bundle. It fails when a region with an environment has no drive, or a drive without claude, codex or pdev. Afterwards a `--production` release deletes the drives that neither an environment row (archived ones too) nor one of the 3 newest released bundles names; one Vercel project holds every database's drives, so no other release deletes any. |
| `bun run src/real-sandbox.ts` | The Sandbox workflow dispatched with `real-sandbox`: one real sandbox from the registry image on this checkout's bundle and its agents drive in the default region, the same probe, a stop and a wake. |
| Sandbox workflow, `production-release` | `bun run release --production` from `main` on a CI runner: the image, the internal golden and production's environment rows. Runs on a merge that touches the bundle and every night; a failed merge or nightly run opens an issue, or comments on the open one. |

Expand Down
3 changes: 0 additions & 3 deletions packages/sandbox/bundle/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
"claude": {
"sha256": "1e08503dbdf3c2cb0d706d32f3408277388d1c76ef108673e8fe42c1b322925b",
"suffix": "",
"dest": "/usr/local/share/superset/media/claude-2.1.280",
"mode": "0755",
"source": "https://storage.googleapis.com/claude-code-dist-86c565f3-f756-42ad-8dfa-d59b1c096819/claude-code-releases/2.1.280/linux-x64/claude",
"version": "2.1.280"
Expand All @@ -25,7 +24,6 @@
"codex": {
"sha256": "a676f5722aae0d86cbe3764332f08eab1fdd89dc0a300b3ad4ffbf48611fa3e3",
"suffix": ".tar.gz",
"dest": "/usr/local/share/superset/media/codex-0.162.1.tar.gz",
"mode": "0644",
"source": "https://github.com/openai/codex/releases/download/rust-v0.162.1/codex-package-x86_64-unknown-linux-musl.tar.gz",
"version": "0.162.1"
Expand Down Expand Up @@ -176,7 +174,6 @@
"pdev": {
"sha256": "5a51852a4de0a40e20c0137b873f83e1b26255b4c5d01b6b0a2f608e152d8535",
"suffix": ".tar.gz",
"dest": "/usr/local/share/superset/media/pdev-0.2.5.tar.gz",
"mode": "0644",
"source": "https://github.com/ParadevAI/pdev/releases/download/v0.2.5/pdev-0.2.5-linux-x64.tar.gz",
"version": "0.2.5"
Expand Down
5 changes: 2 additions & 3 deletions packages/sandbox/bundle/rootfs/etc/profile.d/superset.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ export DISPLAY="${DISPLAY:-${SUPERSET_DISPLAY:-:1}}"
# The box's memory ceiling and CPUs, written by superset-boot from the cgroup.
[ -r "${SUPERSET_RUN_DIR:-/run/superset}/resources.env" ] && . "${SUPERSET_RUN_DIR:-/run/superset}/resources.env"
case ":$PATH:" in *":/usr/local/go/bin:"*) ;; *) export PATH="$PATH:/usr/local/go/bin" ;; esac
# The agent CLIs are pinned by the bundle (install-agent-clis) or run from the
# read-only agents drive; a self-update would drift from the pin and only
# prints a warning every session.
# The agent CLIs run from the read-only agents drive, pinned by the bundle; a
# self-update would drift from the pin and only prints a warning every session.
export DISABLE_AUTOUPDATER=1 OPENCODE_DISABLE_AUTOUPDATE=1 PI_SKIP_VERSION_CHECK=1
26 changes: 11 additions & 15 deletions packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
# install the pinned bundle only when its hash differs from current
# apply-rootfs, and the steps each a hash compare per file and step;
# marked beforeHost the only steps host-service waits for
# link the drive's agents only when the control plane mounted one
# link the drive's agents refused without the drive's manifest
# start host-service as ubuntu first; everything else is a side effect
# then, detached: sync-assets and the other steps, then the desktop;
# the checkout beside them; checkout.ready and the start hook after both
Expand Down Expand Up @@ -118,34 +118,32 @@ if [ -z "${SUPERSET_BOOT_REEXEC:-}" ] && [ "$(self_hash)" != "$BOOT_HASH_BEFORE"
fi
stamp "steps.host.start"; "$BUNDLE/setup" run-steps before-host >> "$SUPERSET_BOOT_LOG" 2>&1; stamp "steps.host.end"

# --- the agent CLIs from the region's agent-tools drive, when the control
# plane mounted one. Without it the image's agents stay on PATH. Mounts are
# per session, so a link into a mount this session lacks points nowhere.
# --- the agent CLIs, only from the region's agent-tools drive; the box has no
# other copy, so a boot without the drive's manifest is refused. Each listed
# agent's link is replaced, so an older bundle's link on PATH cannot win. Mounts
# are per session, so a link into a mount this session lacks points nowhere.
link_agents() {
local dir="${SUPERSET_AGENTS_DIR:-/opt/agents}" linked=0 name sha bin target
local manifest="${dir}/manifest.json"
if find /usr/local/bin -maxdepth 1 -lname "${dir}/*" ! -exec test -e {} \; -print -quit | grep -q .; then
SUPERSET_BUNDLE_DIR="$BUNDLE" bash "$BUNDLE/steps/install-agent-clis.sh" >> "$SUPERSET_BOOT_LOG" 2>&1 ||
stamp "agents.restore.failed"
# What is still dangling has no image copy to restore: opencode, pi.
find /usr/local/bin -maxdepth 1 -lname "${dir}/*" ! -exec test -e {} \; -delete
fi
find /usr/local/bin -maxdepth 1 -lname "${dir}/*" ! -exec test -e {} \; -delete
if ! jq -e 'type == "array"' "$manifest" > /dev/null 2>&1; then
stamp "agents.unavailable no manifest at ${dir}"
return 0
refuse "no agent CLIs at ${dir}"
fi
while IFS=$'\t' read -r name sha bin; do
if ! [[ "$name" =~ ^[a-z0-9-]+$ && "$sha" =~ ^[0-9a-f]{64}$ && "$bin" =~ ^[A-Za-z0-9._/-]+$ && "$bin" != *..* ]]; then
stamp "agents.skipped invalid entry ${name:0:32}"; continue
fi
target="${dir}/${name}/${sha}/${bin}"
[ -x "$target" ] || { stamp "agents.missing ${name} ${sha:0:12}"; continue; }
if [ ! -x "$target" ]; then
rm -f "/usr/local/bin/${name}"
stamp "agents.missing ${name} ${sha:0:12}"; continue
fi
ln -sfn "$target" "/usr/local/bin/${name}.tmp" && mv -T "/usr/local/bin/${name}.tmp" "/usr/local/bin/${name}" &&
linked=$((linked + 1))
done < <(jq -r '.[] | [.name, .sha, .bin] | map(tostring) | @tsv' "$manifest")
stamp "agents.linked ${linked} from ${dir}"
}
drive_links() { find /usr/local/bin -maxdepth 1 -lname "${SUPERSET_AGENTS_DIR:-/opt/agents}/*" -printf '%f %l\n' | sort; }
link_agents

# --- host-service, as the sandbox user, first. A runtime that cannot start
Expand Down Expand Up @@ -342,10 +340,8 @@ checkout() {
checkout_pid=$!
exec 8> /run/superset-steps.lock
flock 8
links_before="$(drive_links)"
stamp "assets.start"; "$BUNDLE/setup" sync-assets >> "$SUPERSET_BOOT_LOG" 2>&1 || stamp "assets.failed continuing"; stamp "assets.end"
stamp "steps.start"; "$BUNDLE/setup" run-steps after-host >> "$SUPERSET_BOOT_LOG" 2>&1; stamp "steps.end"
[ "$(drive_links)" = "$links_before" ] || link_agents
exec 8>&-
as_user env "DISPLAY=${SUPERSET_DISPLAY}" "HOME=${SUPERSET_USER_HOME}" "USER=${SUPERSET_USER}" \
"SUPERSET_SANDBOX_WORKSPACE_ID=${SUPERSET_SANDBOX_WORKSPACE_ID}" \
Expand Down
7 changes: 1 addition & 6 deletions packages/sandbox/bundle/steps.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,5 @@
"discards": ["host-service"],
"beforeHost": true
},
{ "name": "install-cli", "assets": ["cli"] },
{
"name": "install-agent-clis",
"assets": ["claude", "codex", "pdev"],
"discards": ["codex", "pdev"]
}
{ "name": "install-cli", "assets": ["cli"] }
]
57 changes: 0 additions & 57 deletions packages/sandbox/bundle/steps/install-agent-clis.sh

This file was deleted.

7 changes: 4 additions & 3 deletions packages/sandbox/src/agent-tools.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
AGENT_TOOLS,
type AgentToolsManifest,
agentToolsDrivesToPrune,
agentToolsGaps,
Expand Down Expand Up @@ -55,7 +56,7 @@ describe("agent-tools manifest", () => {
expect(parseAgentToolsManifest(JSON.stringify(manifest))).toEqual(manifest);
});

test("the committed bundle has a row for every agent, and opencode and pi only on the drive", () => {
test("the committed bundle has a row for every agent, each only on the drive", () => {
const assets = loadAssets(
join(import.meta.dir, "..", "bundle", "assets.json"),
);
Expand All @@ -66,9 +67,9 @@ describe("agent-tools manifest", () => {
"opencode",
"pi",
]);
for (const name of ["opencode", "pi"]) {
for (const { name } of AGENT_TOOLS) {
expect(assets[name]?.dest).toBeUndefined();
expect(assets[name]?.source).toStartWith("https://github.com/");
expect(assets[name]?.source).toStartWith("https://");
}
});

Expand Down
7 changes: 3 additions & 4 deletions packages/sandbox/src/agent-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,7 @@
*
* A new drive starts as a fork of the region's newest earlier one, so the
* builder fetches only the agents whose sha changed. Each agent fails alone:
* it keeps the earlier drive's version, or is left out and boxes use the
* image's copy.
* it keeps the earlier drive's version, or is left out and boxes have none.
*
* bun run agent-tools --bundle <sha> [--region sfo1 ...]
*
Expand Down Expand Up @@ -411,7 +410,7 @@ export function describeAgentToolsPublish(
return results.flatMap((entry) => {
if ("error" in entry) {
return [
`::warning::agent-tools: ${entry.region} not published; its boxes keep the image's agents: ${String(entry.error).slice(0, 300)}`,
`::warning::agent-tools: ${entry.region} not published; its boxes have no agents: ${String(entry.error).slice(0, 300)}`,
];
}
const { drive, outcome, forkedFrom, agents } = entry.result;
Expand All @@ -427,7 +426,7 @@ export function describeAgentToolsPublish(
]
: agent.outcome === "dropped"
? [
`::warning::agent-tools: ${drive}: ${agent.name} left out; its boxes keep the image's ${agent.name}, if the image has one`,
`::warning::agent-tools: ${drive}: ${agent.name} left out; its boxes have no ${agent.name}`,
]
: [],
),
Expand Down
3 changes: 0 additions & 3 deletions packages/sandbox/src/assets/produce.ts
Original file line number Diff line number Diff line change
Expand Up @@ -466,7 +466,6 @@ async function claude(version?: string): Promise<void> {
claude: {
sha256: hash,
suffix: "",
dest: `${SANDBOX_PATHS.media}/claude-${pinned}`,
mode: "0755",
source,
version: pinned,
Expand Down Expand Up @@ -500,7 +499,6 @@ async function codex(version?: string): Promise<void> {
codex: {
sha256: hash,
suffix: ".tar.gz",
dest: `${SANDBOX_PATHS.media}/codex-${pinned}.tar.gz`,
mode: "0644",
source: asset.browser_download_url,
version: pinned,
Expand Down Expand Up @@ -534,7 +532,6 @@ async function pdev(version?: string): Promise<void> {
pdev: {
sha256: hash,
suffix: ".tar.gz",
dest: `${SANDBOX_PATHS.media}/pdev-${pinned}.tar.gz`,
mode: "0644",
source: asset.browser_download_url,
version: pinned,
Expand Down
Loading
Loading