Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/cloud-sandbox-mismatches.md
Original file line number Diff line number Diff line change
Expand Up @@ -903,6 +903,26 @@ and provider. Keep the workspace id on both sides: a provisioning failure is
recorded against the API and a runtime failure against the sandbox, and that id
is the only thing that joins the two halves of one broken workspace.

**A drive mount lasts one session and a fork does not inherit it — open.**
For organizations with the `sandbox-agent-tools-drive` PostHog flag, the agent
CLIs come from the bundle's `agent-tools-<region>-<bundle sha 12>` drive,
mounted read-only at `/opt/agents`. A machine someone owns keeps a mount; a
sandbox gets one only from `Sandbox.create`, `Sandbox.fork` or
`sandbox.update({ mounts })` before the session starts, so `provisionSandbox`
and `wakeSandbox` both pass it. A drive lives in one region and mounts only in
that region. A session without the mount leaves links in `/usr/local/bin`
that point nowhere; `superset-boot` re-runs `install-agent-clis` to put the
image's agents back.

**A read-only snapshot mount taken while a writer holds the drive is corrupt.**
Measured 2026-11-17: a box that mounted the snapshot while the builder still
had the drive read-write got `ro,norecovery`, `ls` answered "Bad message" and
codex segfaulted. On a disk someone owns, a second reader of a live
filesystem is just slow. Here a drive is never written after it is
published: one drive per bundle, written by a builder that is stopped before
any environment row names the bundle, and the control plane mounts only a
drive that exists and is detached.

## Running this repo's own dev stack inside a sandbox

Reproducing an app bug end to end from a cloud workspace means bringing up
Expand Down
1 change: 1 addition & 0 deletions packages/sandbox/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ The rule of thumb: if you can answer "where is it on the box", put it there.
| `bun run image` | Builds the image: OS, apt lists, vendor tools, the `ubuntu` user, then installs the bundle at its sha and runs every pass once, so a fresh box finds every hash sidecar and step marker matching. A push logs Docker into the registry itself with a credential minted from `VERCEL_SANDBOX_TOKEN`, so there is no login to renew. `--local` builds `superset-sandbox:local` for the boot-twice test; `--dry` prints the Dockerfile. |
| boot (`superset-boot`) | On the box, as root, run by the control plane on every create and wake, after it has written `sandbox.conf`, with the host secret in its env: clears `/run/superset`, installs the pinned bundle if the hash differs, runs the three passes (each a hash compare), starts host-service as `ubuntu` with the secret from the boot env, writes `resources.env`, starts `superset-memguard` (ranks every process for the OOM killer and reaps the largest before the kernel has to), then the desktop and the checkout. Once host-service answers, the environment has been pushed and the checkout is in, the runner asks host-service to run the repo's `start` hook with that environment. |
| `bun run release` | Rewrites the host-service and CLI assets from this checkout and the claude, codex and Chrome rows to their latest upstream release, publishes the bundle, pushes the image under a candidate tag `release-<bundle sha>` (unless `--skip-image`), builds the internal golden from it with the environment's `setup` hook, probes a fork of it (`src/environments/probe.ts`) including a stop and wake, and only after the probe passes moves `latest` to the candidate and writes the environment rows. Every image environment resolves `superset-hostsvc` to `latest`, so a failed release changes no box. |
| `bun run agent-tools --bundle <sha>` | Writes that bundle's agent CLIs to `agent-tools-<region>-<sha 12>`, a drive boxes mount read-only at `/opt/agents` when their organization has the `sandbox-agent-tools-drive` flag. A drive is written once and never again. `bun run release` does this for every region with an environment before it records the bundle, and afterwards deletes the drives no environment names. |
| `bun run src/real-sandbox.ts` | The Sandbox workflow dispatched with `real-sandbox`: one real sandbox from the registry image on this checkout's bundle, the same probe, a stop and a wake. |
| Sandbox workflow, `production-release` | `bun run release --production` from `main` on a CI runner: the image, the internal golden and production's environment rows. Runs on a merge that touches the bundle and every night; a failed merge or nightly run opens an issue, or comments on the open one. |

Expand Down
28 changes: 28 additions & 0 deletions packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
# install the pinned bundle only when its hash differs from current
# apply-rootfs, sync-assets, each a hash compare per file, asset, step
# run-steps
# link the drive's agents only when the control plane mounted one
# start host-service as ubuntu first; everything else is a side effect
# start the desktop and the checkout, detached
#
Expand Down Expand Up @@ -117,6 +118,33 @@ fi
stamp "assets.start"; "$BUNDLE/setup" sync-assets >> "$SUPERSET_BOOT_LOG" 2>&1 || stamp "assets.failed continuing"; stamp "assets.end"
stamp "steps.start"; "$BUNDLE/setup" run-steps >> "$SUPERSET_BOOT_LOG" 2>&1; stamp "steps.end"

# --- the agent CLIs from the region's agent-tools drive, when the control
# plane mounted one. Without it the image's agents stay on PATH. Mounts are
# per session, so a link into a mount this session lacks points nowhere.
link_agents() {
local dir="${SUPERSET_AGENTS_DIR:-/opt/agents}" linked=0 name sha bin target
local manifest="${dir}/manifest.json"
if find /usr/local/bin -maxdepth 1 -lname "${dir}/*" ! -exec test -e {} \; -print | grep -q .; then
SUPERSET_BUNDLE_DIR="$BUNDLE" bash "$BUNDLE/steps/install-agent-clis.sh" >> "$SUPERSET_BOOT_LOG" 2>&1 ||
stamp "agents.restore.failed"
fi
if ! jq -e 'type == "array"' "$manifest" > /dev/null 2>&1; then
stamp "agents.unavailable no manifest at ${dir}"
return 0
fi
while IFS=$'\t' read -r name sha bin; do
if ! [[ "$name" =~ ^[a-z0-9-]+$ && "$sha" =~ ^[0-9a-f]{64}$ && "$bin" =~ ^[A-Za-z0-9._/-]+$ && "$bin" != *..* ]]; then
stamp "agents.skipped invalid entry ${name:0:32}"; continue
fi
target="${dir}/${name}/${sha}/${bin}"
[ -x "$target" ] || { stamp "agents.missing ${name} ${sha:0:12}"; continue; }
ln -sfn "$target" "/usr/local/bin/${name}.tmp" && mv -T "/usr/local/bin/${name}.tmp" "/usr/local/bin/${name}" &&
linked=$((linked + 1))
done < <(jq -r '.[] | [.name, .sha, .bin] | map(tostring) | @tsv' "$manifest")
stamp "agents.linked ${linked} from ${dir}"
}
link_agents

# --- host-service, as the sandbox user, first. A runtime that cannot start
# is logged, not fatal: the desktop and the checkout still come up so the
# box can be looked at, and the control plane marks the workspace failed.
Expand Down
1 change: 1 addition & 0 deletions packages/sandbox/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"pin": "bun run src/pin.ts",
"assets": "bun run src/assets/produce.ts",
"release": "bun run src/environments/release.ts",
"agent-tools": "bun run src/agent-tools.ts",
"record-bundle": "bun run src/environments/record-bundle.ts",
"test": "bun test",
"typecheck": "tsc --noEmit --emitDeclarationOnly false"
Expand Down
275 changes: 275 additions & 0 deletions packages/sandbox/src/agent-tools.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,275 @@
import { describe, expect, test } from "bun:test";
import {
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
type AgentToolsManifest,
agentToolsDrivesToPrune,
agentToolsManifest,
type DriveHost,
parseAgentToolsManifest,
pruneAgentToolsDrives,
publishAgentToolsDrive,
renderBuilderScript,
} from "./agent-tools";
import { type Assets, sha256 } from "./manifest";

const sha = (c: string) => c.repeat(64);
const BUNDLE = "0123456789abcdef".repeat(4);

function row(sha256: string, suffix: string): Assets[string] {
return { sha256, suffix, dest: "/usr/local/share/x", mode: "0644" };
}

const agentAssets: Assets = {
claude: row(sha("a"), ""),
codex: row(sha("b"), ".tar.gz"),
pdev: row(sha("c"), ".tar.gz"),
chrome: row(sha("d"), ".deb"),
};

describe("agent-tools manifest", () => {
test("lists every agent at its asset sha, bin relative to its dir", () => {
const manifest = agentToolsManifest(agentAssets);
expect(manifest).toEqual([
{ name: "claude", sha: sha("a"), bin: "claude" },
{ name: "codex", sha: sha("b"), bin: "bin/codex" },
{ name: "pdev", sha: sha("c"), bin: "pdev/pdev" },
]);
expect(parseAgentToolsManifest(JSON.stringify(manifest))).toEqual(manifest);
});

test("a missing or bad manifest reads as empty", () => {
expect(parseAgentToolsManifest(null)).toEqual([]);
expect(parseAgentToolsManifest("{not json")).toEqual([]);
expect(
parseAgentToolsManifest(
JSON.stringify([{ name: "claude", sha: sha("a"), bin: "../x" }]),
),
).toEqual([]);
});
});

function fakeHost(
drives: Record<string, { manifest?: string; attachedTo?: string }>,
) {
const calls: string[] = [];
const host: DriveHost = {
find: async (name) => {
const drive = drives[name];
return drive ? { attachedTo: drive.attachedTo } : null;
},
list: async (prefix) =>
Object.keys(drives).filter((name) => name.startsWith(prefix)),
create: async (name) => {
calls.push(`create ${name}`);
drives[name] = {};
},
remove: async (name) => {
calls.push(`remove ${name}`);
if (drives[name]?.attachedTo) throw new Error("attached");
delete drives[name];
},
readManifest: async (name) => drives[name]?.manifest ?? null,
write: async (name) => {
calls.push(`write ${name}`);
const drive = drives[name];
if (drive)
drive.manifest = JSON.stringify(agentToolsManifest(agentAssets));
},
};
return { host, calls, drives };
}

describe("publishAgentToolsDrive", () => {
const drive = "agent-tools-sfo1-0123456789ab";

test("creates and writes the bundle's drive in the region", async () => {
const { host, calls } = fakeHost({});
expect(
await publishAgentToolsDrive({
host,
region: "sfo1",
bundleSha: BUNDLE,
assets: agentAssets,
}),
).toEqual({ drive, outcome: "published" });
expect(calls).toEqual([`create ${drive}`, `write ${drive}`]);
});

test("leaves a drive that has a manifest alone", async () => {
const { host, calls } = fakeHost({
[drive]: { manifest: JSON.stringify(agentToolsManifest(agentAssets)) },
});
expect(
(
await publishAgentToolsDrive({
host,
region: "sfo1",
bundleSha: BUNDLE,
assets: agentAssets,
})
).outcome,
).toBe("exists");
expect(calls).toEqual([]);
});

test("recreates a drive an earlier run left without a manifest", async () => {
const { host, calls } = fakeHost({ [drive]: {} });
await publishAgentToolsDrive({
host,
region: "sfo1",
bundleSha: BUNDLE,
assets: agentAssets,
});
expect(calls).toEqual([
`remove ${drive}`,
`create ${drive}`,
`write ${drive}`,
]);
});

test("refuses a drive a writer still holds", async () => {
const { host, calls } = fakeHost({ [drive]: { attachedTo: "builder" } });
await expect(
publishAgentToolsDrive({
host,
region: "sfo1",
bundleSha: BUNDLE,
assets: agentAssets,
}),
).rejects.toThrow(/attached to builder/);
expect(calls).toEqual([]);
});
});

describe("pruning", () => {
test("deletes only agent-tools drives no environment references", () => {
expect(
agentToolsDrivesToPrune(
[
"agent-tools-sfo1-0123456789ab",
"agent-tools-sfo1-aaaaaaaaaaaa",
"agent-tools-iad1-0123456789ab",
"agent-tools-sfo1",
"agent-tools-builder-sfo1-xyz",
],
new Set(["agent-tools-sfo1-0123456789ab"]),
),
).toEqual([
"agent-tools-sfo1-aaaaaaaaaaaa",
"agent-tools-iad1-0123456789ab",
]);
});

test("keeps a drive a sandbox holds for the next release", async () => {
const { host } = fakeHost({
"agent-tools-sfo1-aaaaaaaaaaaa": {},
"agent-tools-sfo1-bbbbbbbbbbbb": { attachedTo: "box" },
});
expect(
await pruneAgentToolsDrives({ host, referenced: new Set() }),
).toEqual({
deleted: ["agent-tools-sfo1-aaaaaaaaaaaa"],
kept: ["agent-tools-sfo1-bbbbbbbbbbbb"],
});
});
});

async function runScript(script: string) {
const proc = Bun.spawn(["bash", "-c", script], {
stdout: "pipe",
stderr: "pipe",
});
const stderr = await new Response(proc.stderr).text();
return { exitCode: await proc.exited, stderr };
}

describe.skipIf(process.platform !== "linux")("builder script", () => {
test("writes verified agents, then the manifest last", async () => {
const work = mkdtempSync(join(tmpdir(), "agent-tools-"));
const src = join(work, "src");
mkdirSync(join(src, "bin"), { recursive: true });
writeFileSync(join(src, "bin", "codex"), "#!/bin/sh\necho codex\n", {
mode: 0o755,
});
Bun.spawnSync(["tar", "-czf", join(work, "codex.tar.gz"), "-C", src, "."]);
const claude = new TextEncoder().encode("#!/bin/sh\necho claude\n");
const codex = readFileSync(join(work, "codex.tar.gz"));
const objects = new Map<string, Uint8Array>([
[sha256(claude), claude],
[`${sha256(codex)}.tar.gz`, codex],
]);
const server = Bun.serve({
port: 0,
fetch: (request) => {
const body = objects.get(new URL(request.url).pathname.slice(1));
return body ? new Response(body) : new Response("", { status: 404 });
},
});
const root = join(work, "drive");
mkdirSync(root);
const manifest: AgentToolsManifest = [
{ name: "claude", sha: sha256(claude), bin: "claude" },
{ name: "codex", sha: sha256(codex), bin: "bin/codex" },
];
try {
const run = await runScript(
renderBuilderScript({
root,
baseUrl: `http://127.0.0.1:${server.port}`,
assets: {
claude: row(sha256(claude), ""),
codex: row(sha256(codex), ".tar.gz"),
},
manifest,
}),
);
expect(run.exitCode, run.stderr).toBe(0);
expect(
parseAgentToolsManifest(
readFileSync(join(root, "manifest.json"), "utf8"),
),
).toEqual(manifest);
expect(
Bun.spawnSync([join(root, "codex", sha256(codex), "bin", "codex")])
.stdout.toString()
.trim(),
).toBe("codex");
expect(existsSync(join(root, "claude", sha256(claude), "claude"))).toBe(
true,
);
expect(existsSync(join(root, ".manifest.json.partial"))).toBe(false);
} finally {
server.stop(true);
}
});

test("a sha mismatch fails before the manifest is written", async () => {
const root = mkdtempSync(join(tmpdir(), "agent-tools-"));
const server = Bun.serve({
port: 0,
fetch: () => new Response("not the agent"),
});
try {
const run = await runScript(
renderBuilderScript({
root,
baseUrl: `http://127.0.0.1:${server.port}`,
assets: { claude: row(sha("a"), "") },
manifest: [{ name: "claude", sha: sha("a"), bin: "claude" }],
}),
);
expect(run.exitCode).not.toBe(0);
expect(existsSync(join(root, "manifest.json"))).toBe(false);
} finally {
server.stop(true);
}
});
});
Loading
Loading