Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/sandbox/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ The rule of thumb: if you can answer "where is it on the box", put it there.
| `bun run image` | Builds the image: OS, apt lists, vendor tools, the `ubuntu` user, then installs the bundle at its sha and runs every pass once, so a fresh box finds every hash sidecar and step marker matching. A push logs Docker into the registry itself with a credential minted from `VERCEL_SANDBOX_TOKEN`, so there is no login to renew. `--local` builds `superset-sandbox:local` for the boot-twice test; `--dry` prints the Dockerfile. |
| boot (`superset-boot`) | On the box, as root, run by the control plane on every create and wake, after it has written `sandbox.conf`, with the host secret in its env: clears `/run/superset`, installs the pinned bundle if the hash differs, runs `apply-rootfs` and the steps marked `beforeHost` in `steps.json` (each a hash compare), starts host-service as `ubuntu` with the secret from the boot env, writes `resources.env`, starts `superset-memguard` (ranks every process for the OOM killer and reaps the largest before the kernel has to). Then, detached: `sync-assets` and the other steps, then the desktop, with the checkout beside them. `checkout.ready` waits for both, and the stamp `steps.end` says the steps are done. Once host-service answers, the environment has been pushed and the checkout is ready, the runner asks host-service to run the repo's `start` hook with that environment. |
| `bun run release` | Rewrites the host-service and CLI assets from this checkout and the claude, codex, Chrome, opencode and pi rows to their latest upstream release, publishes the bundle, pushes the image under a candidate tag `release-<bundle sha>` (unless `--skip-image`), builds the internal golden from it with the environment's `setup` hook, probes a fork of it (`src/environments/probe.ts`) including a stop and wake, and only after the probe passes moves `latest` to the candidate and writes the environment rows. Every image environment resolves `superset-hostsvc` to `latest`, so a failed release changes no box. |
| `bun run agent-tools --bundle <sha>` | Writes that bundle's agent CLIs to `agent-tools-<region>-<sha 12>`, a drive boxes mount read-only at `/opt/agents` when their organization has the `sandbox-agent-tools-drive` flag. A drive is written once and never again; it starts as a fork of the region's newest earlier drive, so only changed agents are fetched, and an agent that fails its download or `--version` check keeps the earlier drive's version or is left out. The drive carries claude, codex and pdev (also in the image) and opencode and pi (only on the drive). `bun run release` does this for every region with an environment, six at a time, before it records the bundle, and afterwards deletes the drives no environment names. |
| `bun run src/real-sandbox.ts` | The Sandbox workflow dispatched with `real-sandbox`: one real sandbox from the registry image on this checkout's bundle, the same probe, a stop and a wake. |
| `bun run agent-tools --bundle <sha>` | Writes that bundle's agent CLIs to `agent-tools-<region>-<sha 12>`, a drive boxes mount read-only at `/opt/agents` when their organization has the `sandbox-agent-tools-drive` flag. A drive is written once and never again; it starts as a fork of the region's newest earlier drive, so only changed agents are fetched, and an agent that fails its download or `--version` check keeps the earlier drive's version or is left out. The drive carries claude, codex and pdev (also in the image) and opencode and pi (only on the drive). `bun run release` does this for every region, six at a time, before it records the bundle. It fails when a region with an environment has no drive, or a drive without claude, codex or pdev. Afterwards a `--production` release deletes the drives that neither an environment row (archived ones too) nor one of the 3 newest released bundles names; one Vercel project holds every database's drives, so no other release deletes any. |
| `bun run src/real-sandbox.ts` | The Sandbox workflow dispatched with `real-sandbox`: one real sandbox from the registry image on this checkout's bundle and its agents drive in the default region, the same probe, a stop and a wake. |
| Sandbox workflow, `production-release` | `bun run release --production` from `main` on a CI runner: the image, the internal golden and production's environment rows. Runs on a merge that touches the bundle and every night; a failed merge or nightly run opens an issue, or comments on the open one. |

An asset URL is content-addressed, so the CDN caches it for hours — including a 404. An image
Expand Down
70 changes: 68 additions & 2 deletions packages/sandbox/src/agent-tools.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { join } from "node:path";
import {
type AgentToolsManifest,
agentToolsDrivesToPrune,
agentToolsGaps,
agentToolsManifest,
type DriveHost,
describeAgentToolsPublish,
Expand All @@ -19,6 +20,8 @@ import {
pruneAgentToolsDrives,
publishAgentToolsDrive,
publishAgentToolsDrives,
type RegionPublish,
referencedAgentToolsDrives,
renderBuilderScript,
} from "./agent-tools";
import { type Assets, loadAssets, sha256 } from "./manifest";
Expand Down Expand Up @@ -284,12 +287,20 @@ describe("publishAgentToolsDrive", () => {
expect(agentLine(scripts[drive], "chrome")).toEndWith(
"'--headless' '--version'",
);
const lines = describeAgentToolsPublish([{ region: "sfo1", result }]);
expect(result.available.map(({ name }) => name)).toEqual([
"claude",
"codex",
"pdev",
]);
const lines = describeAgentToolsPublish([{ region: "sfo1", result }], {
...agentAssets,
pdev: { ...(agentAssets.pdev as Assets[string]), version: "0.2.6" },
});
expect(lines[0]).toBe(
`agent-tools: ${drive} published: claude written, codex reused, pdev kept-previous, chrome dropped`,
);
expect(lines.slice(1)).toEqual([
expect.stringMatching(/^::warning::.*pdev kept 999999999999/),
`::warning::agent-tools: sfo1: pdev 0.2.6 failed; ${drive} keeps 999999999999 from the earlier drive`,
expect.stringMatching(/^::warning::.*chrome left out/),
]);
});
Expand Down Expand Up @@ -341,6 +352,61 @@ test("builder results ignore lines that are not results", () => {
).toEqual([{ name: "claude", outcome: "reused", sha: sha("a") }]);
});

describe("agentToolsGaps", () => {
const published = (
region: "sfo1" | "iad1" | "fra1",
names: string[],
): RegionPublish => ({
region,
result: {
drive: `agent-tools-${region}-0123456789ab`,
outcome: "exists",
agents: [],
available: names.map((name) => ({ name, sha: sha("a") })),
},
});

test("names each needed region whose drive is missing or lacks a required agent", () => {
const results: RegionPublish[] = [
published("sfo1", ["claude", "codex", "pdev", "pi"]),
published("iad1", ["claude", "pi"]),
{ region: "fra1", error: new Error("builder exited 1") },
];
expect(agentToolsGaps(results, new Set(["sfo1", "iad1", "fra1"]))).toEqual([
"iad1: agent-tools-iad1-0123456789ab has no codex, pdev",
"fra1: no drive",
]);
});

test("ignores regions no box needs, and an optional agent left out", () => {
const results: RegionPublish[] = [
published("sfo1", ["claude", "codex", "pdev"]),
{ region: "fra1", error: new Error("builder exited 1") },
];
expect(agentToolsGaps(results, new Set(["sfo1"]))).toEqual([]);
});
});

test("referenced drives: every row's bundle in its region, and the recent bundles in every region", () => {
const older = "1".repeat(64);
expect(
[
...referencedAgentToolsDrives({
rows: [
{ region: "iad1", bundleSha: older },
{ region: "fra1", bundleSha: null },
],
recentBundles: [BUNDLE],
regions: ["sfo1", "iad1"],
}),
].sort(),
).toEqual([
"agent-tools-iad1-0123456789ab",
"agent-tools-iad1-111111111111",
"agent-tools-sfo1-0123456789ab",
]);
});

describe("pruning", () => {
test("deletes only agent-tools drives no environment references", () => {
expect(
Expand Down
57 changes: 54 additions & 3 deletions packages/sandbox/src/agent-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,9 @@ const CHECK_TIMEOUT_SECONDS = 30;

export const AGENT_TOOLS_MANIFEST = "manifest.json";

/** Agents a box used to carry on its own disk; a drive without one of them must not be released. */
export const REQUIRED_AGENT_TOOLS = ["claude", "codex", "pdev"] as const;

const agentToolSchema = z.object({
name: z.string().regex(/^[a-z0-9-]+$/),
sha: z.string().regex(SHA256),
Expand Down Expand Up @@ -270,6 +273,8 @@ export interface AgentToolsPublish {
outcome: "exists" | "published";
forkedFrom?: string;
agents: AgentResult[];
/** What the drive's manifest names once published. */
available: { name: string; sha: string }[];
}

/** The region's newest other agent-tools drive that no sandbox holds read-write. */
Expand Down Expand Up @@ -316,7 +321,13 @@ export async function publishAgentToolsDrive(args: {
const found = parseAgentToolsManifest(
await args.host.readManifest(drive, args.region),
);
if (found.length > 0) return { drive, outcome: "exists", agents: [] };
if (found.length > 0)
return {
drive,
outcome: "exists",
agents: [],
available: found.map(({ name, sha }) => ({ name, sha })),
};
await args.host.remove(drive);
}
let forkedFrom: string | undefined;
Expand Down Expand Up @@ -348,11 +359,13 @@ export async function publishAgentToolsDrive(args: {
tools: args.tools,
}),
);
const agents = parseBuilderResults(stdout);
return {
drive,
outcome: "published",
...(forkedFrom ? { forkedFrom } : {}),
agents: parseBuilderResults(stdout),
agents,
available: agents.flatMap(({ name, sha }) => (sha ? [{ name, sha }] : [])),
};
}

Expand Down Expand Up @@ -393,6 +406,7 @@ export async function publishAgentToolsDrives(
/** One line per region and per agent that did not get the bundle's version; warnings start `::warning::`. */
export function describeAgentToolsPublish(
results: readonly RegionPublish[],
assets: Assets = {},
): string[] {
return results.flatMap((entry) => {
if ("error" in entry) {
Expand All @@ -409,7 +423,7 @@ export function describeAgentToolsPublish(
...agents.flatMap((agent) =>
agent.outcome === "kept-previous"
? [
`::warning::agent-tools: ${drive}: ${agent.name} kept ${agent.sha?.slice(0, 12)} from the earlier drive; the bundle's version failed`,
`::warning::agent-tools: ${entry.region}: ${agent.name} ${assets[agent.name]?.version ?? "of this bundle"} failed; ${drive} keeps ${agent.sha?.slice(0, 12)} from ${forkedFrom ?? "the earlier drive"}`,
]
: agent.outcome === "dropped"
? [
Expand All @@ -421,6 +435,43 @@ export function describeAgentToolsPublish(
});
}

/** One line per region a box needs whose drive is missing or lacks a required agent; empty means release. */
export function agentToolsGaps(
results: readonly RegionPublish[],
needed: ReadonlySet<string>,
required: readonly string[] = REQUIRED_AGENT_TOOLS,
): string[] {
return results.flatMap((entry) => {
if (!needed.has(entry.region)) return [];
if ("error" in entry) return [`${entry.region}: no drive`];
const names = new Set(entry.result.available.map(({ name }) => name));
const missing = required.filter((name) => !names.has(name));
return missing.length
? [`${entry.region}: ${entry.result.drive} has no ${missing.join(", ")}`]
: [];
});
}

/**
* Every drive a box may still mount: each environment row's bundle in its
* region, and the newest released bundles in every region, so a rollback and
* an environment created mid-release still find theirs.
*/
export function referencedAgentToolsDrives(args: {
rows: readonly { region: string; bundleSha: string | null }[];
recentBundles: readonly string[];
regions: readonly string[];
}): Set<string> {
return new Set([
...args.rows.flatMap((row) =>
row.bundleSha ? [sandboxAgentToolsDrive(row.region, row.bundleSha)] : [],
),
...args.recentBundles.flatMap((sha) =>
args.regions.map((region) => sandboxAgentToolsDrive(region, sha)),
),
]);
}

/** Agent-tools drives no environment row's bundle and region name any more. */
export function agentToolsDrivesToPrune(
drives: readonly string[],
Expand Down
12 changes: 12 additions & 0 deletions packages/sandbox/src/environments/probe.ts
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,14 @@ export async function probeBox(args: ProbeArgs): Promise<number> {
);
for (const preset of HOST_AGENT_PRESETS) {
if (!isCloudAgentId(preset.presetId)) continue;
const resolved = (
await run(`readlink -f "$(command -v ${preset.command})"`)
).trim();
check(
`${preset.presetId} runs from the agents drive`,
resolved.startsWith(`${SANDBOX_PATHS.agents}/`),
resolved,
);
const out = await run(
`cd /tmp && timeout 15 ${[preset.command, ...preset.args].join(" ")} </dev/null >/tmp/preset.out 2>&1; echo "exit=$?"; head -5 /tmp/preset.out`,
);
Expand Down Expand Up @@ -244,6 +252,10 @@ export async function checkWakeLog(args: {
check("wake: every step skipped", !/step \S+ running/.test(last));
check("wake: checkout kept", /checkout\.skipped/.test(last));
check("wake: host-service ready", /host\.ready/.test(last));
check(
"wake: agents linked from the remounted drive",
/agents\.linked [1-9]/.test(last) && !/agents\.unavailable/.test(last),
);
return failed;
}

Expand Down
Loading
Loading