Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c2f0335
feat(sandbox): add distinct Modal VM backend identity
ColeMurray Sep 22, 2026
1588b5d
Merge remote-tracking branch 'origin/main' into modal-vm-backend-iden…
ColeMurray Sep 22, 2026
6f1f90d
chore(modal): tighten backend typing and configuration guidance
ColeMurray Sep 22, 2026
db10b2a
style(terraform): trim backend contract test whitespace
ColeMurray Sep 22, 2026
c24c920
fix: close Modal VM lifecycle review gaps
ColeMurray Sep 22, 2026
587dc76
chore: merge main into Modal VM backend branch
ColeMurray Sep 23, 2026
86d77ab
fix: commit Modal VM captures before source retirement
ColeMurray Sep 23, 2026
f827db3
style: format Modal VM capture test
ColeMurray Sep 23, 2026
a168b9b
Log Modal VM health and detect stalled heartbeats promptly
ColeMurray Sep 23, 2026
780e3af
Update lifecycle and bridge log assertions
ColeMurray Sep 23, 2026
8126a94
fix: block prompts during failed sandbox preservation
ColeMurray Sep 23, 2026
1b5a0a9
Merge remote-tracking branch 'origin/main' into port-modal-vm-followups
ColeMurray Sep 23, 2026
ed2a8e6
Add independent Modal VM heartbeat diagnostics
ColeMurray Sep 23, 2026
f781dcc
Capture VM reclaim counters in heartbeat diagnostics
ColeMurray Sep 23, 2026
3cc1578
Log top VM processes during heartbeat pressure
ColeMurray Sep 23, 2026
6312fa8
Cap Modal VM CPU and Vitest workers
ColeMurray Sep 23, 2026
d015988
refactor: remove Modal Dict capture recovery
ColeMurray Sep 23, 2026
926ccee
docs: keep Modal VM investigation drafts local
ColeMurray Sep 24, 2026
2097691
Remove Modal unbound image-build recovery
ColeMurray Sep 24, 2026
29e3b94
Clarify rejected startup allocation cleanup naming
ColeMurray Sep 24, 2026
e163e56
Remove unused legacy Docker response flag
ColeMurray Sep 24, 2026
26bc561
Merge origin/main into modal-vm-backend-identities
ColeMurray Sep 27, 2026
aa70051
fix: keep Modal VM sandboxes running and recover failed saves
ColeMurray Sep 27, 2026
eaddecf
Remove Modal VM heartbeat diagnostics
ColeMurray Sep 27, 2026
142f232
Drop the Vitest worker caps
ColeMurray Sep 27, 2026
d3a19bf
docs: describe when Modal VMs are saved
ColeMurray Sep 27, 2026
5e4d604
Merge origin/main into modal-vm-backend-identities
ColeMurray Sep 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ GITLAB_NAMESPACE=
# ---------------------------------------------------------------------------

SANDBOX_PROVIDER=modal
# Set modal-vm for deployment-wide Docker-capable VMs; see docs/MODAL_DOCKER.md.

# Modal. The API secret is shared with the Modal deployment for HMAC-signed
# endpoint calls; the workspace and environment build the endpoint URLs.
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/terraform.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,13 @@ jobs:
run: terraform test
working-directory: ${{ env.TF_WORKING_DIR }}

- name: Modal module contract tests
id: modal_test
run: |
terraform init -backend=false
terraform test
working-directory: terraform/modules/modal-app

- name: Post Validation Results
if: always() && github.event_name == 'pull_request'
uses: actions/github-script@v8
Expand All @@ -140,6 +147,7 @@ jobs:
| Init | ${{ steps.init.outcome == 'success' && '✅' || '❌' }} |
| Validate | ${{ steps.validate.outcome == 'success' && '✅' || '❌' }} |
| Tests | ${{ steps.test.outcome == 'success' && '✅' || '❌' }} |
| Modal module tests | ${{ steps.modal_test.outcome == 'success' && '✅' || '❌' }} |
${planNote}

*Pushed by: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;
Expand Down
5 changes: 3 additions & 2 deletions docs/GETTING_STARTED.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,8 +160,9 @@ Create an R2 API Token:

### Modal

> Only required when `sandbox_provider = "modal"` (the default, used by the core path). To use
> Daytona, Vercel Sandboxes, OpenComputer, or E2B instead, skip this section and follow
> Only required when `sandbox_provider = "modal"` (the default, used by the core path) or
> `"modal-vm"`. Select `modal-vm` for Docker-capable VMs; see [Modal VM setup](MODAL_DOCKER.md). To
> use Daytona, Vercel Sandboxes, OpenComputer, or E2B instead, skip this section and follow
> [Alternative Sandbox Providers](#alternative-sandbox-providers-optional).

1. Go to [Modal Settings](https://modal.com/settings)
Expand Down
76 changes: 76 additions & 0 deletions docs/MODAL_DOCKER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Modal VM backend

Open Inspect offers two Modal compute backends:

| `SANDBOX_PROVIDER` | Runtime | User Docker workloads |
| ------------------ | ----------------------- | --------------------- |
| `modal` (default) | Existing gVisor sandbox | Not enabled |
| `modal-vm` | Modal VM | Included |

Selection is deployment-wide. There is no Docker checkbox or repository/session override. Mixed use
within one deployment is a follow-up. Both backends share a Modal app, account, credentials,
transport, and implementation; they have separate prepared-image pools.

## Deployment

For Terraform, set `sandbox_provider = "modal-vm"`. The existing Modal module builds and verifies
the Docker image before deploying the worker. No extra provisioning/admission flag is required. For
a standalone control plane, set `SANDBOX_PROVIDER=modal-vm` and provision the data plane first:

```bash
cd packages/modal-infra
BUILD_MODAL_VM_IMAGE=true uv run python deploy.py --build-sandbox-image
BUILD_MODAL_VM_IMAGE=true uv run modal deploy deploy.py
```

These commands create billable resources. Use the intended Modal environment and credentials. Never
deploy `src/app.py` directly. VM launches reject old or incompatible API responses rather than
falling back to standard sandboxes. Existing standard clients can still use the standard endpoints.

## Runtime and resources

The harness, bridge, workspace, IDE, and desktop run directly on the VM host. Docker is for user
workloads such as PostgreSQL, Redis, and container builds. The runtime supervises the local daemon;
user environment variables cannot enable Docker or redirect its readiness probes.

Generic `cpuCores` and `memoryMib` size the **outer VM**, not individual containers. Missing/null
values select the offering defaults (currently 2 cores and 4096 MiB). Positive explicit settings
override them. VM image builds use their scope's configured resources. Standard build sizing is
unchanged. These values are product defaults, not claimed Modal minimums.

## Snapshots and recovery

VM session snapshots are **destructive**: quiesce Docker, capture the filesystem, then confirm VM
termination. Later work restores into a new VM. Standard Modal snapshots remain non-destructive. A
failed or ambiguous capture/retirement must not be reported as a successful checkpoint.

VMs therefore keep running between turns. They are saved and stopped on inactivity, lifetime expiry,
a lost heartbeat, a runtime failure or archive; a VM that stops heartbeating is captured without its
runtime. Cancelling a session stops its VM without saving. While a save has failed or its result is
unknown, new prompts are held and the session offers to retry the save, restore the last saved
state, or discard the VM and start fresh.

Filesystem capture is not process/RAM continuity or an application-consistent database backup.
Containers must use appropriate persistence and restart policies. Live Docker pause/resume is not
provided. Raw daemon logs are truncated after clean preparation before reusable image capture.

Retried VM launches adopt only an exactly owned allocation and recover its original interactive
credentials. A predecessor must be confirmed terminated before launching a replacement. Build
allocations have deterministic backend/build names, so a retried create adopts the allocation an
earlier lost response created. Returned build handles are persisted for cleanup before backend
validation; incompatible builds never start and cannot publish prepared images.

## Switching backends

Changing `SANDBOX_PROVIDER` is an operator cutover, not session migration. Existing sessions and
snapshots may become unusable. Drain/retire active allocations first when feasible, retain
credentials for pending cleanup, and rebuild images under the selected backend. Do not relabel old
artifacts. Rollback to `modal` does not transparently resume VM sessions or prove old VMs have
stopped.

PR #2007's earlier per-session Docker/variant design was not deployed. Its schema additions and
settings are not part of this implementation, so no variant migration is required.

Provider-backed canaries must validate Docker startup, build/restore, access after adoption,
snapshot/retirement, and cleanup in the target deployment before production rollout. Unit tests
alone do not prove those provider behaviors.
4 changes: 4 additions & 0 deletions packages/control-plane/src/image-builds/modal-adapter.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ function createPlan(): ImageBuildPlan {
}

describe("ModalImageBuildAdapter", () => {
it("does not advertise unbound source recovery", () => {
expect("recoverUnboundSource" in new ModalImageBuildAdapter(createProvider())).toBe(false);
});

it("delegates build startup to the Modal provider", async () => {
const provider = createProvider();
const adapter = new ModalImageBuildAdapter(provider);
Expand Down
1 change: 1 addition & 0 deletions packages/control-plane/src/image-builds/modal-adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export class ModalImageBuildAdapter implements ImageBuildAdapter {
await this.provider.triggerImageBuild({
scopeKind: plan.scope.kind,
scopeId: plan.scope.id,
resources: plan.resources,
buildId: plan.buildId,
repositories: plan.repositories,
cloneToken: plan.cloneAuth.type === "credential_helper" ? plan.cloneAuth.token : undefined,
Expand Down
23 changes: 5 additions & 18 deletions packages/control-plane/src/image-builds/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@ import {
formatRepositoryFullName,
parseRepositoryFullName,
} from "@open-inspect/shared/types/repositories";
import type {
ImageBuildScopeKind,
ImageBuildStatus,
import {
IMAGE_BUILD_PROVIDER_IDS,
type ImageBuildScopeKind,
type ImageBuildStatus,
} from "@open-inspect/shared/types/image-builds";
import type { HarnessId } from "@open-inspect/shared/harnesses";
import { z } from "zod";
Expand All @@ -24,21 +25,7 @@ import {
MIN_COMPATIBLE_RUNTIME_GENERATION,
} from "../sandbox/runtime-manifest";

/**
* Providers with image-build support: Modal images, Vercel snapshots,
* OpenComputer checkpoints, E2B snapshots, Daytona snapshots.
*
* Support is the provider's ability to build and boot an artifact. Whether a
* deployment may start new Daytona builds is a separate, operator-owned
* question — see `resolveImageBuildAdmission` in provider-policy.ts.
*/
export const IMAGE_BUILD_PROVIDER_IDS = [
"modal",
"vercel",
"opencomputer",
"e2b",
"daytona",
] as const;
export { IMAGE_BUILD_PROVIDER_IDS } from "@open-inspect/shared/types/image-builds";

export const imageBuildProviderSchema = z.enum(IMAGE_BUILD_PROVIDER_IDS);

Expand Down
1 change: 1 addition & 0 deletions packages/control-plane/src/image-builds/planner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ export class ImageBuildPlanner implements ImageBuildPlannerPort {
callbackUrl: params.callbackUrl,
failureCallbackUrl: params.failureCallbackUrl,
buildTimeoutMs: resolveBuildTimeoutSeconds(sandboxSettings) * MS_PER_SECOND,
resources: { cpuCores: sandboxSettings.cpuCores, memoryMib: sandboxSettings.memoryMib },
userEnvVars: userEnvVars
? prepareLegacyManagedProviderEnv({
exposedSecrets: userEnvVars,
Expand Down
3 changes: 2 additions & 1 deletion packages/control-plane/src/image-builds/provider-factory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ class EnvImageBuildAdapterFactory implements ImageBuildAdapterFactory {
create(provider: ImageBuildProvider, operation: "start" | "existing_session"): ImageBuildAdapter {
switch (provider) {
case "modal":
return new ModalImageBuildAdapter(createSandboxProviderFromEnv(this.env, "modal"));
case "modal-vm":
return new ModalImageBuildAdapter(createSandboxProviderFromEnv(this.env, provider));
case "vercel":
return new VercelImageBuildAdapter(createSandboxProviderFromEnv(this.env, "vercel"));
case "opencomputer":
Expand Down
2 changes: 2 additions & 0 deletions packages/control-plane/src/image-builds/types.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import type { RepositoryShaEntry } from "@open-inspect/shared/types/image-builds";
import type { SandboxSettings } from "@open-inspect/shared/types/integrations";
import type { CorrelationContext } from "../logger";
import type { ImageBuildProviderImageRef, ImageBuildScope } from "./model";

Expand Down Expand Up @@ -32,6 +33,7 @@ export type ImageBuildCloneAuth =
* Every supported provider uses the same create-bind-launch session contract.
*/
export interface ImageBuildPlan {
resources?: Pick<SandboxSettings, "cpuCores" | "memoryMib">;
buildId: string;
scope: ImageBuildScope;
repositories: ImageBuildRepository[];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ const registerBuildSpy = vi.spyOn(ImageBuildStore.prototype, "registerBuild");
const getActiveBuildSpy = vi.spyOn(ImageBuildStore.prototype, "getActiveBuild");
const hasReadyImageSpy = vi.spyOn(ImageBuildStore.prototype, "hasReadyImageForFingerprint");
const markBuildFailedSpy = vi.spyOn(ImageBuildStore.prototype, "markBuildFailed");
const markSourceCreateIntentSpy = vi.spyOn(ImageBuildStore.prototype, "markSourceCreateIntent");
const bindProviderSessionSpy = vi.spyOn(ImageBuildStore.prototype, "bindProviderSession");
const setImageBuildEnabledSpy = vi.spyOn(RepoMetadataStore.prototype, "setImageBuildEnabled");

Expand All @@ -220,6 +221,7 @@ beforeEach(() => {
markBuildFailedSpy.mockResolvedValue(true);
setImageBuildEnabledSpy.mockResolvedValue(undefined);
bindProviderSessionSpy.mockResolvedValue(true);
markSourceCreateIntentSpy.mockResolvedValue(true);
modalClient.createImageBuildSandbox.mockResolvedValue({
providerSessionId: "modal-session-1",
});
Expand Down
Loading
Loading