Skip to content

fix(sandbox): drop on-demand swap from memguard (draft: trade-off for review) - #9843

Closed
saddlepaddle wants to merge 1 commit into
mainfrom
superset/thin-no-swap-draft-978700a6
Closed

saddlepaddle wants to merge 1 commit into
mainfrom
superset/thin-no-swap-draft-978700a6

Conversation

@saddlepaddle

@saddlepaddle saddlepaddle commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Draft: a human decides whether this ships. It trades speed on undersized boxes for smaller snapshots.

What changed

  • superset-memguard: removed grow_swap / shrink_swap, the swap constants (SWAP_DIR, SWAP_STEP_MB, SWAP_IDLE_S), box_mb and the swap cap, and the SwapTotal / SwapFree reads.
  • The reap condition was MemAvailable < floor && SwapFree < SwapTotal/10 + 1. It is now MemAvailable < floor (8% of the box). The old swap check was there to wait for swap to fill. With no swap, MemAvailable alone is the right signal. Ranking, victim choice (500 → 0 → −300), the cooldown and the log lines did not change.
  • Kept: at startup memguard still runs swapoff on any /var/swap/superset-* an older bundle left and removes it. superset-boot still removes the old /swapfile.
  • boot.test.ts: the swap-cap test is replaced with two tests. Nothing in boot or memguard turns swap on, and the stale-file cleanup and the boot /swapfile removal are still there. memguard reaps on available memory alone.
  • docs/cloud-sandbox-considerations.md: the "Swap is on demand" entry is replaced by "No swap", which keeps the history and records the open telemetry question.

The trade

  • Before: an undersized box gets slow but finishes the job, and its snapshots grow by up to 16 GB, permanently. A snapshot is a full block copy of the disk with no discard, so swap written once stays in every later snapshot of the workspace. A box measured today held 8 GB of it, and fleet snapshots of 14-27 GB exist.
  • After: on an undersized box, a dev server or build is killed earlier, with a reap pid=… rss=… available=… floor=… line in memguard.log, and the app's pressure banner shows it. The remedy is a larger machine size.
  • Measured while writing this PR, on this workspace (8 GB box, old bundle): a full bun run typecheck of the monorepo made memguard grow swap 0 → 12 GB (its cap) in about 60 s. free showed 4.3 GB of swap in use afterwards, so at least that much is now written to this workspace's disk and every later snapshot will copy it. Under this change, the same typecheck on the same box would probably have had its largest tsc reaped. Earlier measurements (2026-10-22, 2/4 box): the apps/desktop typecheck needed about 8 GB and finished in 86 s only because of swap. Expect it to be reaped on a 2/4 box now.
  • Expected saving: up to the swap cap (6-16 GB) per snapshot for any workspace that ever came under pressure. Workspaces that never grew swap save nothing. This change does not shrink snapshots that already hold swap: removing the file frees the space for reuse but does not zero it.

Alternatives not taken

  • zram: the sandbox kernel has no zram module.
  • Swap on a Vercel Drive, once workspaces have one: drives are not part of the snapshot, so this would keep swap without the snapshot cost. It is not available yet. This is the likely long-term answer.
  • Zero-filling swap at stop: stop gives SIGTERM and at least 10 s. Zeroing up to 16 GB does not reliably fit, and it needs a lot of new logic in a fail-open script.

What would settle it

How many boxes ever grow swap, and how often swap is what saved a job. memguard.log is box-local today, so we do not know. Shipping the swap.grow count (or the peak swap use) through host-service's existing pressure report (runtime/sandbox-box-pressure) to telemetry for a week, before or instead of merging, would answer it.

Checks

Ran:

  • bun run lint:fix: passed (it reformatted the new test).
  • bun run typecheck (whole monorepo, 42 tasks): passed.
  • bun test in packages/sandbox: 25 pass, 0 fail.
  • bash -n superset-memguard: passed.
  • bun run --cwd packages/sandbox image --local (Docker started on this box): built superset-sandbox:local.
  • src/runner-check.ts: all expectations met.
  • src/boot-twice.ts: memguard starts (memguard.start, memguard.pid present), and the boot, host-service and second-boot skip expectations pass. 3 expectations failed: first boot: checkout done, start hook: the runner asked host-service once the push landed, second boot: checkout marker respected. They come from checkout.clone.failed: the container cannot clone from GitHub here (git ls-remote in the image fails with "server certificate verification failed"; this workspace's egress needs a CA the container does not have). memguard is not involved, but I did not run boot-twice on main in this environment to confirm the same failures happen there.
  • In the built image, as root: I created /var/swap/superset-0 and ran memguard for 3 s. It logged start and removed the file.

Not run:

  • Any real Vercel sandbox (no credentials in this workspace). The reap-instead-of-swap behaviour under real memory pressure has not been measured. The "after" behaviour above is expected, not observed.
  • Snapshot size before and after.

Summary by cubic

Drops on-demand swap from memguard so snapshots stop permanently containing swap data. memguard now reaps when available memory drops below the 8% floor instead of growing swapfiles first, because snapshots are full block copies with no discard: swap written once (up to 16 GB per box) was being copied into every later snapshot of the workspace.

The trade-off

  • Before: an undersized box finished slowly but its snapshots grew by up to 16 GB permanently (a measured 8 GB on one box; fleet snapshots of 14–27 GB exist).
  • After: on an undersized box the largest process is reaped earlier, with a reap line in memguard.log; the remedy is a larger machine size.
  • Startup cleanup is kept: memguard still removes /var/swap/superset-* from older bundles, and boot still removes the old /swapfile.

Alternatives not taken

  • zram: the sandbox kernel has no zram module.
  • Swap on a Vercel Drive once workspaces have one: drives are not part of the snapshot, so this would keep swap without the snapshot cost — the likely long-term answer.
  • Zero-filling swap at stop: stop gives SIGTERM and at least 10 s, and zeroing up to 16 GB does not reliably fit.

How often swap actually saved a job is unknown because memguard.log is box-local; reporting swap.grow counts through host-service's pressure report for a week would settle it.

Written for commit 1718597. Summary will update on new commits.

View guided diff Turn on auto-fix

A snapshot is a full block copy of the disk with no discard, so swap
memguard writes stays in every later snapshot of the workspace. memguard
now reaps on MemAvailable below the floor alone, and still removes any
/var/swap/superset-* an earlier bundle left.
@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

🔗 Preview Links

Service Status Link
Neon Database (Neon) ✅ View Branch
Vercel API (Vercel) ✅ Open Preview
Vercel Web (Vercel) ✅ Open Preview
Vercel Agents Web (Vercel) ➖ Not set up
Vercel Marketing (Vercel) ✅ Open Preview
Vercel Docs (Vercel) ✅ Open Preview

Preview updates automatically with new commits

Copy link
Copy Markdown
Member Author

Closing: swap stays for now. A full typecheck of this repo on an 8 GB box needs it (measured: swap grew to the 12 GB cap in about a minute, and one run was OOM-killed even with it). The clean fix is swap on the workspace drive (#9847), which keeps it out of snapshots.

This branch was previously deployed

1 inactive deployment
preview — 17185974 Deployed Nov 16, 2026 by saddlepaddle via Deploy API #14840
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant