From 7ead4796f8d7083870b84c760e03a507ab2e8f40 Mon Sep 17 00:00:00 2001 From: Satya Patel <14907857+saddlepaddle@users.noreply.github.com> Date: Wed, 18 Nov 2026 01:57:15 +0000 Subject: [PATCH] fix(sandbox): no swap on cloud boxes; memguard reaps early enough without it memguard no longer grows swapfiles. A snapshot is a full block copy of the disk, so swap written once stayed in every later snapshot (up to 1.5x box RAM, 6-16 GB). At start it still turns off and removes /var/swap/superset-* from an earlier bundle. Without swap a late reap is a stall, not a slow box, so the loop changes: - check MemAvailable every 0.25 s with no forks (rank still once a second) - reap when the next check at the current rate of fall would be under the floor, with a fresh scan before choosing the victim - after a kill, wait for the victim to exit (at most 3 s), not a fixed 3 s - kill before logging Measured on 4 and 8 GB boxes under four kinds of pressure: host-service's longest silence 314 ms, no kernel OOM kill. Main (swap, 1 s check) let a 2 GB/s allocator win in 6 of 7 runs, with silences up to 94 s. --- docs/cloud-sandbox-considerations.md | 46 +++++---- docs/cloud-sandbox-mismatches.md | 3 +- .../bundle/rootfs/usr/local/bin/superset-boot | 2 +- .../rootfs/usr/local/bin/superset-memguard | 98 +++++++------------ packages/sandbox/src/boot.test.ts | 24 +++-- 5 files changed, 86 insertions(+), 87 deletions(-) diff --git a/docs/cloud-sandbox-considerations.md b/docs/cloud-sandbox-considerations.md index c690a9a6d2e..170cb936a90 100644 --- a/docs/cloud-sandbox-considerations.md +++ b/docs/cloud-sandbox-considerations.md @@ -197,23 +197,35 @@ reach every tool an agent runs; and it reaps the largest least-protected process when `MemAvailable` drops under 8% of the box, so the agent sees an exit 137 from its tool instead of the box dying. Boot also writes `SUPERSET_SANDBOX_MEMORY_MB` and `SUPERSET_SANDBOX_VCPUS` into every login -shell from the cgroup, so a hook or an agent can size its work. **Swap is on demand -(2026-10-22):** memguard adds 2 GB swapfiles under `/var/swap` when free memory -drops below twice its reap floor, up to 1.5x the box (at least 6 GB, at most -16 GB), at `vm.swappiness=10`, and drops the newest one after ten quiet -minutes if RAM can take its pages back; the cgroup permits it -(`memory.swap.max` is unbounded). A box that never needs swap holds none on -disk. A box that did need it carries those blocks in every snapshot after, -even once memguard drops the file: a snapshot keeps every block the disk ever -wrote (see "Deleting a file does not shrink a snapshot" in -`cloud-sandbox-mismatches.md`). Measured on a 2/4 box: the host-service -typecheck the kernel had killed at 3 GB finished in 22 s with 855 MB swapped; -the `apps/desktop` typecheck, which needs about 8 GB in all, grew swap -0 → 6 GB and finished in 86 s with host-service's health check never above -47 ms, then swap shrank back to nothing. A GC keeps its hot set in RAM, so a -swapped heap is slow, not a stall. Swap is never on without memguard: with a -fixed 2 GB swapfile and no memguard, the desktop typecheck stalled -host-service past 5 s and was killed by the kernel after 233 s. Still open: automatic resume of an agent whose own process was +shell from the cgroup, so a hook or an agent can size its work. **No swap +(2026-11-17):** from 2026-10-22 memguard grew 2 GB swapfiles under +`/var/swap` on demand, up to 1.5x the box (6-16 GB). That let an undersized +box finish a large typecheck slowly, but a snapshot is a full block copy of +the disk, so swap written once stayed in every later snapshot of the +workspace. Vercel's sandbox team recommends no swap and preventing OOMs +instead, so memguard now grows none, and at start it turns off and removes +any `/var/swap/superset-*` an earlier bundle left (that does not shrink a +snapshot that already holds it); boot still removes the older `/swapfile`. +Without swap, a reap that comes late is not a slow box but a stall: with no +memguard, a steady allocator on a 4 GB box stalled host-service for 282 s and +the kernel's killer never fired. Three changes keep memguard ahead of that. +It checks `MemAvailable` four times a second (it ranks once a second) and +forks nothing between checks; it reaps when the next check at the current +rate of fall would be under the floor, not only once it is; and after a kill +it waits for the victim to exit, not a fixed 3 s, because eight parallel +workers refilled 2 GB in that pause and starved memguard itself for 170 s. +Measured on 4 and 8 GB boxes (reserved hugepages on a real box, the image's +boot, host-service probed every 250 ms from outside): a steady allocator, +parallel workers, page cache with an allocator, and a 2 GB/s allocator all +kept host-service's longest silence at 314 ms with no kernel kill. With swap +and the old 1 s check, the 2 GB/s allocator beat memguard in 6 of 7 runs +and the kernel's killer fired; host-service went silent for up to 62 s +(4 GB) and 94 s (8 GB). What a customer loses: on a box too small for the job, the largest +process is killed at about 92% use instead of the box swapping; the remedy +is a larger machine size. Open: Vercel will send webhook events for sandbox +OOMs; once they exist, the API can report them to the realtime hub beside +host-service's pressure report (`runtime/sandbox-box-pressure`), which +cannot report a box whose host-service has stalled. Still open: automatic resume of an agent whose own process was reaped while its terminal survived. ## Untested behaviour diff --git a/docs/cloud-sandbox-mismatches.md b/docs/cloud-sandbox-mismatches.md index 988674e3af3..ffc8144c02c 100644 --- a/docs/cloud-sandbox-mismatches.md +++ b/docs/cloud-sandbox-mismatches.md @@ -577,7 +577,8 @@ forked by every workspace of its environment, so each of its blocks is paid once per fork. **What we did:** the golden setup hook (`packages/sandbox/src/environments/satya-setup.sh`) keeps its npm, Electron and node-gyp download caches in `/dev/shm`. **Open:** workspace snapshots -carry every block a box ever wrote, swap included. +carry every block a box ever wrote. Boxes have had no swap since 2026-11-17, +so swap no longer adds to that; a snapshot that already holds swap keeps it. **A tag-filtered sandbox list is not sorted by status time. Fixed.** The usage reconciler scans by `statusUpdatedAt` and stops at entries older than the maximum diff --git a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot index 175309dc972..4f8e743efa3 100755 --- a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot +++ b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot @@ -237,7 +237,7 @@ if [ -x /usr/local/bin/superset-memguard ]; then stamp "memguard.start pid=$!" fi -# The fixed swapfile of an earlier bundle; memguard adds swap on demand now. +# The fixed swapfile of an earlier bundle; boxes have no swap now. [ -f /swapfile ] && ! grep -q "^/swapfile " /proc/swaps && rm -f /swapfile # The repository's start hook: the services a workspace needs every boot. diff --git a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-memguard b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-memguard index fca19a05138..ba799c9804a 100755 --- a/packages/sandbox/bundle/rootfs/usr/local/bin/superset-memguard +++ b/packages/sandbox/bundle/rootfs/usr/local/bin/superset-memguard @@ -1,6 +1,7 @@ #!/usr/bin/env bash # Keeps the box answering under memory pressure. Started by superset-boot as -# root beside host-service and resident for the session. Once a second: +# root beside host-service and resident for the session. Once a second it +# ranks, and four times a second it checks for a reap: # # rank every process of the sandbox user gets an oom_score_adj for what it # is, not for who spawned it: the kernel copies the value at fork, so @@ -12,14 +13,15 @@ # its window manager and stream), gone until the next wake # 500 everything else, the platform's default: what an agent or a # hook ran, and the first thing to go -# swap below twice the floor, add a 2 GB swapfile, up to 1.5x the box -# (at least 6 GB, at most 16 GB); after ten quiet minutes, drop the -# newest one if RAM can take its pages back. Nothing is on disk until -# a box needs it, and swap never exists without this script. -# reap below the floor with swap at its cap, kill the largest process in the least protected -# class. Without swap the kernel drops code pages before it kills -# anything, and host-service's event loop stalls for minutes while -# every process refaults; the kernel's killer is the last resort. +# reap below the floor, or within one check's fall of it, kill the largest +# process in the least protected class. The box has no swap, so the +# kernel drops code pages before it kills anything, and host-service's +# event loop, and this script, stall for minutes while every process +# refaults; the kernel's killer is the last resort. So the check +# forks nothing, and a reap does not wait for the next second. +# +# No swap: a snapshot copies every block of the disk, so swap written once +# stays in every later snapshot of the workspace. # # One line per action in memguard.log. Fail-open: a box without this script # behaves as before. @@ -27,12 +29,9 @@ set -uo pipefail . /etc/superset/contract.sh FLOOR_PERCENT="${SUPERSET_MEMGUARD_FLOOR_PERCENT:-8}" -INTERVAL_S=1 +CHECKS_PER_SCAN=4 COOLDOWN_S=3 MIN_VICTIM_KB=32768 -SWAP_DIR=/var/swap -SWAP_STEP_MB=2048 -SWAP_IDLE_S="${SUPERSET_MEMGUARD_SWAP_IDLE_S:-600}" exec 9> /run/superset-memguard.lock flock -n 9 || exit 0 @@ -95,76 +94,51 @@ victim_at_or_above() { [ -n "$best" ] && echo "$best" } -mem_total=0 mem_avail=0 swap_total=0 swap_free=0 +mem_total=0 mem_avail=0 read_meminfo() { local key val _ while read -r key val _; do case "$key" in MemTotal:) mem_total="$val" ;; MemAvailable:) mem_avail="$val" ;; - SwapTotal:) swap_total="$val" ;; SwapFree:) swap_free="$val" ;; esac done < /proc/meminfo } -box_mb() { - local max - max="$(cat /sys/fs/cgroup/memory.max 2>/dev/null || true)" - case "$max" in ''|max) echo $(( $(awk '/MemTotal/ {print $2}' /proc/meminfo) / 1024 )) ;; *) echo $(( max / 1048576 )) ;; esac -} -swap_cap_mb=$(( $(box_mb) * 3 / 2 )) -[ "$swap_cap_mb" -lt 6144 ] && swap_cap_mb=6144 -[ "$swap_cap_mb" -gt 16384 ] && swap_cap_mb=16384 - -swap_files=() -mkdir -p "$SWAP_DIR" -for f in "$SWAP_DIR"/superset-*; do [ -e "$f" ] && { swapoff "$f" 2>/dev/null; rm -f "$f"; }; done +# Swapfiles an earlier bundle grew on demand. +for f in /var/swap/superset-*; do [ -e "$f" ] && { swapoff "$f" 2>/dev/null; rm -f "$f"; }; done +rmdir /var/swap 2>/dev/null -grow_swap() { - local f="${SWAP_DIR}/superset-${#swap_files[@]}" - [ $(( (${#swap_files[@]} + 1) * SWAP_STEP_MB )) -le "$swap_cap_mb" ] || return 1 - if fallocate -l "${SWAP_STEP_MB}M" "$f" && chmod 600 "$f" && mkswap -q "$f" >/dev/null && swapon "$f"; then - swap_files+=("$f") - sysctl -q -w vm.swappiness=10 - log "swap.grow ${f} total=$(( ${#swap_files[@]} * SWAP_STEP_MB ))MB cap=${swap_cap_mb}MB available=$(( mem_avail / 1024 ))MB" - return 0 - fi - rm -f "$f"; log "swap.grow.failed ${f}"; return 1 -} +exec {pause_fd}<> <(:) +pause() { read -rt "$1" -u "$pause_fd"; } -shrink_swap() { - local n=${#swap_files[@]} f - [ "$n" -gt 0 ] || return 0 - f="${swap_files[$(( n - 1 ))]}" - [ $(( swap_total - swap_free + floor * 2 )) -lt "$mem_avail" ] || return 0 - if swapoff "$f"; then - rm -f "$f"; unset 'swap_files[$(( n - 1 ))]' - log "swap.shrink ${f} total=$(( ${#swap_files[@]} * SWAP_STEP_MB ))MB" - fi +# A killed process's memory is free once it is a zombie: wait for that, not a fixed pause. +gone_or_cooled() { + local key state _ i + for (( i = 0; i < COOLDOWN_S * 10; i++ )); do + state=Z + while read -r key state _; do [ "$key" = State: ] && break; done < "/proc/$1/status" 2>/dev/null + [ "$state" = Z ] && return 0 + pause 0.1 + done } -last_pressure=$EPOCHSECONDS +prev_avail=0 check=0 while true; do - scan + (( check++ % CHECKS_PER_SCAN == 0 )) && scan read_meminfo floor=$(( mem_total * FLOOR_PERCENT / 100 )) - if [ "$mem_avail" -lt $(( floor * 2 )) ]; then - last_pressure=$EPOCHSECONDS - if [ "$swap_free" -lt $(( SWAP_STEP_MB * 1024 )) ] && grow_swap; then - sleep "$INTERVAL_S"; continue - fi - elif [ $(( EPOCHSECONDS - last_pressure )) -ge "$SWAP_IDLE_S" ] && [ ${#swap_files[@]} -gt 0 ]; then - shrink_swap - last_pressure=$EPOCHSECONDS - fi - if [ "$mem_avail" -lt "$floor" ] && [ "$swap_free" -lt $(( swap_total / 10 + 1 )) ]; then + drop=$(( prev_avail > mem_avail ? prev_avail - mem_avail : 0 )) + prev_avail="$mem_avail" + if [ $(( mem_avail - drop )) -lt "$floor" ]; then + scan victim="$(victim_at_or_above 500 || victim_at_or_above 0 || victim_at_or_above -300)" if [ -n "$victim" ]; then - log "reap pid=${victim} ${CLASS[$victim]} rss=$(( RSS_KB[$victim] / 1024 ))MB adj=${ADJ[$victim]} available=$(( mem_avail / 1024 ))MB floor=$(( floor / 1024 ))MB" kill -9 "$victim" 2>/dev/null - sleep "$COOLDOWN_S" + log "reap pid=${victim} ${CLASS[$victim]} rss=$(( RSS_KB[$victim] / 1024 ))MB adj=${ADJ[$victim]} available=$(( mem_avail / 1024 ))MB drop=$(( drop / 1024 ))MB floor=$(( floor / 1024 ))MB" + gone_or_cooled "$victim" continue fi log "pressure available=$(( mem_avail / 1024 ))MB floor=$(( floor / 1024 ))MB nothing to reap" fi - sleep "$INTERVAL_S" + pause 0.25 done diff --git a/packages/sandbox/src/boot.test.ts b/packages/sandbox/src/boot.test.ts index df240b4fb7f..70bb4641eba 100644 --- a/packages/sandbox/src/boot.test.ts +++ b/packages/sandbox/src/boot.test.ts @@ -36,13 +36,25 @@ describe("superset-boot", () => { ); }); - test("swap is memguard's, grown on demand up to a cap by box size", () => { + test("nothing turns swap on, and an earlier bundle's swapfiles are removed", () => { expect(boot).not.toMatch(/swapon/); - expect(memguard).toMatch(/swap_cap_mb=\$\(\( \$\(box_mb\) \* 3 \/ 2 \)\)/); - expect(memguard).toMatch(/-lt 6144 \] && swap_cap_mb=6144/); - expect(memguard).toMatch(/-gt 16384 \] && swap_cap_mb=16384/); - expect(memguard).toMatch(/swapon "\$f"/); - expect(memguard).toMatch(/swapoff "\$f"/); + expect(memguard).not.toMatch(/swapon|mkswap|fallocate/); + expect(memguard).toMatch( + /\/var\/swap\/superset-\*.*swapoff "\$f".*rm -f "\$f"/, + ); + expect(boot).toMatch(/rm -f \/swapfile/); + }); + + test("memguard reaps below the floor, or a check before a fast fall reaches it", () => { + expect(memguard).toMatch( + /if \[ \$\(\( mem_avail - drop \)\) -lt "\$floor" \]; then\n\s*scan\n\s*victim=/, + ); + }); + + test("memguard's loop forks nothing between checks, and kills before it logs", () => { + expect(memguard).not.toMatch(/^\s*sleep /m); + expect(memguard).toMatch(/pause 0\.25\ndone\s*$/); + expect(memguard).toMatch(/kill -9 "\$victim" 2>\/dev\/null\n\s*log "reap /); }); test("starts host-service after only the before-host steps; the rest run detached after it", () => {