Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 29 additions & 17 deletions docs/cloud-sandbox-considerations.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,23 +197,35 @@ reach every tool an agent runs; and it reaps the largest least-protected
process when `MemAvailable` drops under 8% of the box, so the agent sees an
exit 137 from its tool instead of the box dying. Boot also writes
`SUPERSET_SANDBOX_MEMORY_MB` and `SUPERSET_SANDBOX_VCPUS` into every login
shell from the cgroup, so a hook or an agent can size its work. **Swap is on demand
(2026-10-22):** memguard adds 2 GB swapfiles under `/var/swap` when free memory
drops below twice its reap floor, up to 1.5x the box (at least 6 GB, at most
16 GB), at `vm.swappiness=10`, and drops the newest one after ten quiet
minutes if RAM can take its pages back; the cgroup permits it
(`memory.swap.max` is unbounded). A box that never needs swap holds none on
disk. A box that did need it carries those blocks in every snapshot after,
even once memguard drops the file: a snapshot keeps every block the disk ever
wrote (see "Deleting a file does not shrink a snapshot" in
`cloud-sandbox-mismatches.md`). Measured on a 2/4 box: the host-service
typecheck the kernel had killed at 3 GB finished in 22 s with 855 MB swapped;
the `apps/desktop` typecheck, which needs about 8 GB in all, grew swap
0 → 6 GB and finished in 86 s with host-service's health check never above
47 ms, then swap shrank back to nothing. A GC keeps its hot set in RAM, so a
swapped heap is slow, not a stall. Swap is never on without memguard: with a
fixed 2 GB swapfile and no memguard, the desktop typecheck stalled
host-service past 5 s and was killed by the kernel after 233 s. Still open: automatic resume of an agent whose own process was
shell from the cgroup, so a hook or an agent can size its work. **No swap
(2026-11-17):** from 2026-10-22 memguard grew 2 GB swapfiles under
`/var/swap` on demand, up to 1.5x the box (6-16 GB). That let an undersized
box finish a large typecheck slowly, but a snapshot is a full block copy of
the disk, so swap written once stayed in every later snapshot of the
workspace. Vercel's sandbox team recommends no swap and preventing OOMs
instead, so memguard now grows none, and at start it turns off and removes
any `/var/swap/superset-*` an earlier bundle left (that does not shrink a
snapshot that already holds it); boot still removes the older `/swapfile`.
Without swap, a reap that comes late is not a slow box but a stall: with no
memguard, a steady allocator on a 4 GB box stalled host-service for 282 s and
the kernel's killer never fired. Three changes keep memguard ahead of that.
It checks `MemAvailable` four times a second (it ranks once a second) and
forks nothing between checks; it reaps when the next check at the current
rate of fall would be under the floor, not only once it is; and after a kill
it waits for the victim to exit, not a fixed 3 s, because eight parallel
workers refilled 2 GB in that pause and starved memguard itself for 170 s.
Measured on 4 and 8 GB boxes (reserved hugepages on a real box, the image's
boot, host-service probed every 250 ms from outside): a steady allocator,
parallel workers, page cache with an allocator, and a 2 GB/s allocator all
kept host-service's longest silence at 314 ms with no kernel kill. With swap
and the old 1 s check, the 2 GB/s allocator beat memguard in 6 of 7 runs
and the kernel's killer fired; host-service went silent for up to 62 s
(4 GB) and 94 s (8 GB). What a customer loses: on a box too small for the job, the largest
process is killed at about 92% use instead of the box swapping; the remedy
is a larger machine size. Open: Vercel will send webhook events for sandbox
OOMs; once they exist, the API can report them to the realtime hub beside
host-service's pressure report (`runtime/sandbox-box-pressure`), which
cannot report a box whose host-service has stalled. Still open: automatic resume of an agent whose own process was
reaped while its terminal survived.

## Untested behaviour
Expand Down
3 changes: 2 additions & 1 deletion docs/cloud-sandbox-mismatches.md
Original file line number Diff line number Diff line change
Expand Up @@ -577,7 +577,8 @@ forked by every workspace of its environment, so each of its blocks is paid
once per fork. **What we did:** the golden setup hook
(`packages/sandbox/src/environments/satya-setup.sh`) keeps its npm, Electron
and node-gyp download caches in `/dev/shm`. **Open:** workspace snapshots
carry every block a box ever wrote, swap included.
carry every block a box ever wrote. Boxes have had no swap since 2026-11-17,
so swap no longer adds to that; a snapshot that already holds swap keeps it.

**A tag-filtered sandbox list is not sorted by status time. Fixed.** The usage
reconciler scans by `statusUpdatedAt` and stops at entries older than the maximum
Expand Down
2 changes: 1 addition & 1 deletion packages/sandbox/bundle/rootfs/usr/local/bin/superset-boot
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ if [ -x /usr/local/bin/superset-memguard ]; then
stamp "memguard.start pid=$!"
fi

# The fixed swapfile of an earlier bundle; memguard adds swap on demand now.
# The fixed swapfile of an earlier bundle; boxes have no swap now.
[ -f /swapfile ] && ! grep -q "^/swapfile " /proc/swaps && rm -f /swapfile

# The repository's start hook: the services a workspace needs every boot.
Expand Down
98 changes: 36 additions & 62 deletions packages/sandbox/bundle/rootfs/usr/local/bin/superset-memguard
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env bash
# Keeps the box answering under memory pressure. Started by superset-boot as
# root beside host-service and resident for the session. Once a second:
# root beside host-service and resident for the session. Once a second it
# ranks, and four times a second it checks for a reap:
#
# rank every process of the sandbox user gets an oom_score_adj for what it
# is, not for who spawned it: the kernel copies the value at fork, so
Expand All @@ -12,27 +13,25 @@
# its window manager and stream), gone until the next wake
# 500 everything else, the platform's default: what an agent or a
# hook ran, and the first thing to go
# swap below twice the floor, add a 2 GB swapfile, up to 1.5x the box
# (at least 6 GB, at most 16 GB); after ten quiet minutes, drop the
# newest one if RAM can take its pages back. Nothing is on disk until
# a box needs it, and swap never exists without this script.
# reap below the floor with swap at its cap, kill the largest process in the least protected
# class. Without swap the kernel drops code pages before it kills
# anything, and host-service's event loop stalls for minutes while
# every process refaults; the kernel's killer is the last resort.
# reap below the floor, or within one check's fall of it, kill the largest
# process in the least protected class. The box has no swap, so the
# kernel drops code pages before it kills anything, and host-service's
# event loop, and this script, stall for minutes while every process
# refaults; the kernel's killer is the last resort. So the check
# forks nothing, and a reap does not wait for the next second.
#
# No swap: a snapshot copies every block of the disk, so swap written once
# stays in every later snapshot of the workspace.
#
# One line per action in memguard.log. Fail-open: a box without this script
# behaves as before.
set -uo pipefail
. /etc/superset/contract.sh

FLOOR_PERCENT="${SUPERSET_MEMGUARD_FLOOR_PERCENT:-8}"
INTERVAL_S=1
CHECKS_PER_SCAN=4
COOLDOWN_S=3
MIN_VICTIM_KB=32768
SWAP_DIR=/var/swap
SWAP_STEP_MB=2048
SWAP_IDLE_S="${SUPERSET_MEMGUARD_SWAP_IDLE_S:-600}"

exec 9> /run/superset-memguard.lock
flock -n 9 || exit 0
Expand Down Expand Up @@ -95,76 +94,51 @@ victim_at_or_above() {
[ -n "$best" ] && echo "$best"
}

mem_total=0 mem_avail=0 swap_total=0 swap_free=0
mem_total=0 mem_avail=0
read_meminfo() {
local key val _
while read -r key val _; do
case "$key" in
MemTotal:) mem_total="$val" ;; MemAvailable:) mem_avail="$val" ;;
SwapTotal:) swap_total="$val" ;; SwapFree:) swap_free="$val" ;;
esac
done < /proc/meminfo
}

box_mb() {
local max
max="$(cat /sys/fs/cgroup/memory.max 2>/dev/null || true)"
case "$max" in ''|max) echo $(( $(awk '/MemTotal/ {print $2}' /proc/meminfo) / 1024 )) ;; *) echo $(( max / 1048576 )) ;; esac
}
swap_cap_mb=$(( $(box_mb) * 3 / 2 ))
[ "$swap_cap_mb" -lt 6144 ] && swap_cap_mb=6144
[ "$swap_cap_mb" -gt 16384 ] && swap_cap_mb=16384

swap_files=()
mkdir -p "$SWAP_DIR"
for f in "$SWAP_DIR"/superset-*; do [ -e "$f" ] && { swapoff "$f" 2>/dev/null; rm -f "$f"; }; done
# Swapfiles an earlier bundle grew on demand.
for f in /var/swap/superset-*; do [ -e "$f" ] && { swapoff "$f" 2>/dev/null; rm -f "$f"; }; done
rmdir /var/swap 2>/dev/null

grow_swap() {
local f="${SWAP_DIR}/superset-${#swap_files[@]}"
[ $(( (${#swap_files[@]} + 1) * SWAP_STEP_MB )) -le "$swap_cap_mb" ] || return 1
if fallocate -l "${SWAP_STEP_MB}M" "$f" && chmod 600 "$f" && mkswap -q "$f" >/dev/null && swapon "$f"; then
swap_files+=("$f")
sysctl -q -w vm.swappiness=10
log "swap.grow ${f} total=$(( ${#swap_files[@]} * SWAP_STEP_MB ))MB cap=${swap_cap_mb}MB available=$(( mem_avail / 1024 ))MB"
return 0
fi
rm -f "$f"; log "swap.grow.failed ${f}"; return 1
}
exec {pause_fd}<> <(:)
pause() { read -rt "$1" -u "$pause_fd"; }

shrink_swap() {
local n=${#swap_files[@]} f
[ "$n" -gt 0 ] || return 0
f="${swap_files[$(( n - 1 ))]}"
[ $(( swap_total - swap_free + floor * 2 )) -lt "$mem_avail" ] || return 0
if swapoff "$f"; then
rm -f "$f"; unset 'swap_files[$(( n - 1 ))]'
log "swap.shrink ${f} total=$(( ${#swap_files[@]} * SWAP_STEP_MB ))MB"
fi
# A killed process's memory is free once it is a zombie: wait for that, not a fixed pause.
gone_or_cooled() {
local key state _ i
for (( i = 0; i < COOLDOWN_S * 10; i++ )); do
state=Z
while read -r key state _; do [ "$key" = State: ] && break; done < "/proc/$1/status" 2>/dev/null
[ "$state" = Z ] && return 0
pause 0.1
done
}

last_pressure=$EPOCHSECONDS
prev_avail=0 check=0
while true; do
scan
(( check++ % CHECKS_PER_SCAN == 0 )) && scan
read_meminfo
floor=$(( mem_total * FLOOR_PERCENT / 100 ))
if [ "$mem_avail" -lt $(( floor * 2 )) ]; then
last_pressure=$EPOCHSECONDS
if [ "$swap_free" -lt $(( SWAP_STEP_MB * 1024 )) ] && grow_swap; then
sleep "$INTERVAL_S"; continue
fi
elif [ $(( EPOCHSECONDS - last_pressure )) -ge "$SWAP_IDLE_S" ] && [ ${#swap_files[@]} -gt 0 ]; then
shrink_swap
last_pressure=$EPOCHSECONDS
fi
if [ "$mem_avail" -lt "$floor" ] && [ "$swap_free" -lt $(( swap_total / 10 + 1 )) ]; then
drop=$(( prev_avail > mem_avail ? prev_avail - mem_avail : 0 ))
prev_avail="$mem_avail"
if [ $(( mem_avail - drop )) -lt "$floor" ]; then
scan
victim="$(victim_at_or_above 500 || victim_at_or_above 0 || victim_at_or_above -300)"
if [ -n "$victim" ]; then
log "reap pid=${victim} ${CLASS[$victim]} rss=$(( RSS_KB[$victim] / 1024 ))MB adj=${ADJ[$victim]} available=$(( mem_avail / 1024 ))MB floor=$(( floor / 1024 ))MB"
kill -9 "$victim" 2>/dev/null
sleep "$COOLDOWN_S"
log "reap pid=${victim} ${CLASS[$victim]} rss=$(( RSS_KB[$victim] / 1024 ))MB adj=${ADJ[$victim]} available=$(( mem_avail / 1024 ))MB drop=$(( drop / 1024 ))MB floor=$(( floor / 1024 ))MB"
gone_or_cooled "$victim"
continue
fi
log "pressure available=$(( mem_avail / 1024 ))MB floor=$(( floor / 1024 ))MB nothing to reap"
fi
sleep "$INTERVAL_S"
pause 0.25
done
24 changes: 18 additions & 6 deletions packages/sandbox/src/boot.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,25 @@ describe("superset-boot", () => {
);
});

test("swap is memguard's, grown on demand up to a cap by box size", () => {
test("nothing turns swap on, and an earlier bundle's swapfiles are removed", () => {
expect(boot).not.toMatch(/swapon/);
expect(memguard).toMatch(/swap_cap_mb=\$\(\( \$\(box_mb\) \* 3 \/ 2 \)\)/);
expect(memguard).toMatch(/-lt 6144 \] && swap_cap_mb=6144/);
expect(memguard).toMatch(/-gt 16384 \] && swap_cap_mb=16384/);
expect(memguard).toMatch(/swapon "\$f"/);
expect(memguard).toMatch(/swapoff "\$f"/);
expect(memguard).not.toMatch(/swapon|mkswap|fallocate/);
expect(memguard).toMatch(
/\/var\/swap\/superset-\*.*swapoff "\$f".*rm -f "\$f"/,
);
expect(boot).toMatch(/rm -f \/swapfile/);
});

test("memguard reaps below the floor, or a check before a fast fall reaches it", () => {
expect(memguard).toMatch(
/if \[ \$\(\( mem_avail - drop \)\) -lt "\$floor" \]; then\n\s*scan\n\s*victim=/,
);
});

test("memguard's loop forks nothing between checks, and kills before it logs", () => {
expect(memguard).not.toMatch(/^\s*sleep /m);
expect(memguard).toMatch(/pause 0\.25\ndone\s*$/);
expect(memguard).toMatch(/kill -9 "\$victim" 2>\/dev\/null\n\s*log "reap /);
});

test("starts host-service after only the before-host steps; the rest run detached after it", () => {
Expand Down
Loading