[OnePunchMonk]

Reference copy of the full startup-time work behind radixark#310, kept here so the upstream PR can carry only the final code.

This branch has everything that was used for the measurements:

• STARTUP_TIMING probes in every process (driver, rollout manager, engine actor, train actors) and scripts/parse_startup_timing.py, which rebuilds one cross-process timeline from a ray job log.
• MILES_STARTUP_BENCH_EXIT=1, which stops the driver right after the first weight sync.
• MILES_SGLANG_USE_RUNAI_STREAMER=1, to switch the engine back to the RunAI streamer for comparison.
• The commit history of each step (overlap, gate, lazy imports, ServerArgs in the server process, parallel train ranks).

Results and analysis are in the comments on radixark#310. Not meant to be merged.

[OnePunchMonk]

Two more reference branches: perf/startup-timing-v1 is radixark#310 before the prespawn commit, and exp/sgld-prespawn-probes is the scheduler prespawn experiment with the timing probes (used for the 147 s numbers).

[OnePunchMonk]

Measurement history moved here from radixark#310 so that PR stays readable. The comments below are copied verbatim, in order, with their original dates.

Index:

• Baseline and ladder (env-flag variants, 2x H200, full MiniMax-H3): comments 1 to 6
• Breakdown and profiling of engine.wait_server_healthy and imports: comment 7
• New default (overlapped startup on by default) and its GPU check: comments 8 to 10
• Optimization rounds (lighter actor imports, engine memory reserve, ServerArgs in the server process, parallel train ranks): comments 11 to 15
• PR trimmed to the required code, experiment code kept here: comment 16
• sglang-side prespawn ([diffusion] launch_server: accept pre-started scheduler workers sglang#8): comment 17
• Open items: the first weight sync and train step with the new default have not run on real hardware (Modal blocks CUDA IPC); multi-node is untested.

[OnePunchMonk]

1. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 12:02 UTC:

First GPU result: base variant (RunAI streamer on, no overlap), H3 t2va recipe, 2x H200 on Modal, full MiniMax-H3 weights on local disk. First run in the container, so the page cache was cold.

Time to the end of the first weight sync attempt: 461.6 s.

phase	s
create_rollout_manager	275.7
engine wait_server_healthy	166.2
boot offload	51.0
create_training_models	178.6
train load_fsdp_models	99.1
train super_init	40.5
spawn train actors	36.8
engine onload	2.2

Notes:

• Everything is serial in base, so the manager and training phases add up.
• The first weight sync fails on Modal (pidfd_getfd not permitted, CUDA IPC is blocked in the sandbox), so the endpoint here is the start of that sync. Its own cost is the same in every variant.
• Cold cache inflates this run. The next variants run on a warmer cache, so I will rerun base at the end to compare fairly.
• More variants to follow.

[OnePunchMonk]

2. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 12:03 UTC:

Variant 2: mmap loader (SGLANG_USE_RUNAI_MODEL_STREAMER=false), same setup, second run in the container so the page cache is warmer.

Time to the end of the first weight sync attempt: 452.1 s (base was 461.6 s).

phase	base	mmap
engine wait_server_healthy	166.2	110.1
rm.init_rollout_engines	194.3	134.1
boot offload (rm.offload)	51.0	71.4
create_rollout_manager	275.7	233.7
create_training_models	178.6	210.6
train load_fsdp_models	99.1	112.8
spawn train actors	36.8	48.5

Reading it:

• The engine boot gets about 56 s faster, which is the mmap effect (the doc estimated 23 s).
• The total barely moves because the serial phases around it got slower: offload +20 s, train init +32 s. That looks like run to run noise on this box, not an effect of the flag. One rep each, so treat the totals as rough.
• Cache order also matters here, see the base note above.

[OnePunchMonk]

3. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 12:06 UTC:

Variant 3: overlap (engine init not awaited, boot offload not awaited, budget gate on), on top of mmap.

Time to the end of the first weight sync attempt: 261.7 s (mmap 452.1 s, base 461.6 s).

phase	mmap	overlap
create_rollout_manager	233.7	0.1 (not awaited)
create_training_models	210.6	254.0
engine wait_server_healthy	110.1	126.1
train load_fsdp_models	112.8	95.8
total	452.1	261.7

Reading it:

• Train init now runs next to engine boot, so the driver waits on the longer of the two. The critical path is the engine chain plus the offload it triggers (rm.offload is 195 s because it queues behind engine init).
• The memory gate passed immediately: it saw 138 GiB free, because the engines had not loaded yet. Need was 36 GiB (shard 31.5 + largest tensor 0.5 + 4 margin). So the budget check alone does not order train vs engine allocation. There was no OOM in this run, but that is timing luck, not a guarantee. The next variant (overlap_ready) also waits for engine init to finish, to close that race.
• One rep. Run to run noise on this box is around 30 s per phase.

[OnePunchMonk]

4. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 12:13 UTC:

Variant 4: overlap_ready (overlap plus the train gate also waits for engine init), on top of overlap.

Time to the end of the first weight sync attempt: 333.9 s (overlap 261.7 s).

phase	overlap	overlap_ready
train budget_gate wait	0.0	162.7
train load_fsdp_models	95.8	268.1
rm.offload	194.7	177.2
engine wait_server_healthy	126.1	116.1
total	261.7	333.9

Reading it:

• Waiting for engine init closes the allocation race but costs 72 s. The wait is longer than the engine boot alone: the rollout manager actor handles calls one at a time, so the wait_engines_ready call queues behind the boot offload. The gate effectively waits for boot plus offload, which is the phase based gate from perf(startup): overlap engine boot with train init + mmap loading radixark/miles_diffusion#215 again.
• So the cheap overlap number (261.7 s) is only safe if the train allocation cannot beat the engine load. The next variant removes the boot offload, so the engine wait no longer sits behind it.
• No OOM in either run. One rep.

[OnePunchMonk]

5. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 12:18 UTC:

Variant 5: skip_roundtrip (no startup offload, no startup onload; engines stay resident while train inits), on top of overlap_ready.

Time to the end of the first weight sync attempt: 257.8 s (overlap_ready 333.9 s, ungated overlap 261.7 s).

phase	overlap_ready	skip_roundtrip
train budget_gate wait	162.7	105.9
rm.offload	177.2	0 (skipped)
driver onload	2.7	0 (skipped)
engine wait_server_healthy	116.1	118.1
total	333.9	257.8

Reading it:

• The gate now waits only for engine init (about 106 s), since it no longer queues behind an offload. That is the 76 s saved against overlap_ready.
• The engines and the train shard fit together: the gate saw 69 to 73 GiB free once the engines were loaded, against 36 GiB needed. Train init finished with the engines resident, no OOM. This is the safe version (gated on engine init) and it is as fast as the ungated overlap, within noise (4 s apart, noise here is about 30 s).
• The saving the plan expected from skipping the round trip (about 23 s) is not separable from noise in this data. What it really buys is making the safe gate cheap.
• One rep.

[OnePunchMonk]

6. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 12:24 UTC:

Variant 6 (engines_first: engine init launched before router/tracking/data source, router port polled instead of sleep 3), and the full ladder.

engines_first: 263.6 s to the end of the first weight sync attempt. That is the same as skip_roundtrip (257.8 s) within noise, so this change is not visible at this resolution.

Full ladder, 2x H200, full MiniMax-H3, one rep each, each row adds one change to the row above (cumulative, not isolated):

variant	s	vs base
base (RunAI streamer)	461.6
+ mmap	452.1	-9
+ overlap, budget gate (races train vs engine load)	261.7	-200
+ gate waits for engine init	333.9	-128
+ skip startup offload/onload	257.8	-204
+ engines first, router poll	263.6	-198

Net: the gated, race free stack (last row) is about 43% faster than base here (462 s to 264 s).

What moved the number: overlapping train init with engine boot (about 190 s) and, once the gate waits for the engines, not queuing that wait behind a boot offload. mmap shows up in engine boot (166 s to 110 s) but not in the total at this noise level. The router and ordering change is not measurable here.

What the data says about next steps:

• With the gate waiting for engines, the floor is roughly engine boot (about 118 s from spawn to healthy) plus the train GPU phase after it. The doc's 153 s is not reached here.
• engine.wait_server_healthy is 110 to 126 s in every variant. That is the largest single item and it is still unexplained. The scheduler grandchild probes are in this PR but I have not broken that bucket down yet.

Caveats:

• One rep per row. Noise on this box is about 30 s per phase, so differences under that are not real.
• Row order confounds cache warmth: base ran first on a cold cache. I have not rerun base on a warm cache.
• The endpoint is the start of the first weight sync, because that sync fails on Modal (CUDA IPC blocked). Its cost is the same in every variant.
• 2 GPUs (the CI recipe), not the 4 from the doc, so absolute numbers are not comparable to the doc.
• Raw logs and parse_startup_timing reports are on the Modal volume miles-h3-profile-out.

[OnePunchMonk]

7. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 12:36 UTC:

Breakdown of engine.wait_server_healthy (118 s in the skip_roundtrip run, engine0). From sglang's own timestamped log lines, no extra GPU run:

step	s
launch_server process: spawn + import	27
scheduler process: spawn + import, until ZMQ bind	27
distributed init (NCCL)	4
text_encoder load (24.8 GB per rank)	21
transformer load (30.9 GB per rank)	23
audio_vae load	1
video_vae load (decode-dtype store re-home fails, see below)	11
processor, tokenizer, pipeline, FastAPI start, health poll	3

So it is about half Python process boot and half weight load. The DiT is only 23 s of it.

Import profile (CPU container, same image, python -X importtime):

• import sglang.multimodal_gen.runtime.launch_server: 65 s cold, 38 s warm. The scheduler entry (worker_bootstrap) is 38 s warm too. Each process pays the full chain again.
• The big items: sglang/__init__ pulls in about 20 s, multimodal_gen.configs.pipeline_configs about 10 s (it imports every model config, e.g. cosmos3video, which drags in quantization and linear layers), transformers about 6 s.
• 46% of package dirs in the image have no __pycache__ (2378 of 5215 in dist-packages, 270 of 438 in sglang). That is the 65 s vs 38 s gap on a fresh container, which CI hits every run.

Other notes:

• Load speed stays about 1.2 GB/s per rank even on later runs where the files were already in page cache, so the load looks CPU bound (per tensor work), not disk bound. Not confirmed with a profiler yet.
• video_vae logs "could not re-home video_vae decode-dtype weights ... existing decode-dtype store does not match the module" every run.

[OnePunchMonk]

8. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 13:21 UTC:

Pushed 900c755: the overlapped startup is now the default, not behind env flags.

• Engine init is not awaited, the startup offload/onload is skipped, and colocated train ranks wait for engine init plus the memory budget before their first big GPU allocation (actor, ref and teacher loads).
• If the budget does not fit next to the resident engines, the engines are offloaded once and onloaded again before the first weight sync, so tighter setups fall back to the old memory behavior instead of OOMing.
• --serial-startup restores the old order.
• Dropped the engines-first and router polling changes; they did not measure.
• Dockerfile bakes .pyc files.

Corrections to my earlier comments:

• compileall closes less than I said: a cold first import goes from 65 s to 50 s with it (38 s warm). The rest looks like cold file reads on Modal's lazily pulled image.
• The image I benchmarked (dev-cu129-sglang-main-20260927) has an older sglang than main expects: its launch_server calls run_scheduler_process directly, so the miles scheduler wrapper (and my grandchild probes) never ran there. Worth rechecking on a fresh CI image.

CPU tests: 240 passed on this commit. The new default path has not run on GPU yet; a full run is queued on Modal and I will post it here.

[OnePunchMonk]

9. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 13:39 UTC:

GPU check of the new default (900c755), 2x H200, full MiniMax-H3, one rep.

Time to the end of the first weight sync attempt: 331.9 s. This was the first engine load in a fresh container, so compare it with the cold base run (461.6 s): about 28% faster. The ladder's 258 to 264 s numbers were warm runs later in a container.

• The default path works on GPU up to the first weight sync (which fails on Modal as before, CUDA IPC). The gate waited 173 s for engine init, then saw 69 to 73 GiB free next to the resident engines against 36 GiB needed, so no offload fallback.
• engine.wait_server_healthy was 190 s here vs 118 s warm. Where the extra 72 s went:
  • generating GPU P2P access cache in /root/.cache/sglang/gpu_p2p_access_cache_for_0,1.json: 35 s. Only the first engine in a fresh container pays it, later runs read the file. Every fresh CI container pays it. Baking or caching that file, or skipping the check, is a cheap win.
  • all-reduce setup around it: about 19 s more than warm.
  • cold page cache on the weight load: text_encoder 25 s, transformer 31 s, video_vae 13 s (warm: 21, 23, 11).

Engine experiments:

• Forking the scheduler workers from launch_server instead of spawning them does not work as is: Cannot re-initialize CUDA in forked subprocess. launch_server touches CUDA before it starts the workers, so this needs an sglang change, not a miles one.
• Parallel text_encoder load was not measured: it only ran combined with fork, which crashed first.
• The engine-only runs (cProfile of the loaders) failed on a config error in my bench script (--debug-rollout-only turns off --colocate, which --pickscore-reward-colocate requires). Not related to this PR.

[OnePunchMonk]

10. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 13:44 UTC:

Critical path of the current default stack, warm run (257.8 s to the first weight sync), from the probe timestamps:

time (s)	what happens	s
0 to 32	rollout manager Ray worker starts; mostly Python imports (27 s)	29
32 to 36	router start, fixed sleep(3)	3
36 to 62	engine actors created, their processes start	27
62 to 182	engine boot until healthy; the train side waits at the gate the whole time	120
182 to 257	train GPU phase after the gate: FSDP sharding plus the broadcast from rank 0	74
before 0	driver imports (24.5 s), launcher sleeps, ray stop/start	30+

Five Python processes start one after another on this path (driver, rollout manager, engine actor, launch_server, scheduler), each importing torch and sglang for 25 to 30 s.

Next experiments, in one Modal container:

• let the train GPU phase overlap engine boot when an upper bound on engine memory is known (--startup-engine-reserve-gib)
• lighter imports in the rollout manager and engine actor processes
• threaded text_encoder load in the engine
• a warm rep of the current default as baseline

[OnePunchMonk]

11. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 14:21 UTC:

Pushed a new commit: lighter actor imports, an opt-in engine memory reserve for the startup gate, and a fix for test_ema_optimizer (it execs actor methods in a hand-built namespace and broke on the startup probes; that break is in the earlier commits of this PR too).

• Import time, CPU container: miles.ray.rollout 29 s to 5.5 s, the engine actor module 23 s to 1.4 s. Neither process loads sglang at import any more.
• --startup-engine-reserve-gib: colocated train ranks start their GPU load before engine init finishes if free memory minus the reserve covers the init peak. Off by default.
• Tests: full tests/fast was 436 passed, 1 failed (the test above); after the fix that file and the gate tests pass (21).

First GPU result of this round, cold container, SGLANG_SKIP_P2P_CHECK=1: the P2P step went from 35 s to 2 s and the engine now starts 7 s after the rollout manager instead of about 30 s. But the run was slower overall (500 s vs 332 s cold before): launch_server import 66 s, scheduler 52 s and weight load 94 s were all much slower than in the earlier cold run. My guess is the earlier engine start now overlaps rank 0's cold read of the full model, so both hit a cold disk at once. Not confirmed. Warm runs (old default vs each change) are running now and will tell whether the changes help.

[OnePunchMonk]

12. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 14:41 UTC:

Warm runs from this round, same container, one rep each. Measured from driver process start to the end of the first weight sync attempt, because the lazy imports move the driver's sglang import from before main() into parse_args (so the old in-main totals are not comparable).

run	s	vs old
old_default (900c755)	460.0
new_lazy (lighter actor imports)	421.0	-39
new_reserve (lazy + --startup-engine-reserve-gib 72)	328.4	-132

• Lazy imports: the engine now starts 7 s after the rollout manager starts instead of 53 s.
• Reserve: the train ranks passed the gate immediately (134 to 138 GiB free, 36 GiB needed with 72 GiB reserved for the engine) and ran their GPU load while the engine booted. No OOM. Engine boot got slower under that contention (204 s vs 180 s), but the train phase is no longer after it.
• This container is much slower than the one the earlier ladder ran on: the same old_default code took 460 s here vs about 283 s there (engine boot 174 s vs 118 s, post-gate train phase 119 s vs 74 s). So only compare rows within this table.
• SGLANG_SKIP_P2P_CHECK=1 on all rows.

Threaded text_encoder load and the combined run are still going.

[OnePunchMonk]

13. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 14:50 UTC:

Last two runs of this round (same container, from driver start to the end of the first weight sync attempt):

run	s	vs old
old_default (900c755)	460.0
new_lazy	421.0	-39
new_reserve (lazy + reserve 72)	328.4	-132
new_par (lazy + text_encoder loaded on a thread)	398.7	-61
new_all (lazy + reserve + threaded text_encoder)	314.2	-146

Threaded text_encoder load: drop it.

• In new_par the text encoder came up as 49.54 GB instead of 24.84 GB, i.e. in fp32 instead of bf16. sglang's loaders set torch's global default dtype (set_default_torch_dtype), so two loaders on threads race on it. In new_all it happened to come out right. That is a correctness bug, not a speedup.
• It also barely helps: the component loads took 66 to 74 s on a thread vs 71 to 91 s in sequence, consistent with a CPU bound, GIL bound load.
• The gap between new_par and new_lazy, and between new_all and new_reserve, is within this container's noise.

So the keepers from this round are the lighter actor imports (about -39 s) and the engine memory reserve (about -93 s more), both already pushed. The reserve is opt-in (--startup-engine-reserve-gib). It needs an upper bound on the engine's per-GPU memory; for H3 tp2 the engine holds about 62 GB of weights, so 72 left margin.

One rep per row, on a slower container than the earlier ladder.

[OnePunchMonk]

14. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 15:49 UTC:

Pushed c95ce79 and 85605b1:

• c95ce79: the engine actor no longer imports sglang. The launch_server child builds ServerArgs (it imports sglang anyway); the actor passes it a plain-picklable copy of args plus host and ports.
• 85605b1: all train ranks are created at once and get rank 0's rendezvous address before init (actor_group used to wait for rank 0 to boot before creating ranks 1..N).
• Tests: 440 passed on tests/fast.

GPU, one container (a faster one than the last round), 2x H200, full MiniMax-H3, all rows with --startup-engine-reserve-gib 72 and SGLANG_SKIP_P2P_CHECK=1. Driver start to the end of the first weight sync attempt:

code	run 1	run 2	mean
53a24fd (previous head)	213.6	223.2	218.4
+ c95ce79	189.0
+ c95ce79 + 85605b1	186.1	177.6	181.9
53a24fd, first run in a cold container	257.2

So about -36 s (-17%) for the two changes together.

• c95ce79: the server process now starts the moment engine init begins (spawn at 41 s vs 73 s), and the engine is ready about 35 s earlier.
• 85605b1: every train rank starts at about 64 s (the last rank used to start at 94 s), and train init finishes at 152 to 154 s instead of 187 to 197 s. It barely moves the total because the engine is now the only critical path: train is done about 25 to 30 s before the engine is healthy.
• What remains on the engine path (item1_3 rep 2): driver parse_args to 25 s (sglang import for the CLI), engine init begins at 40 s, then about 136 s of engine boot (launch_server import, scheduler import, NCCL, 60 to 90 s of weight load).

On forking the scheduler workers: the launch_server process already has CUDA initialized before launch_server() runs, in both the old and new layout. So it happens during the sglang imports (or while unpickling the args), not in launch_server itself. Forking the workers would need sglang to stop touching CUDA at import time. I have not pinned down the exact call yet.

[OnePunchMonk]

15. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 15:59 UTC:

Summary so far (2x H200, full MiniMax-H3, H3 t2va recipe):

• Startup to the first weight sync is now about 182 s (mean of two runs), down from 218 s for the previous head on the same box. The original serial startup measured 462 s, but on a slower box, so that is not a same-box comparison.
• About 136 s of the 182 s is spent inside sgld processes: two sglang imports back to back (launch_server, then the scheduler), NCCL setup, and 60 to 90 s of weight loading. The train side now finishes about 25 s before the engine, so the engine alone sets the total.
• What is left on the miles side is small (about 20 to 30 s: caching the sglang CLI args in the driver, CI/launcher settings). Further gains need sgld work: lazy imports, not initializing CUDA at import time (so the scheduler workers can be forked), and a faster weight loader.
• Not validated yet: the first weight sync and train step with the new default. Modal blocks CUDA IPC, so every run stopped at the sync. This needs one run on real hardware.

[OnePunchMonk]

16. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 16:31 UTC:

Trimmed this PR down to the required optimised code, now a single commit (749057e). The experiment code (STARTUP_TIMING probes, scripts/parse_startup_timing.py, the benchmark exit hook, the RunAI streamer toggle, and the step by step commit history) is kept for reference in #1.

The code here is the same as the measured version minus that instrumentation. tests/fast: 439 passed.

[OnePunchMonk]

17. From perf: startup timing probes and env-gated startup optimizations radixark/miles_diffusion#310, 2026-10-11 17:41 UTC:

New commit 7ddb65f: --rollout-prespawn-schedulers (opt-in) starts the sglang-d scheduler workers before the sglang server process imports sglang, so the two imports run in parallel instead of back to back.

It needs a small hook in sglang (launch_server.register_prespawned_schedulers), which I put up as a draft in my fork: OnePunchMonk/sglang#8. Without that hook the flag logs a warning and startup falls back to the old path.

Result (2x H200, full MiniMax-H3, same box, two runs each, with --startup-engine-reserve-gib 72 and SGLANG_SKIP_P2P_CHECK=1):

	run 1	run 2	mean
this PR before the flag	181.5	176.0	178.8
with prespawn	149.9	145.1	147.5

Engine boot (server spawn to healthy) went from about 138 s to 104 s. The engine and the train side now finish at about the same time (143 to 149 s), so the next cuts have to hit both.

The previous version of this PR is kept at OnePunchMonk/miles_diffusion branch perf/startup-timing-v1, and the measured version with probes at exp/sgld-prespawn-probes.

Tests: tests/fast 441 passed; the sglang hook's unit test passes on the benchmark image.

@Rockdu could you also take a look at OnePunchMonk/sglang#8? If the approach looks fine to you, I can open it as an upstream PR in sgl-project/sglang.

[OnePunchMonk]

Updated reference branch exp/sgld-prespawn-probes: the probe-instrumented versions of the CLI arg cache, lazy fsdp_utils, rank 0 shard-copy skip and the sharded checkpoint fill (CPU-tested). The pinned chunk weight reader for sglang stays local; its benefit on H200 is not measured yet.