ssd-streaming: opt-in measure mode — per-token page invalidation + true disk-read accounting (Darwin)

[FabioMalpezzi]

What

On Darwin, sustained-decode benchmarks of --ssd-streaming are silently optimistic: the unified page cache serves repeat expert reads from RAM, so on a machine with more RAM than the working set you are measuring the page cache, not the SSD path.

This opt-in mode (DS4_STREAMING_MEASURE_INVALIDATE=1) makes streaming benchmarks honest:

• invalidates routed-expert pages every N decoded tokens via msync(MS_INVALIDATE) — the only primitive we found effective on Darwin (madvise(MADV_DONTNEED) and friends are advisory no-ops for this purpose);
• purges expert residency at decode start and reports residency before/after;
• reports true disk reads via ri_diskio_bytesread and per-run read accounting;
• reports its own overhead separately, so the corrected rate R = N / (T_decode − T_msync) can be computed.

Validation

Measured on M4 Max 128 GB, DeepSeek 4 Flash (91 GB), --ssd-streaming-cache-experts 39GB, current main (54b36ed): four 1500-token decode runs. Residency at decode start: 0% (verified in-log each run). Disk-read accounting matches cache-miss arithmetic exactly (misses × expert size = bytes pread, to the MiB). The msync-corrected rate is stable across runs whose raw rates differ by ±7% from page-cache state — the correction removes exactly the state-dependent term.

One caveat we observed once in four runs: total disk reads ~2.8× the pread bytes in a single run (not reproduced in an identical rerun); we left a note in the accounting docs — happy to discuss.

Why upstream

Anyone benchmarking or optimizing --ssd-streaming on a Mac needs this to see real I/O. It is opt-in, zero cost when disabled, Darwin-only guarded.

[FabioMalpezzi]

Validation on this exact branch (M4 Max 128 GB, macOS 26, DeepSeek 4 Flash 91 GB, --ssd-streaming-cache-experts 39GB):

Inertness (flag unset) — same command on vanilla main (54b36ed) and on this branch: token-for-token identical output, zero measure-mode lines in the log, throughput equal within noise (15.06 vs 15.36 tok/s warm).

Enabled (DS4_STREAMING_MEASURE_INVALIDATE=1, 1500-token decode) — residency 0% after initial and final purge (verified in-log); disk-read accounting matches cache-miss arithmetic exactly (19,496 misses × 6.75 MiB = 128.51 GiB = bytes pread, to the hundredth); msync overhead reported separately (52.2 ms/token), corrected rate R = N/(T_decode − T_msync) stable across repeated runs whose raw rates differ with page-cache state.

Logs available on request.