cuda: copy model into VRAM on single-GPU via --gpu-resident

[pvaccarello]

Summary

On a single discrete CUDA GPU whose VRAM is large enough to hold the whole model,
ds4 never copies the weights into device memory. It keeps the model
host-mapped (zero-copy) and reads every routed-expert weight across PCIe on each
token. On an RTX PRO 6000 Blackwell (96 GB) running DeepSeek-V4-Flash-IQ2XXS
(~80.76 GiB) this pins decode at ~1 tok/s with VRAM stuck at ~18 GB, even with
--gpu-vram auto (which reports a 90 GB budget).

This PR:

1. Fixes the copy-to-VRAM path, which was dead code on this configuration.
2. Exposes it as a first-class flag --gpu-resident (maps to the existing
   DS4_CUDA_COPY_MODEL_CHUNKED env var).

Result on the hardware above: ~55 tok/s (a ~50× speedup), model fully
resident in VRAM.

Root cause

The single-GPU, no-slice startup path calls ds4_gpu_set_model_map_range():

extern "C" int ds4_gpu_set_model_map_range(...) {
    (void)max_tensor_bytes;
    if (!ds4_gpu_register_model_map_no_copy(model_map, model_size)) return 0; // (1)
    if (getenv("DS4_CUDA_COPY_MODEL_CHUNKED") != NULL &&
        !cuda_model_copy_chunked(model_map, model_size, map_offset, map_size)) { // (2)
        (void)cuda_model_prefetch_range(model_map, model_size, map_offset, map_size);
    }
    return 1;
}

1. ds4_gpu_register_model_map_no_copy() does cudaHostRegister(...) and sets
   g_model_registered = 1.
2. cuda_model_copy_chunked() opens with
   if (g_model_device_owned || g_model_registered) return 1; — so, because (1)
   already set g_model_registered = 1, the copy is always skipped and every
   weight access goes over PCIe. DS4_CUDA_COPY_MODEL_CHUNKED is effectively a
   no-op on this path.

Fix

1. ds4_cuda.cu — in ds4_gpu_set_model_map_range(), attempt the chunked
   device copy before the no-copy host registration, so the guard no longer
   short-circuits it. On failure (OOM, or the existing DS4_CUDA_NO_MODEL_COPY /
   DS4_CUDA_DIRECT_MODEL escape hatches) we restore state and fall back to the
   original no-copy path. Default behavior is unchanged unless the copy is
   explicitly requested.

   Execution already routes weight reads through the resolver
   if (model_map == g_model_host_base && g_model_device_base) return g_model_device_base + offset;,
   so once the copy sets g_model_device_base to the VRAM image, kernels read
   from VRAM with no further changes.

2. ds4_server.c, ds4_cli.c, ds4_agent.c, ds4_bench.c — new
   --gpu-resident flag sets DS4_CUDA_COPY_MODEL_CHUNKED via setenv() before
   engine creation, in each binary's arg parser. (Kept as a thin mapping over the
   env var so no cross-backend symbol/plumbing is added; can be promoted to a
   proper ds4_engine_options field if you prefer.)

3. ds4_help.c — shared help text for the new flag (shown by all binaries).

Benchmark (RTX PRO 6000 Blackwell, 96 GB, sm_120, CUDA 13.0)

Model DeepSeek-V4-Flash-IQ2XXS (~80.76 GiB), make ds4-server CUDA_ARCH=sm_120.

	before	after (--gpu-resident)
VRAM used	~18 GB	~94 GB (model resident)
startup	"no-copy … selective cache"	"CUDA chunk-copying 80.76 GiB … complete in 6.6s"
decode	~1 tok/s	~55 tok/s (200 tok in 3.57s)
GPU util during decode	100% (PCIe-bound)	~92% (compute-bound)
output	coherent	coherent (no corruption)

Both the flag and the raw env var were tested and produce identical behavior.

Run:

./ds4-server --gpu-resident --gpu-vram auto --ctx 64000

Notes for the maintainer

• Opt-in only. Without --gpu-resident (or the env var), behavior is
  byte-for-byte the previous no-copy streaming path.
• Single-tier only. engine_install_per_device_caches() (multi-tier)
  deliberately uses the no-copy variant; untouched here.
• The original cuda_model_prefetch_range() fallback is dropped: it only ran in
  the "env set but copy skipped" case, which no longer exists. Can be re-added in
  the copy-failure branch as a host-page warm hint if desired.
• Headroom: needs VRAM ≥ model + KV + buffers + working set. ~80 GiB on a
  96 GB card lands at ~94–95 GB (--ctx 64000…100000). If it doesn't fit,
  cudaMalloc fails and we fall back to no-copy.
• --gpu-resident is wired into all four CUDA-capable binaries: ds4-server,
  ds4 (CLI), ds4-agent, ds4-bench.

Test plan

• Builds with make ds4-server ds4 ds4-agent ds4-bench CUDA_ARCH=sm_120.
• --gpu-resident on ds4-server (no env var): model copied to VRAM,
  ~55 tok/s, coherent output.
• --gpu-resident on ds4 CLI: triggers the VRAM copy end-to-end.
• All four binaries accept the flag and list it in --help.
• DS4_CUDA_COPY_MODEL_CHUNKED=1 (no flag): same behavior.
• Neither: unchanged no-copy streaming path.
• (maintainer) confirm no regression on multi-GPU / SSD-streaming / Metal /
  CPU paths (all untouched).

---

PS: ci vediamo a Campobello di Licata!!!

[teo-mateo]

Can confirm this on identical hardware RTX PRO 6000 Blackwell 96GB, same IQ2XXS Flash 0731.

On current main (54b36ed) I was getting ~0.7 tok/s with VRAM stuck around 13GB, and DS4_CUDA_COPY_MODEL_CHUNKED did nothing. Traced it to the same early-return in cuda_model_copy_chunked that this PR describes: g_model_registered is already 1 by the time it runs, so the copy is dead code on the single-GPU path.

After patching just that guard locally, the chunk copy runs (80.76 GiB in ~7.7s from warm page cache), the model goes fully to VRAM and decode jumps to 61.5 tok/s at 128k context. So the diagnois here is definitely right, and the numbers match what's reported. Would be great to see this merged.