[cyyever]

Overview

Foundation for reading weights into existing device tensors through the shared cuFile/hipFile loader. Avoid a full CPU assembly buffer when a caller supplies a compatible contiguous destination.

Claims

• Both storage backends accept the same optional destination resolver.
• Compatibility mode writes bounded byte ranges directly into a validated destination and preserves its storage offset. Errors propagate before yielding a partially loaded weight. Negative chunk sizes are rejected before I/O, and direct writes require the current accelerator.
• The existing sharded-state loader supplies destinations using its existing padding logic. Other layouts and P2P retain the existing read path.

Validation

• 15 tests passed: PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 uv run --no-project .venv/bin/python -m pytest --confcutdir=tests/model_executor/model_loader -o addopts= tests/model_executor/model_loader/test_direct_io.py tests/model_executor/model_loader/test_hipfile.py -q.
• All applicable pre-commit hooks passed, including mypy; git diff --check passed.
• New tests exercise offset preservation, partial final chunks, shape rejection before writing, and failed reads. They use CPU-backed reader doubles, not GPU performance evidence.
• No live cuFile/hipFile end-to-end validation or model evaluation yet. No speedup claim. Hardware validation remains pending.

Details

The current compatibility loader assembles oversized tensors on CPU using bounded device reads. A destination resolver avoids that staging when the final device tensor is available. Writes synchronize before and after the external reader; noncontiguous, CPU, or dtype-converting sharded destinations retain the old path.

This does not require converting downloaded checkpoints, but the first caller is the existing sharded-state loader. It DOES NOT yet accelerate the original HF Qwen checkpoint. Ordinary HF weight mapping, fused/reshaped weights, native safetensors buffer/DLPack integration, and hardware correctness/benchmarking remain outstanding. No model-specific layout or new checkpoint format is introduced.

Duplicate search: no open BakeLab/vllm PRs; upstream searches for cuFile destination and hipFile found no matching model-loader change (the cuFile result concerned KV offload). This is a BakeLab fork draft, not an upstream submission.

Remaining work

• Connect original HF loading without bypassing existing weight transformations.
• Reuse safetensors native buffer/DLPack ownership and stream semantics where appropriate.
• Validate CUDA and ROCm hardware paths, output correctness, peak memory and loading time.
• Human review before readiness or upstream submission.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill.
• AI assistance was used during the creation of this PR (OpenAI Codex).
• Design Fit: shared storage logic and existing destination mapping.
• Testing and Validation: unit checks passed; hardware/model validation pending.
• Code Quality and Style: hooks passed; full feature/API review pending.
• Pull Request Contents: limitations and remaining work stated explicitly.