Skip to content

Add model-first provider boundary and split kernel implementations - #628

Open
devteapot wants to merge 2 commits into
antirez:mainfrom
devteapot:t3code/analyze-kernel-core-interface
Open

Add model-first provider boundary and split kernel implementations#628
devteapot wants to merge 2 commits into
antirez:mainfrom
devteapot:t3code/analyze-kernel-core-interface

Conversation

@devteapot

@devteapot devteapot commented Jul 29, 2026

Copy link
Copy Markdown

Summary

  • Add a narrow whole-model provider boundary between the engine core and model integrations.
  • Give DeepSeek V4 and GLM DSA model-first source ownership under models/<model>/.
  • Keep reusable CPU, CUDA, Metal, and ROCm runtime/primitives in their backend directories.
  • Preserve the existing single translation unit per backend and direct calls to tailored kernels.
  • Add live session snapshot coverage and document the extension boundary.

Architectural boundary

The extension point is ds4_model_provider_v1: a whole-model lifecycle
interface, not an abstraction over kernels, tensors, operators, or graphs.

engine core
    │
    ▼
ds4_model_provider_v1
    │
    ▼
model-owned session + graph orchestration
    ├── direct calls to model-owned CUDA / Metal / ROCm kernels
    └── calls to shared backend runtime and low-level primitives

The engine core owns loading, tokenizer/session lifecycle, placement,
checkpoint transport, and provider selection. Each model integration owns its
provider table, CPU/reference path, graph state and scheduling, and custom
backend implementations.

Kernel code remains concrete and specialized. The .inc and .cuh files are
source-ownership units included once by the existing engine/backend entry
points. This adds no virtual kernel dispatch, generic operator layer,
synchronization boundary, or hot-path runtime overhead. Shared code stays
common only where models already use the same low-level contract; duplication
between model-tuned paths remains acceptable.

This creates a strong core seam for future model integrations and leaves open
the option of a small ds4-core with model-specific integrations or forks.

Before and after

Before, ownership followed the translation unit or backend, so model code was
spread across root and backend directories:

ds4.c
models/
├── deepseek_cpu.inc
├── deepseek_graph.inc
├── glm_cpu.inc
└── glm_graph.inc
cuda/
├── deepseek_*.inc
└── glm.inc
metal/
├── deepseek_*.inc
├── glm.inc
└── dsv4_*.metal
rocm/
├── ds4_rocm_<shared>.cuh
├── ds4_rocm_<deepseek>.cuh
└── ds4_rocm_glm.cuh

After, model ownership is the first directory boundary on every backend:

engine core
├── ds4.c
├── ds4_model_provider.{c,h}
└── ds4_model_provider_builtin.h

models/
├── deepseek/
│   ├── provider.{c,h}
│   ├── cpu.inc
│   ├── graph.inc
│   ├── cuda/
│   ├── metal/
│   │   ├── host/
│   │   └── shaders/
│   └── rocm/
└── glm/
    ├── provider.{c,h}
    ├── cpu.inc
    ├── graph.inc
    ├── cuda/
    ├── metal/
    │   ├── host/
    │   └── shaders/
    └── rocm/

shared backend core
├── kernels/   CPU quantization and matmul primitives
├── cuda/      runtime, common dispatch, runtime services
├── metal/     runtime, model I/O, streaming, shared shaders/launchers
└── rocm/      runtime, matmul, quantization, shared MoE/RoPE paths

The Metal mixed shader source was split into model-owned segments while
preserving its original byte order when the runtime assembles the library.

Testing

  • Built and linked the complete CUDA engine on NVIDIA GB10.
  • Built and linked all CPU targets.
  • Live DeepSeek snapshot/restore:
    test_session_snapshot: OK tokens=10 payload=13623728 bytes.
  • CUDA long-context regression passed.
  • Layer packing passed 97/97 checks.
  • Multi-GPU placement passed 98/98 checks.
  • GPU argument parser/CLI passed 44/44 checks.
  • Sampling tests passed.
  • Metal mixed-source reconstruction is byte-identical to the original source;
    Metal compilation is unavailable on this Linux host.
  • ROCm include/dependency paths were checked, but ROCm compilation is
    unavailable because hipcc is not installed on this host.

@devteapot
devteapot force-pushed the t3code/analyze-kernel-core-interface branch from 6a278e3 to 1465498 Compare July 29, 2026 17:26
@devteapot devteapot changed the title Add model provider core and split kernel implementations Add model-first provider boundary and split kernel implementations Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant