Repository navigation
Serve a model from a registry: add OCI and Existing ModelCache sources #512
Description
ModelCache.spec.source is enum: [HuggingFace] (definition.yaml:38), so the only way to serve a model is to have it on HuggingFace. A customer whose weights are already in Artifact Registry, ECR or their own registry has to push them to HuggingFace first, and a customer who has already staged weights onto a volume has no way to point Modelplane at it.
Both are the normal case for anyone with a model they did not download, and the second is the only answer available in an air-gapped fleet.
#505 proposes the shape in design/modelcache-sources.md. Four sources, each with a different thing that mounts it:
| source | what the user names | what mounts it | what the cluster needs |
|---|---|---|---|
HuggingFace |
a repository ID | a volume Modelplane stages | a ReadWriteMany storage class |
OCI |
a container image holding the weights | the kubelet, as an image volume | Kubernetes 1.36 |
OCI |
a model artifact | the model CSI driver | Kubernetes 1.25 |
PersistentVolumeClaim |
a claim the user filled | a persistentVolumeClaim volume |
the claim bound on the cluster |
The work is larger than the enum. Modelplane resolves a reference to find which kind of artifact it is; each cluster entry carries a mount contract of volumes, volumeMounts and env that the consumer copies into the pod without learning the source; InferenceCluster.status.cache gains imageVolumes so a cluster can say whether it can mount one; and compose-model-replica reads the contract rather than assuming a PVC.
Worth splitting once the design lands. Filing it now because nothing tracks it, and #414 and #457 both assume a staging path this changes.
Activity