Repository navigation
Propose OCI and PersistentVolumeClaim ModelCache sources - #505
Open
dennis-upbound wants to merge 10 commits into
Open
dennis-upbound wants to merge 10 commits into
dennis-upbound wants to merge 10 commits into
Conversation
A ModelCache reads from HuggingFace only, stages to a volume on each matched cluster, and publishes a phase. A consumer calculates the name of that volume, and two functions hold a copy of a calculation that one source makes true. This design adds an OCI source and an Existing source, and a mount contract on each cluster entry. The contract holds the volumes, the volume mounts and the environment that a consumer adds to a pod, in the Kubernetes types of those names. The consumer copies it and never learns the source. Section 4.4 gives the faults the current design permits, from the benchmark work: modelplaneai#495, where an engine starts while the cache stages and reads a 464 GB model a second time; modelplaneai#494, where a replaced node strands the staging volume; and modelplaneai#498, which asks for the cause on the control plane. Section 6 says what model delivery should measure, because the performance claim this design rests on has no series behind it today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
The introduction was written before the benchmark reports went into section 4.2, and it still argued the weaker case. It said a user cannot name their own registry, where the reports say the volume those users are forced through is slower than no cache (modelplaneai#204), does not take the size they asked for on GKE (modelplaneai#414), has no measurement it must satisfy (modelplaneai#383), and fails on its own (modelplaneai#494, modelplaneai#221). An OCI source removes that volume, which is most of why it is worth having. The introduction also named three problems where section 3 lists four tasks. Nothing set up the fourth, that a cluster never says why it cannot read the artifact, which is what modelplaneai#495 cost: an engine started while the cache staged and read a 464 GB model a second time. Each task in section 3 now traces to a problem here. Section 5.1 said the first part lets a user name an artifact where it is. That is true and incomplete, for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
"A ModelCache does three things, and each one causes a problem" counts the items before giving them and tells the reader they are problems. Both are the writer talking about the document rather than writing it. The same sentence appeared again in section 5.1 as "The design has two parts", followed by a list and then a paragraph restating the list. The introduction is now plain declarative prose with no bold scaffolding, which is what the rest of the document uses. Section 5.1 drops the list and says the two changes in the order a reader meets them. The vale ai-tells rules pass this construction, so it is one to catch by reading. I have added it to the skill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
force-pushed
the
dennis/design-modelcache-sources
branch
from
October 12, 2026 18:38
b14c48d to
a5deceb
Compare
dennis-upbound
marked this pull request as draft
October 12, 2026 18:38
dennis-upbound
changed the title
This brings in the wording that came out of review and cites the OCI delivery study the design draws on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
The stricter rules flag counted lead-ins, negated subjects and figurative verbs throughout. Rewrite those sentences in plain terms without changing what the design says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
force-pushed
the
dennis/design-modelcache-sources
branch
from
October 13, 2026 16:16
5d914f7 to
fcd05a9
Compare
The doc used numbered sections, a terms table, and scope and limitations sections that the other design docs don't have, and it buried examples and issue reports in long sentences. It now follows the house format, with a summary, background, goals, proposal and alternatives. The volume reports are a table, each source is a table row, and the proposal has whole OCI and Existing examples with the cluster entries they produce. The out-of-scope list is three items, and the prose uses shorter sentences that pass the ai-tells vale rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
The design said an OCI cache has no credential, yet Modelplane reads the reference's manifest, which a private registry won't serve without one. It also promised an engine never starts before its model is readable, which a Ready cache holding the wrong model breaks, and it gave metrics a full section that no goal asked for. oci.imagePullSecrets now names the secrets Modelplane reads the manifest with, and mount lists them for the engine pod. A tag resolves to the digest written into mount. The wait goal is that no engine pod is composed until the cluster entry has a mount. Metrics move to future improvements, the CSI driver install moves into the proposal, and the container image and model artifact forms are explained up front. Claim replaces volume where a PersistentVolumeClaim is meant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
The design required a pull secret for any private registry while saying the kubelet uses the node's identity, put every source's model at the same path, and still let placement require a ReadWriteMany storage class for sources that never stage. It also claimed image volumes need Kubernetes 1.36, said nothing about the claim's namespace or access mode for Existing, and failed a cluster for a claim the user may create later. Pull secrets are now optional, the HuggingFace path differs from OCI and Existing, and only HuggingFace needs a storage class. Image volumes need Kubernetes 1.35 or later with containerd 2.1 or CRI-O. An Existing claim must be in the replica's namespace, needs ReadWriteMany or ReadOnlyMany for a gang, and a missing one is Pending. A tag resolves once per cluster, compose-model-replica reads the ModelCache, and ModelExpress stays HuggingFace only. The proxy future improvement and repeated sentences are cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
The design said Modelplane reads an OCI manifest from each cluster without saying what reads it or with which credential, and it named the user-filled claim source Existing, a word InferenceCluster already uses for something else. A resolver Job on each cluster reads the manifest with oci.imagePullSecrets and writes the digest and media type to a ConfigMap Modelplane observes, as it observes the HuggingFace staging Job. A private registry therefore needs a pull secret even where the kubelet could use the node's identity, which the limits now say. The source is now PersistentVolumeClaim, with a persistentVolumeClaim block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
marked this pull request as ready for review
October 14, 2026 16:42
The mount each cache publishes, and a replica waiting for it, can ship without the new sources: they need only status changes, and they fix an engine downloading the model while its cache hydrates. They now have their own design, modelcache-mount-contract.md, in a separate pull request. Keep the OCI and PersistentVolumeClaim sources here, and say how each writes its mount. Note what several caches per deployment would need. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
4 tasks done
dennis-upbound
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
A
ModelCachereads only from HuggingFace, and stages what it reads onto a ReadWriteMany claim on every matched cluster. On some clouds that claim is slow, costly at small sizes, or fails to bind.This proposes two more sources:
source: OCIreads the model from a registry. The kubelet mounts it read-only at pod start, as an image volume for a container image or through the model CSI driver for a model artifact. A resolver Job on each cluster reads the manifest, withoci.imagePullSecretsfor a private registry.source: PersistentVolumeClaimmounts a claim the user already filled.Each source writes the per-cluster
mountfrom #574, which replicas copy into their pods. That design and the code for it moved to #574, so they can merge without the new sources. This design addsimagePullSecretsto the mount and a writableHOMEfor read-only mounts.It revises
design/modelcache.md.Refs #512, #414, #457
I have:
Runnix flake check(or./nix.sh flake check) and made sure it passes.Added or updated tests covering any composition function changes.git commit -s.🤖 Generated with Claude Code