Repository navigation
Wait for a ModelCache before starting a replica's engines - #574
Draft
dennis-upbound wants to merge 5 commits into
Draft
dennis-upbound wants to merge 5 commits into
dennis-upbound wants to merge 5 commits into
Conversation
A ModelReplica that references a ModelCache can't tell from the cache whether the model is readable on its cluster, or how to read it. It works out the claim name itself, with a copy of the code that names the claim, and mounts it whether or not the hydration Job has finished. Give each cluster entry a mount: the volumes, volume mounts and environment that read the model there. The cache sets it only while the cluster's phase is Ready, so a replica can copy it into its engine pods and wait until it appears. Describe the existing message field, which says why a cluster isn't Ready. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
|
Docs preview: https://modelplane-docs-pr-574.vercel.app (ready once the site's Content workflow finishes) |
2 of 4 tasks
A ModelCache reports each cluster's phase, but not how to read the model there, nor why a cluster is stuck. When a claim waited on a node the cloud had replaced, the cache said Pending for 40 minutes, and finding the cause took someone with access to the workload cluster. Write each cluster's mount once its phase is Ready: the claim as a volume at /mnt/models, and HF_HUB_CACHE so an engine finds the model by its repo ID. These are the volume, mount and variable replicas already use, so pods built from them don't change. Write a message for every other phase. A Pending cluster names the claim it waits on, and the node the scheduler chose for it. A Hydrating cluster names the repo and claim. A Failed cluster gives the Job, how many attempts failed, and the Job's own message. A missing token names the Secret and key. Towards modelplaneai#494. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
force-pushed
the
dennis/modelcache-mount-contract
branch
2 times, most recently
from
October 26, 2026 20:18
b59120f to
b6d3698
Compare
A replica that references a ModelCache started its engines as soon as it was scheduled. On a fresh cluster the hydration Job was still writing the cache, so the first engine found no model there and downloaded all 464 GB itself, in parallel with the Job. Require the ModelCache, and read how to mount it from the entry its status publishes for the replica's cluster. Until that entry has a mount, a new replica composes nothing and its ModelAccepted condition says why, using the cache's own message: hydrating, a claim waiting on a node, a failed Job, a cluster the cache doesn't stage to, or no cache at all. A replica whose engines already run doesn't wait. Until its cache publishes a mount, it keeps the cache volume, volume mount and HF_HUB_CACHE its workload already has, read from the workload it last composed, or no cache if it has none, and warns. So a cache that stops being Ready or is deleted doesn't take a serving model down, and adding a cache or switching to another doesn't restart the engines until the new cache is Ready. MX_MODEL_REVISION and the IPC_LOCK capability follow the mount the engines use. The mount is the volume, mount and variable replicas used before, so running engines don't restart on upgrade. Fixes modelplaneai#495. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
A replica now holds its engines until its cache publishes a mount, but the docs didn't say so, or where to read why a replica is waiting. Say how a replica waits for its cache, the messages it and the cache report while it waits, how soon it notices the cache is ready, and that a running replica keeps the cache it mounts until the new one is ready. Add the design for the mount each cache publishes, the replica's wait for it, and the messages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
force-pushed
the
dennis/modelcache-mount-contract
branch
from
October 26, 2026 20:59
b6d3698 to
c244513
Compare
4 tasks done
Unit tests show what the functions compose, but not that a replica really holds its engines until the hydration Job has staged the model, or that the pod mounts what the cache publishes. Add an end-to-end test that binds the cache's claim to a hostPath volume only after the replica exists. It checks that the replica waits while the claim is Pending, then that the engine mounts the published claim, sets HF_HUB_CACHE, and reads the staged snapshot. Each run uses a new hostPath, so a previous run's completion marker can't make the Job skip. The local pool has one fake GPU, which mock-demo holds, so the test scales mock-demo to zero while it runs. It runs only with E2E_MODEL_CACHE=1, after the rest of the suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
dennis-upbound
force-pushed
the
dennis/modelcache-mount-contract
branch
from
October 27, 2026 03:59
c244513 to
0f71a37
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Fixes #495.
A replica that references a
ModelCachestarts its engines as soon as it's scheduled. On a fresh cluster the hydration Job was still writing the cache, so the first engine didn't find the model and downloaded all 464 GB itself, in parallel with the Job (#495). When a cache gets stuck, it doesn't say why: a claim waiting on a node the cloud had replaced left the cachePendingfor 40 minutes (#494).Each
ModelCachecluster entry now publishes amount, set once the entry isReady: the volumes, volume mounts and environment that read the model on that cluster.compose-model-replicarequires the cache and copies the mount into every engine pod. Until the mount appears, a new replica composes nothing, and itsModelAcceptedcondition isFalsewith reasonWaitingForCacheand the cache's message. An entry that isn'tReadygets amessage. APendingone gives the claim it waits on and the node the scheduler chose. AHydratingone gives the repo, and aFailedone gives the Job and its error. A missing token gives the Secret and key.A replica whose engines already run doesn't wait. Until its cache publishes a mount, it keeps the cache volume, volume mount and
HF_HUB_CACHEits workload already has, read from the workload it last composed, or no cache if it has none, and warns. So a cache that stops beingReadyor is deleted doesn't take a serving model down, and adding a cache to a running deployment or switchingmodelCacheRefto another cache doesn't restart the engines until the new cache isReady.MX_MODEL_REVISIONand the DynamoIPC_LOCKcapability follow the mount the engines use.Upgrading doesn't restart running engines. The published mount is the same volume, mount and
HF_HUB_CACHEthat replicas composed before, and until the cache publishes it the replica keeps what its workload has, which is the same thing. The existing cached build tests pass with their expected output unchanged.The design is in
design/modelcache-mount-contract.md. #505 covers the OCI and PVC sources.Decisions. This fixes #495 rather than working towards it: a new replica waits for the cache, and so does a running replica that gains a cache or switches to another one. A replica reads its cache once a minute, because Crossplane doesn't watch required resources. Its engines start up to about a minute after the cache turns
Ready. File filters for the cache, an immutable source and the offline-loading docs are in a separate PR that builds on this one.Towards #494.
Both functions have new unit tests, including the kept mount for a Deployment and a LeaderWorkerSet. A new end-to-end test,
e2e/tests/test_model_cache.py, checks that a replica waits while the cache's claim is Pending, then mounts what the cache publishes. It runs withE2E_MODEL_CACHE=1, and uses a new hostPath each run so a previous run's data can't satisfy it. It passed on kind, along with manual checks that a running replica keeps its mount when its cache is deleted or when the deployment adds a cache that isn't ready.A running replica whose cache doesn't publish a mount emits the same Warning on every reconcile. That's a follow-up.
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.🤖 Generated with Claude Code