Repository navigation
Conversation
When a node pool's InferenceClass was missing, or had no provisioning block for the cluster's cloud, compose-inference-cluster returned early with only a warning or normal result. It didn't compose the cluster XR on that reconcile, and Crossplane deletes a composed resource that's missing from desired state. Deleting an InferenceClass in use would therefore delete the cluster built from it. This makes those paths return a fatal result instead. Crossplane applies none of a failed reconcile's desired state, so the existing cluster XR is left alone, while the ClusterReady condition still says why. Signed-off-by: Nic Cope <nicc@rk0n.org>
|
Docs preview: https://modelplane-docs-pr-600.vercel.app (ready once the site's Content workflow finishes) |
An InferenceCluster node pool could only consume an AWS Capacity Block, through an EKS-only capacityBlock field that GKE pools silently ignored. There was no way to back a GKE pool with a Compute Engine reservation. This adds CapacityReservation, a cluster-scoped resource that references an existing reservation: a Compute Engine reservation on GKE, or a Capacity Block or On-Demand Capacity Reservation on EKS. A node pool references one by name, consumes only that reservation, and never falls back to on-demand capacity. compose-inference-cluster fails the reconcile if the reservation doesn't match the cluster's cloud, region, or (on GKE) the pool's zone. Removing capacityBlock is a breaking change. Towards modelplaneai#553. Signed-off-by: Nic Cope <nicc@rk0n.org>
Signed-off-by: Nic Cope <nicc@rk0n.org>
5b5b591 to
464b328
Compare
dennis-upbound
left a comment
There was a problem hiding this comment.
Thanks, this reads well and matches the v0.6 scope on #553. The exact-zone check on GKE and the CEL rule that rejects reservationRef on other clouds both look right, and the first commit fixes a real way to lose a cluster. I'd like one change before merge, enforcing the immutability the docs describe, plus two smaller notes inline.
|
|
||
| When a reservation ends, the cloud removes the nodes that consume it. To move | ||
| to a new reservation, add a node pool that references it, ideally before the | ||
| old reservation ends, then remove the old pool. Don't change the reservation |
There was a problem hiding this comment.
We can enforce this instead of documenting it. nodePools is a map list keyed by name, so a transition rule on each pool works. This one rejects changing or removing a pool's reservationRef, and still allows adding one, which an existing EKS pool needs after upgrading from capacityBlock. I tried it against an apiserver:
x-kubernetes-validations:
- rule: "!has(oldSelf.reservationRef) || (has(self.reservationRef) && self.reservationRef.name == oldSelf.reservationRef.name)"
message: A node pool's reservationRef can't be changed or removed. Add a new pool instead.The same change can also come in through the CapacityReservation. Editing spec.eks.id or spec.gke.name would point the pools that reference it at another reservation. A self == oldSelf rule on spec.gke and spec.eks, plus self.provider == oldSelf.provider on spec, closes that path. It fits a resource that stands for one cloud reservation.
| Capacity Reservation (`type: OnDemand`). Modelplane doesn't create, change, or | ||
| delete the reservation itself. | ||
|
|
||
| When a reservation ends, the cloud removes the nodes that consume it. To move |
There was a problem hiding this comment.
This holds for Capacity Blocks and GKE calendar-mode reservations. It doesn't hold for On-Demand Capacity Reservations: when one ends, its instances keep running as on-demand instances (AWS docs). Compute Engine won't delete a specific reservation that VMs still consume (GCP docs). Maybe "When a Capacity Block or a calendar-mode reservation ends, the cloud removes the nodes that consume it."
|
|
||
| return True | ||
|
|
||
| def fail(self, reason: str, msg: str) -> None: |
There was a problem hiding this comment.
Non-blocking: a fatal result is better than deleting the cluster XR. But while a referenced CapacityReservation is missing, the whole InferenceCluster stops reconciling, including changes to its other pools. Deleting a reservation that a pool still uses is the likely way into that state. A ClusterUsage of each referenced CapacityReservation by the InferenceCluster, next to the existing deletion guard, would block it. That could be a follow-up.
Description of your changes
Towards #553.
An InferenceCluster node pool could only consume an AWS Capacity Block, through an EKS-only
capacityBlockfield that GKE pools ignored. There was no way to back a GKE pool with a Compute Engine reservation.This PR adds
CapacityReservation, a cluster-scoped resource that references an existing reservation, and replacesnodePools[].capacityBlockwithnodePools[].reservationRef. It covers the v0.6 scope from this comment: GKE (any Compute Engine reservation, including calendar mode) and EKS (Capacity Blocks and On-Demand Capacity Reservations).A pool consumes only its reservation and never falls back to on-demand capacity. On GKE the node pool uses a specific reservation affinity. On EKS the launch template targets the reservation, as
capacityBlockdid.compose-inference-clusterrejects a reservation for another cloud, another region, or (on GKE) a zone the pool isn't restricted to. It can't check the machine type or, on EKS, the Availability Zone, because Modelplane doesn't read the reservation from the cloud yet.The first commit fixes a bug I hit while adding those checks. When a node pool's InferenceClass was missing or had no provisioning block for the cluster's cloud,
compose-inference-clusterreturned early without composing the cluster XR, so Crossplane would delete the cluster. Those paths, and the new reservation checks, now return a fatal result, so Crossplane leaves the cluster alone. A missing InferenceClass now shows asSynced=Falserather than a waiting event.Removing
capacityBlockis a breaking change. An existing EKS pool that set it will lose its reservation on upgrade and needs aCapacityReservationandreservationRef.I tested GKE and EKS against real clusters. On GKE, with a specific on-demand reservation for one
g2-standard-8inus-central1-c, the GPU node pool came up withSPECIFIC_RESERVATIONaffinity and the reservation reported 1 of 1 in use. On EKS, with an On-Demand Capacity Reservation for oneg6.xlargeinus-east-1a, AWS accepted the launch template and the GPU node launched into the reservation. I didn't test a Capacity Block, which uses the same settings as before, or a GKE calendar-mode reservation. The GKE test also found that a fatal result stops Crossplane fetching required resources. The function now only fails once Crossplane has fetched a class or reservation and found it missing.I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.