Skip to content

Back node pools with CapacityReservations instead of capacityBlock - #600

Open
negz wants to merge 3 commits into
modelplaneai:mainfrom
negz:reserved-seating
Open

negz wants to merge 3 commits into
modelplaneai:mainfrom
negz:reserved-seating

Conversation

@negz

@negz negz commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Description of your changes

Towards #553.

An InferenceCluster node pool could only consume an AWS Capacity Block, through an EKS-only capacityBlock field that GKE pools ignored. There was no way to back a GKE pool with a Compute Engine reservation.

This PR adds CapacityReservation, a cluster-scoped resource that references an existing reservation, and replaces nodePools[].capacityBlock with nodePools[].reservationRef. It covers the v0.6 scope from this comment: GKE (any Compute Engine reservation, including calendar mode) and EKS (Capacity Blocks and On-Demand Capacity Reservations).

apiVersion: modelplane.ai/v1alpha1
kind: CapacityReservation
metadata:
  name: l4-reservation
spec:
  provider: GKE
  gke:
    zone: us-central1-a
    name: l4-reservation
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-reserved
spec:
  cluster:
    source: GKE
    gke:
      region: us-central1
  nodePools:
  - name: l4-pool
    className: gke-l4-1x-g2
    nodeCount: 2
    zones: [us-central1-a]
    reservationRef:
      name: l4-reservation

A pool consumes only its reservation and never falls back to on-demand capacity. On GKE the node pool uses a specific reservation affinity. On EKS the launch template targets the reservation, as capacityBlock did. compose-inference-cluster rejects a reservation for another cloud, another region, or (on GKE) a zone the pool isn't restricted to. It can't check the machine type or, on EKS, the Availability Zone, because Modelplane doesn't read the reservation from the cloud yet.

The first commit fixes a bug I hit while adding those checks. When a node pool's InferenceClass was missing or had no provisioning block for the cluster's cloud, compose-inference-cluster returned early without composing the cluster XR, so Crossplane would delete the cluster. Those paths, and the new reservation checks, now return a fatal result, so Crossplane leaves the cluster alone. A missing InferenceClass now shows as Synced=False rather than a waiting event.

Removing capacityBlock is a breaking change. An existing EKS pool that set it will lose its reservation on upgrade and needs a CapacityReservation and reservationRef.

I tested GKE and EKS against real clusters. On GKE, with a specific on-demand reservation for one g2-standard-8 in us-central1-c, the GPU node pool came up with SPECIFIC_RESERVATION affinity and the reservation reported 1 of 1 in use. On EKS, with an On-Demand Capacity Reservation for one g6.xlarge in us-east-1a, AWS accepted the launch template and the GPU node launched into the reservation. I didn't test a Capacity Block, which uses the same settings as before, or a GKE calendar-mode reservation. The GKE test also found that a fatal result stops Crossplane fetching required resources. The function now only fails once Crossplane has fetched a class or reservation and found it missing.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

When a node pool's InferenceClass was missing, or had no provisioning
block for the cluster's cloud, compose-inference-cluster returned early
with only a warning or normal result. It didn't compose the cluster XR
on that reconcile, and Crossplane deletes a composed resource that's
missing from desired state. Deleting an InferenceClass in use would
therefore delete the cluster built from it.

This makes those paths return a fatal result instead. Crossplane applies
none of a failed reconcile's desired state, so the existing cluster XR
is left alone, while the ClusterReady condition still says why.

Signed-off-by: Nic Cope <nicc@rk0n.org>
@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-600.vercel.app (ready once the site's Content workflow finishes)

negz added 2 commits October 29, 2026 18:34
An InferenceCluster node pool could only consume an AWS Capacity Block,
through an EKS-only capacityBlock field that GKE pools silently ignored.
There was no way to back a GKE pool with a Compute Engine reservation.

This adds CapacityReservation, a cluster-scoped resource that references
an existing reservation: a Compute Engine reservation on GKE, or a
Capacity Block or On-Demand Capacity Reservation on EKS. A node pool
references one by name, consumes only that reservation, and never falls
back to on-demand capacity. compose-inference-cluster fails the
reconcile if the reservation doesn't match the cluster's cloud, region,
or (on GKE) the pool's zone.

Removing capacityBlock is a breaking change.

Towards modelplaneai#553.

Signed-off-by: Nic Cope <nicc@rk0n.org>
Signed-off-by: Nic Cope <nicc@rk0n.org>
@negz
negz force-pushed the reserved-seating branch from 5b5b591 to 464b328 Compare October 30, 2026 01:36
@negz
negz marked this pull request as ready for review October 30, 2026 02:17
Copilot AI balanced review requested due to automatic review settings October 30, 2026 02:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@dennis-upbound dennis-upbound left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this reads well and matches the v0.6 scope on #553. The exact-zone check on GKE and the CEL rule that rejects reservationRef on other clouds both look right, and the first commit fixes a real way to lose a cluster. I'd like one change before merge, enforcing the immutability the docs describe, plus two smaller notes inline.


When a reservation ends, the cloud removes the nodes that consume it. To move
to a new reservation, add a node pool that references it, ideally before the
old reservation ends, then remove the old pool. Don't change the reservation

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can enforce this instead of documenting it. nodePools is a map list keyed by name, so a transition rule on each pool works. This one rejects changing or removing a pool's reservationRef, and still allows adding one, which an existing EKS pool needs after upgrading from capacityBlock. I tried it against an apiserver:

x-kubernetes-validations:
- rule: "!has(oldSelf.reservationRef) || (has(self.reservationRef) && self.reservationRef.name == oldSelf.reservationRef.name)"
  message: A node pool's reservationRef can't be changed or removed. Add a new pool instead.

The same change can also come in through the CapacityReservation. Editing spec.eks.id or spec.gke.name would point the pools that reference it at another reservation. A self == oldSelf rule on spec.gke and spec.eks, plus self.provider == oldSelf.provider on spec, closes that path. It fits a resource that stands for one cloud reservation.

Capacity Reservation (`type: OnDemand`). Modelplane doesn't create, change, or
delete the reservation itself.

When a reservation ends, the cloud removes the nodes that consume it. To move

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This holds for Capacity Blocks and GKE calendar-mode reservations. It doesn't hold for On-Demand Capacity Reservations: when one ends, its instances keep running as on-demand instances (AWS docs). Compute Engine won't delete a specific reservation that VMs still consume (GCP docs). Maybe "When a Capacity Block or a calendar-mode reservation ends, the cloud removes the nodes that consume it."


return True

def fail(self, reason: str, msg: str) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: a fatal result is better than deleting the cluster XR. But while a referenced CapacityReservation is missing, the whole InferenceCluster stops reconciling, including changes to its other pools. Deleting a reservation that a pool still uses is the likely way into that state. A ClusterUsage of each referenced CapacityReservation by the InferenceCluster, next to the existing deletion guard, would block it. That could be a follow-up.

@dennis-upbound
dennis-upbound self-requested a review October 30, 2026 03:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants