[negz]

Description of your changes

Fixes #172.

The fleet scheduler charges a whole node for every pod that claims a GPU, however many of the node's GPUs it claims. A one-GPU replica on a pool of four-GPU nodes leaves three idle, and a prefill and decode replica that fits on one eight-GPU node needs two.

This PR proposes scheduling by free GPUs in design/free-gpu-scheduling.md, and builds it. The serving stack installs the cluster agent on every InferenceCluster. The agent reports a histogram of each pool's nodes by free devices, which the InferenceCluster publishes in status.gpuPools. The scheduler fits each pod against it. A pool the agent doesn't report on falls back to whole-node accounting.

The histogram counts the devices scheduled pods hold as taken. To avoid counting them twice, the scheduler charges a replica's pods on a pool only until the histogram accounts for them. The agent records when it first saw each ResourceClaim allocated, and publishes the latest such time per pool on both the InferenceCluster and the ModelReplica. The histogram accounts for a replica once its ModelPodsScheduled condition is true and its time for the pool is no later than the pool's.

The scheduler can't know which node the cluster's scheduler binds each pod to, so a wrong guess can leave a pod Pending until devices free up. docs/content/architecture/scheduling.md lists this and the other known limitations. InferenceCluster pool names must now be DNS labels, because each is also the name of the PoolObservation the agent reports through.

I tested an earlier version on EKS with one g5.12xlarge (four A10Gs). Four one-GPU replicas shared the node, the scheduler refused a fifth, and scaling down freed a GPU for the next replica. That version stopped charging replicas on a one-minute timer rather than by allocation time, so it needs testing again.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

[Copilot]

🔵 Needs a closer look

The multinode reservation issue remains unresolved in a broad scheduling and infrastructure change.

2 open issues

• Preserve distinct-node reservations for multinode gang members · New
• Update PR status to reflect implemented serving-stack changes · New

What changed in this PR

Implements GPU-aware scheduling using per-node free-device histograms, with cluster-agent observations, schema updates, and documentation.

Changes:

• Adds pool and replica observation resources.
• Updates scheduling to account for free GPUs.
• Installs the cluster agent in serving stacks.
• Updates schemas, tests, and documentation.

File	Summary
schemas/​python/​models/​ai/​modelplane/​inferencecluster/​v1alpha1.py	Adds free-device histogram models.
schemas/​.lock.json	Updates schema lock data.
functions/​compose-serving-stack/​tests/​test_fn.py	Updates serving-stack fixtures.
functions/​compose-serving-stack/​function/​stacks/​common.py	Installs the cluster-agent chart. Moderate finding (4 votes): update the PR description/status to reflect the implementation.
functions/​compose-model-replica/​tests/​test_observation.py	Tests replica observations.
functions/​compose-model-replica/​tests/​test_fn.py	Updates replica composition tests.
functions/​compose-model-replica/​function/​observation.py	Handles replica observations and scheduling status.
functions/​compose-model-replica/​function/​fn.py	Integrates replica scheduling conditions.
functions/​compose-model-replica/​function/​backends/​base.py	Exposes replica-index labels.
functions/​compose-model-deployment/​tests/​test_scheduling.py	Tests histogram-based scheduling.
functions/​compose-model-deployment/​function/​scheduling.py	Implements histogram-based placement. Moderate finding (2 votes): preserve distinct-node reservations for multinode gangs.
functions/​compose-model-deployment/​function/​fn.py	Supplies scheduling time input.
functions/​compose-inference-cluster/​tests/​test_fn.py	Tests pool observation behavior.
functions/​compose-inference-cluster/​function/​fn.py	Publishes pool observations.
docs/​content/​platform/​inference-cluster.md	Documents agent pool labels.
docs/​content/​overview/​how-it-works.md	Updates the scheduling overview.
docs/​content/​architecture/​scheduling.md	Documents histogram accounting.
design/​free-gpu-scheduling.md	Adds the scheduling design.
apis/​inferenceclusters/​definition.yaml	Extends the InferenceCluster status schema.

_{Review effort: Lite}

[dennis-upbound]

Nice approach. The histogram of nodes by free devices is a clean answer to #172, and the allocation times neatly settle when a pool's view counts a replica. Most of this is about what node sharing newly exposes. I've left out what Chris and Copilot already raised.

Design

1. Mixed replica sizes can overcommit a pool permanently, because neither scheduler packs pods. The fleet scheduler predicts each pod goes to the node with the most free devices, and the cluster's scheduler doesn't pack DRA devices. Take a pool of four 4×L4 nodes running four 1-GPU replicas, probably one per node. 12 GPUs are free, but a 4-GPU replica fits on no node. It stays Pending, and Modelplane doesn't move it. Whole-node accounting couldn't overcommit like this. A P/D pair also isn't held to the one 8-GPU node the scheduler counted it against. Could the design state how pods are packed, and consider preferred pod affinity on kubernetes.io/hostname among Modelplane GPU pods, and affinity for P/D pairs? Either way, I'd bring "move a replica whose pods can't schedule" forward from Future improvements, or keep a known-limitations entry in scheduling.md to replace the #172 one this PR removes.
2. Sharing makes the CPU, memory and surge gaps real. Engine pods don't request CPU or memory, so two engines on one node compete for host memory (/dev/shm, offload), and the kubelet evicts them first under pressure. Engine workloads also roll with the default 25% surge, so on a node that sharing has filled to all four GPUs, any template change needs a surge pod with a GPU that doesn't exist. On a pool that can't grow, the rollout sticks. I'd do proportional CPU and memory requests in this PR, and size or disable surge for GPU pods.
3. Growth is counted even when the agent knows the pool can't grow. On GKE and Nebius, DRA pods can't trigger scale-up from zero (#521). A pool with observedNodes: 0 looks like nodes empty nodes. Every replica placed there is Pending until something else grows the pool. The agent already reports ScaleUpBlocked. I'd count no growth while it's True.
4. Other tenants and taints. GPUs that other workloads hold through the device plugin (nvidia.com/gpu) aren't counted as taken. Nodes with NoSchedule or NoExecute taints Modelplane pods don't tolerate, such as ToBeDeletedByClusterAutoscaler, count as free. design.md still says Modelplane has the cluster's GPUs to itself, which this relaxes.
5. Partitioned and shared devices. Free devices are counted by device ID, ignoring shared counters. So a GPU with one MIG slice allocated counts as free, and a device that allows multiple allocations counts as full after one claim. I'd declare both out of scope, or mark such pools as unobservable.
6. The timestamp scheme assumes one agent writes. The chart runs one replica with no leader election. After a node partition, two agents can run at once. Each records allocation times in its own memory, and together they can double-book a replica's GPUs. A Lease, and a sentence stating the single-writer requirement, would cover it.

Code (details inline)

7. The fit depends on the order of a copy's pods.
8. A partly scheduled replica is charged on top of the histogram.
9. PodsScheduled counts finished pods.
10. More than 256 buckets fails the whole status write rather than falling back.
11. A settled replica whose count grows isn't charged for the extra devices.
12. PoolObservation Objects can block InferenceCluster deletion.
13. Smaller ones:
    • A pool name that isn't a valid object name, like GPU_pool, silently falls back to whole-node accounting.
    • A new node whose DRA plugin hasn't published ResourceSlices yet briefly doesn't show any free devices.

Before merge

14. The agent can't be pulled anonymously. oci://xpkg.upbound.io/modelplane/cluster-agent:0.1.0 returns 404, and every InferenceCluster goes NotReady until it resolves.
15. The agent repo is private. An open-source Modelplane would depend on it.
16. The fields this reads exist only on the agent's joint-account branch. Its main has no lastAllocationTime.
17. The branch is behind main. The diff against main reverts dcgm.py and other files.

I reproduced 7 and 8 with small scripts against the scheduler, and checked 14 to 16 directly. The rest is from reading the code here and on the agent's joint-account branch.

[negz]

Replies to the design points in your review. I've replied to the code points inline.

1. You're right that this PR lets a pool overcommit where whole-node accounting refused. The scheduler refuses your four-replica example, because the histogram doesn't show a node with four free. But placing a 4-GPU replica and two 1-GPU replicas on two nodes before any schedule can strand the big one. I'd rather not steer the cluster with preferred pod affinity. The cluster's scheduler only weighs it against its other scores, so a pod can still go to another node, and inter-pod affinity slows scheduling in large clusters. I've added a known limitation to scheduling.md. If mispredictions like this turn out to be common, I'd rather rethink how the fleet scheduler places replicas than move them after the fact.

2. Agreed on both, though neither is new. The cluster could already pack pods onto one node, and a replica claiming a whole node already needed a second node to surge. Proportional CPU and memory requests stay the future improvement the design lists. The agent would have to report each pool's allocatable CPU and memory, and they'd need a policy for how much to request. I'd keep surge too. A Deployment whose surge pod can't fit stalls with the old pod still serving. Disabling surge would take every replica of a deployment down at once, since they all roll together. I've noted both in scheduling.md.

3. The scheduler counted growth this way before this PR too. #521 isn't quite this. The pool did scale from zero, then stopped adding nodes without an event saying why, so the agent would report ScaleUpBlocked Unknown rather than True. ScaleUpBlocked is also True when one pinned pod couldn't use a new node, like one claiming more than a node has, so counting no growth would stop the pool growing for every deployment on it. I'd leave this until we know how often pools stock out.

4. Good catch on taints. The agent now leaves out nodes with a NoSchedule or NoExecute taint that Modelplane's GPU pods don't tolerate, which is anything but nvidia.com/gpu with NoSchedule. That covers ToBeDeletedByClusterAutoscaler. GPUs held through the device plugin are outside what Modelplane supports, and the serving stack turns the plugin off where it installs the GPU operator. With both running, two allocators keep separate books over the same GPUs, which the cluster's own scheduler can't account for either. The agent reports a pool's extended resources, so it's visible. Counting other workloads' DRA claims as taken tolerates them, but Modelplane still assumes exclusive control of the cluster's GPUs, so I'd leave design.md as it is.

5. Modelplane doesn't support MIG or other partitioned GPUs yet, so I'd leave both until it does. Counting by name is right for whole GPUs, which is all Modelplane claims today. A device that allows several allocations counts as taken after the first, which errs toward the pool looking full.

6. Good catch, I've enabled leader election.

[negz]

@dennis-upbound following up on the rest of your review.

On 13, pool names must now be DNS labels, so the InferenceCluster's schema rejects a name like GPU_pool rather than the pool silently falling back. A new node whose DRA plugin hasn't published its ResourceSlices yet counts as a node with nothing free until it does, which errs toward the pool looking full.

On 14 to 17, the agent's v0.1.0 is released and its chart and image pull anonymously, the repo is public, lastAllocationTime is on its main branch, and this branch is rebased on main.

[negz]

I tested this on EKS: a control plane built from this branch with nix run .#run, and one InferenceCluster in us-east-2 with a single g4dn.12xlarge pool (4× T4). Every change went through the control plane, except where a test simulated something outside Modelplane. Everything behaved as designed:

• Four one-GPU replicas shared the node, and the scheduler refused a fifth. It refused both while the pool's report lagged the replicas' and after it caught up.
• A two-GPU replica was refused while one GPU was free, then placed on two GPUs of the same node once two were.
• Scaling one deployment down freed a GPU, and a waiting deployment got it.
• A GPU claimed by a pod that isn't Modelplane's counted as taken.
• A cordoned node dropped out of the pool's capacity, and uncordoning it restored it.
• With the agent scaled to 0, freed GPUs stayed counted as taken. Once it came back they became schedulable.
• Pool names that aren't DNS labels are rejected.

provider-kubernetes stops receiving watch events once the EKS kubeconfig token expires, after 15 minutes (crossplane-contrib/provider-kubernetes#598). After that the agent's reports only arrived on the 10-minute poll, so a freed GPU took about 8 minutes to become schedulable. c2fdafa polls the observations every 30 seconds until that's fixed. I haven't tested that commit on a cluster.

As expected, two deployments waiting on a full pool both placed a replica on the same freed GPU, and one stayed Pending.

[Copilot]

🟡 Changes recommended

Unresolved compatibility, capacity-fallback, replica-accounting, and stale-status issues can cause reconciliation failures or incorrect scheduling decisions.

2 open issues

• Preserve existing pool names during schema migration · New
• Clear stale pool allocations when agent reports none · New

2 resolved since last review

• Update PR status to reflect implemented serving-stack changes
• Preserve distinct-node reservations for multinode gang members

_{Review effort: Lite}

[dennis-upbound]

Thanks for working through the last round. Leaving out tainted nodes and turning on leader election close the gaps I was most worried about, and the known limitations in scheduling.md read well. I agree the scheduler follow-ups can come later. There's one thing I'd fix before merge and one small one, both inline, plus a nit below.

One nit outside the diff: in functions/compose-model-replica/function/backends/base.py, the comment on LABEL_DEPLOYMENT, LABEL_REPLICA and LABEL_ENGINE (line 261) says nothing selects on these labels, but the ReplicaObservation's pod selector now selects on all three. If they're renamed in a telemetry change, every replica on a pool the agent reports on would quietly be charged twice. Worth updating the comment.

[negz]

@dennis-upbound thanks! I've also fixed the comment on LABEL_DEPLOYMENT, LABEL_REPLICA and LABEL_ENGINE. It now says the ReplicaObservation selects pods by them, so renaming them breaks the scheduler's accounting.

[Copilot]

🔵 Needs a closer look

Approval should be deferred because the change introduces complex cluster-wide agent, Helm/CRD, DRA allocation, and fleet-placement behavior that static review cannot validate without live EKS runtime testing.

No open issues 🎉

2 resolved since last review

• Preserve existing pool names during schema migration
• Clear stale pool allocations when agent reports none

_{Review effort: Lite}