AMD GPU pools get whole-node scheduling because the cluster agent only tolerates the NVIDIA taint #601

What problem are you facing?

Modelplane only schedules individual GPUs on NVIDIA pools. On an AMD GPU pool a replica that claims one GPU takes a whole node, so an 8x MI325X node serves one single-GPU replica instead of eight.

Per-GPU scheduling depends on the cluster agent observing the pool. The agent counts a node as schedulable only if every NoSchedule or NoExecute taint on it is nvidia.com/gpu:

https://github.com/modelplaneai/cluster-agent/blob/ca7631d72ef8550a2d2f415f33db1ec52b20c65c/internal/pool/pool.go#L457-L477

The agent would report an AMD pool tainted amd.com/gpu as having no schedulable nodes, so the scheduler couldn't place a replica there. To avoid that, #556 only asks the agent to observe pools whose class claims gpu.nvidia.com:

https://github.com/modelplaneai/modelplane/blob/4693b4b9/functions/compose-inference-cluster/function/fn.py#L264-L272

The scheduler falls back to whole-node accounting for any pool the agent doesn't observe:

https://github.com/modelplaneai/modelplane/blob/4693b4b9/functions/compose-model-deployment/function/scheduling.py#L76-L78

CPU pools have the same limitation for the same reason: engine pods tolerate modelplane.ai/cpu, but the agent doesn't.

How could Modelplane help solve your problem?

The cluster agent should treat a node as schedulable when its taints are ones Modelplane's engine pods tolerate: nvidia.com/gpu, amd.com/gpu and modelplane.ai/cpu today. compose-inference-cluster could then observe every pool rather than only NVIDIA ones.