Repository navigation
EKS InferenceCluster reports only Creating when a node pool can't get instances #579
Description
What happened?
An EKS InferenceCluster whose GPU pool couldn't get instances sat in Creating for over 30 minutes with no reason given. The InferenceCluster, its EKSCluster and the NodeGroup MR only ever reported:
InferenceCluster: Ready=False Creating "Unready resources: eks-cluster"
EKSCluster: Ready=False Creating "Unready resources: nodegroup-<pool>"
NodeGroup: Ready=False Creating
EKS's own node group health was empty. The cause was only visible in the node group's Auto Scaling group activity, via the AWS CLI:
Could not launch On-Demand Instances. InsufficientInstanceCapacity - We currently do not have sufficient g6.12xlarge capacity in the Availability Zone you requested (us-east-1b).
Nothing on the control plane distinguishes a pool that's out of capacity from one that's still provisioning.
How can we reproduce it?
- Create an EKS InferenceCluster with a pool of an instance type that's out of capacity in its zone. g6.12xlarge, g5.12xlarge and g6e.12xlarge in us-east-1a/b/c were all out on 2026-10-26.
- Watch the InferenceCluster's conditions.
What environment did it happen in?
Modelplane version: main plus #542 (e574dcf)
- Crossplane v2.4.0
- provider-aws-eks v2.8.1
- EKS 1.36
Activity