Foreground deletion of an InferenceCluster can finish before its cloud resources are deleted #582

What happened?

kubectl delete ic eks-us-east --cascade=foreground removed the InferenceCluster immediately, while its EKSCluster and about 100 MRs were still deleting. The clean-up docs treat the InferenceCluster disappearing as the signal that Modelplane has deleted the cloud resources and it's safe to delete the control plane. Following them here would have orphaned those resources.

I'd already removed two GPU pools from the InferenceCluster, and their NodeGroups were still deleting (stuck behind in-progress creates). After the delete, the EKSCluster's ownerReference to the InferenceCluster had blockOwnerDeletion: false. It's normally true.

This looks like the Kubernetes garbage collector's cycle detection. When a dependent's own dependents are already being foreground-deleted, it marks the dependent's ownerReferences as non-blocking:

https://github.com/kubernetes/kubernetes/blob/v1.34.0/pkg/controller/garbagecollector/garbagecollector.go#L591-L613

Crossplane deletes composed resources with foreground propagation, so any composed resource that's mid-delete when you delete the InferenceCluster can trigger it.

A later, normal teardown of another InferenceCluster didn't hit this. The InferenceCluster stayed until Modelplane had deleted everything, and blockOwnerDeletion stayed true.

Related: #477.

How can we reproduce it?

1. Create an EKS InferenceCluster.
2. Remove a pool from it, or wait until any composed MR is deleting but not yet gone.
3. kubectl delete ic <name> --cascade=foreground --wait=false
4. The InferenceCluster disappears at once, while kubectl get ekscluster -A and kubectl get managed -A still show its resources.

What environment did it happen in?

Modelplane version: main plus #542 (e574dcf)

• Crossplane v2.4.0
• provider-aws-eks v2.8.1
• Control plane: kind