[dennis-upbound]

Description of your changes

Fixes #496.

A Standalone engine's Deployment used Kubernetes' default rolling update. With fewer than four copies, the controller adds one surge pod and waits for it to be available before it removes an old pod. It cleans up unhealthy old pods only inside that budget (maxScaledDown = allPods - (copies - maxUnavailable) - newRSUnavailable in rolling.go), which is zero while the surge pod is Pending. On a pool with no free device the surge pod can't schedule until an old pod releases its claim. In both #496 incidents the old pod was restarting and not Ready, so the config fix never rolled out and nothing served.

This sets spec.strategy from the Deployment's observed status, which compose-model-replica already reads from the Object's status.atProvider.manifest:

• maxSurge is 25%, as today.
• Once the Deployment controller marks the rollout Progressing False with reason ProgressDeadlineExceeded, maxUnavailable is the number of copies that aren't available, when that's more than the 25% default allows. Otherwise it's 25%.

The controller sets that condition once progressDeadlineSeconds (600 by default) have passed with no progress, using its own clock and requeue (progress.go). It checks the deadline when it next syncs the Deployment, so the condition can come later. On the kind e2e it took 10 to 18 minutes. Any progress, such as a pod becoming ready or an old pod going away, sets it back to True, and the strategy returns to the defaults. So a crash-looping engine's fix rolls out at least ten minutes after it's applied: the controller removes the old pods that aren't serving, and their devices go to the new pods.

Until a rollout stalls, a Deployment renders {type: RollingUpdate, rollingUpdate: {maxSurge: 25%, maxUnavailable: 25%}}. That's exactly what SetDefaults_Deployment in pkg/apis/apps/v1/defaults.go fills in, so on upgrade the patch matches the live spec, and the Deployment's generation and pods don't change. The controller hashes only spec.template for a ReplicaSet (sync.go), so a strategy change doesn't start a new ReplicaSet either.

Backend.build takes the observed resources as a keyword-only argument. The llm-d and Grove backends ignore it. LeaderWorkerSet (maxSurge: 0, maxUnavailable: 1) and Grove (RollingRecreate) replace a gang before starting its successor by default, so they can't deadlock this way.

Decisions:

• The override waits for the progress deadline. Acting on any observed unready copy would let one readiness failure take down a copy that recovers a moment later, and a timer of our own would need a requeue and a timeout to pick. The cost is a wait of at least ten minutes before a stuck fix rolls out. A copy that goes unready after a rollout has already passed its deadline still counts as down. A crash-looping pod that turns Ready between crashes resets the deadline each time.
• A rollout of a healthy engine on a full pool keeps stalling with the old pod serving, as on main. Every copy is available, so none counts as down even past the deadline. That avoids a surprise outage on a config change, at the cost of a rollout that waits for room. Replacing first, with an opt-in rollout: Surge | Replace or a strategy picked from Schedule replicas by free GPUs rather than free nodes #542's free-device histogram, would trade that stall for an outage. Picking surge or replace from the histogram is future work once Schedule replicas by free GPUs rather than free nodes #542 merges. A ModelReady that reports the stall is ModelDeployment and ModelEndpoint report Ready while the engine has no ready pods #497.

I ran this on the kind e2e with the fake DRA driver and a pod that claimed every free device:

• A crash-looping 1-copy engine completed its rollout after a fix. ProgressDeadlineExceeded appeared 10 to 18 minutes after the fix, and the function set maxUnavailable: 1 within 2 seconds of it. The controller deleted the old pod, the new pod became ready, and the strategy returned to 25%. The strategy changes didn't create a ReplicaSet, so the rollout ended with the old one and the fix's.
• A plain Deployment with the default strategy in the same setup kept its new pod Pending for 9 minutes. Patching in maxUnavailable: 1 cleared it in 3 seconds.
• A Ready 1-copy engine stalled with the old pod serving, before and after the deadline. All 328 requests during the stall returned 200, and the strategy stayed at 25%.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

🤖 Generated with Claude Code