[mikemikimike]

With no model routes on an InferenceGateway, Envoy AI Gateway removes its ext-proc sidecar and rolls the Envoy Deployment. A new ModelRoute can report RoutingReady=True before the rollout completes, so requests return HTTP 500.

Fixes #531 (#531)

This keeps a backend-free AIGatewayRoute attached with an exact match on x-modelplane-gateway-anchor: "true", not the model header, so /v1/models lists only real models. GatewayReady waits for the anchor route, the AI Gateway ext-proc template marker, and the Envoy Deployment rollout. The EnvoyProxy pins the Deployment name so readiness observes the fixed Deployment; existing gateways replace the generated Deployment during the upgrade. While the gateway is unready, compose-model-route continues composing the route and keeps RoutingReady false.

Validation: all 631 composition-function tests passed. Ruff formatting and lint checks, the repository's ty checks, function-test structure checks, and docs manifest validation passed. The Nix flake check and kind-based integration suite have not been run on this Windows host.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

[dennis-upbound]

Thanks for picking this up. I applied the anchor route from this PR to a live GKE gateway running Envoy AI Gateway v1.1.0, the version the serving stack pins.

The approach works. With the anchor in place, I removed the gateway's last model route, and the Envoy Deployment kept the same ReplicaSet and pods. Without the anchor, the same step rolled it every time.

1. The anchor shows up in /v1/models. AI Gateway lists every exact match on x-ai-eg-model as a model (gateway.go at v1.1.0). The option to exclude a model from the list came after v1.1.0. With this route applied, GET /v1/models returned ["ml-team/qwen3-8b-uni", "modelplane-system/__gateway_anchor__"], and a request naming the anchor got HTTP 500. Matching on a header that isn't the model header, such as x-modelplane-gateway-anchor: "true", avoids this. Any AIGatewayRoute counts toward keeping ext-proc injected, whatever it matches. After that change, the route was still accepted, /v1/models listed only the real model, and the real model kept serving.

2. GatewayReady still turns True before ext-proc runs. AI Gateway accepts the route within a second, but the sidecar arrives with the Envoy Deployment's rollout, about 30 seconds later in #531's timeline. With the anchor in place, that window happens once, when the gateway is created, instead of with every first model, which is the main fix. A model deployed together with a new gateway can still get 500s while GatewayReady is True. Either say so in the description, or have GatewayReady wait for the rollout. An observe-only Object on the Envoy Deployment, whose readiness checks the rollout finished, would do it.

3. The description is missing the template's checklist, so checklist-completed fails. CONTRIBUTING.md also asks the description to open with the problem before Fixes #531.

[dennis-upbound]

Thanks, the header change fixes /v1/models, and the description reads well. Waiting for the Envoy rollout is the right idea. One blocking problem, and a few smaller ones:

1. Gating compose-model-route on GatewayReady deletes every model's routes whenever the gateway isn't Ready. resolve_inputs() returning False makes the function compose nothing, and Crossplane deletes composed resources the pipeline stops desiring: the AIGatewayRoute, backends and client certificate. GatewayReady now goes False whenever readyReplicas != replicas on the Envoy Deployment, so one Envoy pod restarting after a node drain, an Envoy Gateway upgrade or an eviction takes every model on the gateway offline. That happens while the other replica could still serve. I'd keep composing and gate only RoutingReady: report the route not ready until GatewayReady is True, but don't drop it. Or apply the gate only before the route first exists.
2. The rollout check should also require status.replicas == spec.replicas. During a rolling update, updatedReplicas, readyReplicas and availableReplicas can all equal spec.replicas while old pods, without the ext-proc sidecar, are still running and taking traffic. status.replicas counts those.
3. Set spec.watch: true on the observed Deployment. provider-kubernetes re-reads a not-ready Object every 30s, but a Ready one only on its hourly resync (#497). Without the watch, the gateway can report Ready for up to an hour after a later rollout starts.
4. Renaming the Envoy Deployment to modelplane-inference-gateway-envoy changes it on existing gateways. Does Envoy Gateway create the new Deployment before deleting the old one? If not, upgrading Modelplane briefly drops traffic on every gateway, and the description should say so.

[dennis-upbound]

Returning False here composes nothing, so Crossplane deletes this route's AIGatewayRoute, backends and certificate whenever GatewayReady is False, including when one Envoy pod restarts. I'd let composition continue and gate only RoutingReady on GatewayReady.

[mikemikimike]

Fixed in e0ccd09: when GatewayReady is false, an already observed route remains composed and RoutingReady stays false. Before a route exists, composition still waits. The GatewayRollout case covers this behavior.

[mikemikimike]

Addressed in e0ccd09: the rollout query requires status.replicas to equal spec.replicas, and the observed Deployment enables spec.watch. I removed the PR-added Envoy Deployment name override and observe the existing generated name, so this change does not rename deployed resources. All 630 composition-function tests passed; Ruff lint and formatting passed. The kind integration suite was not run.

[dennis-upbound]

Thanks. e0ccd09 fixes three of the four:

• An existing route keeps composing during a gateway rollout, with RoutingReady False.
• The rollout check now requires status.replicas == spec.replicas.
• The observed Deployment is watched.

One blocking problem with the Deployment name. Envoy Gateway names it envoy-<namespace>-<gateway>-<hash>. On two GKE clusters running this stack, the inference gateway's Deployment was envoy-modelplane-system-inference-gateway-5dbfd6c0. _ENVOY_DEPLOYMENT_NAME is modelplane-system-inference-gateway-5dbfd6c0, without the envoy- prefix. So the observe-only Object looks for a Deployment that doesn't exist, stays not Ready, and holds GatewayReady False. Every route that doesn't exist yet then waits forever, so no new model gets a route. The test fixtures use the same name, so they pass. Adding the prefix fixes it. A test that builds the name the way Envoy Gateway does would keep it from drifting.

[dennis-upbound]

Envoy Gateway prefixes this with envoy-: the live Deployment is envoy-modelplane-system-inference-gateway-5dbfd6c0. Without the prefix the observed Object never finds it, so GatewayReady stays False.

[mikemikimike]

Addressed in 824e861a:

• Pin envoyDeployment.name to envoy-modelplane-system-inference-gateway, so readiness observes the fixed Deployment instead of deriving a generated hash.
• Keep composing model routes while GatewayReady is false; RoutingReady remains false until the gateway can serve.
• Require the observed generation and a non-empty AI Gateway ext-proc config marker. Readiness compares updatedReplicas with the replicas currently reported by the Deployment, and requires at least one available replica. This allows one updated replica to serve while its peer is temporarily unavailable, but keeps readiness false while an old-template replica remains.
• Add regression coverage for rollout states and TLS plus auth reaching GatewayReady.

All 631 composition-function tests passed. Ruff formatting and lint, the repository's ty checks, function-test structure checks, and CEL readiness cases passed. The Nix flake check and kind-based integration suite were not run on this Windows host.

[negz]

Thanks for sticking with this. The anchor route looks like the right fix for #531. A few questions inline, mostly about how GatewayReady behaves now that model routes depend on it.

[negz]

Doesn't Envoy Gateway prefix this with envoy-? At v1.8.4 it builds the name in ExpectedResourceHashedName, which I think gives envoy-modelplane-system-inference-gateway-5dbfd6c0.

Rather than tracking Envoy Gateway's generated name, how about going back to pinning envoyDeployment.name on the EnvoyProxy, as an earlier revision did? Envoy Gateway uses the name when it's set, and a fixed name doesn't need the hash. I think renaming existing gateways is fine before 1.0.

[negz]

With two replicas and envoyPDB.maxUnavailable: 1, won't this turn GatewayReady False whenever one Envoy pod is drained or evicted? Now that compose-model-route follows GatewayReady, that would set RoutingReady False on every model on the gateway, and hold new routes, while the other pod keeps serving.

If I'm reading AI Gateway right, the sidecar is added by a pod webhook, so a replacement pod gets it anyway. The 500s would only come from old pods that are still running. Would checking observedGeneration, updatedReplicas == replicas == spec.replicas and Available be enough?

Suggested change

	"has(object.status.readyReplicas) && object.status.readyReplicas == object.spec.replicas && "
	"has(object.status.availableReplicas) && object.status.availableReplicas == object.spec.replicas && "

[mikemikimike]

Fixed in 824e861a. Readiness now compares updatedReplicas with the Deployment's currently reported replicas and requires at least one available replica. If the desired count is two but only one updated replica is currently reported and available, GatewayReady stays true while the peer is temporarily unavailable. If an old-template replica is still included in replicas, the counts differ and readiness stays false. The CEL acceptance cases cover both states.

[negz]

Do we need to hold composition until the gateway is Ready? With the anchor in place, I'd expect a route created during a rollout to be harmless. If so, always composing and gating only RoutingReady would let us drop this special case.

[negz]

Nit: with TLSAndAuth repurposed, does anything still cover a TLS+auth gateway reaching GatewayReady=True? Maybe worth keeping both cases.