E2E Clouds placement audits can run before the serving stack's pods exist #569

What happened?

E2E Clouds' placement audits can run before some of the serving stack's pods exist. A pod the audits don't see can't fail them, so a misplaced or over-tolerant workload can pass CI.

#567 probably got through this way. #539 enabled nvsentinel's NIC health monitor on AKS without adding it to the audits' allowlist, and #539's AKS run still passed. #542's two AKS runs failed on the monitor's pods. #542 doesn't change the nvsentinel config or the harness, but it does add the cluster-agent chart to the stack.

The audits wait only for the ServingStack to go Ready, then list pods once:

A Ready ServingStack means each chart with wait has rolled out its own workloads, and Helm has applied each chart without it. Pods can still appear after that in three ways:

1. A chart without wait applied late. Its DaemonSets' pods appear seconds after it's applied. On AKS, nvsentinel, the DRA driver, and prometheus-adapter are among the last charts applied, and none of them sets wait.
2. Workloads an operator creates. I believe the GPU operator creates its operand DaemonSets after its own pod starts. Helm's wait doesn't cover them. In Schedule replicas by free GPUs rather than free nodes #542's run they appeared about 13 seconds after the gpu-operator Release.
3. Workloads that select on labels a controller applies at runtime. nvsentinel's health monitors select nodes on labels its labeler applies, such as nvsentinel.dgxc.nvidia.com/driver.installed. Until it does, their DaemonSets target no nodes and count as rolled out, so wait can't help.

nvsentinel's monitors hit all three. The nvsentinel chart has no wait:

In #542's failing run, the monitors' pods existed about 90 seconds before the ServingStack went Ready. I haven't confirmed that #539's run audited before the labeler acted. The cluster-agent chart #542 adds is the only other difference I know of, and I can't see how it would affect the labeler.

The fan-out tests poll their DaemonSets, so they don't race. They only poll the DaemonSets in a cloud's expected_daemonsets, which includes none of nvsentinel's.

How can we reproduce it?

The race doesn't fail on its own. It hides other failures, like #567, depending on timing. To see it:

1. Remove an nvsentinel monitor, such as ^gpu-health-monitor, from GPU_WORKLOADS in e2e/clouds/test_placement.py.
2. Run nix run .#e2e -- --cloud aks a few times.

If the race is real, the audits will fail on some runs and pass on others.

What environment did it happen in?

Modelplane version: main (69ae1d7)

nvsentinel v1.25.0, on the KWOK-faked AKS leg of E2E Clouds. The other legs share the audits and the Ready wait, so they can hide failures the same way.