Repository navigation
AKS E2E Clouds placement audits fail on nvsentinel's NIC health monitor #567
Description
What happened?
The AKS leg of E2E Clouds fails on every PR that touches the serving stack. The placement audits flag nvsentinel's NIC health monitor:
FAILED e2e/clouds/test_placement.py::test_tolerations_are_scoped
'nvsentinel/nic-health-monitor-bhtnq tolerates the GPU taint'
'nvsentinel/nic-health-monitor-jm487 tolerates the GPU taint'
FAILED e2e/clouds/test_placement.py::test_only_gpu_workloads_on_the_gpu_pool
"nvsentinel/nic-health-monitor-bhtnq doesn't belong on fake GPU node kwok-gpu-1"
"nvsentinel/nic-health-monitor-jm487 doesn't belong on fake GPU node kwok-gpu-0"
#539 enabled the NIC health monitor in the AKS stack, and only there:
modelplane/functions/compose-serving-stack/function/stacks/clouds/generated/aicr/aks.py
Line 491 in 69ae1d7
| "nicHealthMonitor": {"enabled": True}, |
Like nvsentinel's other health monitors, it belongs on GPU nodes and tolerates the GPU taint. But it isn't in GPU_WORKLOADS or in AKS's gpu_workloads. The audits treat any pod missing from both as misplaced:
modelplane/e2e/clouds/test_placement.py
Lines 39 to 52 in 69ae1d7
| GPU_WORKLOADS = ( | |
| "node-feature-discovery-worker", | |
| "prometheus-node-exporter", | |
| "kubelet-plugin", | |
| "^gpu-feature-discovery", | |
| "^nvidia-dcgm", | |
| "^nvidia-mig-manager", | |
| "^nvidia-operator-validator", | |
| "^nvidia-cuda-validator", | |
| "^gpu-health-monitor", | |
| "^syslog-health-monitor", | |
| "^metadata-collector", | |
| "^platform-connectors", | |
| ) |
modelplane/e2e/clouds/__init__.py
Line 115 in 69ae1d7
| gpu_workloads=("^nvidia-toolkit-hardening",), |
E2E Clouds only runs on PRs that touch functions/compose-serving-stack/** or e2e/**, so main never ran it after #539 merged. #542 is the first PR to hit it (run).
#539's own AKS run passed, probably because the audits race nvsentinel's labeler. #569 tracks that.
How can we reproduce it?
- On main, run
nix run .#e2e -- --cloud aks.
What environment did it happen in?
Modelplane version: main (69ae1d7)
nvsentinel v1.25.0, on the KWOK-faked AKS leg of E2E Clouds.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
Allow nvsentinel's NIC health monitor on the AKS GPU pool