Skip to content

AKS E2E Clouds placement audits fail on nvsentinel's NIC health monitor #567

Closed

Description

@negz

What happened?

The AKS leg of E2E Clouds fails on every PR that touches the serving stack. The placement audits flag nvsentinel's NIC health monitor:

FAILED e2e/clouds/test_placement.py::test_tolerations_are_scoped
  'nvsentinel/nic-health-monitor-bhtnq tolerates the GPU taint'
  'nvsentinel/nic-health-monitor-jm487 tolerates the GPU taint'
FAILED e2e/clouds/test_placement.py::test_only_gpu_workloads_on_the_gpu_pool
  "nvsentinel/nic-health-monitor-bhtnq doesn't belong on fake GPU node kwok-gpu-1"
  "nvsentinel/nic-health-monitor-jm487 doesn't belong on fake GPU node kwok-gpu-0"

#539 enabled the NIC health monitor in the AKS stack, and only there:

Like nvsentinel's other health monitors, it belongs on GPU nodes and tolerates the GPU taint. But it isn't in GPU_WORKLOADS or in AKS's gpu_workloads. The audits treat any pod missing from both as misplaced:

GPU_WORKLOADS = (
"node-feature-discovery-worker",
"prometheus-node-exporter",
"kubelet-plugin",
"^gpu-feature-discovery",
"^nvidia-dcgm",
"^nvidia-mig-manager",
"^nvidia-operator-validator",
"^nvidia-cuda-validator",
"^gpu-health-monitor",
"^syslog-health-monitor",
"^metadata-collector",
"^platform-connectors",
)

gpu_workloads=("^nvidia-toolkit-hardening",),

E2E Clouds only runs on PRs that touch functions/compose-serving-stack/** or e2e/**, so main never ran it after #539 merged. #542 is the first PR to hit it (run).

#539's own AKS run passed, probably because the audits race nvsentinel's labeler. #569 tracks that.

How can we reproduce it?

  1. On main, run nix run .#e2e -- --cloud aks.

What environment did it happen in?

Modelplane version: main (69ae1d7)

nvsentinel v1.25.0, on the KWOK-faked AKS leg of E2E Clouds.

Activity

  1. added
    bugSomething isn't working
    on Oct 26, 2026
  2. moved this from Backlog to In progress in Modelplane Roadmapon Oct 26, 2026
  3. self-assigned this
    on Oct 26, 2026
  4. added a commit that references this issue on Oct 26, 2026
    b6cbae1
  5. moved this from In progress to In Review in Modelplane Roadmapon Oct 26, 2026
  6. moved this from In Review to Done in Modelplane Roadmapon Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions