Repository navigation
Conversation
0cb3a0d to
360923b
Compare
haarchri
left a comment
There was a problem hiding this comment.
thanks taking care, can we add the same to vultr ?
One thing around failing e2e tests:
e2e/clouds/test_placement.py L39 (GPU_WORKLOADS)
The new DCGM pods are called mp-dcgm-exporter-xxxxx. The pattern ^nvidia-dcgm does not match this name, so two e2e tests will fail on Nebius. Can you add dcgm-exporter (without ^) to this list?
e2e/clouds/test_placement.py L57 (GPU_ONLY)
Please add dcgm-exporter here too. This list makes sure GPU-only pods never run on the normal (non-GPU) node. If the affinity breaks later, the test will catch it.
e2e/clouds/__init__.py L198 (Nebius expected_daemonsets)
Please add "dcgm-exporter/mp-dcgm-exporter": 2. Right now no test checks that the new DaemonSet actually runs on the two GPU nodes.
e2e/clouds/__init__.py L192 (Nebius labels)
Your comment in nebius.py says real Nebius GPU nodes have pci-0302_10de, not pci-10de. But the fake nodes here use pci-10de. Can we change this to feature.node.kubernetes.io/pci-0302_10de.present: "true"? Then the e2e test checks the real Nebius case. The DRA kubelet plugin will still be scheduled, because its default affinity also has the 0302 label.
8cf44c3 to
fbba5a7
Compare
On Vultr: VKE already runs an exporter. I checked on a live VKE cluster. VKE's managed GPU Operator ( But it isn't quite at parity with Nebius after this PR, though. It exports the operator's default field list rather than the generated CSV, and it has no DRA attribution. If you want parity, I can either update this pr or a follow-up PR that handles the exporter the way I will say that getting a VKE cluster up at all took a workaround. VKE now rejects the hard-coded system pool plan |
Signed-off-by: L. Dalorion Johnson <ldalorion@users.noreply.github.com>
Signed-off-by: L. Dalorion Johnson <ldalorion@users.noreply.github.com>
fbba5a7 to
17ef5ea
Compare
Description of your changes
On a Nebius
InferenceClustereverymodelplane_gpu_*series andmodelplane_energy_joules_totalis empty, because nothing on the cluster exports DCGM (#583). This adds NVIDIA's standalonedcgm-exporterchart tostacks/clouds/nebius.py, with DCGM embedded, since no host engine runs on these nodes. The collector finds the pod by name and port, so the ServiceMonitor is off. The collector's comment that Modelplane doesn't install DCGM is updated to match.It uses the standalone chart rather than the gpu-operator Civo enables in #526. Nebius's driver preset already installs the driver and a device plugin and labels its nodes
nvidia.com/gpu.deploy.operands=false, so the operator would be there only to run the exporter, with every other operand switched off.The counters are the shared
generated/aicr/dcgm.pyCSV, imported the waycivo.pydoes, not the chart's own default list. The default covers every fieldmetrics.pymaps, but it's 24 fields to the shared list's 43, so Nebius would have no SM activity, NVLink or pipe-level series where every other cloud has them. A test pins the chart'scustomMetricsto the same independent literal Civo's cases use.Validation: we've run the same chart and version, with a Nebius node selector instead of the NFD terms, on a Nebius cluster with 8 RTX PRO 6000 Blackwell GPUs since Oct 13, and every series
metrics.pymaps arrives, profiling ratios included, as do SM active and SM occupancy.Fixes #583
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.