Skip to content

Install a DCGM exporter on Nebius clusters - #584

Open
ldalorion wants to merge 2 commits into
modelplaneai:mainfrom
ldalorion:nebius-dcgm-exporter
Open

ldalorion wants to merge 2 commits into
modelplaneai:mainfrom
ldalorion:nebius-dcgm-exporter

Conversation

@ldalorion

@ldalorion ldalorion commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Description of your changes

On a Nebius InferenceCluster every modelplane_gpu_* series and modelplane_energy_joules_total is empty, because nothing on the cluster exports DCGM (#583). This adds NVIDIA's standalone dcgm-exporter chart to stacks/clouds/nebius.py, with DCGM embedded, since no host engine runs on these nodes. The collector finds the pod by name and port, so the ServiceMonitor is off. The collector's comment that Modelplane doesn't install DCGM is updated to match.

It uses the standalone chart rather than the gpu-operator Civo enables in #526. Nebius's driver preset already installs the driver and a device plugin and labels its nodes nvidia.com/gpu.deploy.operands=false, so the operator would be there only to run the exporter, with every other operand switched off.

The counters are the shared generated/aicr/dcgm.py CSV, imported the way civo.py does, not the chart's own default list. The default covers every field metrics.py maps, but it's 24 fields to the shared list's 43, so Nebius would have no SM activity, NVLink or pipe-level series where every other cloud has them. A test pins the chart's customMetrics to the same independent literal Civo's cases use.

Validation: we've run the same chart and version, with a Nebius node selector instead of the NFD terms, on a Nebius cluster with 8 RTX PRO 6000 Blackwell GPUs since Oct 13, and every series metrics.py maps arrives, profiling ratios included, as do SM active and SM occupancy.

Fixes #583

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

@ldalorion
ldalorion force-pushed the nebius-dcgm-exporter branch from 0cb3a0d to 360923b Compare October 27, 2026 01:37

@haarchri haarchri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks taking care, can we add the same to vultr ?
One thing around failing e2e tests:
e2e/clouds/test_placement.py L39 (GPU_WORKLOADS)
The new DCGM pods are called mp-dcgm-exporter-xxxxx. The pattern ^nvidia-dcgm does not match this name, so two e2e tests will fail on Nebius. Can you add dcgm-exporter (without ^) to this list?

e2e/clouds/test_placement.py L57 (GPU_ONLY)
Please add dcgm-exporter here too. This list makes sure GPU-only pods never run on the normal (non-GPU) node. If the affinity breaks later, the test will catch it.

e2e/clouds/__init__.py L198 (Nebius expected_daemonsets)
Please add "dcgm-exporter/mp-dcgm-exporter": 2. Right now no test checks that the new DaemonSet actually runs on the two GPU nodes.

e2e/clouds/__init__.py L192 (Nebius labels)
Your comment in nebius.py says real Nebius GPU nodes have pci-0302_10de, not pci-10de. But the fake nodes here use pci-10de. Can we change this to feature.node.kubernetes.io/pci-0302_10de.present: "true"? Then the e2e test checks the real Nebius case. The DRA kubelet plugin will still be scheduled, because its default affinity also has the 0302 label.

Comment thread functions/compose-serving-stack/function/stacks/clouds/nebius.py
Comment thread functions/compose-serving-stack/function/stacks/clouds/nebius.py
Comment thread functions/compose-serving-stack/tests/test_stacks.py
@ldalorion
ldalorion force-pushed the nebius-dcgm-exporter branch from 8cf44c3 to fbba5a7 Compare October 28, 2026 02:56
@ldalorion

ldalorion commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

thanks taking care, can we add the same to vultr ?

On Vultr: VKE already runs an exporter. I checked on a live VKE cluster. VKE's managed GPU Operator (gpu-operator-v26.3.0) runs nvidia-dcgm-exporter on the GPU node, and it matches what the collector's modelplane-gpu job looks for: a pod named nvidia-dcgm-exporter-<hash> with a port named metrics (9400). Its /metrics serves real GPU data. Adding ours would double every GPU series.

But it isn't quite at parity with Nebius after this PR, though. It exports the operator's default field list rather than the generated CSV, and it has no DRA attribution. If you want parity, I can either update this pr or a follow-up PR that handles the exporter the way compose-vultr-cluster already handles the device plugin: label GPU pools nvidia.com/gpu.deploy.dcgm-exporter=false, and install the same exporter as Nebius.

I will say that getting a VKE cluster up at all took a workaround. VKE now rejects the hard-coded system pool plan vc2-6c-16gb (desired: vcg-a40-24c-120g-48vram) with Invalid NodePool plan (422). I'll open a separate issue for that. Created: #589

Signed-off-by: L. Dalorion Johnson <ldalorion@users.noreply.github.com>
Signed-off-by: L. Dalorion Johnson <ldalorion@users.noreply.github.com>
@haarchri
haarchri force-pushed the nebius-dcgm-exporter branch from fbba5a7 to 17ef5ea Compare October 30, 2026 08:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nebius cloud doesn't install DCGM exporter, so modelplane_gpu_* and energy metrics are empty

2 participants