Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
8825816
docs: draft GPU Operator notes for DCGM Exporter 4.9.0 metric names
nccurry Oct 22, 2026
9e0e294
docs: update DCGM Exporter metric names for dcgm-exporter 4.9.0
nccurry Oct 22, 2026
5180a2a
docs: document custom DCGM Exporter metrics files in the GPU Operator
nccurry Oct 22, 2026
da9e0c8
docs: add missing dcgmExporter values and shared GPU metrics steps
nccurry Oct 22, 2026
27b587a
docs: extend draft release notes for DCGM Exporter 4.9.0 upgrades
nccurry Oct 22, 2026
b261a42
docs: limit the DCGM Exporter page to what the GPU Operator does
nccurry Oct 23, 2026
fe917b6
docs: retire the manual GPU telemetry and Prometheus setup pages
nccurry Oct 23, 2026
c850189
docs: shorten the draft GPU Operator notes for DCGM Exporter 4.9.0
nccurry Oct 23, 2026
e0bcbb6
docs: use the usual release-note sections in the draft GPU Operator n…
nccurry Oct 23, 2026
47b900c
docs: correct DCGM Exporter 4.9.0 details in the GPU Operator pages
nccurry Oct 26, 2026
b9960d8
docs: say DCGM Exporter 4.9.0 needs DCGM 4.7.0
nccurry Oct 28, 2026
94ad36b
docs: point to the tested DCGM version for DCGM Exporter 4.9.0
nccurry Oct 28, 2026
94474bf
docs: drop the life-cycle table TODO; the release adds its own column
nccurry Oct 29, 2026
b00923f
docs: describe DCGM Exporter metric names and DCGM pairing without na…
nccurry Oct 29, 2026
bee9c68
docs: drop the release-notes draft; the GPU Operator release notes ar…
nccurry Oct 29, 2026
768fb0c
docs: link to DCGM Exporter release notes for metric names and explai…
nccurry Oct 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion gpu-operator/getting-started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -421,8 +421,10 @@ To view all the options, run ``helm show values nvidia/gpu-operator``.
- ``{}``

* - ``dcgmExporter.enabled``
- By default, the Operator gathers GPU telemetry in Kubernetes using `DCGM Exporter <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html>`_.
- By default, the Operator gathers GPU telemetry in Kubernetes using `DCGM Exporter <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html>`_.
Set this value to ``false`` to disable it.
For all DCGM Exporter values, such as custom metrics files and the ``ServiceMonitor``, refer to
`DCGM Exporter Helm Values <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html#dcgm-exporter-helm-values>`_.
Available values are ``true`` (default) or ``false``.
- ``true``

Expand Down
29 changes: 28 additions & 1 deletion gpu-operator/gpu-sharing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,38 @@ the mixed MIG strategy.
Limitations
===========

- DCGM-Exporter does not support associating metrics to containers when GPU time-slicing is enabled with the NVIDIA Kubernetes Device Plugin.
- The Operator does not monitor changes to a time-slicing config map.
Refer to :ref:`time-slicing-update-config-map`.


.. _time-slicing-gpu-metrics:

GPU Metrics with Time-Slicing
=============================

By default, DCGM Exporter does not associate GPU metrics with the pods that share a GPU through time-slicing or MPS.
To associate them, set the ``KUBERNETES_VIRTUAL_GPUS`` environment variable to ``true`` in ``dcgmExporter.env``.
For example, in the Helm values:

.. code-block:: yaml

dcgmExporter:
env:
- name: KUBERNETES_VIRTUAL_GPUS
value: "true"

Or patch the cluster policy.
The patch replaces the whole ``env`` list, so include any variables that you already set:

.. code-block:: console

$ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type=merge \
-p '{"spec": {"dcgmExporter": {"env": [{"name": "KUBERNETES_VIRTUAL_GPUS", "value": "true"}]}}}'

For the other DCGM Exporter settings in the Operator, refer to
`DCGM Exporter with the NVIDIA GPU Operator <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html>`__.


Changes to Node Labels
======================

Expand Down
48 changes: 18 additions & 30 deletions gpu-telemetry/about-telemetry.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,39 +20,27 @@
About GPU Telemetry
###################

Monitoring stacks usually consist of a collector, a time-series database to store metrics and a visualization layer.
A popular open-source stack is `Prometheus <https://prometheus.io/>`_ used along with `Grafana <https://grafana.com/>`_ as
the visualization tool to create rich dashboards. Prometheus also includes an `Alertmanager <https://github.com/prometheus/alertmanager>`_,
to create and manage alerts. Prometheus is deployed along with `kube-state-metrics <https://github.com/kubernetes/kube-state-metrics>`_ and
`node_exporter <https://github.com/prometheus/node_exporter>`_ to expose cluster-level metrics for Kubernetes API objects and node-level
metrics such as CPU utilization.
The NVIDIA GPU Operator deploys NVIDIA DCGM Exporter on every GPU node by default.
DCGM Exporter reads GPU metrics from `NVIDIA DCGM <https://developer.nvidia.com/dcgm>`__
and exposes them for `Prometheus <https://prometheus.io/>`__.
You can view the metrics in Grafana or, on Red Hat OpenShift, in the web console.

This documentation covers what the GPU Operator does for GPU telemetry:

To gather GPU telemetry in Kubernetes, its recommended to use DCGM Exporter. DCGM Exporter, based on `DCGM <https://developer.nvidia.com/dcgm>`_ exposes
GPU metrics for Prometheus and can be visualized using Grafana. DCGM Exporter is architected to take advantage of
``KubeletPodResources`` `API <https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/>`_ and exposes GPU metrics in a format that can be
scraped by Prometheus. A ``ServiceMonitor`` is also included to expose endpoints.
* :doc:`dcgm-exporter` describes the resources that the Operator creates,
the Operator values for DCGM Exporter, custom metrics files, Prometheus scraping, and a separate DCGM host engine.
* :external+ocp:doc:`enable-gpu-monitoring-dashboard` describes the GPU dashboard in the OpenShift web console.

.. note::
DCGM and DCGM Exporter are deployed by default with the `NVIDIA GPU Operator <https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/>`_.
For DCGM Exporter itself, such as standalone and Helm installs, the configuration file, command-line options,
and the list of metrics, refer to the
`DCGM Exporter documentation <https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html>`__.


*************************
Benefits of GPU Telemetry
*************************

Understanding GPU usage provides important insights for IT administrators managing a data center.
Trends in GPU metrics correlate with workload behavior and make it possible to optimize resource allocation,
diagnose anomalies, and increase overall data center efficiency.
As GPUs become more mainstream, users would like to get access to GPU metrics to monitor GPU resources, just
like they do today for CPUs.

*************************
***********
Quick Links
*************************
***********

* `DCGM Exporter GitHub repository <https://github.com/NVIDIA/dcgm-exporter>`_
* `DCGM Documentation <https://docs.nvidia.com/datacenter/dcgm/latest/>`_
* `DCGM Exporter Documentation <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/dcgm-exporter.html>`_
* `Integrating GPU Telemetry into Kubernetes <https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/integrating-telemetry-kubernetes.html>`_
* `NVIDIA GPU Operator Documentation <https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/>`_
* `NVIDIA GPU Operator documentation <https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/>`__
* `DCGM Exporter documentation <https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html>`__
* `DCGM Exporter release notes <https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/dcgm-exporter.html>`__
* `DCGM Exporter GitHub repository <https://github.com/NVIDIA/dcgm-exporter>`__
* `DCGM documentation <https://docs.nvidia.com/datacenter/dcgm/latest/>`__
Loading
Loading