Skip to content

cpick-kv-computedomain-crd-sync-main #2864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tariq1890 wants to merge 10 commits into from
Closed

cpick-kv-computedomain-crd-sync-main #2864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
cec9863
Mount host /lib/modules for precompiled drivers on SUSE
abrarshivani Aug 21, 2026
316f55b
Avoid advancing state on cleanup errors
kvalliyurnatt Aug 21, 2026
1b16164
Merge pull request #2793 from NVIDIA/backport-2782-to-release-26.7
rahulait Aug 24, 2026
37a3246
Merge pull request #2791 from NVIDIA/backport-2784-to-release-26.7
rahulait Aug 24, 2026
37a3f2b
Skip device plugin alert when devicePlugin is disabled in ClusterPolicy
harche Aug 26, 2026
3b8a06e
Merge pull request #2818 from NVIDIA/backport-2238-to-release-26.7
rahulait Aug 27, 2026
2346486
Stop defaulting deprecated cdi.default in ClusterPolicy
dentiny Aug 25, 2026
3483cc1
Merge pull request #2819 from NVIDIA/backport-2796-to-release-26.7
rahulait Aug 27, 2026
1715984
revert the CUDA samples version bump to v13.2 (#2848)
github-actions[bot] Sep 3, 2026
6e7ae70
Sync ComputeDomain CRD with DRA driver v0.5.0 schema
karthikvetrivel Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion api/nvidia/v1/clusterpolicy_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1966,7 +1966,6 @@ type CDIConfigSpec struct {

// Deprecated: This field is no longer used. Setting cdi.enabled=true will configure CDI as the default mechanism for making GPUs accessible to containers.
// +kubebuilder:validation:Optional
// +kubebuilder:default=false
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.displayName="Deprecated: This field is no longer used"
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors.x-descriptors="urn:alm:descriptor:com.tectonic.ui:booleanSwitch,urn:alm:descriptor:com.tectonic.ui:hidden"
Expand Down
5 changes: 4 additions & 1 deletion assets/state-node-status-exporter/0800_prometheus_rule_openshift.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@ spec:
- name: Alert on node deployment failure
rules:
- alert: GPUOperatorNodeDeploymentFailed
# There is no GPU exposed on the node,
# There is no GPU exposed on the node.
# When the device plugin is intentionally disabled in the ClusterPolicy
# (devicePlugin.enabled: false), the metric is set to -1, so this
# alert will not fire in that case.
expr: |
gpu_operator_node_device_plugin_devices_total == 0
for: 30m
Expand Down
1 change: 0 additions & 1 deletion bundle/manifests/nvidia.com_clusterpolicies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,6 @@ spec:
used in the cluster
properties:
default:
default: false
description: 'Deprecated: This field is no longer used. Setting
cdi.enabled=true will configure CDI as the default mechanism
for making GPUs accessible to containers.'
Expand Down
5 changes: 3 additions & 2 deletions bundle/manifests/resource.nvidia.com_computedomains.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,13 @@ spec:
- resourceClaimTemplate
type: object
numNodes:
default: 0
description: |-
Intended number of IMEX daemons (i.e., individual compute nodes) in the
ComputeDomain. Must be zero or greater.

With `featureGates.IMEXDaemonsWithDNSNames=true` (the default), this is
recommended to be set to zero. Workload must implement and consult its
recommended to be set to zero (default). Workload must implement and consult its
own source of truth for the number of workers online before trying to
share GPU memory (and hence triggering IMEX interaction). When non-zero,
`numNodes` is used only for automatically updating the global
Expand All @@ -91,10 +92,10 @@ spec:

The `numNodes` parameter is deprecated and will be removed in the next
API version.
minimum: 0
type: integer
required:
- channel
- numNodes
type: object
x-kubernetes-validations:
- message: A computeDomain.spec is immutable
Expand Down
10 changes: 9 additions & 1 deletion cmd/nvidia-validator/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,15 @@ func (nm *NodeMetrics) Run() error {
go nm.watchStatusFile(&nm.cudaReady, cudaStatusFile)

go nm.watchDriverValidation()
go nm.watchDevicePluginValidation()
if os.Getenv("DEVICE_PLUGIN_ENABLED") != "false" {
go nm.watchDevicePluginValidation()
} else {
// Set to -1 so the alert (expr: == 0) does not fire.
// The gauge is auto-registered by promauto and defaults to 0,
// which would be a false positive.
nm.deviceCount.Set(-1)
log.Info("metrics: DevicePlugin is disabled in ClusterPolicy, skipping device plugin validation")
}
go nm.watchNVIDIAPCI()

log.Printf("Running the metrics server, listening on :%d/metrics", nm.port)
Expand Down
1 change: 0 additions & 1 deletion config/crd/bases/nvidia.com_clusterpolicies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,6 @@ spec:
used in the cluster
properties:
default:
default: false
description: 'Deprecated: This field is no longer used. Setting
cdi.enabled=true will configure CDI as the default mechanism
for making GPUs accessible to containers.'
Expand Down
8 changes: 8 additions & 0 deletions controllers/object_controls.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,8 @@ const (
NvidiaDisableRequireEnvName = "NVIDIA_DISABLE_REQUIRE"
// GDSEnabledEnvName is the env name to enable GDS support with device-plugin
GDSEnabledEnvName = "GDS_ENABLED"
// DevicePluginEnabledEnvName indicates whether the device plugin is enabled in the ClusterPolicy
DevicePluginEnabledEnvName = "DEVICE_PLUGIN_ENABLED"
// MOFEDEnabledEnvName is the env name to enable MOFED devices injection with device-plugin
MOFEDEnabledEnvName = "MOFED_ENABLED"
// GDRCopyEnabledEnvName is the envvar that enables injection of the GDRCopy device node with the device-plugin
Expand Down Expand Up @@ -2568,6 +2570,12 @@ func TransformNodeStatusExporter(obj *appsv1.DaemonSet, config *gpuv1.ClusterPol
obj.Spec.Template.Spec.Containers[0].Args = config.NodeStatusExporter.Args
}

devicePluginEnabled := "true"
if !config.DevicePlugin.IsEnabled() {
devicePluginEnabled = "false"
}
setContainerEnv(&(obj.Spec.Template.Spec.Containers[0]), DevicePluginEnabledEnvName, devicePluginEnabled)

@coderabbitai coderabbitai Bot Sep 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Prevent NodeStatusExporter.Env from overriding DEVICE_PLUGIN_ENABLED.

NodeStatusExporter.Env is applied after this assignment. A ClusterPolicy that sets DEVICE_PLUGIN_ENABLED=true overrides the controller value when DevicePlugin.Enabled=false. The validator then starts device-plugin validation and can emit the false alert condition. Apply this reserved variable after user environment variables, or reject the reserved name.

Proposed fix 
-	setContainerEnv(&(obj.Spec.Template.Spec.Containers[0]), DevicePluginEnabledEnvName, devicePluginEnabled)
-
 	// set/append environment variables for exporter container
 	if len(config.NodeStatusExporter.Env) > 0 {
 		for _, env := range config.NodeStatusExporter.Env {
 			setContainerEnv(&(obj.Spec.Template.Spec.Containers[0]), env.Name, env.Value)
 		}
 	}
+	setContainerEnv(&(obj.Spec.Template.Spec.Containers[0]), DevicePluginEnabledEnvName, devicePluginEnabled)


// set/append environment variables for exporter container
if len(config.NodeStatusExporter.Env) > 0 {
for _, env := range config.NodeStatusExporter.Env {
Expand Down
2 changes: 1 addition & 1 deletion controllers/state_manager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -965,13 +965,13 @@ func (n *ClusterPolicyController) step() (gpuv1.State, error) {
if (n.stateNames[n.idx] == "state-driver" || n.stateNames[n.idx] == "state-vgpu-manager") &&
n.singleton.Spec.Driver.UseNvidiaDriverCRDType() {
n.logger.Info("NVIDIADriver CRD is enabled, cleaning up all NVIDIA driver daemonsets owned by ClusterPolicy")
n.idx++
// Cleanup all driver daemonsets owned by ClusterPolicy while keeping the
// running driver pods available until NVIDIADriver rolls replacements.
err := n.cleanupAllDriverDaemonSets(n.ctx, metav1.DeletePropagationOrphan)
if err != nil {
return gpuv1.NotReady, fmt.Errorf("failed to cleanup all NVIDIA driver daemonsets owned by ClusterPolicy: %w", err)
}
n.idx++
return gpuv1.Disabled, nil
}

Expand Down
74 changes: 74 additions & 0 deletions controllers/state_manager_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,80 @@ func TestGetGPUNodeOSInfoListError(t *testing.T) {
require.Contains(t, err.Error(), "unable to list nodes with GPU present")
}

func TestStep(t *testing.T) {
expectedErr := errors.New("step failed")
driverCRDPolicy := &gpuv1.ClusterPolicy{
Spec: gpuv1.ClusterPolicySpec{
Driver: gpuv1.DriverSpec{UseNvidiaDriverCRD: ptr.To(true)},
},
}

testCases := []struct {
name string
controller ClusterPolicyController
expectedStatus gpuv1.State
expectedIndex int
expectedError error
}{
{
name: "normal state advances on success",
controller: ClusterPolicyController{
controls: []controlFunc{{func(ClusterPolicyController) (gpuv1.State, error) { return gpuv1.Ready, nil }}},
stateNames: []string{"test-state"},
},
expectedStatus: gpuv1.Ready,
expectedIndex: 1,
},
{
name: "normal state does not advance on error",
controller: ClusterPolicyController{
controls: []controlFunc{{func(ClusterPolicyController) (gpuv1.State, error) { return gpuv1.NotReady, expectedErr }}},
stateNames: []string{"test-state"},
},
expectedStatus: gpuv1.NotReady,
expectedIndex: 0,
expectedError: expectedErr,
},
{
name: "driver cleanup advances on success",
controller: ClusterPolicyController{
ctx: context.Background(),
client: errorListClient{},
singleton: driverCRDPolicy,
stateNames: []string{"state-driver"},
},
expectedStatus: gpuv1.Disabled,
expectedIndex: 1,
},
{
name: "driver cleanup does not advance on error",
controller: ClusterPolicyController{
ctx: context.Background(),
client: errorListClient{err: expectedErr},
singleton: driverCRDPolicy,
stateNames: []string{"state-driver"},
},
expectedStatus: gpuv1.NotReady,
expectedIndex: 0,
expectedError: expectedErr,
},
}

for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
status, err := tc.controller.step()

if tc.expectedError != nil {
require.ErrorIs(t, err, tc.expectedError)
} else {
require.NoError(t, err)
}
require.Equal(t, tc.expectedStatus, status)
require.Equal(t, tc.expectedIndex, tc.controller.idx)
})
}
}

func TestGetGPUNodeOSInfoNoGPUNodes(t *testing.T) {
scheme := runtime.NewScheme()
require.NoError(t, corev1.AddToScheme(scheme))
Expand Down
29 changes: 29 additions & 0 deletions controllers/transforms_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3334,6 +3334,35 @@ func TestTransformNodeStatusExporter(t *testing.T) {
Name: "dummy",
Image: "nvcr.io/nvidia/cloud-native/node-status-exporter:v1.0.0",
ImagePullPolicy: corev1.PullIfNotPresent,
Env: []corev1.EnvVar{
{Name: DevicePluginEnabledEnvName, Value: "true"},
},
SecurityContext: &corev1.SecurityContext{
RunAsUser: rootUID,
},
}),
},
{
description: "node status exporter with device plugin disabled",
ds: NewDaemonset().
WithContainer(corev1.Container{Name: "dummy"}),
cpSpec: &gpuv1.ClusterPolicySpec{
NodeStatusExporter: gpuv1.NodeStatusExporterSpec{
Repository: "nvcr.io/nvidia/cloud-native",
Image: "node-status-exporter",
Version: "v1.0.0",
ImagePullPolicy: "IfNotPresent",
},
DevicePlugin: gpuv1.DevicePluginSpec{Enabled: newBoolPtr(false)},
},
expectedDs: NewDaemonset().
WithContainer(corev1.Container{
Name: "dummy",
Image: "nvcr.io/nvidia/cloud-native/node-status-exporter:v1.0.0",
ImagePullPolicy: corev1.PullIfNotPresent,
Env: []corev1.EnvVar{
{Name: DevicePluginEnabledEnvName, Value: "false"},
},
SecurityContext: &corev1.SecurityContext{
RunAsUser: rootUID,
},
Expand Down
1 change: 0 additions & 1 deletion deployments/gpu-operator/crds/nvidia.com_clusterpolicies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,6 @@ spec:
used in the cluster
properties:
default:
default: false
description: 'Deprecated: This field is no longer used. Setting
cdi.enabled=true will configure CDI as the default mechanism
for making GPUs accessible to containers.'
Expand Down
5 changes: 3 additions & 2 deletions deployments/gpu-operator/crds/resource.nvidia.com_computedomains.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,13 @@ spec:
- resourceClaimTemplate
type: object
numNodes:
default: 0
description: |-
Intended number of IMEX daemons (i.e., individual compute nodes) in the
ComputeDomain. Must be zero or greater.

With `featureGates.IMEXDaemonsWithDNSNames=true` (the default), this is
recommended to be set to zero. Workload must implement and consult its
recommended to be set to zero (default). Workload must implement and consult its
own source of truth for the number of workers online before trying to
share GPU memory (and hence triggering IMEX interaction). When non-zero,
`numNodes` is used only for automatically updating the global
Expand All @@ -91,10 +92,10 @@ spec:

The `numNodes` parameter is deprecated and will be removed in the next
API version.
minimum: 0
type: integer
required:
- channel
- numNodes
type: object
x-kubernetes-validations:
- message: A computeDomain.spec is immutable
Expand Down
7 changes: 5 additions & 2 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.

ARG CUDA_SAMPLES_VERSION=13.2
# R580 is the minimum driver branch supported by gpu-operator and
# its backcompat stretches all the way back to the Maxwell architecure.
# CUDA SAMPLES v12.9 is used as that is the latest version with Maxwell support.
ARG CUDA_SAMPLES_VERSION=12.9

FROM golang:1.26.6@sha256:0d1d3a794be25f809dd2cb3160d8c73276c4056a9f8242a138e908ddeee7b6b6 AS builder

Expand Down Expand Up @@ -64,7 +67,7 @@ WORKDIR /build
ARG SAMPLE_NAME=vectorAdd

RUN curl -L https://codeload.github.com/NVIDIA/cuda-samples/tar.gz/refs/tags/v${CUDA_SAMPLES_VERSION} | \
tar -xzvf - --strip-components=1 --wildcards */${SAMPLE_NAME}/* --wildcards */Common/* --wildcards */cmake/* && \
tar -xzvf - --strip-components=1 --wildcards */${SAMPLE_NAME}/* --wildcards */Common/* && \

@coderabbitai coderabbitai Bot Sep 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🔎 Supported by static analysis

🤖 get_repo_knowledge executed:

get_repo_knowledge NVIDIA/gpu-operator /tmp/coderabbit-repo-knowledge/nvidia-gpu-operator-eed98df4/conventions

Length of output: 1663

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- docker/Dockerfile (relevant range) ---'
nl -ba docker/Dockerfile | sed -n '1,90p'
printf '%s\n' '--- relevant diff ---'
git diff -- docker/Dockerfile
printf '%s\n' '--- archive-related references ---'
rg -n -C 3 'SAMPLE_NAME|cmake/Modules|cudaNvSci|cuda-samples' docker .github Makefile README.md 2>/dev/null || true
printf '%s\n' '--- upstream v12.9 sample contract ---'
curl -fsSL https://raw.githubusercontent.com/NVIDIA/cuda-samples/v12.9/Samples/4_CUDA_Libraries/cudaNvSci/CMakeLists.txt | sed -n '1,180p'
printf '%s\n' '--- upstream v12.9 module paths ---'
curl -fsSL https://api.github.com/repos/NVIDIA/cuda-samples/git/trees/v12.9?recursive=1 |
  jq -r '.tree[].path' | grep -E '(^|/)(cmake/Modules/FindNVSCI\.cmake|Samples/4_CUDA_Libraries/cudaNvSci/CMakeLists\.txt)$'

Repository: NVIDIA/gpu-operator

Length of output: 3538

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- docker/Dockerfile ---'
sed -n '1,82p' docker/Dockerfile
printf '%s\n' '--- exact diff ---'
git diff --no-ext-diff --unified=12 -- docker/Dockerfile
printf '%s\n' '--- v12.9 archive paths used by the extraction ---'
curl -fsSL https://codeload.github.com/NVIDIA/cuda-samples/tar.gz/refs/tags/v12.9 |
  tar -tzf - |
  grep -E '(^|/)(Samples/4_CUDA_Libraries/cudaNvSci/CMakeLists\.txt|Common/|cmake/Modules/FindNVSCI\.cmake)' |
  head -20

Repository: NVIDIA/gpu-operator

Length of output: 4053

Preserve cmake/Modules for cudaNvSci.

When SAMPLE_NAME=cudaNvSci, its CMakeLists.txt loads FindNVSCI.cmake from cmake/Modules. The extraction omits this file, so CMake can omit the target and the later cp ${SAMPLE_NAME} command can fail. Restore */cmake/* extraction or restrict supported SAMPLE_NAME values, and test each value.

cd $(find /build/Samples -iname "${SAMPLE_NAME}") && \
cmake . && \
make && \
Expand Down
Loading
Loading