Skip to content

Move nvidia drivers to use overlayfs for the 6.1 and 6.12 kernels - #566

Open
vigh-m wants to merge 6 commits into
bottlerocket-os:developfrom
vigh-m:nvidia-multi-driver/kernel-kit
Open

vigh-m wants to merge 6 commits into
bottlerocket-os:developfrom
vigh-m:nvidia-multi-driver/kernel-kit

Conversation

@vigh-m

@vigh-m vigh-m commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes:

❯ diff packages/kmod-6.12-nvidia-r595 packages/kmod-6.18-nvidia-r595
diff packages/kmod-6.12-nvidia-r595/Cargo.toml packages/kmod-6.18-nvidia-r595/Cargo.toml
2c2
< name = "kmod-6_12-nvidia-r595"
---
> name = "kmod-6_18-nvidia-r595"
12c12
< package-name = "kmod-6.12-nvidia-r595"
---
> package-name = "kmod-6.18-nvidia-r595"
68c68
< kernel-6_12 = { path = "../kernel-6.12" }
---
> kernel-6_18 = { path = "../kernel-6.18" }
Only in packages/kmod-6.12-nvidia-r595: kmod-6.12-nvidia-r595.spec
Only in packages/kmod-6.18-nvidia-r595: kmod-6.18-nvidia-r595.spec


❯ diff packages/kmod-6.12-nvidia-r580 packages/kmod-6.18-nvidia-r580
diff packages/kmod-6.12-nvidia-r580/Cargo.toml packages/kmod-6.18-nvidia-r580/Cargo.toml
2c2
< name = "kmod-6_12-nvidia-r580"
---
> name = "kmod-6_18-nvidia-r580"
12c12
< package-name = "kmod-6.12-nvidia-r580"
---
> package-name = "kmod-6.18-nvidia-r580"
68c68
< kernel-6_12 = { path = "../kernel-6.12" }
---
> kernel-6_18 = { path = "../kernel-6.18" }
Only in packages/kmod-6.12-nvidia-r580: kmod-6.12-nvidia-r580.spec
Only in packages/kmod-6.18-nvidia-r580: kmod-6.18-nvidia-r580.spec
  • The diff between the 6.1 and 6.18 drivers is larger since 6.1 doesn't have nvidia-imex support
❯ diff packages/kmod-6.1-nvidia-r595 packages/kmod-6.18-nvidia-r595
diff packages/kmod-6.1-nvidia-r595/Cargo.toml packages/kmod-6.18-nvidia-r595/Cargo.toml
2c2
< name = "kmod-6_1-nvidia-r595"
---
> name = "kmod-6_18-nvidia-r595"
12c12
< package-name = "kmod-6.1-nvidia-r595"
---
> package-name = "kmod-6.18-nvidia-r595"
43a44,53
> url = "https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/x86_64/nvidia-imex-595.91.07-1.amzn2023.x86_64.rpm"
> sha512 = "94db467f6f50cbfdde5e4c4ccc46e576cb9c276aa64bf242356815b1c230869c6de5d98e944e5e18afdb38bcf7687a4eaecd226cc0a813afc95afd39352a5dd2"
> force-upstream = true
>
> [[package.metadata.build-package.external-files]]
> url = "https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/sbsa/nvidia-imex-595.91.07-1.amzn2023.aarch64.rpm"
> sha512 = "7d6233f5fc5b34a6388ac33955a555f44778941d7487381a346761267e37356b3a81a0719699eba13dd2054d409faecda7ff8f7645c39432b68862e97efb46e6"
> force-upstream = true
>
> [[package.metadata.build-package.external-files]]
58c68
< kernel-6_1 = { path = "../kernel-6.1" }
---
> kernel-6_18 = { path = "../kernel-6.18" }
Only in packages/kmod-6.1-nvidia-r595: kmod-6.1-nvidia-r595.spec
Only in packages/kmod-6.18-nvidia-r595: kmod-6.18-nvidia-r595.spec
Only in packages/kmod-6.18-nvidia-r595: nvidia-imex-default-channel.conf
Only in packages/kmod-6.18-nvidia-r595: nvidia-imex-tmpfiles.conf
Only in packages/kmod-6.18-nvidia-r595: nvidia-imex.cfg
Only in packages/kmod-6.18-nvidia-r595: nvidia-imex.service
diff packages/kmod-6.1-nvidia-r595/nvidia-pb-overlay-config.service.in packages/kmod-6.18-nvidia-r595/nvidia-pb-overlay-config.service.in
15a16
>     /etc/nvidia-imex \
17c18,20
<     /etc/.nvidia-config-overlay/work
---
>     /etc/.nvidia-config-overlay/work \
>     /etc/.nvidia-imex-config-overlay/upper \
>     /etc/.nvidia-imex-config-overlay/work
18a22,24
>
> # Writable so downstreams can supply the IMEX node list (nodes_config.cfg).
> ExecStart=/usr/bin/mount -t overlay nvidia-imex-etc --options rw,nosuid,nodev,noexec,noatime,context=system_u:object_r:etc_t:s0,lowerdir=__NVIDIA_ROOT__/etc/nvidia-imex:/etc/nvidia-imex,upperdir=/etc/.nvidia-imex-config-overlay/upper,workdir=/etc/.nvidia-imex-config-overlay/work /etc/nvidia-imex
diff packages/kmod-6.1-nvidia-r595/nvidia-tesla-pb-build-config.toml.in packages/kmod-6.18-nvidia-r595/nvidia-tesla-pb-build-config.toml.in
9c9
< link-objects = ["nvidia.o", "nvidia.mod.o"]
---
> link-objects = ["nvidia.o", "nvidia.mod.o", ".module-common.o"]
15c15
< link-objects = ["nvidia-modeset.o", "nvidia-modeset.mod.o"]
---
> link-objects = ["nvidia-modeset.o", "nvidia-modeset.mod.o", ".module-common.o"]
18c18
< link-objects = ["nvidia-uvm.o", "nvidia-uvm.mod.o"]
---
> link-objects = ["nvidia-uvm.o", "nvidia-uvm.mod.o", ".module-common.o"]


❯ diff packages/kmod-6.1-nvidia-r580 packages/kmod-6.18-nvidia-r580
diff packages/kmod-6.1-nvidia-r580/Cargo.toml packages/kmod-6.18-nvidia-r580/Cargo.toml
2c2
< name = "kmod-6_1-nvidia-r580"
---
> name = "kmod-6_18-nvidia-r580"
12c12
< package-name = "kmod-6.1-nvidia-r580"
---
> package-name = "kmod-6.18-nvidia-r580"
43a44,53
> url = "https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/x86_64/nvidia-imex-580.178.04-1.amzn2023.x86_64.rpm"
> sha512 = "89f4b38004f5d621da2b79e15f10ff46365544d418fdd1349dfa5598dd980738cf35f223953b7ce8fe061c29e64025c604696613985e72b08fef52816a3f286f"
> force-upstream = true
>
> [[package.metadata.build-package.external-files]]
> url = "https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/sbsa/nvidia-imex-580.178.04-1.amzn2023.aarch64.rpm"
> sha512 = "39e50f9bbf7d65c7926305757d0b71fec1f8529690d6f42dc66029bc05fd7db7f18027f4b2284ea00905313d32e908f0c53462edd0e3dfecc0609191a718594b"
> force-upstream = true
>
> [[package.metadata.build-package.external-files]]
58c68
< kernel-6_1 = { path = "../kernel-6.1" }
---
> kernel-6_18 = { path = "../kernel-6.18" }
Only in packages/kmod-6.1-nvidia-r580: kmod-6.1-nvidia-r580.spec
Only in packages/kmod-6.18-nvidia-r580: kmod-6.18-nvidia-r580.spec
Only in packages/kmod-6.18-nvidia-r580: nvidia-imex-default-channel.conf
Only in packages/kmod-6.18-nvidia-r580: nvidia-imex-tmpfiles.conf
Only in packages/kmod-6.18-nvidia-r580: nvidia-imex.cfg
Only in packages/kmod-6.18-nvidia-r580: nvidia-imex.service
diff packages/kmod-6.1-nvidia-r580/nvidia-lts-overlay-config.service.in packages/kmod-6.18-nvidia-r580/nvidia-lts-overlay-config.service.in
15a16
>     /etc/nvidia-imex \
17c18,20
<     /etc/.nvidia-config-overlay/work
---
>     /etc/.nvidia-config-overlay/work \
>     /etc/.nvidia-imex-config-overlay/upper \
>     /etc/.nvidia-imex-config-overlay/work
18a22,24
>
> # Writable so downstreams can supply the IMEX node list (nodes_config.cfg).
> ExecStart=/usr/bin/mount -t overlay nvidia-imex-etc --options rw,nosuid,nodev,noexec,noatime,context=system_u:object_r:etc_t:s0,lowerdir=__NVIDIA_ROOT__/etc/nvidia-imex:/etc/nvidia-imex,upperdir=/etc/.nvidia-imex-config-overlay/upper,workdir=/etc/.nvidia-imex-config-overlay/work /etc/nvidia-imex
diff packages/kmod-6.1-nvidia-r580/nvidia-tesla-lts-build-config.toml.in packages/kmod-6.18-nvidia-r580/nvidia-tesla-lts-build-config.toml.in
9c9
< link-objects = ["nvidia.o", "nvidia.mod.o"]
---
> link-objects = ["nvidia.o", "nvidia.mod.o", ".module-common.o"]
15c15
< link-objects = ["nvidia-modeset.o", "nvidia-modeset.mod.o"]
---
> link-objects = ["nvidia-modeset.o", "nvidia-modeset.mod.o", ".module-common.o"]
18c18
< link-objects = ["nvidia-uvm.o", "nvidia-uvm.mod.o"]
---
> link-objects = ["nvidia-uvm.o", "nvidia-uvm.mod.o", ".module-common.o"]

Testing done:

Tested with a custom AMI per kernel version for the following instance families: g4dn, g5, g6, g7

  • Both driver trees, /usr/share/.nvidia/lts and /usr/share/.nvidia/pb, are in the image.
  • Branch selection: g4dn, g5, and g6 select lts. g7 selects pb. Each node had exactly one /run/nvidia/branch-<branch> marker, and the loaded driver matched it.
  • Units: nvidia-select-branch and the selected branch's overlay-driver, overlay-config, and ldconfig units were active.
  • systemctl --failed was empty.
  • the host ldconfig -p lists the NVIDIA libraries, and libnvidia-ml.so.1 is at /usr/lib64
  • IMEX: kernel 6.1 has no nvidia-imex.service on either branch. Kernel 6.12 has the unit on both branches.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@vigh-m
vigh-m force-pushed the nvidia-multi-driver/kernel-kit branch from fe04084 to c80bb86 Compare September 24, 2026 18:28
@vigh-m
vigh-m force-pushed the nvidia-multi-driver/kernel-kit branch from c80bb86 to 0deca9c Compare October 5, 2026 17:16
vigh-m added 6 commits October 5, 2026 10:26
Declare mutual Conflicts among the kernel-6.1, kernel-6.12 and
kernel-6.18 packages.

Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
@vigh-m
vigh-m force-pushed the nvidia-multi-driver/kernel-kit branch from 0deca9c to 1ebb9cc Compare October 5, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants