Skip to content

feature: GPU node and Proxmox LXC check scripts - #702

Open
Flegma wants to merge 11 commits into
mainfrom
feature/gpu-node-check-scripts
Open

Flegma wants to merge 11 commits into
mainfrom
feature/gpu-node-check-scripts

Conversation

@Flegma

@Flegma Flegma commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Adds two check scripts for GPU nodes next to setup-gpu-nodes.sh, and a README for setting up a dedicated machine, a VM or a Proxmox LXC container.

  • gpu-node/gpu-node-check.sh checks a GPU node: driver, firmware, container runtime, CPU governor, disk and k3s version.
  • gpu-node/proxmox-gpu-lxc.sh CTID checks the Proxmox host and the container's settings.
  • Both only report unless --apply is passed, exit 1 while a check fails, and never touch GPU power or clock settings.
  • gpu-node/README.md covers NVIDIA and AMD, one vendor per panel as setup-gpu-nodes.sh sets it up.

gpu-node/gpu-node-check.sh checks a GPU node (driver, firmware, container
runtime, CPU governor, disk, k3s version) and gpu-node/proxmox-gpu-lxc.sh checks
a Proxmox host and its LXC container. Both only report unless --apply is given.
The README walks through dedicated machines, VMs and Proxmox LXC containers for
NVIDIA and AMD, one vendor per panel as setup-gpu-nodes.sh sets it up.
@lukepolo

lukepolo commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Reviewed at bb035b0. Report mode really is read-only, there's no injection path, nothing blacklists modules or writes GPU power or clock settings, and the LXC lines match the api script. --apply and the guidance need work before this goes in.

Must fix

  1. --apply can wipe the container config and still report success (proxmox-gpu-lxc.sh:144-159, :448). mktemp and the awk redirect aren't checked before cat "$tmp" >"$file". With awk failing, as on a full /tmp, the conf ends up 0 bytes with rc=0, so it prints [FIX ] and exits 0. backup_conf also ignores a cp failure. Check both, refuse an empty result, and abort if the backup fails. pve-common's own file_set_contents writes a temp file in the same directory and renames it, which contradicts the comment at :153, so please check that and do the same.
  2. The k3s pin does nothing with the current join command (README.md:18-25, :43, gpu-node-check.sh:551, :561-566). The panel now shows curl -fsSL https://get.5stack.gg | sudo sh -s -- join …. sudo drops an exported INSTALL_K3S_VERSION and the agent doesn't set it, so get.k3s.io installs latest, the skew the README warns about. It needs to go through sudo: curl … | sudo INSTALL_K3S_VERSION=v… sh -s -- join …. Please test that one, I've only read the path.
  3. proc:rw goes in without the host-wide warning (proxmox-gpu-lxc.sh:95, README.md:88-104). The api's node script warns that it lets the kubelet set vm.overcommit_memory=1, kernel.panic=10 and kernel.panic_on_oops=1 on the whole host, which then reboots 10 seconds after an oops. On a shared Proxmox host that takes every guest down. Put the warning in the README and print it from --apply.
  4. Wrong advice with a discrete card and an iGPU (gpu-node-check.sh:233-234, README.md:67-68). It says to select the node in the integrated list for any integrated device. Ticked, a node with a discrete card renders on the iGPU and ignores the card. servers/gpu-nodes/amd.md says to leave those unticked, so only say this when the integrated GPU is the only AMD GPU.
  5. The driver install's result message goes to /dev/null (gpu-node-check.sh:404-405). The >/dev/null sits on the apply_step call, so it swallows fixed and fail too, and "reboot, then run this again" never prints. Move the redirect into a wrapper.
  6. False PASS on /proc/sys (gpu-node-check.sh:599). It tests /proc/sys/net/ipv4/ip_forward, but LXC's proc:mixed leaves /proc/sys/net writable while /proc/sys/vm is read-only. Test /proc/sys/vm/overcommit_memory, as the api script does.
  7. The README points at flows that are gone (README.md:39, :44, :46, :83, :106). "Settings, Nodes" isn't a page, node creation is under /game-server-nodes. On a service install it's sudo 5stack run setup-gpu-nodes.sh, not ./setup-gpu-nodes.sh. There's also no step for getting the scripts onto a GPU node or Proxmox host, which won't have a panel checkout.
  8. Drop the amdgpu.runpm=0 advice (gpu-node-check.sh:243-244, README.md:64-66). The AMD docs say there's no kernel parameter to add, and it's GPU power management, which these scripts said they'd stay out of.

Smaller

  • proxmox-gpu-lxc.sh:383-386 compares disk size with a free-space requirement: size=120G passes, but the join needs 120 GB free. The pct resize … 128G hint likely falls short too.
  • proxmox-gpu-lxc.sh:370-375, :433: --apply on an unprivileged container records the FAIL and still adds all the lines. Stop there instead.
  • proxmox-gpu-lxc.sh:220-224: the nomodeset FAIL also fires on NVIDIA hosts, which gives an exit 1 that can't be fixed.
  • proxmox-gpu-lxc.sh:65-74: size=131072M reads as 0.
  • Use 5stack-cpu-governor.service, the name the panel's tuning uses, so a node doesn't end up with two units. enable --now also does nothing on an already-active unit, so its [FIX ] can be false.
  • README.md:13 says every script takes --apply, but setup-gpu-nodes.sh doesn't.

Please test

  • Run --apply on a real Proxmox host and a fresh node. Right now only the pure functions have tests.
  • Check gpu-node-check.sh:414 on a headless NVIDIA node. It requires /dev/nvidia-modeset, and the nvidia-modprobe -u -c0 hint won't create it (that needs -m).

Conflict: only .github/workflows/panel-scripts.yml. Keep both sides: add gpu-node/*.sh to main's shellcheck line, which now includes service.sh and utils/choose_node.sh, and add proxmox_gpu_lxc_test.sh after choose_node_test.sh.

…k-scripts

# Conflicts:
#	.github/workflows/panel-scripts.yml
- The config is rewritten like pve-common's file_set_contents: a temp file
  beside it, checked (awk result, not empty, every line kept), then renamed
  over it. Any failure leaves the config as it was and reports a FAIL.
- The backup is checked; when it fails, --apply changes nothing.
- --apply stops at an unprivileged container instead of adding the lines.
- proc:rw now comes with the warning that the kubelet then sets
  vm.overcommit_memory, kernel.panic and kernel.panic_on_oops on the whole
  host.
- The disk check asks for 120 GB free: measured inside a running container,
  from the size (140 GB with the system) in a stopped one. Sizes in M, K or
  bytes are read right (131072M is 128 GB).
- nomodeset and the amdgpu kernel arguments only fail an AMD host.
- The CPU governor uses 5stack-cpu-governor.service, the panel tuning's
  unit, is restarted rather than enabled with --now, and only reports a fix
  once every CPU runs performance. The NVIDIA device unit is checked the
  same way.
- An integrated AMD GPU is only ticked in setup-gpu-nodes.sh's integrated
  list when it is the node's only AMD GPU; beside a discrete card it stays
  unticked so the card renders.
- The amdgpu.runpm=0 advice is gone: there is no kernel parameter to add.
- The NVIDIA driver install keeps its own output quiet, so its FIX or FAIL
  line and the reboot note print again.
- The missing device node hint names nvidia-modprobe -m for
  /dev/nvidia-modeset.
- The k3s pin goes after sudo in the join command
  (sudo INSTALL_K3S_VERSION=... sh -s -- join), since sudo drops an
  exported variable.
- The LXC check tests /proc/sys/vm/overcommit_memory, which proc:mixed
  leaves read-only, instead of /proc/sys/net.
- The CPU governor uses 5stack-cpu-governor.service like the panel tuning,
  restarted and verified.
- The script can be sourced for its functions; gpu_node_check_test.sh
  covers them and runs in the panel scripts workflow.
- How to get the scripts onto a GPU node or Proxmox host.
- The k3s pin goes after sudo in the get.5stack.gg join command.
- Nodes are created on the Game Server Nodes page; setup-gpu-nodes.sh runs
  as sudo 5stack run setup-gpu-nodes.sh on a service install.
- Only the two check scripts take --apply.
- The proc:rw host-wide warning, the 120 GB free disk rule, the
  discrete-card rule for the integrated list, nvidia-modprobe -m, and no
  amdgpu.runpm=0.
- A running container with CS2 installed needs the existing install
  minimum (60 GB) free, not the fresh one; the disk messages name both
  panel settings.
- The governor unit also sets GOVERNOR="performance" in
  /etc/default/cpufrequtils when that file exists, as the panel's CPU
  governor tuning does.
- --apply with a missing or unprivileged container turns itself off
  before any host step, so a run that cannot finish changes nothing.
- A failed pct set or LXC line change no longer asks for a restart.
- Without nvidia-modprobe, --apply says it cannot add the device node
  unit instead of promising one.
- The host governor check reads CPU_SYSFS like the rest of the script.
- Only an AMD card bound to amdgpu counts as a discrete GPU for the
  integrated list advice; one on vfio-pci or without a driver does not.
- The governor unit also sets GOVERNOR="performance" in
  /etc/default/cpufrequtils when that file exists, as the panel's CPU
  governor tuning does.
- The k3s pin hint says it is untested: INSTALL_K3S_VERSION only takes
  effect if the join passes it through to the k3s installer.
- A test drives install_nvidia_driver through check_nvidia.
check_container stops after the unprivileged failure when --apply was
refused, like it does under --apply, so the report no longer offers fixes
that a rerun with --apply would refuse again. gpu-node-check.sh's
apply_step returns 1 on failure, like the Proxmox script's.
@Flegma

Flegma commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. All of it is fixed, first in a4b38de (with main merged in at d04a158 and the conflict resolved the way you described), then a second pass in f7f0f67, 615fc10, bb0b1e1 and 069b8b0 (head 069b8b0). The line numbers below are for 069b8b0.

Must fix

  1. The config rewrite now works like file_set_contents: a temp file beside the config, the awk result checked, an empty or short result refused, then mv over it. A failure leaves the config as it was and gives a FAIL (proxmox-gpu-lxc.sh:162). backup_conf checks the copy with cmp, and --apply stops if the backup fails (:609, :574, :594).
  2. The pin now goes after sudo: curl -fsSL https://get.5stack.gg | sudo INSTALL_K3S_VERSION=v… sh -s -- join … (README.md:39, gpu-node-check.sh:580). I haven't verified that it reaches k3s, though: the join hands off to the 5Stack agent and the panel's node script, and the variable only takes effect if they pass it through to the k3s installer. The README and the script's hint now say that and mark it as untested (README.md:32-36, gpu-node-check.sh:584).
  3. The proc:rw warning is in the README (README.md:139-144) and printed whenever the line is missing, in report mode and under --apply (proxmox-gpu-lxc.sh:513, :591).
  4. "Tick this node" now only shows when the integrated GPU is the only AMD GPU on amdgpu. With a discrete card it says to leave the node unticked (gpu-node-check.sh:201, README.md:92). A discrete card on vfio-pci or with no driver no longer counts as discrete, since it renders nothing on that node (gpu-node-check.sh:217).
  5. The redirect moved into an install_nvidia_driver wrapper, so the FIX or FAIL line and the reboot note print again (gpu-node-check.sh:382, :448).
  6. The LXC check now tests /proc/sys/vm/overcommit_memory (gpu-node-check.sh:31, :679).
  7. The README now uses the Game Server Nodes page (README.md:63), shows sudo 5stack run setup-gpu-nodes.sh for a service install (:69), and has a section on getting the scripts onto the node or host (:17).
  8. The amdgpu.runpm=0 advice is gone. The README says there is no kernel parameter to add (README.md:89).

Smaller

  • In a running container the disk check measures free space (pct exec … df). It now uses the same two minimums as the node setup: 60 GB once CS2 is installed (/opt/5stack/serverfiles/game/csgo exists), 120 GB before. The FAIL hint names both panel settings ("Required disk space", fresh and existing install). A stopped container still goes by size: under 120 G fails, with a note that a container that already has CS2 needs only 60 GB free once started; under 140 G warns. Resize hints are based on what is missing (proxmox-gpu-lxc.sh:473).
  • --apply with a missing or unprivileged container now turns itself off before the first host step, so such a run changes nothing on the host either (proxmox-gpu-lxc.sh:626), and the report no longer offers --apply fixes for that container (:534). check_container still stops there too (:529-538).
  • A failed pct set or LXC line change no longer asks for a container restart (proxmox-gpu-lxc.sh:576, :596).
  • With --apply and no nvidia-modprobe, the NVIDIA check now says it cannot add the device node unit and points to the .run installer, instead of promising a unit (proxmox-gpu-lxc.sh:318).
  • The nomodeset and amdgpu argument checks moved into check_host_amd, so they only run on AMD hosts (proxmox-gpu-lxc.sh:265).
  • size_gb now handles K, M, G, T and plain bytes, so 131072M reads as 128 (proxmox-gpu-lxc.sh:85).
  • Both scripts install the same governor script and 5stack-cpu-governor.service as the panel's CPU governor tuning, and, like it, set GOVERNOR="performance" in /etc/default/cpufrequtils when that file exists. They use enable plus restart, and only give [FIX ] once every CPU reports performance (proxmox-gpu-lxc.sh:409, gpu-node-check.sh:536). The host governor check now reads CPU_SYSFS like the rest of the script (proxmox-gpu-lxc.sh:396). The NVIDIA device unit gets the same restart and check (proxmox-gpu-lxc.sh:337).
  • README.md:13 now says only the two check scripts take --apply.

Tests

  • proxmox_gpu_lxc_test.sh has 34 tests. They cover failing, empty and cut-off awk rewrites (through a PATH shim), a read-only directory, replace by rename, a failed backup, a stopped unprivileged container, --apply and report output, disk free and size with a stubbed pct (fresh and with CS2 installed), the --apply refusal before the host steps (through main too), no restart after a failed change, the NVIDIA unit without nvidia-modprobe, nomodeset on NVIDIA, the cpufrequtils rewrite, and the governor fix failing when nothing changed.
  • New gpu_node_check_test.sh with 20 tests: the integrated and discrete cases (including a discrete card on vfio-pci), the nvidia-modprobe hints, the join hint and its caveat, the driver install result with a stubbed ubuntu-drivers, both directly and through check_nvidia, a read-only /proc/sys/vm probe, the governor unit and cpufrequtils, and usage errors. It is in the workflow. To make the script testable, it now runs main unless GPU_NODE_CHECK_LIB is set.
  • I reverted each of the second-pass fixes in a scratch copy, and each one turns at least one test red.
  • shellcheck 0.10.0 is clean on the workflow's file list. Of the 16 workflow test scripts, 14 pass on my machine. system_upgrade_test.sh (2 tests) and write_overlays_test.sh (2 tests) fail there only because it has no kubectl. CI has it.
  • I tested the sudo part in a throwaway ubuntu:24.04 container with a stand-in for get.5stack.gg. An exported INSTALL_K3S_VERSION is dropped by sudo sh, and the form after sudo reaches a child process.

Not tested

  • --apply on a real Proxmox host and on a fresh node. It is only covered by the tests above.
  • The full pinned join through the real 5stack agent. I couldn't check that the agent and the node script pass INSTALL_K3S_VERSION on to get.k3s.io, which is why the docs now call the pin untested.
  • The /dev/nvidia-modeset case on a real headless NVIDIA node. The hint now names nvidia-modprobe -m, but I haven't run it on hardware.
  • Report mode of gpu-node-check.sh was only run on one machine, an unprivileged LXC on an AMD APU host. There it correctly flagged the read-only /proc/sys/vm.

@lukepolo

lukepolo commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

I trimmed the comments, README and description myself in 0a02779, so pull before your next push. Script behaviour is unchanged; only some hint text moved or went.

The k3s pin caveat is gone from the scripts, the README and the test. I traced it on feature/agent: install.sh execs the agent, join runs in that process, and the runner passes its environment on, so INSTALL_K3S_VERSION reaches get.k3s.io. That's from reading the code, not a run.

Still needed before merge: --apply on a real Proxmox host and on a fresh node, plus one pinned join checked with k3s --version. The rest of your "Not tested" list stands too. I moved the --apply warning here from the description, since it applies until those runs are done.

…ff PATH

proxmox-gpu-lxc.sh counted a container that gets its GPU through Proxmox's
devN: entries (dev0: /dev/kfd,gid=993, dev1: /dev/dri/renderD128) as missing
the /dev/dri and /dev/kfd binds, and --apply would have added them a second
time. A devN entry for the device, or for a render node in the case of
/dev/dri, now counts.

gpu-node-check.sh looked for k3s on PATH only, which pct exec and some sudo
setups leave without /usr/local/bin, so an installed k3s read as not
installed.
@lukepolo

lukepolo commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Reviewed 6572980: the devN: passthrough and the k3s PATH fix are right, and the new tests fail without them.

Not merging yet: --apply still hasn't run on a real Proxmox host or a fresh node, and a pinned join hasn't been checked with k3s --version. Post the results of those runs here and this can go in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants