A small Linux kernel module that gives NVIDIA GPUs behind PCIe switches a full-size BAR1 (e.g. 32 GB on an RTX 3090), even when the BIOS has no Resizable BAR option and the kernel's own resize fails. A full-size BAR1 is what the community P2P-enabled NVIDIA driver needs for GPU peer-to-peer (P2P) on GeForce cards.
It was written for a Gigabyte G292-Z20 (AMD EPYC, 8× RTX 3090 behind four PCIe switches, two GPUs per switch) and has run in production there since October 2026. It should work on other boards with the same pattern: root port → switch → two (up to four) GPUs.
⚠️ This module rewrites live PCIe bridge windows and BAR addresses. A wrong layout makes devices unreachable until the next reboot. Always run the dry run first, and keep out-of-band access (BMC/IPMI) at hand.
The P2P path of the patched NVIDIA open kernel modules requires BAR1 to map the GPU's whole VRAM (RTX 3090: 24 GB VRAM → BAR1 32 GB). On our server:
- The BIOS offers no Resizable BAR, neither in legacy nor in UEFI mode. The GPUs boot with a 256 MB BAR1.
echo 15 > /sys/bus/pci/devices/<gpu>/resource1_resizefails with "No space left on device". The kernel only releases the GPU's own bridge window. It does not grow the switch and root port windows above it while the neighbouring GPU still sits in them.- Remove + rescan of the root port does not help either. The kernel sizes the switch window at ~64 GB, but two 32-GB BARs with 32-GB alignment need ~96 GB. The second GPU then gets no resources at all.
pci=reallocandpci=hpmmioprefsize=do not help, because the ports are not hot-plug capable.
For every root port given in rp=, barfix builds the 64-bit prefetchable window itself:
Root port window = switch upstream window (~96 GB, 32-GB aligned, free range found in the root bus window)
├─ downstream port A: [GPU A BAR1 32 GB][GPU A BAR3 32 MB]
└─ downstream port B: [GPU B BAR1 32 GB][GPU B BAR3 32 MB] (BAR1 again on a 32-GB boundary)
Steps in fix_root_port():
- Detect the topology: root port → switch upstream port → downstream ports → NVIDIA GPU (function 0). It aborts
if a downstream port has no GPU, or (without
dry_run) if a driver is bound to a GPU. - Per GPU, pick the largest BAR1 size its ReBAR capability supports, up to
max_log2_mb(default 15 = 32 GB). A GPU that only supports a small BAR1 keeps it, and the layout adapts. - Find a free, aligned range in the parent window with
allocate_resource(). - Swap the resources: turn off the GPUs' memory decoding, release the old resources bottom up, and insert the new
ones into the kernel resource tree top down (
request_resource()). - Program the hardware: the bridges' prefetchable windows (like
pci_setup_bridge_mmio_pref()), the ReBAR control register, the BAR1/BAR3 addresses, then restore the command register.
Only the 64-bit prefetchable window is touched. 32-bit windows (BAR0, the GPUs' audio functions, switch management)
stay as they are. The kernel functions this would normally use (pci_setup_bridge, pci_update_resource) are not
exported to modules, so barfix writes the config space directly.
Requires the headers of the running kernel (on Proxmox: proxmox-headers-$(uname -r); on Debian/Ubuntu:
linux-headers-$(uname -r)).
make # builds barfix.ko against the running kernel
make KVER=6.8.12-4-pve # or against another installed kernelFind the root ports above your GPUs, for example with lspci -tv or
readlink -f /sys/bus/pci/devices/<gpu-bdf> (the first bridge below the root complex). Then:
# 1. dry run: only prints the plan to dmesg, changes nothing (works with the driver loaded)
insmod barfix.ko rp=0000:00:03.1,0000:40:01.1,0000:80:03.1,0000:c0:01.1 dry_run=1
dmesg | grep barfix
rmmod barfix
# 2. for real: unload the NVIDIA driver first (stop everything that uses the GPUs)
rmmod nvidia_drm nvidia_modeset nvidia_uvm nvidia
insmod barfix.ko rp=0000:00:03.1,0000:40:01.1,0000:80:03.1,0000:c0:01.1 dry_run=0
rmmod barfix # the module has no function afterwards; the new layout stays
dmesg | grep "BAR1 now"
# 3. load the (P2P-enabled) driver again and check
modprobe nvidia # or insmod the patched modules
nvidia-smi -q -d MEMORY | grep -A1 "BAR1 Memory Usage"
nvidia-smi topo -p2p rexamples/enable-p2p.sh puts steps 2–3 together for a boot script.
| Parameter | Default | Meaning |
|---|---|---|
rp= |
– (required) | comma-separated root ports dddd:bb:dd.f whose GPUs get a large BAR1 |
dry_run= |
1 |
1 = only print the plan, 0 = apply it |
max_log2_mb= |
15 |
largest BAR1 size as log2(MB): 15 = 32 GB, 14 = 16 GB, … |
Example dry-run output:
pcieport 0000:40:01.1: barfix: plan (dry_run): window 0x... (98336 MB), alignment 32768 MB
nvidia 0000:43:00.0: barfix: DSP 0000:42:00.0 window 0x...+32800 MB, BAR1 0x... (256 MB->32768 MB), BAR3 0x...
nvidia 0000:44:00.0: barfix: DSP 0000:42:01.0 window 0x...+32800 MB, BAR1 0x... (256 MB->32768 MB), BAR3 0x...
Gigabyte G292-Z20 (BIOS R45, no ReBAR option), AMD EPYC 7532 (NPS4), 8× RTX 3090 behind four PCIe switches, Proxmox kernel 7.0, NVIDIA open kernel modules with the community P2P patch:
- All 8 GPUs with a 32 GB BAR1 (
nvidia-smi: BAR1 Total 32768 MiB). A switch with two GPUs needs a 98,336 MB window. In an earlier setup one GPU advertised only 256 MB in its ReBAR capability; barfix kept it small and adapted the layout (65,568 MB window for that switch). - P2P between GPUs on different switches: 26.4 GB/s per direction with
cudaMemcpyPeer(51.2 GB/s bidirectional). Data checked over 2 GiB with copy engines and kernel stores. - NCCL all-reduce over P2P: 23.9 GB/s instead of 15.6 GB/s over shared memory.
- Not persistent. A reboot restores the BIOS layout, which is also the way back. Run it from a boot script before the NVIDIA driver loads (blacklist the driver's autoload, then barfix, then the driver).
- No undo inside the module. If the result is wrong, reboot (possibly via BMC/IPMI).
- At most four downstream ports per switch. Every downstream port must have an NVIDIA GPU (function 0); the module aborts otherwise.
- Only tested with NVIDIA GPUs (RTX 3090) behind Microchip/Microsemi PCIe switches on AMD EPYC.
- NUMA side effect: with a large BAR1 we saw NVML report a wrong NUMA affinity for the GPUs. Tools that bind by it
(e.g. SGLang's
SGLANG_AUTO_NUMA_BIND) should not use it. - For NCCL across root complexes you may need
NCCL_P2P_LEVEL=SYS. - A GPU that keeps a small BAR1 must not be used for BAR1 P2P. The P2P driver may still report "P2P OK" for it, and
NCCL then fails with
NV_ERR_INVALID_STATE. We patched the driver to require BAR1 ≥ VRAM for both GPUs. - sysfs
resource1keeps its boot-time size after the BAR was resized. Read the live size from/sys/bus/pci/devices/<bdf>/resource(line 2 = BAR1), ornvidia-smi -q -d MEMORY.
- aikitoria/open-gpu-kernel-modules and tinygrad's P2P patches – the P2P-enabled NVIDIA open kernel modules for GeForce cards, which need the full BAR1.
GPL-2.0, see LICENSE. No warranty: this modifies live PCIe configuration; use at your own risk.