bug(gpu): cuInit segfaults in DGX OS 7.5 sandbox while plain CDI passes

[senthilr-nv]

Description

On a DGX Station GB300 running DGX OS 7.5, a GPU-enabled OpenShell 0.0.85
sandbox can load the injected driver library and call cuDriverGetVersion, but
cuInit(0) segfaults. The same library and GPU pass on the host and in ordinary
Docker CDI containers, including as the sandbox UID/GID.

This blocks qualification of DGX OS 7.5 for the NemoClaw Station golden path.
The existing generic Ubuntu path remains the supported control.

Related: #1486

Environment

• Hardware: DGX Station GB300, ARM64
• OS: DGX OS 7.5.0-GB300ws-GB200ws / Ubuntu 24.04.4
• Kernel: 6.17.0-1014-nvidia-64k
• NVIDIA driver: 595.58.03
• Driver API: cuDriverGetVersion=13020
• Docker: 29.2.1
• NVIDIA Container Toolkit: 1.19.0
• OpenShell: 0.0.85
• NemoClaw commit: b47c81e0cc2b3dc22e07921167e48ad0a4ff7c1c

Reproduction

In a GPU-enabled OpenShell sandbox:

openshell sandbox exec -n my-assistant --no-tty -- \
  python3 -c 'import ctypes; lib=ctypes.CDLL("libcuda.so.1"); rc=lib.cuInit(0); print(f"cuInit(0)={rc}")'

Observed during fresh onboarding and in a separate bounded diagnostic:

GPU proof passed: nvidia-smi when available
GPU proof passed: /proc/<pid>/task/<tid>/comm write
GPU proof inconclusive: cuInit(0) via libcuda.so.1
Segmentation fault (core dumped)

The isolated diagnostic produced:

sandbox_libcuda_load: loaded=libcuda.so.1, exit 0
sandbox_driver_version: cuDriverGetVersion=0,version=13020, exit 0
sandbox_cuda_init: no output, exit 1

Controls

All of the following passed with cuInit(0)=0:

• the DGX OS host;
• an ephemeral network-disabled Docker CDI container as root;
• the same CDI container as UID/GID 998:998;
• a current generic Ubuntu 24.04 ARM64 control rebuilt with the same NemoClaw
  commit and OpenShell 0.0.85 (kernel 6.8.0-124-generic, driver 610.43.02).

The DGX OS host and sandbox used the same injected driver file:

sha256 ba8d543fee78b6e7932592d55abba877a0df226d48631e746bd50beb05916dac
/usr/lib/aarch64-linux-gnu/libcuda.so.595.58.03

The managed vLLM container, direct inference, routed agent inference, exact
response, tool-use artifact, restart/reuse, and ECC checks all passed. This
narrows the failure to the OpenShell sandbox boundary on the DGX OS
kernel/driver stack; the current evidence does not isolate kernel versus driver.

Expected Behavior

A GPU-enabled sandbox should complete cuInit(0) when the host and equivalent
plain CDI container pass. At minimum, the sandbox must fail deterministically
with an actionable diagnostic rather than continuing with CUDA marked
unverified after a segmentation fault.

[elezar]

📋 triage-agent

Triage Assessment

Classification: needs-investigation

Summary

The report has substantive diagnostics and strong controls: host CUDA and plain Docker CDI pass, while an OpenShell GPU sandbox using the same injected driver library reaches cuDriverGetVersion but segfaults in cuInit(0). I do not see this as a duplicate of #1486 because #1486 covered the generic /proc and static GPU baseline failure; this report says the /proc/.../comm write and Ubuntu control path now pass.

Investigation

• crates/openshell-driver-docker/src/lib.rs selects Docker CDI devices and submits DeviceRequest { driver: "cdi" }; it does not derive inner sandbox policy from the selected CDI spec.
• crates/openshell-sandbox/src/lib.rs currently detects GPU presence via /dev/nvidiactl or /dev/dxg, adds static GPU paths, enumerates /dev/nvidia[0-9]*, and promotes /proc to read-write for CUDA compatibility.
• architecture/security-policy.md documents that GPU baseline enrichment is path-based and startup-time, and architecture/plans/cdi-derived-gpu-filesystem-policy-issue.md already identifies the design gap: non-standard CDI layouts can inject correct devices, mounts, libraries, or groups into the outer container while the OpenShell inner workload still lacks the corresponding Landlock or process setup.
• Existing GPU e2e coverage (e2e/gpu/README.md, e2e/rust/tests/gpu/workloads.rs) validates the generic cuda-basic path, but not this DGX OS 7.5 / GB300 / ARM64 stack or CDI-derived platform shapes.

Recommendation

Route as a spike for platform-specific GPU sandbox investigation. The spike should collect the selected CDI spec/container edits from DGX OS 7.5, compare the OpenShell effective filesystem/process setup against a passing plain Docker CDI container, and decide whether the fix belongs in CDI-derived sandbox requirement enrichment, CUDA runtime diagnostics, or a separate DGX OS compatibility path. Do not apply state:agent-ready until a human approves the investigation or a concrete build issue.