Canonicalize the path returned by current_binary_dir()

[GoodOlClint]

Proposed changes

Re-submitting #3945 with the demonstration that was missing. Thanks for the quick look — the table in the original PR was the wrong evidence, and your reading of it was right: those three rows all name the same directory, so on its own that table shows nothing but a cosmetic difference.

The problem is not the value current_binary_dir() returns. It is that the two JIT include-path lookups immediately take .parent_path() of it, and parent_path() is a lexical operation — it strips the last component of the spelling, not of the directory. Two spellings of the same directory therefore have different parents.

mlx/backend/cuda/jit_module.cpp (include_path_args()) and mlx/backend/cpu/jit_compiler.cpp (get_preamble()) both do:

auto root_dir = current_binary_dir();
#if !defined(_WIN32)
root_dir = root_dir.parent_path();
#endif
auto path = root_dir / "include";   // ... then / "cccl"

So:

dli_fname	current_binary_dir()	root_dir	resulting include root
/p/lib/libmlx.so	/p/lib	/p	/p/include
/p/lib/./libmlx.so	/p/lib/.	/p/lib	/p/lib/include
./tests	.	""	include, resolved against the CWD
tests (found on PATH)	""	""	include, resolved against the CWD

Row 2 is the direct answer to your question: /p/lib/. and /p/lib are the same directory — that is exactly why the defect is invisible until something takes the parent. The trailing . absorbs the "go up one level", and the include root silently moves into the wrong tree. Nothing relative is involved and both paths are absolute.

Rows 3 and 4 are worse: parent_path() of a single-component path is empty, so the include root degrades to a bare relative include, resolved against whatever the process's working directory happens to be rather than against the binary. The result is also cached in a function-local static and consumed lazily, long after startup, so a later chdir() moves it again.

The other two callers don't take the parent, but still consume the raw value: cuda/delayload.cpp does fs::absolute(current_binary_dir() / relative) (a relative value roots that at the CWD) and metal/device.cpp appends the metallib name to it.

The failure this actually caused

CUDA build on an aarch64 GB10 / sm_121 box, configured -DMLX_BUILD_CUDA=ON -DMLX_BUILD_TESTS=OFF, run from the build tree. NVRTC:

mlx/backend/cuda/device/indexing.cuh(3): catastrophic error: cannot open source file "cuda/std/tuple"

strace showed NVRTC never received a CCCL --include-path at all — it only ever probed the toolkit copy:

openat(AT_FDCWD, "/usr/local/cuda/include/cuda/std/tuple", O_RDONLY) = -1 ENOENT

Both branches of the CCCL lookup missed. The MLX_CCCL_DIR fallback was compiled out because that define sits inside if(MLX_BUILD_TESTS) — that is the separate one-liner in #3944, and with it applied this becomes a silent fallback rather than a hard error. The primary branch missed because of this bug.

Isolating it by invocation at the time, with a fresh MLX_PTX_CACHE_DIR per run (the PTX cache masks the effect and produced one false PASS before we controlled for it):

CCCL dir staged at	./example1 from build/	absolute path, from /tmp
both locations	PASS	PASS
only build/include/cccl	PASS	FAIL
only build/../include/cccl	FAIL	PASS
neither	FAIL	FAIL

The two middle rows are exactly complementary. The same binary looks in two different directories depending on how it was invoked, and each invocation fails when only the other one's directory exists — the search path is computed from argv[0] as typed, not from where the binary lives.

And the patch closes it

A second, tighter experiment on a fresh GB10 while preparing this re-submit. The one above varies the invocation; this one holds the invocation fixed and varies only the patch — before is main at fb5133e1, after is that same commit with this PR applied. Identical install layout (prefix/bin/mlxprobe, CCCL staged only at prefix/include/cccl, prefix/bin/include deliberately absent), identical relative invocation from prefix/bin, fresh PTX cache each. The probe is built as MLX's own example target so CMake owns the CUDA link line. -DMLX_BUILD_CUDA=ON -DMLX_BUILD_TESTS=OFF, CUDA 13.0.88, sm_121, aarch64.

##### before (main @ fb5133e1) #####
staged : /workspace/prefix-before/include/cccl -> .../cccl-src/include
absent : /workspace/prefix-before/bin/include/cccl (deliberately)
confirmed: cuda/std/tuple present at the staged path
--- ./mlxprobe from /workspace/prefix-before/bin ---
  what():  Failed to compile kernel: mlx/backend/cuda/device/indexing.cuh(3):
           catastrophic error: cannot open source file "cuda/std/tuple"
  1 catastrophic error detected in the compilation of "gather_int32_int32_1.cu".
RESULT[before]=FAIL

##### after (fb5133e1 + this PR) #####
staged : /workspace/prefix-after/include/cccl -> .../cccl-src/include
absent : /workspace/prefix-after/bin/include/cccl (deliberately)
confirmed: cuda/std/tuple present at the staged path
--- ./mlxprobe from /workspace/prefix-after/bin ---
JIT_OK array([1, 3, 5], dtype=int32)
RESULT[after]=PASS

The headers are present and readable in both runs, at the path the canonicalized lookup computes. Unpatched, MLX doesn't look there.

Reproducing it without a GPU

No CUDA device is needed to watch the include root move. Against a plain CPU-only build of this tree (Ubuntu 24.04, aarch64, gcc 13), with a ten-line program that calls mlx::core::current_binary_dir() and then computes the same root_dir / "include" / "cccl" the two JIT lookups compute.

Layout is a normal install tree — prefix/bin/mlxprobe and prefix/include/cccl. The correct answer in every run is /out/prefix/include/cccl, and it exists the whole time. One binary, one directory, four invocations:

===== 1. invoked by absolute path =====
argv[0]              = /out/prefix/bin/mlxprobe
current_binary_dir() = '/out/prefix/bin'
root_dir             = '/out/prefix'
cccl candidate       = '/out/prefix/include/cccl'      -> exists() = YES

===== 2. same binary, same dir, spelled with a trailing /. =====
argv[0]              = /out/prefix/bin/./mlxprobe
current_binary_dir() = '/out/prefix/bin/.'
root_dir             = '/out/prefix/bin'
cccl candidate       = '/out/prefix/bin/include/cccl'  -> exists() = NO

===== 3. same binary, invoked relatively from its own dir =====
argv[0]              = ./mlxprobe
current_binary_dir() = '.'
root_dir             = ''
cccl candidate       = 'include/cccl'
  -> absolute        = /out/prefix/bin/include/cccl    -> exists() = NO

===== 4. same binary, found on PATH, cwd elsewhere =====
argv[0]              = mlxprobe
current_binary_dir() = ''
root_dir             = ''
cccl candidate       = 'include/cccl'
  -> absolute        = /tmp/include/cccl               -> exists() = NO

Three of the four look for the headers outside the install tree; run 4 leaves it altogether and lands in /tmp.

With the patch, runs 1–3 all report /out/prefix/include/cccl, exists() = YES.

Run 4 is not fixed, deliberately. When a statically-linked executable is found via PATH, glibc reports dli_fname as the bare argv[0] (mlxprobe, no separator) — the real location simply isn't in the value, so there is nothing to resolve. Closing it needs /proc/self/exe, which is Linux-specific and a bigger change than belongs here; happy to add it if you'd prefer. It also doesn't arise for the ways MLX ships as a library: when MLX is a shared object or a Python extension, dli_fname is the loader's path and always contains a separator.

That case does need an explicit guard rather than being left to weakly_canonical, because the two standard libraries disagree about it:

input	libc++	libstdc++
""	current_path()	""
"bare-name" (relative, missing)	$CWD/bare-name	"bare-name"

Left ungoverned, libc++ would resolve a bare argv[0] against the current working directory and hand back a confidently wrong absolute path — worse than the empty result it replaces, because exists() no longer fails cleanly. So the patch checks the input for a directory component and returns the empty parent unchanged when there is none. Measured on macOS/libc++ and in a gcc:13 container.

The fix

std::filesystem::weakly_canonical on dli_fname before taking the parent. It normalizes away . and .., roots a CWD-relative path, and resolves symlinks, so the value stops depending on how the process was started. It is computed once inside the existing static initializer, so the added stat traffic happens at most once per process.

Two guards around it, both for cases I could reproduce:

• The no-separator case described above, to stop libc++ inventing a CWD-derived answer.
• The error_code overload rather than the throwing one. The throwing overload is reachable: the value is computed lazily in a function-local static, so a relative dli_fname whose working directory has since been removed makes weakly_canonical throw where the old purely-lexical code could not fail. Reproduced on both libc++ and libstdc++. It matters most in cuda/delayload.cpp, which runs inside a Windows delay-load hook, where an escaping C++ exception terminates rather than failing the load cleanly. On error the patch keeps the old lexical answer.

Symlink resolution is intentional, and is a behavior change worth calling out. For an install reached through a symlinked lib directory, the sibling include/ tree sits next to the real path, not next to the link — so resolving is what makes the lookup find it. If you'd rather keep the old symlink semantics, absolute(...).lexically_normal() fixes the reported bug without that part; I went with weakly_canonical deliberately but it is an easy swap.

The declaration also gains MLX_API. The library builds with hidden visibility (mlx/CMakeLists.txt:32-34), so without it the new test fails to link in a shared build — which is what the macOS CI job's -DBUILD_SHARED_LIBS=ON reconfigure does.

This is a no-op on macOS, which is why the defect is Linux-only. dyld reports a fully resolved dli_fname in all four invocations above — including the bare-PATH one that glibc cannot recover — so on macOS weakly_canonical is handed an already-canonical path every time. Re-verified on arm64 macOS 15 while preparing this re-submit.

Verification

tests/utils_tests.cpp gains a case asserting the result is absolute and already in normal form. Both checks are purely lexical — no filesystem access, and it does not re-apply the function under test. It returns early on an empty result, since that is the legal outcome for the bare-argv[0] case above.

Applying only that test file to the base, so it runs against unpatched code:

###### unpatched code + this PR's test ######
--- absolute invocation ---
[doctest] test cases: 1 | 1 passed | 0 failed
--- relative invocation from the test dir ---
tests/utils_tests.cpp: ERROR: CHECK( dir.is_absolute() ) is NOT correct!
  values: CHECK( false )
[doctest] test cases: 1 | 0 passed | 1 failed

With the patch, absolute and relative invocations both pass, and a bare-PATH invocation runs zero assertions via the early return.

Being explicit about the limitation: the test only fails on a non-canonical invocation, so CI, which invokes the binary by absolute path, is green either way. It guards against reintroduction rather than catching this in CI. A test that failed regardless of invocation would have to re-exec itself, which seemed worse than the guard is worth — happy to be told otherwise.

Full C++ suite on Linux (aarch64, Ubuntu 24.04, gcc 13, CPU-only), this commit against its base:

base (fb5133e1):  244 test cases | 244 passed | 0 failed   (3238 assertions)
this commit:      245 test cases | 245 passed | 0 failed   (3240 assertions)

Also built with -DBUILD_SHARED_LIBS=ON -DMLX_BUILD_TESTS=ON to confirm the tests target links against a hidden-visibility libmlx.so — it does not without the MLX_API decoration.

I did not re-run the Metal suite for this re-submit; the change is a no-op on macOS for the reason given above.

Checklist

Put an x in the boxes that apply.

• I have read the CONTRIBUTING document
• I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
• I have added tests that prove my fix is effective or that my feature works
• I have updated the necessary documentation (if needed)