Skip to content

Put mt.exe on PATH for Windows ROCm builds - #36

Closed
sawansri wants to merge 1 commit into
lemonadefrom
sawansri/windows-rocm-mt-exe
Closed

sawansri wants to merge 1 commit into
lemonadefrom
sawansri/windows-rocm-mt-exe

Conversation

@sawansri

@sawansri sawansri commented Nov 2, 2026 •

Copy link
Copy Markdown

Problem

Every scheduled release since b12013 (2026-10-30) has shipped without ROCm archives. The windows-rocm builds fail at link time:

lld-link: error: unable to find mt.exe in PATH: no such file or directory
gmake.exe[2]: *** [...] bin/llama-llava-cli.exe] Error 1

The release job then drops each incomplete ROCm family, Ubuntu tarballs included, even though the Ubuntu builds pass qualification. Lemonade's llama.cpp auto-update therefore can't bump rocm-stable (see lemonade-sdk/lemonade#4036, stuck at b11836).

Cause

Upstream ggml-org#31329 (d63fbfe79, landed between b12003 and b12013) embeds an app manifest in every Windows executable. With TheRock's clang in MSVC mode, lld-link merges that manifest by invoking mt.exe, which lives in the Windows SDK. This job builds with Unix Makefiles outside a VS developer shell, so mt.exe isn't on PATH. The workflow commit was the same in the last passing run and the first failing run.

Fix

Before configuring, the Build step finds the newest Windows SDK x64 bin folder that contains mt.exe and appends it to PATH. Appending keeps the SDK from shadowing anything already on PATH. Then it runs mt.exe /? and fails with a clear message if the loader can't start it.

  • Whole folder, not just mt.exe: I first tried copying only mt.exe into a private folder, but lld-link couldn't run it there (ExecuteAndWait failed with no output). The tradeoff is that CMake may now find the SDK's rc.exe ahead of llvm-rc; that's Microsoft's own resource compiler and works with clang in MSVC mode.
  • Why not vcvarsall: it would prepend MSVC tools ahead of TheRock's LLVM, and its environment can't be inherited by a pwsh step.

Validation

PR run 41535886862:

  • windows-rocm 7.14, 10.0 and 10.1 all build.
  • qualify-tier0 and qualify-tier1 (Strix Halo) pass for every Windows and Ubuntu ROCm version.
  • ubuntu-22-openvino fails, unrelated: upstream's OpenVINO backend no longer compiles against the pinned OpenVINO 2026.0, and that only drops the OpenVINO archive.
  • The release job is skipped on PRs, so publishing will be confirmed by the first scheduled run after merge.

🤖 Generated with Claude Code

@sawansri
sawansri force-pushed the sawansri/windows-rocm-mt-exe branch from f866eef to 87440cb Compare November 2, 2026 22:16
Upstream ggml-org#31329 embeds an app manifest in every Windows
executable. With ROCm's clang in MSVC mode, lld-link has to merge that
manifest by invoking the Windows SDK's mt.exe, which isn't on PATH outside
a VS developer environment. Every windows-rocm build since b12013 failed at
link time, and the release job then dropped the whole ROCm family, Ubuntu
archives included.

Enter the VS developer shell before building so the MSVC-ABI toolchain
gets the SDK tools it expects, and keep building with the runner's CMake
instead of the copy VS bundles.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sreeram-11

Copy link
Copy Markdown
Collaborator

Thanks @sawansri for the PR. However, I'd go with the PATH approach. I don't think we should switch to the dev shell, for a few reasons:

  • The dev shell changes too much of the build environment. It sets INCLUDE, LIB, and LIBPATH, and it puts MSVC tools and VS's own CMake first on PATH. That's exactly why you had to add the cmake workaround, and it can also change what ROCm's clang compiles against.
  • The PATH approach has been proven end to end. Run 41535886862 built all three Windows ROCm versions and passed every Tier 0 and Tier 1 check. The log shows it's using C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\mt.exe, and the 10.1 build step succeeded.

So the dev shell adds risk. It could quietly change which headers and libraries the build uses with no real gain over a fix that's already working.

@sreeram-11

sreeram-11 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Closing in favor of #37.

@sreeram-11 sreeram-11 closed this Nov 3, 2026
@sreeram-11
sreeram-11 deleted the sawansri/windows-rocm-mt-exe branch November 4, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants