[MIGraphX EP] The HIP device is not selected on the threads that run a session

[fwyzard]

Describe the issue

The MIGraphX execution provider selects its HIP device (hipSetDevice(device_id)) only:

• in the MIGraphXExecutionProvider constructor, i.e. on the thread that creates the session;
• in the RegisterCreateStreamFn callback, when a new stream is created.

It does not register a SetDeviceFn in RegisterMIGraphXStreamHandles(), and OnRunStart() is a no-op. So StreamExecutionContext / RunSince() never select the device before executing the nodes.

As a result, Run() executes with whatever device is current on the calling thread. That happens, for example, when a session is run from a thread other than the one that created it, or after the application has selected another device on the calling thread.

If there is a different device current on the calling thread:

• MIGraphXAllocator::Alloc() / Free() call hipMalloc() / hipFree() on the wrong device.
  In Debug builds MIGraphXAllocator::CheckDevice() catches this with an ORT_ENFORCE; in Release builds the memory is silently allocated on the wrong device.

• The MIGraphX program is compiled and run with the wrong device current, while the stream belongs to device_id.

For comparison note that the CUDA EP does set the device explicitly in cuda_stream_handle.cc:

stream_handle_registry.RegisterSetDeviceFn(device_type, [](OrtDevice::DeviceId id) { CUDA_CALL_THROW(cudaSetDevice(id)); });

This was found by code inspection while reviewing #32870: Copilot pointed it out for the new user-stream path, but the default path, where the EP creates its own streams, has the same problem.

To reproduce

On a machine with at least 2 AMD GPUs:

1. Create a session with the MIGraphX EP and device_id = 0.
2. Run it from a new thread that has first called hipSetDevice(1).

In a Debug build the allocator check fails. In a Release build the intermediate buffers are allocated on device 1.

Urgency

No response

Platform

Linux

OS Version

AlmaLinux release 9.7 (Moss Jungle Cat)

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

a11b4e5

ONNX Runtime API

Python

Architecture

X64

Execution Provider

MIGraphX

Execution Provider Library Version

ROCm 7.14.0

[fwyzard]

This is fixed by #32870 as part of implementing #32850.