Add ROCM GPU support

[TibixDev]

Hey, it'd be really nice if we could run this model API on AMD cards, and it would provide a very nice speedup over CPU. Do you think you could add this when you have some free time? It shouldn't be too hard since afaik ONNX models can run with ROCM just fine.

[remsky]

We're refactoring a few things to more easily support different architectures, though I haven't worked with AMD cards yet. Will try adding them to the onnx providers but may need some community testing to get right

[Reed-Schimmel]

Would ONNX work on the intel Arc GPUs?

[bgs4free]

I've got it to work with rocm. The problem is that the docker image becomes huge: Around 22G when I use rocm/dev-ubuntu-22.04 as a base image. I might get it down with a slimmer base image, but I don't expect it to come even close to ghcr.io/remsky/kokoro-fastapi-gpu, which is about 5.7G.

https://github.com/bgs4free/Kokoro-FastAPI/tree/add-rocm-support

[bgs4free]

Would ONNX work on the intel Arc GPUs?

Theoretically, yes, but I had difficulties compiling it on linux.
Edit: misread your question. Thought this was about AMD.

[gitchat1]

I've got it to work with rocm. The problem is that the docker image becomes huge: Around 22G when I use rocm/dev-ubuntu-22.04 as a base image. I might get it down with a slimmer base image, but I don't expect it to come even close to ghcr.io/remsky/kokoro-fastapi-gpu, which is about 5.7G.

https://github.com/bgs4free/Kokoro-FastAPI/tree/add-rocm-support

What version of rocm did you get it working with I have a 5700xt and that thing only supports rocm 5.2 unofficially. Any chance you could get it working on that? Or maybe get direct-ml working with the docker file.

[bgs4free]

What version of rocm did you get it working with I have a 5700xt and that thing only supports rocm 5.2 unofficially. Any chance you could get it working on that? Or maybe get direct-ml working with the docker file.

The Dockerfile is based on 6.3.1. Works on 7900XTX.

https://github.com/bgs4free/Kokoro-FastAPI/blob/b064066b9ef80ee09afefb8ff53e2ffbee1387de/docker/rocm/Dockerfile#L1

I'm using the pytorch index for 6.2, because I had issues with the 6.3 version (version mismatch seems to be no issue AFAIK).

https://github.com/bgs4free/Kokoro-FastAPI/blob/b064066b9ef80ee09afefb8ff53e2ffbee1387de/docker/rocm/pyproject.toml#L23

Change it to 5.2, try and see if it works. Make sure you have enough disk space.

# in docker/rocm
docker compose up --build

[gitchat1]

Okay I tried that and it gave me the following error
Error response from daemon: error gathering device information while adding cust
om device "/dev/kfd": no such file or directory

Acording to this thread https://stackoverflow.com/questions/73229163/amd-rocm-with-pytorch-on-navi10-rx-5700-rx-5700-xt you have to Set the environment variable.HSA_OVERRIDE_GFX_VERSION=10.3.0 I just added the variable to the docker file under env. Any idea what else i could try?

[bgs4free]

device "/dev/kfd": no such file or directory

devices get passed through from the host. See: https://github.com/bgs4free/Kokoro-FastAPI/blob/add-rocm-support/docker/rocm/docker-compose.yml#L20

What is your host system? This was tested on Linux and I can't speak for any other OS.

[gitchat1]

Oh, okay its windows. I thought that wouldn't reaööy matter since the docker runs an Ubuntu image.

[bgs4free]

Oh, okay its windows. I thought that wouldn't reaööy matter since the docker runs an Ubuntu image.

https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html

Don't see anything about docker for windows here.

[gitchat1]

I checked my docker config it is running through wsl so that is technically Ubuntu so it should work.

[remsky]

Taking a shot at this shortly, will post links if anyone is able to test it out

(I don't have an AMD card unfortunately, so can just skeleton it up)

[bgs4free]

Taking a shot at this shortly, will post links if anyone is able to test it out

(I don't have an AMD card unfortunately, so can just skeleton it up)

My fork works on my linux machine. Didn't make a PR, though, because I personally didn't find it convincing enough to pursue. Please feel free to use my stuff if you see any value.

[TheMrCodes]

Would ONNX work on the intel Arc GPUs?
Theoretically yes (with the oneDNN executor it should work OpenVino not because of unsuported Ops)

But for Arc I already implemented a version with ipex (intel extention for pytorch) support that uses nearly the exact code as CUDA GPUs
https://github.com/TheMrCodes/Kokoro-FastAPI

For more on Arc please use Issue#106