Background: This Nomic blog article from September 2023 promises better performance in GPT4All for AMD graphics card owners.
Run LLMs on Any GPU: GPT4All Universal GPU Support
Likewise on GPT4All’s GitHub page.
September 18th, 2023: Nomic Vulkan launches supporting local LLM inference on NVIDIA and AMD GPUs.
Problem: In GPT4All, under Settings > Application Settings > Device, I’ve selected my AMD graphics card, but I’m seeing no improvement over CPU performance. In both cases (AMD graphics card or CPU), it crawls along at about 4-5 tokens per second. The interaction in the screenshot below took 174 seconds to generate the response.
Question: Do I have to use a specific model to benefit from this advancement? Do I need to install a different AMD driver? What steps can I take to troubleshoot this?
Sorry if this is an obvious question. Sometimes I feel like the answer is right in front of me, but I’m unsure of which key words from the documentation should jump out at me.
My system info:
- GPU: Radeon RX 6750 XT
- CPU: Ryzen 7 5800X3D processor
- RAM: 32 GB @ 3200 MHz
- OS: Linux Bazzite
- I’ve installed GPT4All as a flatpak
I am somewhat new to Linux and hosting local LLMs, but I think I had to install AMD ROCm for LLMs to work with my GPU.
https://rocm.docs.amd.com/en/latest/about/release-notes.html
Can gpt4all use ROCm?
Rats—according to their System Requirements (Linux) page, they don’t support Fedora. Even if I were to switch to a supported distro, it looks like only a small set of graphics cards are supported, and unfortunately, mine is not one of them. 😢
Supported graphics cards:
Supported operating systems:
Thanks anyway for the tip!
Actually I run fedora and have ROCm working. In fact it’s in the default package manager.
You can see 'em all by running:
sudo dnf list | grep rocmTo get your GPU working you can look up: HSA_OVERRIDE_GFX_VERSION And maybe HIP_VISIBLE_DEVICES
Though this is the techy route. If you get LM studio running, or even better llamacpp, you’ll have access to much better quantization formats than q4_1.
So, you’ll be at the same speed or even faster than Vulkan, and with high quality outputs.
Thanks for the info—maybe I’ll give this another whirl when I have some more time.
Which card are you running on?
I use the 24GB 7900 XTX.
I wonder why ROCm 6.4 doesn’t support you, but ROCm 6.3 does. Maybe there is a way to downgrade. Also that override_gfx environment variable may be enough to get 6.4 working for you. Not sure though.
I’d say an easy route (if it works lol) would be using dnf to install ROCm, and then use LM studio’s installer to get the rest.
I wonder why ROCm 6.4 doesn’t support you, but ROCm 6.3 does.
Wait, where are you reading that 6.3 supports the 6950 XT? I dug up the System Requirements (Linux) page for 6.3 and it lists the same cards as the 6.4 page. Is there another document out there that covers this topic?
Okay I rechecked and it looks like 6.4 and 6.3 have similar compatibility/incompatibility with certain cards.
Here are the gfx versions of different amd cards:
https://rocm.docs.amd.com/en/develop/reference/gpu-arch-specs.html
Here are the supported versions of 6.4
https://rocm.docs.amd.com/en/docs-6.4.0/compatibility/compatibility-matrix.html
So given this extra bit of research it looks like you may be able to run ROCm on a 6950XT but I’m not sure about a 6750XT.
From my experience ROCm supports more than they say they do. They say they support the cards they’ve tested, but other’s still may work. I was running ROCm on my 7900 XTX before they officially supported it.
I don’t have a clue, I only tried LM Studio and Automatic1111.
What card are you running on?
7900 XTX. Sorry, forgot that ROCm only supports some cards.
My best guess would be that you installed the flatpak version of gpt4all, and somehow that messed with it’s ability to use the GPU.
Vulkan should work with your GPU, the model you chose should fit in your GPU if it is q4_0 or q4_1 which is the default in gpt4all I think.
I’ll try installing non-flatpak GPT4All in a distrobox and see if I get a different result. Thanks for the idea.
For sure!
Update: I thought I’d report back on my progress. I tried installing GPT4All in distrobox containers, several different images (Ubuntu 24.04 and 22.04, and Fedora 41), but in every case, the installation script fails due to missing dependencies. I can’t get to the installer GUI. Upon further investigation, it appears that GPT4All does not support Wayland. There is an open feature request from last year, but I’m not holding my breath. I did some cursory searches for workarounds, but couldn’t figure it out in the time I had available today.
[me@UbuntuTestingGpt4All ~]$ ./gpt4all-installer-linux.run ./gpt4all-installer-linux.run: error while loading shared libraries: libxkbcommon-x11.so.0: cannot open shared object file: No such file or directoryI wonder if I would have the same issue if I tried this while running an X session on the host machine. I’ll post another update if I test this scenario.
Anyway, thanks again for the tip.
For sure!
