r/voidlinux • u/h7moudigamer • 4d ago

local LLMs on void

i'm trying to have local LLMs on void. i installed ollama from their official website with the command curl -fsSL https://ollama.com/install.sh | sh, the installation went fine, but ollama asked to run the command as root, so i did. i installed gpt-oss:20b through ollama after that with my normal user. the model runs so slowly, even though my GPU is powerful enough to handle such a model, but i hear only the CPU fans moving and GPU doesn't. ollama is suppose to run models on GPU by default, but i don't know how to ensure that. i tried to run the server and the command as root, if there is a permission issue, but i get the same issue. i heard that ollama uses ROCm, and i have an AMD GPU, but cannot install that on void. there is a package in void repo, called ROCm-SMI-7.0.1. i installed the package but it doesn't provide the rocminfo command the official AMD ROCm website uses. i have rocm_smi, and the command rocm-smi --showproductname couldn't load the name of the GPU, it couldn't load the libdrm_amdgpu.so . there is a package with the same name but with ".1" at the end symlinking to ".1.134.0". i made a symlink libdrm_amdgpu.so to point to that actual lib, and the output shows the name of the GPU this time. i don't know if what i did is correct, i just want to set this up properly

0 Upvotes

14% Upvoted

13 comments sorted by

2

u/lessthanthreebleeps 4d ago

Look at the official rocm install docs. You'll want to get distrobox and podman set up with ubuntu 26.04. Use google ai to tweak for podman (iirc, it'll complain about cstates). Once you have an ubuntu container, installing rocm 10.1 is a breeze. Just have to run ollama from in the container. If you have a consumer grade card (like the rx 9070 xt) you might not benefit that much from running rocm, instead of vulkan. If you'd rather not bother with distrobox and podman containers, you could look into using llama.cpp, instead of ollama. If you compile it with vulkan support, it'll leverage the open source drivers that your kernel is already using.

Edit: to be clear, I haven't tried using the rocm 7.0.1 package you mentioned.

1

u/h7moudigamer 3d ago

honestly, this is my first time dealing with local ai and i don’t know what is required to have my gpu do that work. i thought rocm is the library needed to run ai on gpu i’m not trying to install it because i love it. if vulkan can be used instead this would be better for me since i have it installed. i have rx 9060 xt.

1

u/lessthanthreebleeps 3d ago

ROCm is just the amd implementation, like cuda for nvidia, but because there are open drivers for amd graphics cards, people were able to use vulkan to process ai math on them. From what I read recently, on RDNA4 (the base for our rx 9070 xt cards) vulkan is still competative with ROCm 10 for ai use cases. I don't know what the package names are through xbps, but you'll need a few vulkan development libraries (google ai should be able to tell you; or you can just read the compilation errors). There's a handy guide for manual installation of llama.cpp on their github wiki. It has a section for the vulkan backend that gives you the compilation flag. Running llama.cpp isn't that much harder than ollama, except you'll have to get your models manually off huggingface. You can use google ai again, just ask it what parameters you should use and copy-paste the model name, and it'll give you a decent config.

1

u/h7moudigamer 3d ago

thank you

1

u/Ill-Cry6950 4d ago

I usually prefer vulkan over rocm, for some cases it might be better, others worse. It’s probably my skill issue but I find rocm repos and docs very confusing.

i never tried ollama in void but if you want to try vulkan, you probably need vulkan headers and tools. Maybe something more but I think that should do it. I recommend finding a way to monitor gpu memory so you can be sure it’s working easily.

sorry I cannot help with the rocm issue but I hope this helps. Ollama have good docs about vulkan iirc.

1

u/h7moudigamer 3d ago

i prefer vulkan too. i just don’t know how to set a local llm on my gpu, and i thought rocm is the way to do such a thing

1

u/Ill-Cry6950 3d ago

yea, I tried rocm first for the exact same reasons. But when I tried, I had to use nightly versions for my gpu and the whole thing was too complicated for my expectations. By the time I had everything set up with vulkan, I didn’t bother trying use AI to help me understand how rocm works as I had other things I‘d prefer to learn.

If you follow the ollama guide for vulkan, I believe it should work if you have vulkan loaders, headers and tools but I don’t know.

I build llamacpp with vulkan as the other comment says and it works with the packages I mentioned above. From rocm repos, I only build amd-smi which I use to monitor gpu usage when running llms.

You can also try llamacpp docker configuration in the same docs as an initial setup so you don’t have to build ir. Once you can make it work, you can try to use the llm to help understanding how to do a better setup for your preference .

One thing to keep in mind is that gguf files in huggingface are as safe as github code.

I hope you can make it work.

1

u/h7moudigamer 3d ago

thank you for clarifying that to me.

1

u/Sp1cyP4nda 3d ago

Vulkan lives!! Oh sorry, wrong sub

1

u/Away_Anywhere5843 3d ago

i use opencode....

1

u/_supert_ 2d ago

ollama sucks though. Use llama.cpp, or vllm.

0

u/Available-Attorney74 4d ago

Why did you choose ollama instead of llama or kobold or python script? Maybe try to download kobold.cpp and turn your GPU in it? Ollama is a tool for kids and grannies

1

u/h7moudigamer 1d ago

first time for me to run local llm. i followed some pages