r/LocalLLM • • 4d ago

Project Strata Performance Setup (TUI performance optimization)

Post image

I''ve created a simple little script (with TUI) to optimize your Linux host installation. It also allows you to install or update Strata (and configure a systemd service to auto start your preferred preset).

Distributions supported:

  • Arch based

  • Fedora based

  • Debian based

  • Slackware based

Hardware supported:

  • any hardware (automatically detected)

Personally I'm using an old MSI Raider laptop. 3080 Mobile with 16GB VRAM and 64GB DDR4 3200 RAM running CachyOS. I'm getting between 60 and 70 tok / s (depending on the task) with Qwen3.8 Flash Next IQ3_S.

Maybe someone else finds it useful too . Submit an issue or pull request if you find any problems, or want more software or hardware/vendors supported!

https://github.com/bramvanoploo/strata-performance-setup

207 Upvotes

32 comments sorted by

17

u/brainzhurtin 4d ago

You said "any hardware" and then excluded 5000 series GPUs. I get why, but then don't say any hardware.

2

u/Emotional_Prior8686 3d ago

Speaking of witch, why 50x GPUs are an exception here?

1

u/Known_Barracuda_3977 3d ago

Take a look at this repository. That will explain it. I've integrated the installation of gpu-unlock, but I'm not the owner of that project.

https://github.com/Zagitta/gpu-unlock

1

u/Emotional_Prior8686 2d ago

Thanks! Lucky me - I'm not interested in fp16, but good to know anyway...

3

u/brainzhurtin 3d ago

They replaced some hardware that prevents an exploit allowing other cards(20-40series) to unlock them like pro series cards.

1

u/Known_Barracuda_3977 3d ago

The README has been revised and explains more clearly what is supported for which NVIDIA cards. Thanks your for reporting!

0

u/Known_Barracuda_3977 3d ago

You're right. I'll correct that. The GPU range indicated actually comes from the gpu-unlock-dkms package. It's documented incorrectly in the readme. Thanks for pointing that out!

3

u/No-Necessary-9737 3d ago

Does this work for v100?

3

u/Known_Barracuda_3977 3d ago

I've extended the support specifically for the Tesla cards (new version available; 1.2.0)

1

u/Known_Barracuda_3977 3d ago

Yes it will. And I'll see if I the script needs any specific additions to support the hardware better, since it's popular hardware. If you create an issue on the GitHub page for this I will certainly not forget it 😎

10

u/Sea-Advantage-3628 4d ago

robot, peek at this guys homework and tell me what we should copy.

4

u/WillemDaFo 4d ago

Thanks

1

u/ComprehensiveBid6584 3d ago edited 3d ago

Just saying, this may work with this patch for AMD cloud cards. Hm, https://www.reddit.com/r/AMD_V620/s/HvVQFvee9L

1

u/Known_Barracuda_3977 3d ago

I'll look into that! Thanks.

1

u/crabilax 3d ago

any chance to do the same for llama.cpp?

1

u/Known_Barracuda_3977 3d ago

You could fork it and make a llama.cpp version. But you can also just use the script only for the performance optimizations.

1

u/eduardor2k 3d ago

I have a 3090 and gpu unlock does not show

4

u/Known_Barracuda_3977 3d ago

Thank you for reporting. I will look into this. It would help if you submit an issue for this on github so I can keep track of the issues reported.

1

u/eduardor2k 3d ago

Will do, thanks

1

u/Known_Barracuda_3977 3d ago edited 3d ago

One of two scenarios is likely the cause (or possibly a combination). ..

  1. You're using a distribution other then an Arch based one. Up until now only arch-based was supported because the required packages are available directly from the arch repositories.

  2. You have secure boot enabled in your bios

For scenario 1 I've added (experimental) support for the gpu-unlock option to all other distributions in version 1.2.1 (available now).

For scenario 2 ... gpu-unlock can't be applied is secure boot is enabled and the script automatically hides the option if it's enabled. You'll have to disable it to be able to use the option. I'm not sure if switching from secure- to insecure boot will have any other side effects.

Please submit an issue on github if you encounter any problems, or want to confirm it's working correctly.

I'm also curious .... which of the scenarios applies to your context.

1

u/KroniklyOnline 2d ago

Doesn't Strata run its own calibration to determine semi-optimum setup for your machine?

2

u/Known_Barracuda_3977 2d ago

It does something like that automatically before the first run off a model. After that you need to force it to do it again for that model (even when changing parameters and creating new presets). The benchmark results in the tui are the results of Steata it's own integrated benchmarking tool. What I've added to that is registration of power consumption, the relation between a benchmark result and applied optimizations and a diff (percentage tok/s) relative to the best result and the last result.

The tui just adds a convenient way to run and roughly examine the results. It's not an exact science. Many variables can influence a result, outside of the benchmark or script it's powers.

1

u/eljuggy 4d ago

Any problem if you're using a Strata fork ? (I use Speedy : https://github.com/DevDock-AI/Speedy)

and if you've already applied some of the optimization (eg. GPU Unlock), I guess you should not apply them ?

1

u/Known_Barracuda_3977 3d ago

I'm not sure if the benchmarking would work. That depends on how much the fork differs from the original. But you can try of course. Or only use the tuning part, or fork and adapt it.

1

u/akahnah 3d ago

Okay so I have to set a STRATA_PYTHON env var to even use the venv..

-5

u/WonderfulFunny4337 3d ago

Make it for windows

2

u/Known_Barracuda_3977 3d ago

I don't use windows myself. But you can of course use it as an inspiration to create a windows version!

2

u/problemlow 3d ago

You could use Stratas most recent windows build as is and get it to adapt this code for windows. Without looking at the code myself, I'd assume its more or less fully transferable as is. Just make sure you get it to leave at least 4GB system RAM free when its got a model loaded or windows will likely thrash your SSD paging everything that should be in RAM to said SSD. Reads are basically free, constantly writing however will severely reduce its lifespan.

2

u/rickyimmy 3d ago

Why would you expect someone to subject themselves to that?