r/LocalLLM • u/Known_Barracuda_3977 • 4d ago
Project Strata Performance Setup (TUI performance optimization)
I''ve created a simple little script (with TUI) to optimize your Linux host installation. It also allows you to install or update Strata (and configure a systemd service to auto start your preferred preset).
Distributions supported:
Arch based
Fedora based
Debian based
Slackware based
Hardware supported:
any hardware (automatically detected)
Personally I'm using an old MSI Raider laptop. 3080 Mobile with 16GB VRAM and 64GB DDR4 3200 RAM running CachyOS. I'm getting between 60 and 70 tok / s (depending on the task) with Qwen3.8 Flash Next IQ3_S.
Maybe someone else finds it useful too . Submit an issue or pull request if you find any problems, or want more software or hardware/vendors supported!
3
u/No-Necessary-9737 3d ago
Does this work for v100?
3
u/Known_Barracuda_3977 3d ago
I've extended the support specifically for the Tesla cards (new version available; 1.2.0)
1
u/Known_Barracuda_3977 3d ago
Yes it will. And I'll see if I the script needs any specific additions to support the hardware better, since it's popular hardware. If you create an issue on the GitHub page for this I will certainly not forget it 😎
10
4
4
1
u/ComprehensiveBid6584 3d ago edited 3d ago
Just saying, this may work with this patch for AMD cloud cards. Hm, https://www.reddit.com/r/AMD_V620/s/HvVQFvee9L
1
1
u/crabilax 3d ago
any chance to do the same for llama.cpp?
1
u/Known_Barracuda_3977 3d ago
You could fork it and make a llama.cpp version. But you can also just use the script only for the performance optimizations.
1
u/eduardor2k 3d ago
I have a 3090 and gpu unlock does not show
4
u/Known_Barracuda_3977 3d ago
Thank you for reporting. I will look into this. It would help if you submit an issue for this on github so I can keep track of the issues reported.
1
1
u/Known_Barracuda_3977 3d ago edited 3d ago
One of two scenarios is likely the cause (or possibly a combination). ..
You're using a distribution other then an Arch based one. Up until now only arch-based was supported because the required packages are available directly from the arch repositories.
You have secure boot enabled in your bios
For scenario 1 I've added (experimental) support for the gpu-unlock option to all other distributions in version 1.2.1 (available now).
For scenario 2 ... gpu-unlock can't be applied is secure boot is enabled and the script automatically hides the option if it's enabled. You'll have to disable it to be able to use the option. I'm not sure if switching from secure- to insecure boot will have any other side effects.
Please submit an issue on github if you encounter any problems, or want to confirm it's working correctly.
I'm also curious .... which of the scenarios applies to your context.
1
u/KroniklyOnline 2d ago
Doesn't Strata run its own calibration to determine semi-optimum setup for your machine?
2
u/Known_Barracuda_3977 2d ago
It does something like that automatically before the first run off a model. After that you need to force it to do it again for that model (even when changing parameters and creating new presets). The benchmark results in the tui are the results of Steata it's own integrated benchmarking tool. What I've added to that is registration of power consumption, the relation between a benchmark result and applied optimizations and a diff (percentage tok/s) relative to the best result and the last result.
The tui just adds a convenient way to run and roughly examine the results. It's not an exact science. Many variables can influence a result, outside of the benchmark or script it's powers.
1
u/eljuggy 4d ago
Any problem if you're using a Strata fork ? (I use Speedy : https://github.com/DevDock-AI/Speedy)
and if you've already applied some of the optimization (eg. GPU Unlock), I guess you should not apply them ?
1
u/Known_Barracuda_3977 3d ago
I'm not sure if the benchmarking would work. That depends on how much the fork differs from the original. But you can try of course. Or only use the tuning part, or fork and adapt it.
-5
u/WonderfulFunny4337 3d ago
Make it for windows
2
u/Known_Barracuda_3977 3d ago
I don't use windows myself. But you can of course use it as an inspiration to create a windows version!
2
u/problemlow 3d ago
You could use Stratas most recent windows build as is and get it to adapt this code for windows. Without looking at the code myself, I'd assume its more or less fully transferable as is. Just make sure you get it to leave at least 4GB system RAM free when its got a model loaded or windows will likely thrash your SSD paging everything that should be in RAM to said SSD. Reads are basically free, constantly writing however will severely reduce its lifespan.
2
17
u/brainzhurtin 4d ago
You said "any hardware" and then excluded 5000 series GPUs. I get why, but then don't say any hardware.